Webinar – Immersive technologies & their use in agriculture research & extension

Webinar – Immersive technologies & their use in agriculture research & extension

My name’s Luke Beange & I’m with NSW DPI & I’m bringing you this webinar today on behalf of GRDC Communities. 2 parts of GRDC Communities, Crop Nutrition & Field Crop Diseases. We are coving today the tips tricks and traps about filming,
processing, editing & then using this technology, the technology of immersive
technologies for project extension. Matthew Cox from Agriculture Victoria
and Michael Moody from Frontier Farming Systems Immersive Ag, we are going to hear their experiences with 360 filming. Matt Cox has been involved for many
years with the Victorian Resources Online website and has been involved in
setting up 360 degrees footage of Agriculture Victoria’s Smart Farm
initiative Michael Moody is the farming systems RD&E consultant and has been using the 360-degree footage for a few years
to further extend information presented at field days throughout the Mallee. Over to you Matthew. Alright thank you very much Luke. So, thank you for
the opportunity to have a chat to you & demonstrate some of the
interactive ways we’ve been trying to I should have mentioned my
name is Matthew Cox from Agriculture Victoria Introduce some of the ways
that we’ve been using 360 panoramas and videos for enhanced communication and
knowledge management. As you can see here I’m presenting on behalf of my former boss Mark Imhof who I’ve been working for for 15 years. He recently
took a couple years long service leave so he’s left me with the opportunity
to present this to you. I hope I can do justice to his presentation. In about 2009, we had a poster presentation that we used to put up in
in conferences and it was a soil and landscape assessment in the Upper Bet Bet Creek. The Upper Bet Bet Creek in Victoria is a little bit away from from the general populous areas so
it takes a little while to get up there and we’re finding that we were spending
the majority of these days traveling to these sites and then
trying to rush through what we needed to present and then rushing back home again
and we felt that we weren’t getting the most out of those trips. We had seen some other examples of using 360 degree panoramas to disseminate information knowledge and information and we thought that we could potentially use that idea to replace this poster presentation.
So we developed a concept of a virtual poster and the URL for that
virtual posters is there at the bottom of the screen and in the chat
notes, so if you wanted to have a look at that a little bit later at your own time
there’s a fair bit of information on there about that area. One of the reasons we did this, we took a, we decided to
carry out a bit of a research project about this and we split a group of
students up, from memory it was about seven seven students in each category we
actually took the seven students to the we showed the virtual poster to
seven students and got them to familiarise themselves with the Upper Bet Bet Creek and all the information that we had on the website and then we took
the full group up to the Bet Bet and asked some questions at the end of the
trip to find out which group had absorbed the most information and it was
pretty clear it was the those people that had seen the virtual poster had
a far better understanding so we kind of gave us to this idea that we could extend the 360 panorama solution out to other
topic areas that we had on Victorian Resources Online at that time. So, in 2011 I have been checking my notes earlier I think was a little bit
before then we used to go out to UM country Victoria with a regular instant
camera and a tripod and we took around 200 photos. We stitched those together
using some software and published these these online. Now to take 200 photos and
stitch them together it was quite a laborious and time-consuming process. It
takes sometimes take up to about an hour for each one of the 360 panoramas to be
joined together and we had quite a lot of problems with parallaxing. Which is
where you’re trying to stitch together two images that don’t match up and
that’s a common mistake when you’re starting out doing 360 panorama, that you try and carry out this work without the right equipment. So, a
couple years ago we actually invested in a zero nodal tripod head and digital SLR
camera and we’ve managed to get the workflow down from 200 shots to four
photos and we do those, we take high-dynamic-range photos and then pull
all those photos together to give an example of the equirectangular image
below that shows that there’s no parallaxing. So, in a parallax image you’d
see there the handrails would all be there be lots and lots of handrails
where it didn’t really stitch them together very well. So using the new
tripod head and digital SLR we can take a panoramic photo in about four
or five minutes and it takes about 15 minutes to stitch them together
and you get some really really great great results. So some of the examples of
how we’ve used panoramas or 360 panoramas, immersive experiences to
explain the soil landscape units in Victoria. So again these are places
throughout Victoria and that are reasonably remote, not as remote
as they could be but does take a little bit of time to get there so we go out
there and we take the panoramic photographs and we embed information
products inside these panoramas. Now I’ll let you, if you want have a look at
that example here of the Maffra map. Just under on the right hand
side you can see a gentleman, there’s a video he’s explaining the channel of an
old stream there and that’s a good colleague of as Ian Sargent and
unfortunately Ian passed away a couple of years ago after a long illness and he
he gave his life to mapping land form units throughout Victoria and we were
lucky enough to spend some time with him so we went out and videoed him in the
landscape that he loved and characterised and you know, things could
have been had done a lot better given the technologies that we know now. But
we were able to capture a bit of a legacy for Ian. Him talking about some of the most interesting landform characteristics in
Victoria. Here’s a few comments just showing some feedback from
colleagues that actually do use these panoramic or immersive experiences to
share information. Another URL there to look at if you want to. We captured a number of long-term trials throughout Victoria. These are
scientific experiments some of them that have been going on for quite a number of
years and capturing the scientific results from those experiments, some of the characterisations of the soil and some
other information products about those it really allows you to put that
information product into the context of where it’s been, where it happened
This is this example here these long-term trials they were
done as outputs from projects and those projects have finished a little
time ago. This was our first attempt to do some, so a lot of these
long-term trials we’re all developed using flash based technology
which we know is starting to be little less contemporary and this IT world that we live in, but it shows
that we’ve been doing it for a little bit of time and and the information
still there if people do want to to read about it. Now more recently, kind of in the last 12 to 18 months were creating Virtual Smart Farm experiences across Victoria, so in Agriculture Research we have some Smart
Farms scattered across the state right from Horsham on one side of the state,
right across to Ellinbank, in Gippsland on the other side of the state. It does
take a little bit of time to travel that distance and the virtual Smart Farm
experience that we’ve created unfortunately, it’s an internal product like an intranet for internal AVR staff. This shows some of
the more recent techniques that were using with high dynamic
range videos and information products. We’re finding it a very good way of
showing things that are happening across the state from the place where
we’re demonstrating. In the top right hand corner, you can see a TV screen. That’s me on the left hand side and I’m actually showing an example of the
virtual smartphones experience to two deputy secretaries in the department.
We wouldn’t have had the opportunity to do that if we hadn’t
created these virtual experience. I’m just going to zoom across to this
screen, hopefully it works. So I just wanted to show you a live example of
this. This is taken at our Tatura smart farm, it’s taken from the roof of a
comms unit, a shipping container that you can climb up the side and it
allows us to take some photos from a little bit higher up. You can get some reasonably good photos. As you can see, it might be moving
a little bit quickly. I’ll just wait for you to catch up. Where I’m showing
you on the ground, that’s where the tripod and camera are, so with the software
and hardware that we’re using you can actually erase any bits of me or any bits of the horizon that you don’t want to see. So we can even
get up to a point where I can was able to replace the tin on the
top of that roof. That can be quite helpful if you’re
working in reasonably confined areas I must have PowerPoint open twice. So we’re finding this virtual Smart Farm experience quite an
immersive way of showing examples of places in Victoria that you can’t get to
and that’s how we’re finding it finding it to be very useful. You can show
things that people can’t get to or are not allowed to get to. I wanted to talk
a little bit more about 360 degree video. So, still panoramas, 360
panoramas are very very useful but we found to provide an even more immersive
experience 360 degree videos are a great great example. So interviewing
researchers in the context of where their science is happening, interviewing
research scientists who do tours of facilities and they get a little bit
tired of doing the same tour over and over again. So we record the tour once
and you can view these videos. Obviously using 360 degree videos on a
flat-screen, a traditional TV, it’s kind of monoscopic, as has been written there. It’s kind of one set of images so you kind of have to zoom around
yourself. We’ve had some quite good experiences of actually using some of
the Google cardboard technology or hardware and also if you view these
videos on a mobile device you can actually move the tablet or mobile
device around and you actually get to see the direction that that video is
being taken in. More recently we’ve moved into kind of virtual reality, so using 360 degree videos for virtual reality. Unfortunately that means that you
need to have these headsets but it allows you to be fully immersive inside the experience that’s happening. The colleague that you can see on
that screen actually runs one of the research facilities in Victoria. We
actually interviewed him and he was able to do a few takes and we got some
quite good results with using these experiences. Now, it’s not
for everybody, I find sometimes if you spend quite a lot of
time inside these headsets, it’s not a particularly comfortable experience but
for short amounts of time like a few minutes I find them find them a very
good way of communicating something that you perhaps wouldn’t be able to show. I’ve just got a quick slide up there showing another example of
demonstrating the Smart Farm on the left-hand side in that picture, on
the right-hand side of the screen our Minister for Agriculture is there and I
was giving her a demonstration. Thank you to Julio for taking that
picture for me as well. Everybody can kind of see these
screens but I think this is actually a duplicate slide of something that
was on earlier so I do apologise for that. The last thing I do, I’m just
checking that I haven’t gone too far over, but we’re moving into kind of fully
immersive AR/VR and mixed reality experiences. So I’ve got an example where
we’re capturing high-resolution aerial imagery from drones to capture
a digital elevation model, a DEM. We’re using these DEM to create photorealistic
models of landscapes. So, we’re doing that as a first cut for all about smart farms.
The image you can see on the bottom lower bottom right is at the Ellinbank
smart farm set so quite a good example of what can be achieved. Now
the picture above you can see there’s three people standing around a
holographic image. So, what we’re hoping to do is actually move these digital
elevation models into this fully immersive experience that you can
actually interact with these mixed reality sessions from any location. You
can also embed live data feeds. So API’s is that you can actually virtually
interact with the physical world. So, I’m not saying that we’re there yet but
we’re hoping at some stage you’ll be able to push a gate and it
would actually have some sort of smarts within that system to actually
close the gate on the farm or check the levels of troughs remotely. We’re hoping that this can be used to support remote learning and
knowledge transfer. These mixed reality headsets, how they differ from
the headset in the previous screen which was the oculus that actually get to see the surroundings of the room that you’re in and this
hologram is actually displayed in front of you. So it’s all a bit like the
Star Wars scene where the hologram appears and that’s the
approach that we’re actually working on at the moment and we think it’s actually
going to be realised within about the next 12 months. That we’ll actually be
able to provide fully immersive experiences for all of our Smart Farms. I’m just going to move this over. So, just in
conclusion we’ve found that the interactive 360 panoramas are able to add rich information to static web pages and support more
interactive and dynamic user exploration As I mentioned before with our colleague Ian Sargent It provides a means for capturing
expert knowledge and enhancing place-based understanding to capture
context and transfer the information and A lot of our specialists are reaching the end of their careers and this is an
opportunity that, one of the only opportunities that we might have to
capture this information. We think with this generation some of the information
if not capture is going to be lost. You can integrate a range of data products eg. the videos, the data displays, API feeds and webpages as well. So we’ve got
examples of loading up static HTML pages into the panoramas that provide some
some information about that In conclusion the 360 degree videos offer a very good immersive experience and we
strongly believe that kind of these virtual mixed reality experiences are
going to be pivotal in the way that we disseminate information going forward. Well thank you everyone for having me here today I’ll say straight up not a technology whiz or anything like that. I’m just a
humble of Agronomist looking for ways to do our job better and communicate
what we do to our end uses which more often they’re not the farmers
that are out there that are paying their levees give those projects to help them
become more profitable. So, that really is the starting point where we’re
coming from and yeah I guess where we’re more at the
the consumer end of products. So products and are cheap enough for us to get out
there and utilise and it also work well enough that we can develop some products
relatively cheaply and easily to get the information across. So I’m going to concentrate today on the virtual field day development that we’ve been working
on for a couple years. To do this I work I’m a consultant with a business called
Frontier Farming Systems. We work right across the spectrum of research
development and extension. So I guess this project is really coming from that
extension end and how do we get the research that we’re doing across to
the farmers. But we work very closely with Mallee Sustainable Farming,
which is a farmer group working across the tri-state Mallee area. We’ve
put together this little collaboration which we’re calling Immersive Ag. So
Immersive Ag being the sort of branding front for the development of these
virtual field days and also the dissemination of this product out to the
farmers So, how did how did this sort of come
about or how did we get involved in this The fundamental question I suppose is,
the traditional way of extending out information to growers relied a lot on
basically their presence and also communication of our trials at field
days. I guess working with Mallee Sustainable Farming, would run in
excess of ten different events across the region in any given year. But over time we’re finding farmers becoming much more poorer to physically
attend these events and so we’ve been thinking for quite a while, what are some
of the, what are the other ways that we can actually reach farmers when they do
have time and I guess one of my thoughts was farmers these days are very
automated in what they’re doing at farming level. Perhaps one
of the opportunities is actually to be able to get the information to them while
they’re sitting in the tractor. Most of our guys are farming very large paddocks,
using auto steer and technology to do a lot of the work for them and a lot of the
farmers are now spending a hell of a lot of time on Twitter and other things.
Which is demonstrating that they’ve got lots of time available within
the tractor cab, not so much, a lot of time available to be spending days at
the time looking at all the trials that we do and also because we’re covering a
very large region as well. You could never ever give that trial experience to
every farmer within the region You’re really just targeting the local
people in the vicinity of the research that you’re doing So, I guess as I said before not being technology savvy or anything like that How are we going about doing
these virtual trial site tours Well the first thing is to capture a
series of photos of the trial sites and to do that we’re using consumer-level
360 degree cameras So these cameras, as opposed to I guess the
process that Matt was doing which may give a higher quality result. We’ve got one-click cameras that have a lens on each side and and basically you take
that camera out to where you want to take a photograph. Hook it up to
your via bluetooth to your phone and you can operate that camera remotely to take
pictures. So we’re going within trials to each end of the plot taking a picture.
Sometimes we might even go into the plot take a picture there, but I guess at this
stage we’re using consumer-level cameras which costing anywhere up to about a
thousand dollars depending on the brand This space, doing a bit of
background research and taking an interest in it is moving very quickly and I think I’ve already up to the third camera that we’ve bought because
every year there seems to be a better and more effective model coming
out. So the space in 360 camera development is only getting better and
getting a much better product because so it’s quite a new industry. What we then
do is for each photo, so we might take up 40 photos in a trial which could
represent one replicate for example. Annotate features and we saw some of
that Matt’s presentation. So we can bring those photos
into some software commercially available software that is used widely
in the real estate industry. You would have seen, no doubt advertisements these days for open houses which are in the virtual space You could walk through on your computer.
So we’re using some of that technology at the moment to stitch our photos
together in a different context. Within those we can annotate each photo the features, so that might be some descriptions about the plot you’re
looking up, descriptions about neighboring plots, it could be features within the view that you think are important and also
data and methodologies and other things and venues on that trial and also
hyperlinks. So the hyperlinks to take you to an external source of information or
the hyperlinks allow you to move in a a virtual way between the photographs that you see, that you are looking at We then use the technology or use the software to stitch the panorama photographs together and again this is commercial software and we’re I guess Immersive Ag came into it, when we
first created our first tours we found them very hard to share.
So trying to share through Facebook initially, but we found it quite a difficult exercise to do. So, what we’ve done is we’ve built
a website that is specific for hosting these virtual tours. So hence bringing about Immersive Ag which is really our platform to deliver to growers and I
guess we see there’s a role for Immersive Ag in the future, not just our own virtual tours but perhaps become a bit of a YouTube of
Ag virtual tours, whether there trial sites
from other groups and researchers or interactive tours of other things of
interest to ag. So, we’re really keen to see that platform grow into the
future. So, there’s a few challenges that we’re currently wrestling with to
improve the process of bringing these virtual tours about. One of these is
the pitfalls of capturing 360 degree photos in an open paddock. The last
thing that you want is somebody standing in the shot two or three meters away
from the camera, so we want a bit of an open view as if somebody’s
standing there in the middle of paddock and looking around in a 360 degree direction
themselves. So getting out of the shot is is becoming time-consuming and I’m
currently working on some methodologies to perhaps try and improve the workflow
there to capture pictures much quicker and faster. At the moment we’re
parking a vehicle some way away out of the view and walking to & from
the tripod to move it each time, so that takes up a bit of
time. Annotating & hyperlinking within each of the pictures is also a
time constraint as well because sometimes, most often you actually
the same feature might appear in four or five, six different photos. So in each of
those photos you’ve got to go in and label the same feature & label the
hyperlink, which plot you’re looking at & what data. So it’s a lot of repetition
and so looking at ways to solve that, looking at other computer software that
may solve that is something that were also keen to pursue and I guess this time consumption at the moment is limiting the number of plots. So having a
lot of detail within the tour is limiting the number of plots So, at the moment we might just focus on
one replicate, but I guess if we can either find a way to speed up how we add
the detail in or reduce the need for detail within the tour then we can
actually stitch more photos together and have more complete virtual tours of our
trial sites. So, some of the opportunities I suppose in virtual tour development to
improve the process, but also improve the product. We’ve done not just trial sites
but we’re focused on doing a bit of a collection of a whole array of different
tours that might be of interest to people. So, we’ve dug some soil pits, collected some data from soil pits and also put
those in a 360 tour to give that immersive experience, but also coupled up
with some of the landscape stuff as you’ve seen with Matt’s presentation
before. Giving the user a feel for not only what’s underground but also
above ground. We’ve have looked at some 360 video with machinery and machinery
operations and put some videos together mounting the camera on a deep ripping
machinery at the start of the year. We’ve used it to look at some
on-farm NRM and document on-farm NRM issues. One that is specific to the Mallee
is dune seepage and salinity using 360 photos to show the landscape and also the breakout points of salinity. We’re currently developing some
tours of stock containment areas, given there’s parts of our region in very severe
drought currently. People are looking to have better options to feed livestock
and conserve paddocks So, we’re currently busily
developing up some 360 tours of containment feedlots that people have done well
and also there’s lots of opportunities I think in education & awareness, which we’ve seen plenty of examples of those in the past as well. Another opportunity I feel
strongly about, is I think there’s a market for 360 virtual tour development
in trial site Quality Assurance and a lot of projects these days are
requiring collaboration even at a national level. Quite often the research
scientists that might be in charge of a project may be based several thousand
kilometers away, so I see putting together these virtual tours of the
trial sites is a good market there for those research scientists to not only document but they will have a great resource to go back to when
they’re trying to analyse and write up their results and even from a funding
organisation, they may be a great resource to actually document
that the research has been done to a high quality level and have a
record of that that’s tangible. The opportunities as I
discussed before trying to overcome some of those time constraints so we’re
currently looking at capturing the photographs aerially with drones.
I’ve also tried in the past mounting a 360 camera on a remote
control car and driving that through the trials to try and capture the
photos without being in the shop and that was to the amusement of my wife
when she saw the invoice for a remote-controlled car from ebay but you’ve got to try these things. One of the the areas I’m quite interested in pursuing is 360 versus three-dimensional, so I guess was 360 what we have is we have a
two dimensional photograph that we can look around in a 360 sphere, but 3d which is 3-dimensional is another level again and looking at LiDAR
and color point clouds it’s possibly a way to capture that in the future which
will now allow us not only to to look around the scene but also look through
the scene. I see some great opportunities with that technology and with some
autonomous cars being such a a focus in R&D at the moment , that’s very
similar technology that was required in that industry. So we’re seeing senses
becoming cheaper for that type of application. So just to finish off I thought
I’d give you a bit of an example by going through Immersive Ag and I’m looking at a
tour that we have developed recently So this is the Immersive Ag webpage, so currently I guess we’re busily trying to develop
content to grow this page but also improve on the content that we
currently do have. So I’m going to take you now into a tour that was captured a month or two
ago, which is, we’re part of a project called the Management of early sown
wheat. It’s a GRDC trial site with a trial
network across the southern region in Australia, but I think this particular
trial illustrates the potential benefits of utilising this technology
and showcasing things of value. So we click in, we’re now into this
virtual tour, you’ll see some of the stuff that we can we can embed, we’ve got
this about screen here which is giving a bit of background to what you’re
looking at in the trial site, it’s also a great way to recognise project funders
and also project collaborators that you’re working with on the trial. We can
build in information around the methodology that was used. In this case,
this trial we have four times of sowing from the 19th of March through to the
29th of April and at each time of sowing we have a range of varieties that are
both normal spring wheats, that would currently grow but we’re very interested
in the potential for winter type wheats, which have a fertilisation
requirement. So we want to see whether they offer some potential technology
there to be able to sow earlier and and get some benefits out of that.
So this trial is located 30 kilometers west of Mildura.
So we can click out of that screen there and what we can do is we’ve got a trial
map here, allows us to look at a bit of a plan view of what we’re going to look at in this trial. So, I want to go to time of sowing four and look at the
variety Cutlass. So, time of sowing four was in our normal sewing window. This
trial was captured, this virtual tour was captured in July. So, during the
growing season and what we’re really after is to show, we’re looking
for wheats that can be sown early but also flower within their optimal flowering window. So, if you see any varieties that are currently
flowering or have their head emerged currently this is way too early and really not
applicable for this region. So, Cutlass in time of sowing four, you can still see just at the end of the vegetative phase there, which is what we’d expect. So Cutlass being a mid spring wheat currently grown within the region.
If we click down to the next plot on. What we have is a early released variety
called Longsword. It is a winter type wheat, so it has a fertilisation
requirements and you can also see when sown at a normal sowing period for us, so the 29th of April. It’s also in a vegetative phase. If we look to the right,
we can see a plot here that’s clearly not the vegetative phase. It’s got it’s
head out and well into the reproductive phase of its development. So this is Scepter. This is Scepter sown on time of sowing one which is in the middle of
March. You can see this is way too quick
for our environment and it’s likely that this plot isn’t going to come to any yield. It’s going to be subject to frost and not yield very
well. If we go and see what Cutlass is doing at that time of sowing one. So to come back
to time of sowing one, you can see Cutlass. OK, Cutlass also at time of sowing one is way too quick. It’s around that head emergence stage in July. So again it’s going to be frosted
and probably not yield very well. If we turn around though and look at the plot directly behind it. OK, we can see we’ve got a variety here called DS Bennett.
It’s a mid slow winter type. So, it’s fertilisation mechanism is
holding it up from rushing through to the flowering stage and it’s still
well into the vegetative phase. Which in the middle of July is what you want and if we go to the next variety further down and we turn around again. This is
Longswords, which is the first variety that we looked, one of the winter types we
looked at before. Again, it’s in it’s vegetative phase and is at a stage development where we want it. So, I guess this trial hasn’t
had a field day here at this site this year, but this is a great way I
think of demonstrating the key points of what we’ve seen through the
growing season at this site. Just to look at another example that we’ve done. We looked at one of our break crop research sites. This is one of the first tours
that we put together It was done last year. Here at this site
we’ve got a range of break crop options In front of us this is field peas. We can
see we’ve got Butler, if we look to our right you can see we’ve got Casper. Look above us, we can see
there’s a variety called Twilight. If we click over here, we can see it’s a pretty low rainfall season. When the stroll was run. About a 100 mm of growing season
rainfall only. But as the trials been completed now. We’ve been able to build in
some results tabs. So just before we are looking at Butler and then we’re looking
at Twilight and you can see the final yield result for those varieties was about 0.6 of a tonne per hectare Which was a pretty good result given the
season that we had. So, just to finish off on another example. I talked before
about utilising the videos as well This may or may not work. It worked before
but we’ll see how we go. These videos, earlier in the year we had
a deep ripping demonstration day. So, deep ripping of our sand dunes is a practice that it’s really taking off in our region. We work with Agrivision
Consultants to put on a day that compared some commercial
machines. Farmers can come along and see how they looked. But, I guess we were able
to give the farmers a bit of a unique perspective of how some of these
machines worked. If you weren’t there on the day or even if you were there on a
day as well So, start this video up So, I guess the real benefit here
is being able to create your own view of what you’re interested at. So here we
have the camera mounted underneath the machine. That’s not a view that you can
have even if you’re out the field day Some of the things that farmers are
really interested in are the level of soil disturbance and also trash throw handling.
Might be a good considerations to think about. So, we can have a look under under
this machine and it let’s see how things are going. So, I think that’s frozen for
my end but you’re more than welcome to go and have a look at the website,
have a look at some of those videos and how they work. So, just a just another
idea I suppose of how the 360 cameras can help generate some content that we
don’t usually deal with. So, I guess Luke that’s the end
of what I had to say

Posts created 37718

Leave a Reply

Your email address will not be published. Required fields are marked *

Related Posts

Begin typing your search term above and press enter to search. Press ESC to cancel.

Back To Top