Good. Afternoon, everyone I, think. First of all thank. You and thank you and thank you for staying to the last session I, think, we should give everyone a round of applause for SATA data. And. So. Today what we're gonna share with you is really this whole topic of a I feel good you, saw one. Of the initiative, being launched during, Starsky know let, me just switch. And. So, today is basically out of the three programs we're gonna share with you some, of the work that we have been doing in using AI for advancing. Some, of toughest, challenges, right and therefore if you recap, what you have seen earlier. Ignite. This year we shed about AI for humanitarian, action but. Prior to this if you've been looking at the past 18, months we. Actually have been launching a lot of AI for good initiative, and the, gold is really to empower many. Of us right me you everyone. In. Being. Able to engage and use AI in, more meaningful ways and the. Goal is very simple right to create a more sustainable and, more accessible. Work and if. You look at the breadth of technologies, that you have seen across the five days of ignite whether. It's AI where. This advancement. In the cloud where, this advancement in hardware, and so, on and so forth this. Is changing the way that we live we. Learn and we work and one, of the fascinating thing about this transformation. Is, it's actually creating tremendous, opportunities, of course for the work that we do for our companies. By, the same time if you reflect on it we, all have a huge responsibility right, we are all technologies and therefore. Is important, for us to be able use technology, to, both, you, know meet of course your KPIs which is really important, by the same time thing about how we can use it to solve, some of these challenges right. Where there's in a free time whether, it's with a nonprofit that you work with and so on and so forth and. So if you boys out everything, that you have seen in the past five days it's, really three things that you, should think about there. Has been tremendous, advancements. In algorithms. And this. Pace, of innovation in, algorithms, is just going to happening. Right. At the same kind of cost all, these kind of technology, conferences, you, see advancements, in Hardware right. Whether it is with GPUs. Whether it's with FPGAs. And so, on and so forth hardware is continuously. Innovations. Is happening in that space and of. Course the third piece is really the fact that hey know we, all collect a lot of data right what is in the companies that we work with the, nonprofit, that we work with and so, on and so forth, there's tremendous amount of data been collected not just about enterprises. Right but data collected, about the species around the world data, collected, around water, climate. And so on and so forth and therefore. If we put all this together it, will allow us to use, technology to solve really some of these challenges, and today what, we're going to share with you is some of the projects that we've been working on jointly. With ESRI, but, at the same time we also share with you where we're heading, right, and if you're interested in some of these projects we'll give you a call to action on how we can participate in it sounds, good. Right. And so the story let me first start off with a story and all, these discussions, about AI that you've been saying are specifically in deep learning really, started off of this thing, that is not new right and. A. Convolution. You learn at work today in the context of deep learning really allows you to do a lot of amazing stuff from, being able to do image classification to. Even audio classification. And people have been using it and, continuously. Innovating, in this space now, so a trivia, for, everyone in the room is.
After. All the session that we've been to and all the tech conferences, that all the AI discussions, that you might have how. Long does it take to train a deep learning model like a convolution, your innate that you saw earlier. Less. Than a minute awesome, that's, the aspiration that we're getting PUFA, thank you right. And of, course we want to get to less than a minute and we're actually getting close to that right, and one. Of the reasons why I asked this question is the following just, to illustrate and, provide, you with a data point on how, fast, the space is innovating. Now. If you speak to me about before 2017. And. You say hey I want to train a really deep convolution, you're a network. It'd probably take days, right. But 2017. Was an amazing, year, in. April. 2017. Facebook. With some of the open, source hardware called, Big Basin. Demonstrated. That you could train, imagenet. Which is a very. Classical. Data set used for image classification or, the basis for image classification in. An hour and. In fact that piece of work in April, 2017. Kick off a whole, conversation. And, innovation. In the distributed, deep learning space. Now. A few months later, another. Group which, is a university, sure, that without GPUs, you could train an image, net model for 31, minutes which, is fascinating, and of. Course we think the year 2017. Is going to end up that way no, right, it continued innovating, and. Towards. The end of 2017. Image. Net training took just about 15, minutes and, this is done by a group in Japan and, the. Point I were trying to get to is just look at the pace. Of innovation it's, what happened in 2017. Alone. For. A really simple task not exactly but let's suppose it's a really simple task of training, a deep neural network, for image classification. The. Time is half every time there's an innovation, but, and in fact as this, gentleman call out 2018. Just in July this year a, group. In China right, demonstrated. That they could do image net training, using. Rest net in 6.6. Minutes and I'm sure we the year have not ended right it's just September, we'll get to the goal of one minute but, what is also fascinating, is that following today, we. Used to think that hey all these innovations, that's happening, here needs, to be a big. Research lab right, or a big, company that's doing that and there's, already been a lot of science, in the community, that shows that today, we public, Hardware, without, so many GPUs, and without so much compute, you, are able to do it as well meaning we could do it on our way, back to wherever we come from right.
And You should be able to train it within you. Know 15, 31 minutes and that is the point the point is the power is, no longer and, skills to say hey you know we can't have all this infrastructure, because we don't have it right, the. Power is now available, for all of us to do this and that's why it's important, because. Then. We can use all this to solve a lot of these fun problems, right and if AI for Irv was really a program that. Was conceived, by the chief environmental officer. Lucas, Jabba many. Years ago in fact he was a Microsoft, Research it, was at Microsoft Research for many years before he became, the chief environmental officer. And it's, a program that looks at one, providing. Access to, access to, all this technology so. That all of us can solve, some of these challenges, what. Is more important, is we can, dream and we. Are willing to do it right. Do. Is of course education, being. Able to work with every, one of us in this room to, understand, why a lot of the other resources, are being depleted. Production. Is not going as fast as we want it to be and why, this is a pressing problem and why, technology. And AI can solve all this right, in the right way and if you use in the right hand and the. Last piece of it is well, what. Does deep learning community, or any of this technology communities. That we are all a part of, we. Have tremendous creativity, and therefore we can continuously, innovate, and through. All this innovation is gonna move the state of art right. And so what is AI for Earth and. Why is this important, right AI for Earth is really a program that. Is designed to, empower nonprofit. Right, and it not even be a nonprofit organization, it, could be an individual, that cares about the, earth and saving the earth with. Technology, with grunts, with. Access, to some of the engineering, teams at Microsoft, so, that you can empower, both, people, as well as the nonprofit organization. To solve some of this global, environmental. Challenges, and, it. Really focuses on a few areas one. Which is agriculture. How many of you have been to the Expo and been. Down to the expo you see the expo at the center of the Expo does this huge, booth right. And you can actually practice precision. Agriculture, there. Through, a project called farmers and. Ever we've been spending a lot I'm on agriculture, I'm don't share view some of the projects in the area and we, actually getting a lot of interesting. Benefits from, there for example right today, oh that's neat to eat in fact after this session I'm sure we all be hitting off for lunch, we, are all hungry, and but, the point is today if you want to fit the world rapidly, growing population. It's. A no-brainer right the, farmers, and organization. That's producing, the food regardless of what kind of food must, produce more food, but. We live on planet earth, and on. Planet of this only limited land earth it's not going to double the size in two. Years right, and therefore with. Limited, resources how can we then and able, all these farmers around the world to. Be able to so more, effectively, to be able to create more production, on others limited land and we.
Believe That AI can, help a lot in there and fun businesses, actually want the project in the area now. The other area that we look at is really water. Now. Water is very important, and therefore how can we conserve, and protect all this fresh water supply, that we have it's super important and top of our my as well and. Biodiversity. And we're gonna show you a few projects. I've been working on understanding, the biodiversity, on earth but more importantly, protecting, the animals that might eventually get em dangerous and. What is also interesting there. Is we, might think about protecting, endangered animals, that's not our business someone, else business, but. We've seen and we work with a lot of people, whether it's within Microsoft, and outside. On Microsoft, there. Has been using technology to solve this problem even, though. You. Know it is not their day job right, which is fascinating and of. Course the last piece which I think all of us have seen is climate, change has, indeed, been. A issue, that we all need to think about and how, can we use AI to reduce, all these challenges, right, of. Climate, change on communities, so if you look at AI for a program, in general is really divided, into this four areas and what, is interesting since this program start that we are about one and a half years old now we. Have given grants, across the world right, to about 100, plus grantees, and the, list is still growing. And today. Every, one of us in this room if you have a passion, for using, AI to solve some of these challenges we, invite everyone to apply for a grant of course, you have to put in the justification. There's a proposal right, but we, give grants not just a nonprofit, organization we. Give grants not just, to universities, and so on and so forth we give grants to anyone, that has the passion to, solve this problem with us right and so. To. Put all this in perspective and, just to show you why, AI is such an important, component, of. Saving, the world, let's. Try a simple game or, and to do that I'm going to share of you the story of the Snow Leopard how many of you have had a Snow Leopard before. Now. In fact I hear. I hear some laughter in, fact what happened is this, whole snow leopard trust was really started, and of course there's a lot of such organization, around the world this, was started by, as a result of two captain, brought to the Woodland.
Zoo In Washington State. And. In fact last October, we celebrated, the first birth of a cop and everyone was there cuddling. The cop and so on and so forth but Snow, Leopard is an interesting species, in. The areas where they operate around the world, Snow. Leopard is known as the ghost of the mountain and four, specific groups. In the world they actually seen a sacred, animus right. And as sacred animals they are really a really important, component of the entire ecosystem, or both. The low curses in the area but at the same time they. Tremendously. Important. To the Fox or the people. In the Himalayan, region right. Now and, so. We work, with the Snow Leopard trust to, use AI in this case image classification models. To, detect the Snow Leopard because, in order to protect a species like this you must find them and understanding. The habitats, where they operate and so on and so far and so just to illustrate to you why a is important, and why, you know, we. All want. To experience, what a Snow, Leopard researcher. Does let's, play a simple game now. Somewhere, in this picture, there, is a Snow Leopard. Now. Every time I do this someone, was saying that Snow Leopard might be hiding behind a rock. Anyone. Can find Snow Leopard. Now. As they are looking at Li let me continue the story, today. If I was under the group that we work wave at one of the researchers is called co-star he's. Been about six to, eight years on his PhD just on snow leopard right. And every, single day he has to paw through lots, of photos like this to. Identify, the snow leopard so that he can then understand, their habitat, how they move and so on so forth and so. In fact when we work with him last year one, of the things that he shared with us is. That he, actually spent time looking at the snow leopard photos then family photos, because, he need to finish his ph.d and. The. Point is when we met him we said well what if technology can help and what if AI can help and it, turns out that in order to be able to detect this in the deep learning community, this is known as a fine grain computer. Vision problem and if. We can start teaching algorithm, to know that this is the bounding box where. The snow leopard is. Then. The next time I show you the photo again now, all of us I can actually detect snow leopards right because, I have taught you how to do that and the.
Point Years well. What if he has all this every time he does a survey, and he has all these traps. Which I'm going to show you in a while. This. Traps are distributed, all around the area where he is doing his study each. Time, he collects millions, of images, and we, call it the snow leopard going up to take the selfie now in extra is not right, they. Go they. Might pass, by it you might get glimpse, of it and so on and so forth and he. Brings all these images, down so, that he can analyze them so one of the things that we built with him last year was. That he took all these photos that he has in his collection over the six to, nine years that he was doing his ph.d he uploaded, it to Asha storage, and what, we did was it that's, the Asha function, right that gets triggered when all this gets uploaded right. And as a result, by, the time he uploads, right. The time the most likely time is really the uploading, then. In his blob storage he gets two folders one, high. Probable. Snow leopards. - no. Snow letters so it reduces the, huge amount that, you have to do and process. And view -, maybe less than twenty percent of it right, and in fact one of the things that is fascinating as she was doing his research is, that all the photos that you have has. Other animals, accepts knowing that but guess what kind of animals he see most often. Goats. Mountain, goes right. And somehow the mountain goats are just curious, about the camera traps so, the mountain go go out look at the camera and so on and so forth and so this helps him reduce write the time taken for him to identify the Snow Leopard and of course each, one of the photos that gets taken right is Joe Peck - where the camera trap is the, time the, location are all there and so, now he can understand, the habitats better now sounds familiar, now whether this is useful snow. Leopards or whether this is useful for example manufacturing. The, technology, is the same being, able to detect, and being. Able to classify, I'm. Gonna show you something what they've been doing in this area especially. On biodiversity. Monitoring. So, one of the companies that we work with is. A company, called I naturalist, in fact I naturalist, is a fascinating, app that you can download whether it's on iOS and Android, today is that a great deal for you to download today and it, allows us to say well all of us can play a part in trying. To understand, the species diversity on, the planet in, fact when I came to Orlando right. I had my i natural is app I can understand all the fauna or all the flora and the flowers, there's, around this area that's interesting, I as. I walked back to my hotel in the night and I see an interesting thing I can take a picture of it and contribute. To this growing. Database. If you will of, information. About animals flowers. And so on and so forth right, and the point there is once you have dead you, now have a really diverse, data, set of trying, to understand, animal. Population, across the world right. And so, I naturalist, has been working with us for a while, the AI for a program, on trying, to turn this into API say all of us can use now. Just to give you a perspective I naturalist. Today is a non-microsoft company. That. We work with it. Has an online community, of what we call citizen, data scientists. There's been recording. Data on the, distribution of, of biodiversity. Greater. Than right now that's about 500,000. People 6. Million, observations. Growing. By the day and about. 120,000. Distinct species identify. Right, and, this is fascinating and, so this, app, we. Have been working with them in order to improve. It now, so this app shows you how we could provide, information using. The I naturalist, app and just to give you an illustration of, how it works, anytime.
You Bring the app up and you see it any more whether it's a rabbit your, backyard you, know a beer, that you see when you're driving back home or some. Animals that you see somewhere, you. Take a picture of here it's gonna ask you for a location it's. Gonna ask you to say hey what do you think this animal is you're, gonna give your best judgment right. It's gonna share with you what are the other people who have seen similar animals, life right. And. Then they're going to provide more inputs, so this crowdsourcing. Way of doing data labeling, right, being able to then use all these Labor's, as have to, be able to train algorithms. Becomes, something now easily achievable, and all of us can do it right, and, once. You are done with that of course then this, is provided back to the community, in fact I naturalist. Every for the past two years that's been running this big. Competition at a conference called cvpr, which, is the premier, computer, vision. Conference. And this. I naturalist, database, is just growing now to show you why, I naturalist, is important, I wanted, to show you a demo right. Now so. This was an app that we build on. Once. You have all this data that's laid but using an app like I natural is what, can you do with it. So. Right now what you see here is, the. Sample site that we build right. And what. Is this sample say about this sample size really about well if you have all these AI naturalist, data how. Can you train a machine learning model, to. Be able to de understand, the animals, that you see and so on and so forth right so I could go explore or, I could upload a picture there, from. And so, right now this picture you see here every. Single image here is not taken by us we pull it from big using, the Bing API, you. Can search for any any month that you want any any months that you're interested in, anyone. Sorry. Octopus, oh that's a hot one that's. A good one but it's a hard oops. Tapas. That's, about Corey. No. We couldn't find out to post any other anymore. Sorry. Kiwi. Oh wow. That's interesting. Looks. Let. Me just refresh the site. Yeah. We didn't find Kiwi. Let's. Do one last one if not I'm gonna find my favorite rabbit anyone. Lima. Oh hello ima. LEM. You, are sorry. Who. Okay we did find some Lima right. And so, once. This is done we. Can click on each one of this and when we could predict.
Anyone, Of this what. Is happening is it sends to a back-end API that we have constructed and you. See that now we identify, the bounding, box around the lima and. More importantly, we tell you this could be a koala because. The. Algorithms, not prefer it could be a monkey it. Could be a long year our it could be a baboon and it, could be a spider monkey right, and the reason why we continuously, work with organizations. Like for example I natural. Is is this. Is gonna get better right we're not saying this is perfect and that's the reason why we did not launch this as a product but, this is a species, classification, API, that, today after this conference if you're interested, to be against drop, us an email and we can actually make it available to you as. A way for you to play with stars right. But more importantly, this allows you to then. Be. Able to do species classification, very, quickly and so, to show you how it looks like. Any. One of you familiar cognitive. Services. Are. Your view some of it yeah so cognitive services is really a cell pre-trained, models that Microsoft has. Rep, around, api's, restful, api is that you can access now. The same species recognition. API that you see here it's also rap as well and therefore if you download any of the sample code for cognitive service, and if, you change it to this AI 4 of API URL. With. A subscription key, now. You can suddenly now detect. And immerse. In the programs that you write right, and so you could create other, kinds of like AI naturalist, and so on and so forth so this is going, to read all from a foul call, demos. AI for Earth pick, number 2 and let's see what it is. So. This is a picture of zebras, that was taken at, one of the zoo's right. So what we're going to do here is we're. Going to send this for this, picture that you see here through. The air for species classification. API, and if. We go to the API itself let me do a right-click and go to the definition of it you'll, see that I put a few break points, which. Is the point where after we send it to an API it comes back with ourselves results, and, let me try to run this, so. Right now what is happening, is we're. Invoking API, just that how you invoke, any of this cognitive services is, coming. Back with a set of results and it's, sorted, based, on the most probable, animal, or species, that you have seen right, and we also give you other suggestions, as well so, if I step through this. You'll. See that we have identified a top species, as the. One that is highest in the enumeration, and. If you look at the result we think it is a zebra. But. At the same time this, enumeration, that comes back also shows you the top five results so, if I continue running this you'll, see that while we think it's a zebra that's the most highest, probability animal. Or species that were found it, could be a Mustang. It could be an ostrich, it could be a kudu and the, point is as more data we have from, all this crowdsourcing, effort the, algorithms, will get better right and that's, the reason why we say hey know this, api's are available, today and we really invite everyone of, you to, use the I naturalist, app to, help us continue advance, the state of art in this right. So. With that let's switch on to another important, topic which. Is. Precision. Conservation. And before. We get the precision conservation. And what it is we. Wanted to show you one, of the videos from the customer, that we've been working with for a while Koch has to pee Bay Conservancy, and this. Was really a joint project between Microsoft.
S3, And Chesapeake. Bay Conservancy, and let's roll the video. I've. Been a Waterman, in the Chesapeake Bay since, 1995. When. You're out there on the water every day for that long you see how things have changed. It's. Hard knowing there's nothing you can do but watch. My. Name is Bob Ingersoll and, on the farmer, the. Demand for our kind of work is only going on and, we're constantly looking. At how to expand, our yield without, stripping the land for, polluting the bay. The. Chesapeake Conservancy has, been focused on creating tools that. Help answer some of these questions in, the. Infancy of the Chesapeake, Bay program scientists. Built a scale physical, model of the bay to understand, how processes worked, and to simulate potential. Solutions, a lot. Has changed since then and technology. Has been the catalyst, the. Chesapeake. Conservancy has, been a pioneer, in the field of precision, conservation. Getting, the right practices, in the right places but it hasn't always been easy, until. Recently, land cover data was only available at, 30 meters resolution, and represented. What the landscape looked like seven years ago not. Great for precision planning, we. Raised the support and spent 18 months working, with our partners to create a one meter land cover database for the Chesapeake Bay Program this. Unprecedented, project. Took a lot of effort in massive, computing, power now. We are working with Microsoft and using, AI and, deep learning to accelerate, our work both, in the Chesapeake, and across the country our. Collaboration. Is aimed at providing partners, with the information, they need to make informed decisions. Now. So one, of the things that you'll see, that. Is enabled, by all this is, really this concept called land cover mapping to, dis to it out into a simplest, term it, is you have all this satellite, imagery, and. Today there's, a lot of manual efforts by expert, right. Sounds, familiar remember, the snow leopard story earlier, there's. A lot of this manual efforts by expert, to understand, hey you know this part of the land is really a road this, for the land is really vegetation, this, way where all the water area is and so on and so forth and so, in order to be able to do this more efficiently AI, can help and to the AI has, tremendous. Capabilities. To be able to use deep learning algorithms. To understand, the respective, pictures in, a satellite image and, most of this satellite image are for Ben which is RGB, and near-infrared. And, so with all this information you can actually use a lot, of Technology, and AI to, be able to understand each of the every. Single pixel right. And so. This, was really the case when. We started working with Chesapeake, Bay is they're. Trying to produce about, all, trying to analyze about 100, K Mouse or square, miles of their, what is. Known as a watershed area, is about 2 terabytes, of information, it, took them 18, months, by experts, to manually, create the map now, by the time they finish this in December 2016. The map is outdated because Len. Has continued, to evolve and, so we think hey that's a better way to do it right. And so one of the things that we did with them is really, to build a land cover mapping, solution I wish. They don't we're going to show you in, a while but, the point here is being able to use algorithms. To process, all the satellite, imagery, at an amazing, speed so. That it doesn't take months, but. Rather it takes days or even hours to. Produce a fully automated, map of not, just Chesapeake, Bay the whole of the United States and of course our aspiration. Is eventually, to be able to automate, that land cover for the rest of the world we're not there yet, this, is something that we're working towards, now, if you take. All this information. I'm sharing with you right now and distill.
It To it's very core, element. This is essentially a very exciting, area which. Has both. AI, for good interests, you, also have, interesting. Commercial, interests as well and so this space is really to say well you have all this data this. Data arjo, spatial, data. You. Have, massive. Amount. You available, in a cloud and you. Have AI, algorithms, if you combine all of this together, you. Will be able to do amazing, things and of course here shows you a project lifecycle of what. Is happening under the hood and so s3 and Microsoft, together we, have been thinking deeply about this problem and more, importantly making this available, to you to get started right and therefore we did I'm going to invite rags to, show you show, you some of the work there doing together, on trying, to use deep learning a. VM. In the on Azure we should call the Jo a aja. VM. Right. Altogether, in order to help you do, land cover mapping in this case this was a use, case that we did with Noah and I'm gonna hand it over to Rex to share with you. Great. Thanks we. Great. So some of the some. Of the knowledge that we gained when working on the Chesapeake Bay can just a Conservancy project, and working with Microsoft and some of the technology, around. That we've, been able to take some of that information and apply it to a sort of a broader extent and so, part of that effort is part of this deep learning a lanky for classification effort, with Noah working. With artists, ArcGIS the artist platform which is a ezra's platform, for. Working with GIS data and. With Azure, and working on some of the azure capabilities, that are available as, part of Microsoft's, platform. So. Just to start I want, to take a look at a problem an. Issue that a problem that Noah has right now where they. They want to be able to take existing, imagery, that they have available and they, want to want to take that imagery and they want to be able to provide a mechanism or currently.
They're Using, a mechanism to sort. Of manually and through a variety of different mechanisms. Variety, different workflows create. A land cover mapping, and so basically. We, needed a study area to talk to, target and so we started with a relatively, large region, well over 2,000, square miles this, is gnomish County just north of Seattle. They're. Coming with a variety of. Basically. Nationwide high-resolution, imagery at 1 meter resolution. NAIP. Imagery so 4-band imagery as we talked about earlier. This is imagery that's available for download today, from. NOAA's website and. So they want to be able to take this, imagery. And. They, want to be able to optimize the process for land cover for defining land cover classifications, currently. They have a process for labeling. Billions of pixels within this, study area and this would be just just over 11 billion pixels, using. An ensemble, of different techniques right to produce this land cover map some. Of this process is automated some of this is manual and generally, takes weeks actually for, this size of study area to come up with a link of a mapping solution to, come up with basically, an image or a raster with labeled pixels that, define really 60 an output glasses that, they can use than, in further analysis, what. They're looking to be able to do is to, optimize and, to reduce the amount of time to come up with a land, cover map. And so. This, is input data in our study area, with NOAA and we, want to be able to take a look at the approach so how can we tackle this with, deep learning so. The the approach that were gonna take here is working with a deep fully convolutional, neural network right a CNN where you CNN in future, slides here to reference, this process. And, we'll be able to segment this imagery and utilize. Our ground truth basically, the the. Labeled pixels that NOAA uses today that, take a couple of weeks to come up with I want to use that as really a baseline, and so, we want to be able to build, out a network. Using deep learning capabilities. To. Create. A model that we can use to create or to basically, accelerate, this process so. That the it's not a manual or a combination of automated and manual processes, and that we can generate a result much faster, so, there's a variety of different techniques. That we're using here one. Is a residual, unit architecture, that, we're leveraging Microsoft, Research in this case this. Is a the. Same type of technique that's, used in, medical imaging it's, good to find cancer and. Also be good to classify. Land-cover so. It's a very, rich. Model or rich mechanism. For using, using. Within a deep learning process, we've, got a set of different connections we want to make here when we're training our model I wanta, be able to add those together so that we can supplement this model as we train it we've, got a variety of different layers that we're bringing in as part, of convolution, so we're not losing any data this is a lossless process, right.
So The number of input out input pixels and out provincials are the same, the. Goal here is that we want it to be faster if we don't want to take weeks want to take much less than that actually, and we'd like to we'd like to optimize the amount of time it takes to work in this case. We. Also want more, efficient, use of resources and so instead of taking both, physical, and virtual resources, want to be able to optimize the use of resources and cost associated with that too ideally, we'd like the results, to be just. As accurate if not better and so we want to be able to take that into, consideration, the. Key here is that, this, is all being done on AGOA, IVM. That. Is hosted as part of a Chiron the azure plan. Includes. ESRI technology, as for as well as a variety of other different technologies, for building out your building. Out your your. Model training, your model and then viewing the results we'll, take a look a closer look at that here in a little bit. So. What were the results just really quickly what were the results that we saw well if we're using the input imagery, the NAIP imagery that's coming from NOAA we're, using their ground truth it took a couple weeks to come up with and then, we evaluate that as a baseline and trying to figure if once, we train our model to get it to a point where we have better than ninety percent accuracy, that's our first goal and so we were able to get an overall accuracy of about 92% it took about about. Two to three days to train the model to, get to a point where we had this level of accuracy, this. Was this was this was good enough actually to continue and continue to process and at least reevaluate. Additional, inputs that we had within the model. There, were there were some notable, differences here that we need to tackle going. Forward but this is an initial initial, review of being. Able to evaluate and use these inputs to determine or to come up with a model that we could use there. Were some confusion, here between different type of classification types, just forests and scrub some. Of the roads that were covered by, forested, areas but vegetation, that were difficult the model wasn't, able to to determine that quite, so easily, but, one thing to keep in mind here is that there were errors in the ground truth right the baseline there were some errors there too so. We need to account for and consider that as well so. With that let's go ahead and take a look at some of the results here initially, this, is gonna animate through really. Just the the two inputs and. Then the output from the model so we have aerial imagery ground. Truth which, is the labeled labeled, pixels, from. NOAA's current process and then we also have the CNN classified. Output which is the output from a model. So. In some cases we had some good agreement between these two right the ground truth and the CNN model actually came up quite well this is a subdivision we. Can see some of the forested areas there in green some, of the impervious, services, and white and then some open. Areas actually in Brown we can see that there's a pretty good match there in, most cases and so we had some pretty good accuracy there for. Some. For most of the most of the portions. Of the aerial image now, in this case what. We can see is that some. Of the. Open. Areas that, actually have either, open or had scrub. Were. Not classified correctly as part of ground truth and so the. Our. Neural network our deep learning process, our model was able to accurate and more accurately classify, that based upon the results, from the image and using the training that we provided so, this is a better a better scenario here and we can see the difference here between ground truth and classified. There. On the right. There's. Another issue I mentioned before that it was a little more difficult for. The the model itself to recognize roads that were covered by vegetation and so we can see here there's, a road that goes top. To bottom in this image and that the. Ground truth actually sees with. Ground truth we're, taking in a variety different inputs and we can see that that impervious, surface which is white is, fairly a fairly, clean line with. Our model, we're seeing that some of the vegetated areas, that are covered by vegetation are, a little more difficult to see so. Something that we need to take a look at to, further train our model oh, this. Is kind of interesting there. Are a set of piers on a lake on this, area and in. The. Ground. Truth it missed the piers were missed there they weren't considered, part of the land cover with, the with. The model I was able to see those and classify those as structures. So. It was interesting that our model actually detected, that. Also. During, the training of the model we were able to determine or at least filter out different areas that.
That. Were, shadowed but, we were able to determine whether or not there was a shadow based upon the structure within. The image and we were able to accurately classify. This in this case as an open area in. Some, cases though models, were miss miss classified in this case some, of the, shadows. Were misclassified, in, the model and so, in this case the the shadow was, classified as open water and so against maybe some additional training some additional inputs, here to train our model in a better way to, more accurately represent what. We're. Seeing in the imagery. Right. We can see here is that. We. Have some basically. A discrepancy. Here in the type. Of vegetation. Our looking at will scrub and grass and grassland, and, we can see that the. Ground, truth areas. Accurately, classified. This and we need to further train maybe look into other ways of training our model to get better accuracy here - so discrepancy. And. Then one last comparison, here. We. Can see that there's a bit of a sparse forest, within. The the aerial image and. Then as we see the, the. Ground truth compared to the scene. And classified. It. Looks like there's a discrepancy here it's unclear, exactly what. The actual vegetation, is and so being, able to better accurate or more accurately describe this or maybe, get some additional. Ground. Based information to determine a better, model or a better result in this case. What. We found overall though is that the result, of this evaluation. Was pretty good. There's, a couple things that we could use to improve this now as I said it took about. Two. Or three days to actually train the model right to get to a point where we had at least 90% accuracy, once. We had the model in place it took about one to two hours to actually generate. Land. Cover classification, raster, right labeled pixels so, compare, that to two or three weeks one. To two hours, considerably. Less and just, over 90 percent accuracy what. We could do though is we could actually get better input data so. Being able to work with updated. Ground truth maybe. Better input, data to train the models going forward. We. Could also work with other different. Other surfaces. Other. Areas. Or regions that have different types of land cover classification, citizen pervious surfaces, land, cover imagery, that actually includes roads to, better train our models to recognize some, of the other classification, types that it had some difficulty, with. We'd. Like to actually supplement our model probably with a deeper. Understanding of, some of the other structures that, are within the aerial imagery to give it a better understanding, - so there's some some, things that we can do here to actually deliver a really. A optimizer, model for a more, accurate or a better result. So, the key here is that we were able to basically, come, up with a reasonably. Accurate or highly. Accurate result, and. Reduce, the amount of time actually, to deliver this labeled land cover, classification. Raster. Now, one thing I want to highlight here is that although, I walked through really, the process here and some of the results that we saw and we, saw that we actually we were able to optimize. This process and, bring reduce, the amount of resources that were necessary, to, be able to deliver this this. Result I want to highlight the fact that this was all built on top of the. Azure platform and, it was built on top of some.
Of The ESRI technology, that's available to be able to process geographic. Information and imagery so. The one the one item here that we worked with in. This specific, process, was. The, GUI a IVM, this is a data science of PM that's on Azure it's, available today so you can spin one of these up today and on that VM you get a set of applications. And, frameworks, installed, pre-installed, for you and so that you can start working with and building out your models, it's, got, basically. Has a variety. Of different technologies. Including, Python, the. Cognitive toolkit, the. Additional. Ideas. To be able to take advantage of and work with your Python environment to train your model it also includes ArcGIS Pro which. Is Ezra's, desktop product for being able to process, and visualize the geographic, information. Art. Pi is also a technology, that we use to, work with in Python to be able to generate results. In access geographic, data all. This is all this work that we did today was built on top of the cognitive toolkit, microsoft's cognitive toolkit and. All. The results in this case were actually built on on top of this vm that's. Available. Within hazard one. Thing I want to note here is although that was done on a desktop the. Work that was that was achieved here it could also be done through the. Azure batch ai process. Or you, could actually take this model. That you've created and you, could deploy this in a docker container, hosted. In an azure kubernetes, cluster and. Then that model would be available for additional insight and then, be able to access access, that model from another ESRI product artists Enterprise artist, Enterprise is really our server product it's, also available and can be hosted in, Azure. Well. Our Thea's artists Enterprise you can create a service endpoint that enables, access to the model that you've created, you might provide a mechanism for inputting, imagery and generating results land cover classification, results and, making those available to. You through your enterprise, or your. Constituents, so, a variety of different capabilities here, all built on top of a jar all built on top of Microsoft. Technology that Ezra provides for, you to be able to take advantage of some, of these deep learning capabilities, with, that I'm going to hand it back to weave. Who's. Gonna take a look at a demo, of getting started with a geo AVM, so. Before we get into the demo I think, you have heard a lot about some of the work they've been doing on land cover mapping and say hey know what if I want to get started is it like only s3, or Microsoft, can do it or today if we have access meant as your VM can all of us do it that's, one two, is. Even. If we give you a VM you're gonna say hey you know how do you go and go about doing this thing called pixel ever or being able to do land cover mapping, is, it really hot is there. Something that you know tutorials, that's gonna help me get started and so, on and so forth and so I'm going to show you next is really the fact that hey you know today, if you have access to a VM we, actually all can get started, and. So. If you do not have an azure account, I'm, sure everyone will know that hey you know you could get a free account for a limited period of time and once. You have an azure account, and you, log on to the azure product, you. Can do also some wonderful stuff, of Asia but one of the things that you can do is you, can provision, a Joe AI VM so if you search for Joe AI you'll. See this thing here called, a Joe a I data, science, virtual. Machine and one. Of the things that we also realize, is that even if we give you a VM with others deep learning tools that's, on end you. Might say hey you know what if I need a key is right. So between Microsoft, and ESRI we've been spending a lot of time trying. To think about how can we make this experience, frictionless. And so, on this VM once you provision it which we're gonna show you in a while you're, gonna get all this in together, with, all the script and other tutorials. On one of the fouls or one of the folders on the VM and therefore. You, should be able to get started and being able to do, land-cover mapping right and so if I was to start Joe a I did our science, your machine, you read through what it is and, if. You click create, it's. Gonna ask you for some information so, I could say hey you know this is ignite. Joe. A IVM, and.
We'll Give it some name in. This case let's call it AI for us mean. It's. Gonna ask me for some password, I'm gonna copy some password here and, of, course it's gonna say what subscription, you're gonna use it I'm gonna use the AF of engineering, subscription, it's. Gonna say what we saw screw right so in this case I've created one resource group which I call ignite. I think let, me look for it, if. Not I'll create a new resource group okay, let me create a new resource group let's, call this. Ignite. 2018. Demo. -. The. Back-up plan and if, you click OK what is going to happen right now as we offer me live virtual. Machine creation, right. Is it's gonna ask you here you know what, kind of things you do you need in this case because there goes if you're gonna play with the VM you might need to train some deep learning algorithm, so, one of the things that we default to is a, ds4, but, you could then choose nc6, or whatever it's, needed if it's available in the region and once. This is done. Right. I'm gonna click OK, as you. Know move on to the next step to say hey know am I ready to create. Now. Once all this validation, is done and I click create now, in minutes, as we are already familiar with is going. To provision. A virtual. Machine for you in the cloud right and. In this case this is a Joey IBM, and if, I click create the VM gets created, right. And once you have the VM gets created, you, can then RDP, into it and you, see a VM that looks like this right it has all the tools for you to get started with deep learning for you to do data science, but, at the same time all the tutorials are all loaded under the VM and one, of the things that you notice is that gos, is already default, installed, on it and then, once you build the model the tutorials, gonna guide you on how we can consume the model using. ArcGIS, Pro and, with that let, me show you what it looks like. Once, you, start working with this right and with that I'm gonna hand it back to rags, to. Show what is on the VM so. Jo AI data, science virtual machine is a, virtual. Machine that's available today and I. Encourage. Everyone to get started and this comes with all the tools Visual. Studio code data. Ingestion, - connecting. Two different kinds of data platforms. Right, what does cosmos DB over this as your sequel and so on and so forth but it also provides, you with arcgis, pro installed, on the machine ready, for you to get started and let's see what Comanche you can do once. You have all these things installed a goe idea. Great. Thanks me so, we saw there is just a really a practical way for getting started with your, sort of deep learning process, here working with Microsoft technology, building on top of azure but taking advantage of ESRI technology, here when working with your geospatial, data we're. Looking at right now is a this is a Geo AI VM I just created a couple days ago this. Is the same VM that we used to work. Through the the, workflow or the process, that I talked about earlier with Noah alright this is also the same VM we used for Chesapeake Bay right so it's the same technology. The same capabilities, all the tools that are available to you are, already pre-installed and, available in this VM so it's ready for you to go it's, already building on top of a an, NB. Jeep, in NV, VM, in. Azure so, I already, pre enabled. For GPU compute so, you can take advantage of those capabilities right off the bat, what. I'd like to do here is I'm just going to walk through a, couple of the sort. Of the technical, capabilities. That why I talked, about in the NOAA project, some. Of the the. Work that was done here to be able to craft the. Gathering. Of the data the, basically building of the input data sets and then training, the model so, what I've opened up here is Visual Studio code now this is some Python code that we used now, in the actual process itself we ran Python from from. The command line granted, you could you, know you could bring this up you could use this in a variety different IDs, we, happen to do it from the command line I'm bringing up a Visual Studio code here so that you can actually see that this is another mechanism for, visualizing, your, Python code and being able to, manage. Me plate and craft this. Logic now, this is this. Is a Python, script the. Created and his user uses art pipe art pie as part of a Zuri's platform, for being able to access the work with geospatial data what, it's doing is it's taking in NAIP. Imagery from, NOAA it's, also taking. In the ground truth, labeled.
Raster, That we talked about before that they derived after a couple of weeks of work it's, basically. Breaking that up into bashes and it's going to prepare that data for. Training and so. Once. We you have those patches created, we're, gonna bring in another Python, script this actually comes from Microsoft, Research what I talked about before it, brings in some of the technology and the knowledge there and it integrates that within a Python script that allows us to iterate through our training process right. So this training process in this case will. We'll. Use. The. The model, provided, by Microsoft. Research it's also built on top of the cartridge of toolkit Microsoft's, cognitive toolkit and so we'll see a number of different references there and leveraging that for deep learning. Once. This process is running and as I mentioned this takes about 2 to 3 days we're, working on a VM. Here that's got 4 GPUs, right of course if you throw more hardware at, it it's gonna operate, a lot faster we saw that a little bit earlier in Wiis presentation, sort. Of more hardware at some of these processes to reduce the amount of training time of. Course we're looking at cost-effectiveness too right so want to make sure that we, have a nice cost-effective, process here as well so. This is the the script that will be running in Python, as. It's running a log. Is generated, and what I've got here is a tensor. Tensor, board dashboard. To show the results, of that training process over time what, we can see here is that we. Want to be able to see how well these predicted, values the percentage of those predicted values are, correctly predicted, over time and so as we get to a point here where our metrics looking. At close, to what, below below. 10% basically. Here so we're getting over 90%, accuracy. We're, looking at a good model something that we want to work with right so there's one way of keeping, track of as your models working and processing, you can actually see how it's progressing and how well it's how well it's doing of.
Course If it wasn't progressing well we can stop it we could readjust some of the variables and we could done we. Could make make changes as needed now. As, I mentioned this took a couple of days but once we had an output what, I'd like to do what I could do at that point is I could then visualize, this information, within Arches, Pro I'll. Do next is I'll bring up our Chia's Pro this preinstalled this. Is just an example obviously. It's a professional. GIS desktop application. So it has the ability to work with a variety of different data formats what. I'm showing here is sort, of a practical look at some of the inputs that were used this, first layer here that we're looking at right now is one of the nape images or the four-band images that NOAA, provides today. And. I can see this is an area here that we covered actually in the slides earlier what had asset appears you can see them off to the right if. I hide, that we can take a look at the image. That, was created, so. This is the ground truth labeled. Pixels. That was created by no over two to three weeks period we, can see the the. Different labels, applied to different areas. Based upon their manual automated process, and then, we want to be able to, really. Evaluate and, compare, how this looks with in. This case a. Tiff. This. Is the output from, our from, our evaluation, when, using the model so once we evaluate use. The model we input, aerial, imagery we. Output or we generate, a. New. A. New new raster with. Labeled pixels so that's our land cover classifications. From our model and we can see that this is another raster we've not lost any resolution. Here it's the same number of pixels we, just have different classifications. And so if I tick this on and off we, can see here is right now we're looking at ground truth, right. If I turn off the ground truth now we're looking at the the result of. Our evaluation, using our model right and. So we can see this very close our relationships, here now, what's interesting is this was just for Snohomish County. We. Took that model and we applied it to a number of other counties and, if, I flip over to a map here so Snohomish, is just north, of just north of Seattle we've, added in another account a number, of other counties here such as island in King County now. I said it took about 90, minutes issued, it to complete, this analysis, for Snohomish, it, was about a one, to two hours for the other counties too so you can imagine you know if each of those counties took 2-3 weeks previously. Well now you're looking at 1 to 2 hours so something that might have taken you, know almost two months now it takes a day right. So we can see there is that if you, know if we're getting a reasonable amount of accuracy and, this actually is able to use be able to use much, more efficiently. Then you can see that being able to determine land cover much. More accurately over a shorter, period of time be able to evaluate change detection other, evaluations, that go on with land cover land. Cover value land cover classification, evaluation, you can see that this reduces, the amount of time it takes to actually generate results, that are usable going. Forward with. That I'll hand it back to we thank, you. So. Thanks a lot Rex, now. At this point every. Single thing that we shot you from. Being able to do land cover mapping, the work that we had with Chesapeake, Bay to, the point where you can actually do exactly, what we showed you on stage this, is all available on the Jo AI VM and we encourage, everyone to get started, now, the question I wanted to get to now is well we have seen all this and if. We say well we have all the satellite imagery, how. Fast do you think and you do land cover mapping, for. All the data that's in the auto satellite, imagery which, we call name imagery, right across the whole the United States how fast do you think we can do land cover mapping remember early on it was eighteen months we. Show you some of this this might takes several. Hours to train but. If once, we have trained the model and we want to apply this model what. Is the, state of art and how far can we push the limit how much so, just to give you some data points, data. For the whole of the United States from a satellite imagery perspective, ranges. From about 20 terabytes to, about 40 terabytes depending, on how you cut the data question. To you is how, fast do you think an do inference on a and do line cover maybe, one. Minute. We. Are working, toward Ed as well what. Other. Guesses. Days. One. Day that's, pretty, good, estimate, thank. You now. And one, thing that you've been hearing about right since, the big conference this year is we, are now some of the work they've been doing on this project called project with brain wave which. Is the ability for you to then take all these deep learning models, and get, them onto the FPGAs. And epi, jess is going to n able you to be able to do inference really quickly and, therefore we took the same piece of work that you saw, earlier right and we started applying it, using.
Project Brain wave and we load the mod on the project brainwave and. It, took ten minutes to. Process about 20, terabytes of data which is about 200 million images right. And this is how the architecture looks like. Now. What, is also interesting is this what is reproducible. If you had any imagery right. If, you go to the URL I'm, going to share with you in the previous slide. You'll. See a kms, yeah, for a brainwave, land-cover. Mapping, if you go there again will provide you with all the technical information on how we can reproduce this on using. The FPGA that's already available on Asha right. So that's pretty exciting. Now. The next thing that will not show before we end of length of the mapping is really some of the work that we've been doing. Experimental. Projects, internally that we have been playing with with some of the garage team and this is work done with the vancouver. Garage and. Remember. We started off this session by saying hey a I and, asked me in technologies, right, we're able to use both AI and technology advanced and solve, some of these toughest challenges, or it, fixes that little. Sprinkle. Of creativity. On it so what can you do given all this power on your hand given. All the satellite imagery that you have what other things can you do with it so the next video, I'm gonna show really a short one it's, really some work they've been doing a really early experimental. Project to say well if you have satellite, imagery, I could. Do an app like this and what does this app do is. It. Can look at satellite imagery and today it's a image or it could look something like this, and. Then. Now you're able to use it to be able to detect not. Just the animals that's on the ground you could use it to detect the boats you could use it to detect the roads and so on and so forth and this, has tremendous. Impact. Especially. Around humanitarian. Needs right. Today after, one of this fury can heal one, of the things that you really need to do is to go in and figure out the damage at the area right. And so technologies, like this is now doable and more importantly, what you're seeing is an, early prototype that we view that, allows you to then deploy, them or they're not in the cloud because if you go to a lot of these areas where, a hurricane, hits or some situation, or curse, Wi-Fi. Is not a given, and therefore, you want models to run on the intelligent, edge and this, was just an early prototype that we have you just to illustrate there, and this was built by a team of about, six interns from the Vancouver garage and these, are just the possibilities, and the point I'm showing you this, not. Because, this project is interesting. It's. The fact that all of us we build applications we do web applications, with you iOS will, be Android, and all kinds of application, now, today, together as, a community we, can actually view applications, like this and make this available, right and now we can truly use AI and technology, to advance a lot of this needs right and you say I feel good so. That was a quick Pig let, me quickly switch, gear to thumbies if. You've been to the expo you've seen a lot of information, about fun bits just. To provide you of a little bit of data point, if. You look at some of the data from say the UN by. 2050. One of the things that, we know is that the demand for food is, expected. To outpace, production. By. Over 70%, and. Ever, we, need to empower farmers with, the ability, to have, of course more data but, more importantly, to process all this data that the farmer has and the, goal is the following to maximize, that efficiency right. And also maximize, that you so. For example we work with farmers in a state, called Andhra Pradesh in India, and. They are ground up farmers and we. Had a control group that has no machine learning algorithms, and one, that is empowered, by machine, learning algorithms, so one of the things you might be thinking, about is well, most of the farmers doesn't have access to a PC yeah another log on and say hey should I do this or dead right not possible, but one of the things that the farmer has is a mobile phone and so, for the group that is empowered, by the machine learning algorithms, we, will give them short messages, to say hey this is the time you should prepare the land this is the time you should so this, is the time you should have this and these are the things you should do right, and we, did this comparison, about one and a half years ago with, this group of ground that farmers, and what happened after the project in, fact we took a long time to collect the final.
Results Just because you need to wait for the ground nuts to dry before. You know that you just. The, group that uses, machine, learning, has, a 30%. Improvement. In. Crop you and we, also duplicated, the same thing, for carrots, in one of the thumb up. In connection in Washington State and again, they see a similar, improvement, in you right, and therefore fun bits is really a project that, and how, was farmers to do all of this by doing two things one. By providing. Them with low-cost, sensors. But. Not more, most importantly, telling. Them where they should see to it the sensor so that they get the best representative. Coverage for the farm whether it's more fun big farms or possible. In fact we've been looking. At even oil plantations, as well and so. Fun bits is really a project that was born out of that need right, one is today. You could use drones and you fly it across, acres, and acres of farmland for big farms and, you get, satellite. Imagery not Saturday I mean your drone imagery, of the, farmland so you know the you, know the. Terrain. Of the land and so on and so forth and then, if you fuse this data with. The data that's coming from the sensors, it could be temperature it could be humidity. It could be Neutron it could be wind speed and so on and so forth you create, a heat map and we all familiar of heat map for enterprise, use cases this, will become a heat map for the land and this, heat map is going to help the farmer exactly, which, other areas of the land that needs more attention which. Are the areas of the land right, that they should you know do something about it and so on so now farm bills right now is a project not, just a Microsoft research project is ready for use by, everyone in fact we. Have been, working with. Folks. Globally. On deploying, thumbies, and therefore, if you own a farm like I was talking to some of you, during lunch. Contact. Us if you think that farm bits can help you right, in trying to use, AI to. Do, precision. Farming. So. This just shows an architecture, diagram for farm bits, which. Is a combination of drones some, of the advancement, that we have in the networking, space and so on and so forth, and this. Was just a, deployment. That we had in upstate New York as well as in Washingt
2018-10-31