Using pre-built AI to solve business challenges | AIML20
All. Right good morning everybody Walker, has everybody doing today, yay. Awesome. I'm, David Smith I am one of the cloud advocates, from Microsoft, and this, is the second talk in the artificial. Intelligence, and machine learning learning, path now, who was in this room just a few minutes ago for Seth's first talk in this series awesome. Great talk wasn't it I've got some bad news I'm not nearly as funny as if I'm sorry, but. We are going to cover some really cool stuff today. We're. Going to actually be wading into those waters of artificial, intelligence at about knee height for, this session we're. Going to be using pre-built. Artificial. Intelligence, services, that you can plug into your applications. Without any. Need to understand, the, the. The mathematics, behind artificial. Intelligence, but I am going to tell you a little bit about those mathematics, because it is useful to understand. How, they can go wrong and more importantly, how, you can make them go right. Now. There are going to be lots, and lots of links and information, and, resources, and demos, in this talk today but. Don't worry everything. You need to recreate, everything, that I do is available. Online particularly. At the github repository that, you can see at the second link now right there the. Demos I will be giving are going to be pre-recorded and I'll run you through them as you go you'll, understand why they're pre-recorded, if you've tried to use the Wi-Fi here this week, but. Like I said you'll be able to actually find, all the data and, all the scripts, and even, a single click to deploy to as your button for the entire application I'll be using so, you can verify that everything I'm doing you, can do yourselves as well I'll. Show this slide again at the end of the presentation, so if you've missed it now don't worry it's going to come back. So. In this talk I'm going, to be using those pre-built, artificial intelligence services, dad, human-like. Capabilities. To applications. But. What does human-like, mean and, there. Are lots of things that you could do to add applications. To your apps that, have like, the things that humans can do for, example being, able to add the power of speech to applications. Being. Able to make your application, see, so you could do things like being able to search for photos or search photos by content, you. Could give your applications, human-like. Intuition. In a way that they could automatically. Rearrange. Elements, of user interfaces, to, meet expected, human, needs. You. Could give your app the power of speech to, be able to translate text or speech and even in other people's languages, and international. Internationalize. Your applications. Like that or. You, could automate, the human process, of looking at data rather. Than having a human look, at a secret a sequence of numbers and trying to identify spikes. Or outliers you could have a machine do they process and then scale it up to really really huge volumes as well so, just some examples, of the human-like things that you could do with applications. But, does it take a lot of expertise, to, implement, these kinds of things and what, I hope to show you in this presentation today is if that answer is no because. We have in Asia as your, cognitive, services that. Provides simple, API calls, that you can plug into any application, to put these human-like capabilities. Into your applications. And. Repugnant, in. Two. Dozen api's, but. At the broadest level they offer capabilities. Within these categories, of human-like capability.
Vision. So that's all about understanding the continent of photographs, and drawings text. And hand writings even video. Looking. At speech so, we provide tools that enable you to recognize. Speech and generate natural, human-like. Spoken, voice even in your own voice if you wanted to do that in. The language side being up understand, the content, and the. The. Content of written documents, and text and also, translate, between human, languages, we saw a lot of that in Seth's talk in the previous session. Decision. Now that's a brand new category for Azure cognitive services which. Is all about making human-like. Choices, about data contents. And application, user interfaces, we're going to be going for that into that one in a bit of detail today, and. Finally this search which as you saw in Seth took just a moment ago which. Is all about being able to understand, and answer, natural, language questions about the content of large unstructured. Data stores. So. As I said the search category, we covered in the previous talk, but in this particular talk we're going to cover just three of those cognitive services. That's, computer, vision which is all about how. We're going to use these, AI services, to recognize, the content of a photograph, for, a product based application. And then, we're going to enhance that with something called computer, vision so we can enhance that customer, vision to recognize, specific, objects, that. Our retailer, sells and we'll, be using the brand new personalized, a service, that just went into availability. This week which, will automatically, adapt the layout of a unit user interface, by, observing, customer, preferences, that activity, and present, the best product, that any user, might be most likely to click at even, if they're an anonymous, user but. Remember, for across all of, the azure cognitive. Services the principles, of setting things up and using cognitive services are the same for all the AP is so. What you learn in this session today you'll be able to apply to any of the cognitive services API, as you see listed there. So. First we're going to take a look at pre-built. AI for, computer, vision and how, we can use, cognitive. Services to give an application the ability to see and then, customize, it for our particular needs and. We're going to do that in the context, of tailwind. Traders now. You might have seen tailored traders in other presentations, this week it's a fictitious retailer we developed, to, give some story behind these applications. Detail, when traders website, is like any e-commerce, retail. Website it's got the ability to search for products, make. Purchases, and. It's got one particular AI feature, that we're going to enhance in here as well. Once again I even, though this is all fictitious I'm, going to give you all the source code behind everything I show you here at the link at the end of this slide so you can duplicate this yourself. So. Let's go live, to the Te'o and traitors website and one. Of the hey I enabled, features that we have in here is this shot by photo feature that you can see in the center and this. Feature allows the customer, to upload a photo of a product they might want to purchase and the idea of the Apple then tell them if it's available and where. To find it so go ahead and upload this photo. Real and you, can see it's being uploaded the search now the, application recognizes. That was a photo of drill a drill and tells, us we have this drill in stock and also, where to find it in the store.
Now. Let's go ahead and try that same service again with a different picture click on the upload a photo button right there and this. Time let's choose that picture of a pair of pliers there on the right hand side we'll. Upload that to the website and it. Hasn't quite worked because it, has recognized, that particular picture of pliers as a hammer, which is not what we wants it to do so. What we're gonna do now is dive, in a little bit to figure out what went wrong there and then show you how you can fix it so. To. Do that it's going to be helpful to dive in to a little bit that of that theory I mentioned, to understand, how computer, vision works don't. Worry there's not gonna be very much math in this in fact very little at all but. Understanding, a little bit about how computer, vision works is gonna help us understand, what went wrong just there. Now. Not long ago, being. Able to give a computer, a photograph, and then, it being able to give us useful information about, what was in that picture, I mean that was literally the stuff of science fiction you would see it on Star Trek. But. This xkcd. Comic was, published, almost, exactly five years ago in September 2014, and now. Just five years later a computer, can easily, tell whether. A photo is of a bird or any other kind of an animal thanks. To the advent of big data the. GPU computing, and, convolutional. Neural networks, and that's, what I want to show you now. This. Explanation. I'm about to give is adapted, with permission from Brandon, rora Brandon's. Got a really excellent blog that. Has a bunch of video tutorials, with. In-depth, explanations, of many aspects, of AI and machine learning do. Check out Brandis blog for all the details because I'm only going to be able to give a quick sketch in the time I've got right here. So. You've all probably heard that, AI, is powered, by something called deep, learning now. That deep in deep learning that does not mean profound, or anything, that all. It means is, that we, take an image and we pass it through a neural network that's what you see on the screen like they're right there with many layers so, the deep just means deep, numbers, of layers in a neural network and that's it, now. What we have here though is a very, very simple neural network it, has only five layers as you can see right there and. Real world vision systems, will have you, know many dozens maybe hundreds. Of layers and this. One that you see on the screen is designed to take an image as input, and then. Classify, that image is one of exactly, four objects, either, dog. Bicycle. Apple. Or tennis, ball and that's, it. It can't tell any other types of objects except the one that has been designed to detect. So. When in Europe neural, network has been trained and we'll talk about what that means in a second, it passes. The input image through. The network layer by layer at each layer transforming. The image into different, smaller images. Now. Each layer recombines. The, images generated by the prior layer and then the images get smaller, and smaller until at the very end they, just become a single, pixel with, a value between 0, and 1 and that. Number between 0, & 1 represents. The confidence, that the neural network has detected. That particular object, that's labeled there, on that right-hand side and the more confident, is the higher that number is closer to 1 so. In this case you can see we've inputted a picture of a bicycle, on the left-hand, side of the picture it passes, through all those nodes to the far end until. The number for the node bicycle, the right-hand side is 0.93, that's. The highest number so, this particular neural network, has correctly, identified, that that is a bicycle, those input there on the left. But. How do you train, a neural network you know how do you figure out what. To do in each of those stages to get those numbers there at the end. The. Trick is that each of those round circles, those, nodes in the neural network that's, basically just a filter you. Know same kind of idea as a snapchat filter, or an Instagram filter but instead of you, know making the image sepia, or putting bunny ears and all the faces it's doing something very different and if I think something did that we don't even know ahead of time it, actually learns what, kind of transformation to make in such, a way that the label is there on the right hand side give the right answers, through, that training process so, let's see what that is.
It's. All done through these filtering so, let's start with a simple, image so this might have been the image of a bicycle, but just to make things simple let's, let's do this very simple image of a cross it's. Nine pixels by nine pixels and the white pixels are the plus ones and the black pixels or minus ones so. We're going to apply a filter, to that image just like what happens, in each, of the nodes of a neural network. So. To transform that image we're going to apply a three by three grid of weights and that represents, our filter and actually, small grids like this you know three by three or five by five that's pretty typical for computer, vision systems, the, only difference is that in real computer vision systems the weights on round numbers, like this minus one and plus one that I have in this grid they'll, typically be numbers in that range but they look pretty random. So. To apply that, grid of weights to this particular image of a cross we, overlay, the grid of the weights centered, on a particular pixel, in the image and, then, we multiply each. Of the weights by. Each of the pixel value values, and take, the average and, that average. Then, becomes the pixel, in the transformed, image of the next layer down in, the. Neural, network pretty simple right, one. Thing you've probably noticed is that we can't use. The, very edges, of the source image because the weights grid wouldn't overlay without missing, the edges of the picture and that's, one of the reasons why the picture, gets smaller and smaller as we go through the network and there are the reasons as well but that's, basically why we get smaller images as we go down. Here's. Just another example so. I've moved that grid of whites to pixels to the right and to pixels down overlaid. It over the images again now. When we multiply the weights, by the pixel value so we get a different, average. This times 0.5, 5 so that would mean in our output, image we'd get a gray pixel, and, then we will repeat that process over all that entire image and for, all the copies of the image that flow, through the neural network now.
By The way this process of sweeping, the, weights across. The image robe, over and column by column there's. A fancy mathematical, term for that called convolution, but. It's a really simple process as we've just seen but this is why they're called convolutional. Neural networks. So. Now that we know that, each of those nodes each of those circles, in. The neural network is a transformation. Of its input images and that, in turn is determined, by a grid of weights and so. The trick therefore. To train in a neural network to recognize, objects, is to. Pick those weights in such, a way that the right numbers, come out the end and the, way we do that is with training, data, we. Provide this, neural network with, lots and lots of images in our case lots. Of images of dogs lots of images of bicycles, lots, of images, of apples and lots, of images and texts of tennis balls and we know ahead of time what. Each of those images represent. Because a human, has looked at them and labeled. Them like that and, then. All we need to do all we need to do is pass. All of those images that we know what they are through the neural network and then, choose the all those weights in such, a way that the right numbers, curl, the right labels come out with the highest number at the end each time often. That's, not possible but, at least we can choose weights so that the images at the end are labeled correctly most. Of the time and that's the training process. Now. In the real world. There might be millions, of weights to choose and there, might be millions of labeled images to process, through that. Neural, network so how do we compute, all of those weights and. This. Is the points where you know much talks on machine learning and most books on machine learning will start diving straight into the mathematics, this. Is where we'd start talking about things like back propagation, and learning rate and cost function but, unless. You're an AI researcher. You, can ignore all of that for. A couple of reasons first. If you, did want to actually dive down into this math there are lots and lots of great tools that will help you do those computations, and. They'll do that by taking advantage of, big computing, world's resources, and things like you compute processes, you, might have heard of things like of tools like tensorflow or pi torch that, can do those kinds of thing I'm, not gonna talk about those but. The, AI ml 40 talk later in the day by re Bronstein, will so check that out if that's what you're interested in. Also. Even if you want to make of it make use of all those tools you, would, need some good computing, resources available, to you as I just mentioned, you'd, also need a team of AI researchers. Who could sort of make use of them all but. Instead of setting, up that team and having all those resources, you, could just make the use of somebody, else's research department, that.
Has Spent a lot of time and a lot of expertise, and a lot of data to, train up and you will network just like we saw and then you could just use that yourself and that's, exactly what we've done with, cognitive services computer, vision and made that available to you through an API so with, that you get access to a pre trained convolutional. Neural network, so all the weights have already been determined for you so, all you need to do is pass in your image to the neural network or in our case through, the API it. Will do all those computations for, you and then, give you the value that, represents that particular image, and, we can even get more complicated, in that even. Go beyond just classifying. An image as one object, type we, can actually do things like detect multiple. Objects in a single image and even tell you where they are and that's available for you as well so. Now. Let's actually try, out one. Of these pre trained AI models, with cognitive services computer, vision so. This is a service that will take an image, that you provide you know either as an image file or a URL directly to an image on the web and then, it will give back to you tags, or, classifications. For, the objects that it detects and those. Are the labels that. We saw in the neural network you saw a minute a minute ago associated. With the most probable, objects, that it's a text. But. Instead of doing this if you're, using a powerful new or network that we've developed at Microsoft, they can classify thousands. Upon thousands of objects, there's. A really simple web-based user interface you can use to try it out and I'm going to show it to you just in a moment and academic. / try - computer vision, we're. Going to start with that that's just an easy way to see how it works but, then we're gonna dive in to have a look at it doing it programmatically. Alright. So this is the cognitive, services computer, vision webpage so, if you go to that page and just scroll down a little bit you, can see there's this nice web-based, form where you can upload an image for analysis either, from the web or as a file so. I'm going to click on the Browse button there to choose a file. From my desktop and, choose that picture, of a, man in a hard hat and once. It's analyzed that image which just takes a few milliseconds. It. Will come back and. Show us the object detected, in the image on the left hand side so there's two objects there the purple one on the blue one but, on the right is the stuff that we actually want to get access to which is the JSON output with, a detailed analysis, that, gives us things like all the names and the locations, the objects detected in the image a list. Of tags or, labels of, things associated with that image and also, things like a praying plain language description, of the image which is useful for say photo apps this, one comes back with a man wearing a helmet and there's lots of other useful information that comes out of that as well. So. We can see here in this screenshot of. The object section of the output that two, objects, have been detected in this particular image. We've, got, head. Where and, person. So you can see that right up there at the top. We're. More interested in the tag section, which provides a classification, for the overall image which is what we need for our app coming up in a minute along. With the associated confidence, scores those numbers from the right hand of the neural network, the. First thing it detects is man but, we're looking for product so let's look a bit further down but, the second, image it classifies, it is at is, headdress. And that's. Not really what we're looking for because we're building an app for a hardware store which does not sell headdresses. As. It turns out this particular API, is not not, trained, to detect hard, hats it does detect helmet, but not hard hat but, remember, a neural network can only detect those objects that has been trained for in the first place so, it doesn't work for us in this particular case now.
We've. Done, this but just by using a web form and uploading a picture and looking at the results on the screen but. If you want to embed this, capability, into an app you, want to do it programmatically. And so let's look at how we can do that now. So. You can interface, into. The cognitive services API is using. Any language, that can connect to an HTTP, endpoint but. What I have here is a batch script, that. I that I use to connect to the azure CLI, to. Create some resources, and the keys and things that I need to access it and then connect, to the computer vision API using, curl, now. Of course you can install the azure CLI, in your local shell but here I'm using the Azure account extension. In Visual Studio code, to. Launch a cloud shell which, means I hacked in so anything on the local environment but. Once that shows ready I can actually execute. Commands, like this directly, from the script, so. With this command I'm, just creating a resource group which are going to hold the used to hold the keys which I need to access cognitive, services and, now. With a Z cognitive, services I'm generating, those keys so it's got to run that code directly in the cloud shell and. Now. I just need to display those keys so. I can get access to them and as, Seth said I've been validated all these keys so no need to copy those down. And. Let's. Just copy that first key you can use either those those two keys and I'm, gonna take that key and put it into an environment, variable for use in the next part of the script I also. Need the end point where. I can access the. Cognitive, service online and that's actually provided, to us by the service itself. And. That depends on which region you're running it in but. Now I can grab the URL of an image and that's, the same image, of, the man in the hard hat we saw just a moment ago so. I'm just grabbing it off the web and set it off of a file this time and. Then. Passing, the key, to. The endpoint with. The, URL, of the image as a JSON, string and then. Having a look at the results that we get back from curl and. You can see all we have there and that's exactly the same output that you saw in the web interface a moment, ago were. The first two things were detected, were man and head dress, now. You can see them, so. Let's actually try that again just with a different image because we're doing things, programmatically, this time, so. All I need to do is change my environment variable for the image, to. A different image and in this case we're. Going to point to a web, version of. That picture, of a, pair of pliers, so, now of the drill, and. Now. Let's pass that same that different image into the same code to run, the cognitive services on that new image and if, we just scroll up to the top to have a look at this output here. You'll. See what the cognitive services. API. Returned for us there. We go and looking. At the tags of the image you can see the first one that has classified, for us is camera, and that's also incorrect we wanted to gabbed rule back from that so we could search for it in the API, so. What went wrong there and how. Can we fix it. Long. Story short the basic problem is is that using too powerful, and your network it can detect too many different, kinds of objects and it, can mistake one thing for another, but, in our particular case we just want to build an app that can recognize the specific objects.
That Tear win traitors cells. Let's. Say was only five different objects, so. Let's train a specific. Neural network the ingest event to detect, those kinds of things and won't, get things mixed up. Fortunately. We, can fix it using. Neural, networks once again. So. What if I told you that, there's a way that we can start with a vision model that's been trained for many many thousands, of images just as we store but. Then adapt it to, identify. Just those objects that you're interested in for your application. Even. If those objects. Weren't. Part of the original training, even if the original, neural. Network wasn't even able to detect those kinds of objects there might be something completely unique like. Your own brand or a brand new product you. Can actually train one of these computer vision systems to detect even, those let's. See how it works. Let's. Go back to the neural networks we've, got the same. Volution, or neural, network from before but, something is different as you can see we've, stripped up that final layer so, that final layer where the object classifications. Has gone, and. What we're left with is just the images, that were, generated at that second-to-last, layer I've labeled them f1 f2 f3 and so forth. Let's. Even forget the fact of their images you know rather than being three by three images by the time we got down to there let's just think of that of data as data nine, data, points. So. What we can do is, feed. A new image there on the left hand side do, all the same, calculations. With the same weights as before but. At the end of the day what we get out is not a, classification. But. We get a sequence, of nine data. Vectors, that somehow represent, that, image now we, don't really know what. These features are in, any real sense all. We know that those features were useful, in classifying, those, millions, of images that the original neural network was trained to classify, and it turns out they're also useful for classifying. Other kinds, of images as well. Things. Those features, might do just as examples is one. Feature might detect greenness. In an image and that feature, turned. Out to be useful in detecting things like tennis balls or lawns, but. It also is useful detecting, other kinds of green things, another. One of those features might have detected, circular. Features, in, an image and that turned out to be useful detecting, bicycles, but, it's also useful for the detecting hula hoops and other kinds of things as well so, we can use that fact and it's a trick we, can take those features, that are generated, by the old you'll network, collect.
The Values with new images and use that to build a new more new model as we'll see here. So. Let's suppose that we, wanted to train a brand new model based on the old neural network to, identify hammers. And hard hats so. We can paste in an image of how an image of a hammer on the left run. It through the neural network, collect, all the features on the right so, that'll give us eight data vectors one. For each feature. And. It also gets us a binary variable a 1, or a 0 and we'll get for each image that we put through. Same. Thing with hot hats take a picture of a hot hat process. It through the image collect, the features get, another binary variable, 1 or 0 for hard hat or not let's. Do that for a few. Dozen few hundred images so. What do we get. Now. If you're a. Statistician. Or a data scientist, you probably know where I'm coming from this after. We process each of our few hundred images we. Get a sequence of values features. A binary. Indicator what. Could we do with that we. Could build a logistic regression, model we. Could build a one layer neural network a very simple, predictive model we could take from those generated features in our binary values, build, a new protector, just from that and. It turns out that that works really well you. Don't even need a lot of data a few are a few dozen images, will often do the trick as long as the things you're trying to detect are visually. Distinct, and those features can can, identify them, you. Also need a lot of computing power it's, a simple, logistic regression, or a one-layer neural network so you don't need a lot of power to do those computations now. This is obviously a toy example and you'll likely want to identify more than two objects, and the underlying neural network will generate many more than just to eat features, at its, second to the last layer but the principle remains you. Can do this with modest data and modest computing power and it often works really well and. Of, course you don't have to build that transfer, learning model that's what it's called by yourself you. Can use the advanced vision models, we just saw in computer, vision as the. Base and then provide your own images and classifications. And use, the service called custom. Vision now. Just like computer vision you, can train transfer learning learning models programmatically, using web-based UI and that's what I'm about to do here, but. You can also do it programmatically, using curl, as, well through directly through the API so. Anyway now I'm going to use Microsoft. Cognitive, services custom. Vision to. Train a model for that shot by photo feature we saw earlier on in the, Tailwind traders website. So. Here I am in the custom vision web-based, interface which gives us this nice UI we. Can provide new images for the transfer learning and analysis and as you can see in this project, I've already uploaded a number of pictures we've. Got here pictures of screwdrivers. Further. Down we've got pictures of pliers I've, also got drills and hammers and. I'm gonna use those to train my custom model let's. Also add one. Other classification. Which is hardhats. So. I'm gonna click on the add images, button up there to provide some new images directly. From a folder that I have onto my hard drive. There. We go and I'm gonna browse to, the hard hats folder just press ctrl-a, just select them all. Click. Open and now it's uploading those images of the hard hats to the service as well, but I've got to label them as a human so, gonna put the hard hat label, associated.
With Those images as well and that will be used in the training process. It's, gonna take it just a few moments to upload those files but while it's doing that just have a look there on the left hand side where, you can see the counts of the images, associated with the five categories. Or labels that I'm working with here once. Those hardhats have loaded there's only gonna be about 180 images, you know just a couple of dozen per. Classification. But, it's got to see has you'll see in a moment despite the low number of images it's gonna work pretty well. So. Now that we've got our images, in there all we need to do is click that train button it's, gonna go ahead and click train quick, training works really well you can expend a bit more time with the advanced training but it's not necessary, in this case but. As you can see in just a few seconds it. Has done is built that transfer, learning model and generated. Something with an overall. Performance. Of 99 percent, now. When you actually use this in practice there's something called this probability, threshold here on the left and that's really important, and, that determines, when you react the. The. Amount of the probability, that we have to get to before. We'll assign any, classification. At all now. If it's low that. Means you'll give a classification even, if the model is not sure but, you'll make a lot of mistakes if. It's high that. Means a lot of the time you just won't give any classification, at all but you can be pretty sure it's right when, you give one now, choosing, that is really important, and depends. On the application, if. This was an application saver, detecting, cancers, in an x-ray you, probably want to have a pretty low probability. Rather a pretty high probability threshold, because. You don't want to make false diagnoses, but, for this app a low one is fine, anyway. I've clicked on the train the test button to go ahead and train this model so, I can upload a new picture it hasn't the scene it seen before so, I'm gonna browse in here to my test images and, choose. A brand, new image and see how it does on this I'm. Gonna choose that picture of the man in the hot hat that we've already seen before and you, can see it classifies, that as hardhat, not, headdress, with, probability, 99%, so that's looking good for what we want to do here let's. Actually try it on a couple of other images. Choose. Here at image of a drill see, how this one does okay. And it. Gets ninety, four point five percent. Confidence. Classification. For that drill as a drill also very good lastly. Let's try that picture of the pliers we had so much trouble with before. Ok. Classifies, that as clients, with 99.9. Percent confidence so this is all looking good this, looks like a model that we came in bed into. Altai wind traders app. Now. The way we embed, these models into applications, is by exporting, them, so. Now that I'm happy with the model. I'm. Gonna go ahead and click on the export button there now, you can export in, a format good for Android applications, or for iOS applications, I'm, doing a web application here, so I'm gonna export an integer in a universal, format called onyx and then. Download, that onyx representation. Of our new model for detecting hard hats and so forth directly. To my hard, drive as I've done right there. Just. A quick aside so, we just exported, our model in the onyx format, onyx. Or open, neural network exchange it's. An open standard that was launched by Microsoft, and Facebook to, promote the free exchange and deployment.
Of Artificial intelligence models, and it's, supported by a huge, range of applications, and tech vendors. So. Now that we've trained our custom vision model we're going to integrate it into that to--when traitors web app that we saw before and for. That we're going to use onyx runtime, which, is an open source inference, engine that. Allows you that gives you functions, that allow you to generate predictions from, you images, that. You've based, on the model that you provide in the Onyx format. So. Now that we've created our custom foot custom, model we're, going to call it in the app using its API the. Way this is going to work is that we, create a new inference, session, based. On the onyx file and that's what that file path variable is referring to that the onyx file we just generated. We're. Then going to generate a classification, label. From the uploaded, image as a string there's a bit of work here that I didn't show about sort of reforming that image to the right size but you can see that in the code but. Then all we need to do is take, the classification. That came from onyx runtime, and then pass that to the existing, search capability. In the tail end traders app and that's what I'm given going to give us the results of the user sees. So. Let's see how that works now that, model we just exported from custom, vision is actually a zip file that contains, the Onix file so you've got to open up the zip file and you'll find there's a man a manifest, file there we don't need that all we need is that model dot onyx fire which is actually the representation. Of the newly trained, neural, network but. Within the Tailwind traders website in. The file tree there's. A file called products - onyx which is driving the existing, image recognition system. Which isn't working so well so. To fix it all I need to do is take the model to our next file we just exported, rename. It as products to onyx and then upload it into the Tailwind traitors application. So. Let's go ahead and do that now. Now. The Tailwind, traitors website, we, created in app services, so, I'm here I'm just going into the azure portal, having, a look at the App Services app which, drives the tailwind traders website and. If you haven't seen it there's a cool feature within App Services which is the advanced, tools which. Will give us a cruder interface to, interact with the running website. So. Just launching, that right now. Ok. And from that you can go to the debug console, I'm. Actually gone not going to use command or PowerShell what I'm going to do here is just browse through. The file system itself to. Science W routes. Stand-alone and, then, the onyx models folder which is where that existing, products the onyx file lives, then. I'm gonna grab the one that I just exported and extract. It from the zip file. From. My file system there it is and then, just drag that over into kudu which is going to replace that existing, file, into. The file system. Once. That's done all. They need to do is go ahead and restart the website. Let's. Browse through the app services page there and click the restart button and then, restart the app and then within about 60 seconds or so the. App will be ready to use again with. Its new capabilities, so let's have a look at that now. By. The way before. We go back to the hive look at the app I just want to show you a real cool little tool for those of you want to dive a little bit into the details. There's. An application, called Natron. Written. By a guy called lutz rotor which. Allows you to inspect, the. Contents of the neural networks that exist within onyx files so, if you just go to his web app I. Can do that there click open model and load, in that products to onyx file that, we just generated, and. Have a look what is actually doing so. We can see the details of that neural network if. We scroll in a little bit on the top there and you can see that the input is an image it's. A 3 layer there's the 3 to 24 by 224, image, that's. A little bit of a dirty secret about computer, vision models, their. Inputs, are actually pretty small to 24 by 24 hours about the size of a Windows icon, so, computer vision systems actually have pretty bad vision, but they work really well because they can analyze the contents of those images so well. So. Now let's zoom out a little bit and we can scroll through all the layers of the neural network the custom. Vision generated, for us so this is the same representation, of, those, neural networks that we haven't looked earlier on and you can see all the layers as we scroll through that there. And, as we get to the bottom we. Can see how, it generates the outputs one. Thing it generates is a sequence, of five numbers and that's the probabilities, associated with the five classifications. That we train that model on so. That's just the classifications, of those values we, also get another vector which is the loss function which is those probabilities and you, will need to choose your own probability.
Threshold Over. Which those probabilities need, to be before. You assign a classification, or, not and that as I said earlier on that depends on the application but now I'm back at the table when traders website let's try out our new products, at onyx, file. So. I'm gonna browse again, to that image that was giving us trouble earlier on. Once. Again you have these images yourself in the repository, and let's. Choose again that pliars thing that gave us trouble before. And. Now. Today when traders has correctly identified that as pliers and now I shot by photo feature. In the app is working. All. Right I've really got time for one more quick, example of. Pre-built AI and this one comes from the decision category, in cognitive services there's. Something called the personalized, ER service, that, lets us customize, the interface of apps in real time balancing. On what the user is most likely to do a couple, of the things that we would like them to be. Doing. So. We can see how this might work in the Tailwind traders website you might have seen that recommended, section on the screenshots, I showed you earlier on we, chose a selection, of the departments, in, the store there's this large hero, image on the left and a few other smaller images, for the other departments, and. What we're going to do is have the personalized, a service, choose for us choose, for us the, order in which those sections, appear through. An AI technique, called reinforcement learning. Now. Personalized, has actually been in development and in use of Microsoft for many many years. If anybody of you any of you have an Xbox the, home screen on the Xbox and whether you see a new game or an invitation to join a twitch byxis. Dream or. Something. You might want to purchase from the store all. That is determined, for you by. The personalize a service, and since that was introduced, to xbox the. Person the Xbox team has seen a significant, lift in the engagement metrics, of players on the Xbox Service. Personalizes. Also used to optimize the, placement, of ads in Bing search and the, way that news appears. In MSN News again, with. Great improvements, but. Now you can, use personalize, it in your own applications, as well. Personalize. Ur implements. An AI technique, called reinforcement learning, and let's see how that works, so. Suppose we want to display a hero action, to the user and the user probably, isn't sure what to do next. So we could display one of several different suggestions, and in the gaming. App that might be play a game or watch a movie or join a clan and, then. Based on that user's history, and other contextual information like. You know their location, or, the time of day and the day of the week the. Service, will rank the possible actions and then suggest the best one to promote so, in this case that's joined, and. Now, hopefully that user will be happy but how can we be sure and that. Depends on what the user does next, and whether that was something we wanted them to do so. We can get to define our own business, logic where, we can assign, it something called a reward score between 0 & 1 - what happens next for. Example if we want people to spend more time playing games or reading articles or, spending more time in the store we, would assign a higher reward score to that and then. Personalize, it feeds that information, back into the ranking system for the next time we, need to feature an activity. But. This isn't just a recommender, system, recommender. Systems have that unfortunate, feature they just kind of tell the user what they've already done if, you research for shoes on, the web and then every product ad you get is for shoes that's recommendation. Systems, in action this, is different, so. What personalized, it does is it's, it's. Usually in something could explore more Explorer, mode rather where, it recommends, the best activity, based on the history and all we just saw but. Sometimes, it enters something called Explorer mode and presents, the user with new things they may not have otherwise seen and it's, kind of like an automated a be test system. But with more than two branches, and all being tested in real time so. You can try what the percentages, of the time that. Under. The Explorer mode is activated and that controls. How often the user will see new things they have in other ways seeing so. What we did as you can see in the source code for the Tailwind traders app is that, for anonymous, users we're going to use the time of day the day a week and the browser operating, system is context, to influence those rankings and for. The rewards score we use whether or not they clicked on the hero panel. So just a zero one and so. What will happen then is over time, personalized. That will determine what is the best category. To present to anonymous, users, based. On their day of time. Of day day of week in their operating system, based, on that activity, and then 20% other time try, other things to.
See If we can get users, to engage with other stuff. So. To see that in action I'm gonna go back to the tailwind traitors website here. And. What I didn't mention before is the list of products you see down there on that recommended, section is has been determined by the personalized service, in this, case electrical, department is the one that's appearing there first and we. Can see what's going to happen with the Explorer behavior, if I click refresh. Now. Turns out was already an explorer mode and now it's showing me the one that it actually recommends. The most which is garden, center turns, out the people that visit the website most we're clicking on garden center a lot so that's the one that appears more most often but, just keep on refreshing this a few times eventually, it'll go back into Explorer, mode again and we'll. See that our app based. On personalized, is going to choose a different Department to feature you in the hero section, and I. Think from my recording, that happens the next time there we go plumbing appears as one, so we can see how users engage with that super. Easy to implement super, easy to use now check out the code that I have on the website. Just. To wrap up if you're. Thinking, about putting these, AI services. Into production there are lots of things you might want to be thinking about probably. The first thing you want to think about is the cost the. Short story here is that if you're just testing this out as a developer, most, of the service that our services, I've shown you are free but. If you do want to go into production volumes, you know talking about millions, of hits. A second or very large documents, to analyze the. Pricing for that there, my service and region and have a look at that bottom link on the screen to, find out what the specific, cost is for your application. You. Might also want to think about what your data is going and how it's used. For all these inference, services, will provide data and we get back a result the, data is uploaded to Azure and then just immediately discard, it you, can get all the details about that in this compliance website that I have right here but. If. Your app if you're working in a low bandwidth environment, we're sending up a big image across, the web is an issue or you, work in a regulated, environments.
Well You can't send your data out of your firewall. For whatever reason, there, is another option that's available to, you which, is to deploy, with, containers, for. Some of the azure, cognitive, service they are available, as downloadable, containers, which you can put behind your own firewall, so. You send the data to them it sends it results back nothing, ever leaves your system except. For billing data and the. Cost is exactly the same as if you'd used as your cognitive services online, and. Lastly, I've, saved the most important, consideration, for last, you. Know you've seen how easy it is to integrate AIO, capabilities, into your apps but as. They say with great power comes great responsibility. It's. Really important to understand, the impact, your application, has on people and consider the ethical implications. So. If you're working with AI technologies, like these or any others you should be working within it within an ethical framework that, focuses, on enabling, people to achieve more is. Inclusive, of all types of users and is fair and transparent, now. If you don't have an ethical framework set up already a great place to start is Microsoft's, own principles, for AI which. You can read more about at, that link right there, all. Right to. Finish I hope, you've seen that with these pre bought AI you can solve 80%, of your needs for capabilities, in apps really quickly Peter, said in the remaining 20% and building your own apps but. Your own AI systems, rather have a look at the next two talks in this series I, hope. You've learned that pre bull AI isn't magic, it's just data and numbers, but, understanding. How that data is used is really key to understanding, its capabilities, limitations, and you. Don't need to be a rocket scientist, to hook up to these API, and get AI into your systems but do think about how your app will be used and impact. People a few. Slides just with some links if you'd, like to check out all the documentation, of the cognitive services, it's available at that link right there if, you'd. Like to run, through some courses on MS learn about the AI systems, there's a great collection here and that link is in the resources, and. Have. A look at the rest of the talks in this session to learn about all the other things you can do with, artificial intelligence as well thanks. For your time thank you very much.
2020-01-19 10:57
