AI for business optimization: Leveraging custom speech vision and search to optimize - BRK3309
But, welcome, thanks for coming, I know, day three oh it's. A thing. So. Welcome glad, to have you my, name is Noel a charity, uhm, I and I am joined with a bunch of friends you're going to see on stage today we're. Going to spend about the next hour, talking. To you about. Something, pretty important, but I wanted to start off with a really, important. Public. Service. Announcement. Right. Ever. Nobody. Wants, to do the thing you are now about to learn how to solve. For. Your customers, for, your company. Nobody. Wants to push paper I will tell you I have four, children and I. Have just recently moved to Washington, State and. I had to go and sign them up for the dentist and doctors. Do. You know how many times I filled out the same kind, of piece of paper I would. Have gladly had, a bot, to. Do this I. Offer, many times and there's lots of challenges to that field and I get it but there's no customer, on the planet who's like please, don't make this easier for me don't make it more convenient right like they want what, we are about to share with you so I encourage you to think about all of the things that we're about to share and how it might, apply to your, world, and. Also. You. Know think. About these. Pre-built. Models, that are available to you so yes some of these problems can be solved by writing your own machine, learning models and that's, awesome and exciting but, if you don't have a PhD in data science or data, wrangling, and, you don't want to or have the skill set and. Don't want to go higher for, lots, and lots of money data scientists, help you solve these problems there are actually quite a bit of services, available at your fingertips today to. Build these pieces so. My, role at Microsoft, is to, help. Developers. Understand. And use this technology, so at the end of our session today I will show you a bunch of communities, that, I'll encourage you to join. Because. It's how you will get access to our tutorials, and our samples, but, there is no, other time like to, for developers c-sharp, developers Java developers, node, Python. Go to. Be able to leverage AI infuse. It into their apps without actually having to re, you know I always encourage people to go build a model once because it's cool and it's fun and you can say you did it and like walk on but. You don't have to and you can create incredibly, powerful, software so. What. Is it right, cognitive. Services these pre-built, models, there's many, of them 30 of them plus we.
Build More all the time but they are the reason that they're so interesting, is that it's, not new, to us or. It's new to you maybe, some of you actually show, of hands who, not including. What you've learned so far this week have, used cognitive, services before ooh. Nice fans right, we're friends kindred, thanks good good to meet you, so these, services, are all available to you and we actually wanted, so I always like to show it just you see the capabilities, this doesn't list the models by name because. As a developer I'm like I can figure that out I want to know what my models can do for me so I thought this was a unique way, to present the information you. Can tell we're actually gonna focus in on just a couple of these use cases due to the time frame that we have but. Amazing. Stuff right OCR, handwritten. OCR, poof, like business process, automation all over that Alright, computer, vision and the ability to identify. Document. Types the, ability to organize and, file the ability to provide. A bot that allows you with your voice to ask for things and have it just show up right. The ability to even type in a search and have what you need, just show up that's, the kind of power you want to give to your customers and your employees and we're gonna dive into some of those examples today. But. First I want to just share with you we actually did, a study you, know it's funny sometimes we go into and you might do this in your own companies right as a developer, we just kind of dive right into building stuff we're like I I, like this I'm sure everyone, else on the planet likes this but, we thought this year we'd do something different which is be extremely. Data-driven and our approach to building solutions so. We went out and interviewed we we didn't my it's often not we hired a third party company to, be completely, objective and, go and ask the, Fortune 1000 what do you what are you doing what are you investing. In an AI what, are you building. What, are you currently building what are you gonna build right and these we, ended up with about 500. Use cases. 400, and high number, and they. All fell, into one of these 16, boxes it. Would be awesome if I could say to you right now that a developer, experience, we're building it all but. I had to pick what can I do this year and so we picked four boxes, one of those his, business process automation, internal. Bots you can actually come and see build a bot in an hour tomorrow we're, not going to focus on that today but BPA. We, are gonna focus on so. I just want to share with you just a few of the use cases that came out of that study, and you. Can see them here right image, creation, handwriting. Analysis, that's huge all those forms, I filled out at. Least someone doesn't have to manually, type them in that's step one I'd. Eventually like to not fill the forms out if any of you are in that world. But. At least let's, start to automate, parts, of this process you, can also see you, know reputation. Management right, the ability to do keyword. Identification. Entity, extraction. The. Ability to do, PCI, redaction. Right, all of these things are like PCI, some, of you shook, your head are familiar with this right sometimes we have, really important, data and in the customer. Obsession, we write, it on Docs and we don't realize oh darn, we probably shouldn't scan that in for, all eternity, so, it makes sense for us to come up with technology, that can not only protect our customers but can also protect. Our companies. So. There, are a few real-world problems, I wanted, to just share I couldn't share the company names not yet anyway but. I wanted to give you some real-world, examples, of the types of things that we are doing so, there's, a few entertainment. And media companies, that are using cognitive, services to, do transcription. And translation of, programming, live you can actually see it right now right. Up on top of the screen right we have PowerPoint translator, translating. I'm saying along, with the text so. You can see how entertainment, companies would want to do that our, translation, service does, this in 60, languages. And text why wouldn't you automatically.
I'm From Miami why. Wouldn't every company have the ability for someone to speak English and Spanish and sometimes at the exact same time we, do that there. So. It's really important. For us to leverage these tools because, you don't have to build it from scratch you, can use it as a web service and, it's it's incredible, healthcare right, being able to identify emails. That alone can save huge, amounts of, time redact. PCI data or PII data, that. If we're, going to talk about that in just a minute. News. And information right, being able to do custom, site searches, and being able to provide. News and, clips. And content, from our organizations. And bubble it up to the surface we call them top-level intents I just want to ask for what I want and I want it to show up how, hard is that turns out it's actually pretty hard but. Now we have services at your fingertips, to let you do this so, what. I want to do now is just hand it over for our first demo. Of the day and really talk to you about a very prevalent, an example, I just shared with you right, what happens when you have all of this data. And on. Some of that data you've got people handwriting, stuff they should in hand right or you had business processes, that didn't care if you put credit card numbers on things right, how, do you fix that how do you make yourself you're compliant, how do you ensure that your business is not in jeopardy, so I'm gonna welcome to the stage Sean and George and they're gonna talk to you all about it I'm a round of applause. Thank. You Noel. So. George and I are excited, to be here today to share this solution. Area. So. First we'll talk about what is redaction, so, when we talk about redaction, it's kind of what we're showing in the picture here where somebody's went through a document and kind, of used black, sharpie or something to take out a bunch of the content, so, they do that for either security. Or legal purposes and so. How, this came to be is I was in a meeting one day and somebody came up to me and said John. Does anybody in Microsoft, or does, Microsoft, themselves, have a solution that can help us go, through a large amount of documents, and redact content, from them I said. Well I think. I've seen something about that but give me a day or two and we'll come back and, I'll let you know what we have. And. The reason I said that is I had recently seen a demo, so. Just by a show of applause has anybody seen the JFK files demo stuff. Yep. Awesome. So. What it does is it. Uses our cognitive search capabilities, and you can put in any, keyword in, this case we have Oswald. And. You put in the search box and you click Next and, what that's gonna do is go through all the content that. We have stored, there, whether it's typed, whether, it's handwritten, or whether. It's photos and it's gonna find any relation. To the word Oswald, and highlight, it for you and so, first case I'm like well this can't be that hard let's just, highlight it black and we'll probably be good with that so, I called, up George and I said hey George. Just. Would approach with this problem as, is something you think we should work on and of course you said yeah let's do it but. Let's. Look for something fun and exciting let's look for a credit card right, so not, something that was too easy so we. Started to start looking at it and, what. Made it hard is we're. Not looking for a word we're not looking for a synonym, we're, looking for a pattern right, we're not going to put into that search engine, a series. Of numbers for a credit card cuz we don't know if the credit card exists or not and we're talking about millions of documents we want to go through so. We, set out to try and solve the problem using existing, intellectual.
Property And just expanding, that. And so what we did is, we, went to the internet and we grabbed all kinds. Of documents that we could find all kinds of forms and we started putting in information, in the forms so, on the right hand side here we have an example where there would be a credit card box, on the form and we put in the credit card number there but, also the person may have been talking on the phone at the time and they. May have just you know handwritten, it at the top where it shouldn't have been and it. May or may not be a credit card and I probably should say right now none, of the credit card numbers up here a valid they're. All generated. To. Look like or past certain checksum, algorithms, but, they. Won't actually work just. For, notes people take them down here, and. So, yeah there could be written on it on the right hand side somebody. May have put it on a post-it note because, they're taking notes while they're on the phone put, it on top of that document scanned it in thinking that they're preserving it for all the right reasons and. Then. There it is it's there for forever we, also decided to take a list, of numbers just a whole bunch of numbers put them all on to. Sequentially. On a list there and see can it find good, numbers and bad numbers in there and, even use cases where somebody had previously tried to redact it before they scanned it in but then through the. Contrast and scanners, today you can actually kind of read it through there and. So, we, also wanted to make sure that it wasn't always on the same spot, so it might be on page one it might be on page eight it, could be a twenty, page form and we wanted to be able to go through millions. Of pages to do this, and. So it took about two days for us to sit. Down and talk through it and, work. Something out and after that we had a demo which is kind of similar to what we're about to show you here, and. In this demo what we've done is we've keyed it to search for credit cards obviously but through. Thinking and through ideation, we. Believe that we could extend this to other things like send, numbers passports. Healthcare numbers potentially. There's. A lot of other information or PII that, we think we could find with it but enough about the background who wants to do the demo hey.
Everybody So what, you see on the screen here is our page for our computer, vision cognitive, service and this is the basis of what, we were using oh sorry. There you go thank you. There. You go all right so this is our computer, vision cognitive, servus page. I'm. Sure many, of you have used this so. The, first service. That we've got up here the first component, of that is image. Detection, or aspects. Of image so, if you click on that image it will give you some descriptions, of what's in that but where we were, focusing, was. Down on, this. One where. You can pass. In an image with. Handwriting in it and it. Can extract, from that image the. Content, of the the text which is within it so, as you can see the, text is here but what was. Important, to us was this, aspect, the bounding, box so. It actually gives you the coordinates of where that text is within within, the image so. That's the basis of what we were looking at so. If I just show you a couple of these files that we'll put through as as, as. A as a demo, so. Some handwritten, text here with, a valid card valid, credit card number here and a valid credit card number here but. A, set, of numbers that just happened to look like a credit card number in the middle there we've. Also got a PDF document because, most of the documents, that we were looking at when we did this solution was web pdf based obviously. Handwritten and scanned and they were looking back over many years so. What we have here is just a, credit. Card number written in in, a random spot on the form but we also have, predictable. Areas on a form a zone if you want to call it that where a user is supposed to be filling. In their, credit card information. Another. Scenario we had was where, we, would find something that looked like a credit card number but but wasn't we. Might want to highlight that to the, business user so that they could put that through a manual process and determine if, you. Did actually if it was a credit card number but the OCR failed, to read it so. That they could process it manually so, as you can see in that case there's it's. Clearly an invalid credit card number. This. Was the test case we had where we had the post-it note on there so literally. We. Felt, that businesses would have, this scenario whether it, would be written anywhere. On the form random locations, or put on as, a post-it note and, here's a fun one I took one of Shawn yesterday, working a, the experts, booth, right. So, in. Our configuration options. What we've got in the solution here is the ability to select which type of credit card we're interested in redacting. So. JCB, Discoverer Amex, Visa so, we're, just focusing, on a few here we've. Also got, this. Concept of tells we called them so looking for specific, text on a form and there, and therefore identifying, what the form was and what.
The Associated, predictable. Zone was that we were looking to redact from, that from. That form so. Let's. Go in and we'll run this now so, here's the code. Let's. Just fire, it up and I'm going to give it the. Import. Folder where we're picking up files from. And. The. Output, folder. Oops. And, the. Output, folder sorry, the configuration, file that we're pointing it to and. We'll let that chug, away. So. That while that's doing its thing it will go. Up and run through each of the individual. Files and start to output them over here when. We were looking at doing this on large scale we were looking at using Azure batch to process, in parallel, and get through these documents really quickly but. Let's start having a look at some of the outputs of this thing. So. There. You can see the document, that we put in initially so it recognized, the two, credit. Card numbers that were valid and it. Bypassed. The invalid, one and it does that on two levels it uses a regex pattern to firstly identify, the credit card a potential. Credit card number and then, it uses a checksum, algorithm as a second, pass to validate that it is actually a true. Credit card so. We we were looking at scenarios where that you didn't, obviously want to redact something, that might have been a driver's license number or some. Sort of number that was relevant to an industry that they actually did want to keep. We. Can have a look at the PDF structure. Here and as you can see it grabbed the credit card number at the top and the zone down, at the bottom. This. Is the example of the fuzzy logic where. It wasn't a credit card but it thought it was in this case we did it based on proximity, to, the words American Express or Amex or visa, whatever, so, if it was a close to a credit card we could highlight that to the user and they could then process, that manually. As. Shaun, booth. So we managed to grab that credit card number and finally. The. Post-it note where it grabbed from there, the. Last piece in the puzzle that we had in this solution was tracking, the actual activity, of what went on in in. Processing, so as you can see here we're using cosmos. Db2. Structure, or store, the output of our runs so. As you can go through you can see that in this particular document, or. Image it, processed, one page and it found an, Amex card and a MasterCard on on that page didn't, find any fuzzy matches, so.
That's A way of like. Tracking progressing and monitoring. What's actually going on with within, the solution. We're. Back to you Sean, right. So. Just to kind of recap what we just went through there, from. A solution point of view on, the left-hand side we have a, PDF file that's being brought in we. Separate, that out into the subsequent pages which is what you see under the preprocessor, under. The processor, what we're doing is that OCR technology to, bring, all that data in and then, what we do is remove all the white space within that so it becomes one contiguous string, when, we go into the pattern match controller, we're actually looking for any of those those, check sums as George, was talking about and. We identify, any of those that are potentially, within there as, we go over to the master we're bringing that binding. Box with. It to, be able to say these are the ones that we want to remove and this is where they are within the page the, post processor, actually puts that bounding. Box around that highlights, it in black at this point and then puts all those pages back together and Reese Aves it as a, PDF document. That was awesome, like seriously, yeah the first time I saw this I was, like so every, government. Business. Legal, healthcare, like anyone who has Doc's and humans, that is, a thing. Extremely. Interesting I am personally, like. I said the first time I saw it I was like how do we do that so one of the questions we often get is like awesome how do I do that, so my, team like I said we're involved in making this available to developers, and so we have a developer, community as I mentioned we show, you those at the end and, we make this available, basically. On github so we're working on it right now and kind of making it something that is usable for you guys but like. Productizing, and making sure it's templatized, but. That's coming soon so I certainly hope you'll join us and, I'd love to share that with you because I'm sure there are people in here that might find that beneficial, now, as we move to the next piece I, think it's really, interesting to, start thinking about all of the use cases so I know previously. We showed you a slide that had all of these cases on it the good news is is that this is being recorded so you'll have a chance to go back and see it you can also connect with any of us on LinkedIn we'd, be happy to share with you you know the deck that is that's, provider you can specifically, tweet, me and.
I'd Be happy to share that but, we. Can't talk about all of them right but there are a couple that you can go to market with today, and so I want to just take the last few moments we, have to, go through an example. Really. A set of examples around. Search and so, I'm gonna welcome, to the stage Brian. Who. Leads our Bing search. Ecosystem. All. Right thanks so much. Wasn't. That last demo awesome, I was sitting in my seat going like that was so cool, I loved it. Thank, you all for coming. To, Orlando and joining. Us today let me see if I can make this work here, at once - okay. Great. So. So. Yeah here we are in Al and Orlando, and. And. I see some of you in the audience who have come by and visit us at our booth thank you so much for that, what, we're going to talk about a bit today, is, about. The. Data that exists in the world and. Great. Businesses, run on great, data, and. While. I've been talking to you here, there, have been literally millions, and, millions. Of changes, out in the world in terms of data in terms of web documents in terms of tweets. In. Terms of SMS. Messages, and all of those what. We do at Bang is we we, we follow all of those we follow if we see about 76. Billion, URL. Changes, a day and we index, a good, percentage of those and add them to the 35 trillion, that we've seen already. And. What. Our api's, can do is connect, you to that world of changing data so today I'm going to show you some scenarios and, some of the capabilities, that we have that. You can leverage for your business today now. What. Does that look like well. I'm. Sure you can imagine that, you. Know you know Bing the search engine, it's got lots of capabilities in, terms of finding results for things here we have Uncle Bill on. The on the screen and I, can see, deep. Links about them I can see images I can see the latest news I can, see videos all, of that great stuff and you. Might be thinking yourself well that's really great if I want to build a search engine and some, of our great big partners, do but, beyond. That you. Can use that information for, your purposes, for your business as well. The, web search API just, to give you a little bit of background because. I know many of you are quite, technical. We. Provide search responses, through our Web API that come back in the form of JSON and, we, can provide all of that query content, and web pages and news and images and videos as various. Objects, and structures that you can walk through in your JSON, and. We. Have partners like Yahoo that used that to make. Their their Sur pages, we've. Got partners, like AOL he used that to make their serve pages but, we also have people who use that information to do lots of very very interesting things like for example, grammerly. Gravely, provides a service that, allows. For plagiarism, detection and, they. Use the, corpus, of a republic. Document, on the internet and can, compare, that to. Submissions. From students, and you know lots of submissions, from students, they can help, them find those references, that they meant to put in their footnotes. For. Where they found these great ideas that they might have been adding to their document, and. In other cases they can guide them to to, to additional information, but, here's, an example where you're, using the output of a search algorithm and the, many many thousands, of classifiers. That we have and the machine learning behind, that to. Find that right information. Anywhere. On the planet that's publicly available and, use, it for plagiarism, detection.
We've. Got companies, like Irby and B who use search to manage, and and help with their listings and make sure that their listings stay great, we've. Got companies. Like P, Lowe who. Create. This, little medicine. Robot, that helps seniors take their pills and, they. Make it interactive so, when you talk to peel o and you say hey okay pillow, how. Many calories are there in a banana he, can tell you because. We're, providing those entities we're providing that information through our entity API so. That pillow, can tell you whether or not that that banana is good for you today or not but. We have many other verticals, as well we have a news vertical, a video vertical, an image vertical, and a. Much newer which I'll go into a little bit more, deeply in a moment a visual, search vertical. Because. While it's great searching. With. Text. To, bring back things like you. Know what's. The real-time news related. To my company, or, to my product, and we'll talk about that and how you can use that for reputation, management or. Or. Being, able to find the latest videos, on, a particular, item or images. And they're, there they're, all text-based, entry, but what if I've got a picture and, I, want a query using, that picture, well, then great, we'll search allows me to do that and bring back lots of great information and we'll talk about that a little bit more, so. Here's. An example a, scenario, in terms of reputation, management so, Matt, you all of you have businesses. You have customers, you have products, you, probably really care about what people think about your business in your products on a day-to-day basis, and, here. I've run a query on Microsoft, and office 365 and. This, is example output, that you today, can go to the page in. Cognitive services, and you can try out this API and you can do it for your company and for your products and see what comes back from our API but. Imagine. Programmatically, tracking, these sentiments so if, I, put in the query for Forza. Horizon for, I can, take the output, of that news, item and I, can tie it into our text analytics and, I can see that oh this, review that we had on one of the game websites has, a, super, super positive sentiment, I'm feeling really great about that what. Are the ones I'm really looking for though where. It says negative and it's 4% and maybe my PR team wants, to look at that and I really understand, hey, what's going on there did somebody have a really bad experience we, should probably get on that so. While, most of you might be out there searching, in your favorite search engine for, your products, and looking, on various new sites for that kind of thing imagine if you could do that programmatically, you, can today, this, is a solution, that you can create I would, say in under an hour. And, and. You can do that across many languages, we've got quick. Starts I'm not going to go into this much today but I will give a plug for for. Our documentation, or sdk team who, have made these quick starts available, five-minute, quick starts you know in many many languages, and she's c-sharp in Python and you can try try them out it there right there really really great so, so that's that's one. We're using it we've, got adobe spark. Using. Our. Image search to. Bring results into their product, and they're really great because they make their products available, to, students really really cheaply and one. Of the things that they wanted to do was hey, how can I find images, easily the, could use in their website so we, have a filtering. Ability, in our image search that, allows you to scope things down to images, that are available in Creative Commons, nice.
This. Is available for commercial use I don't have to pay for this I want more of that that's fantastic so they. Can bring that in and those, students can can use that and anybody can use it even though the people who actually pay for Adobe spark which, is really really fantastic and our API allows, them to manifest. This search right there in the product it isn't a case of I need, to go out I need, to find it I need to save it on my machine you, can see how now the process, is much more automated, much, faster, creating, delight for all their users and it's really really great it's, a really good product I'd suggest you try to whatsapp. Uses, it to fine you know when you want to put in that queue if for, that lovely logo. For your group, you've probably made groups while you were here at ignite I know we did I'm probably unlike, five, different ignite, groups and having, trouble keeping up with with everyone the in the evening but it's great. And. Here. I'm using, a, search. Based on on Xbox since. I'm feeling comfortable about using these we were just talking about Creative Commons, but. In whatsapp they use our image search again to bring. Back those and and, to sort of take. Their product to that next level in terms of feeling, connected and having those capabilities. Copyright. Management, so many of you have, media. And and and the world was talking about some news and media companies, earlier and you've got great content, you've. Got your. Logo you've got your logos associated, to a new product you've, got information and, you, can use our search capabilities. To go and detect. Copyright. Infringement who's. Using my logo that shouldn't be using it it who's, copying my product in a way that they shouldn't be or trying to pretend your state that they're me this. Is a real, issue for many many companies today as the web grows and grows and grows and people sort of take liberties, and wish they didn't but there you go it happens. And. I can find out where, were, my logo, or my information, is being used so I can remind that person that maybe they should be giving me a loyalty, or some thing or maybe not using it at all but. Another. Way that you can you can manage your information. I, told, you earlier that I tell you more about visual, search so this one's really exciting, I'm. A huge, fan of our, visual search capability, and a picture. Really is, worth. A, thousand, words and. There's, many things that I can't describe like, I went, to my friend's house the other day and he.
Had This really cool sort of light and chandelier, and I couldn't, tell you how to describe that if I was going in to search for it but I can take a picture of it with my phone and using, visual search I can find similar objects, on, the, web, here. I'm showing a. Couple, of other features including. Web annotations, so imagine for example you're. A company that allows, the, uploading. Of products or images, and. Maybe. People describe things really well maybe they don't describe things really really well and well. Our web annotations, based on this picture. Here will, actually come back with yeah this is living room furniture and, we're actually it's it's yellow living, room furniture on top of that and the kind of searches, that we see that would bring somebody to an image like this or these related, searches or maybe. You're running you're. In I talked to a lovely gentleman earlier. From, a university, and they. Were creating. A new library catalog, and her, visual, search could be used to take. Those barcodes, in books. And actually return what is the book so. Again. Back, to Uncle Bill and. And his book. We. Can help you create those, you. Know there's a lot of new applications, being. Created on mobile devices today and people are more and more interested, in using, a camera and using that mode and we're actually seeing you. Know significant, growth especially in Asia for. People, using their camera. On their phone to. To query, we. Have capabilities, where we, can, take. Your photograph, you. Could create a shopping experience we, can actually identify items. In that picture that are very very visually, similar and. Go out onto the web and. And provide that experience. Or anti-d, identification. Or find. Similar images it's sort of like that's not exactly what I was looking for but if you could find me something that was kind of like that that would be great and how many times have you asked somebody in a store that and, here. You, have a capability, to do that so this. Is this is technology, today that you can integrate into your mobile, applications, and tie, it to a camera, and. And, work from there, so. I'm. Going to switch gears a little bit now because, I'm. Gonna talk about Bing. Custom Search now, earlier. I was talking about how. All of your products, and. Services. Are really, important to your businesses, and connecting, customers. With. Your products and services is, very very important. Being, custom search allows. You to create, an instance. Of a search that's. Defined. To a specific, domain now. That could be your business you, might have multiple domains for your business, or. You could even create a vertical, search where, you have domains, across an entire industry create, the best healthcare search financial.
Search Legal, search whatever, is your interest area today, I'm going to talk about this in terms of, of. A, search, specific, to your business so if you think about all that public content, that you have available and. If somebody comes to your site today how. Easy is it to find your products and services, maybe. It's really great and you, have something there maybe it's a lot of overhead to manage, if. You come down to our booth to us I guarantee you I will create a great custom search for you in 30 seconds, or you can go and do it yourself I promise, you and I'm gonna show you that here. Today. Here's. An example of a partner that's, using our custom search so we. Create a JSON, output or, we create a hosted, UIL I'll demo that for you in a moment. But, here's, here's. Temple University, who's using our outputs to, connect their students, with, courses. And inviting. Information, key information on their site. I. Promise. A few announcements as, part of our talk today I should have said that up front. We're going to announce, a few things as we go through here and we have some interesting, giveaways, for a Q&A at the end as well so we'll, get to that but. At, ignite. Here you're hearing it for the first time we. Are launching, custom, search v3, so since. Last October, we've gone through three versions which just goes to show the, speed and scale at where we want to advance this technology. And. We've, added some new capabilities, to custom search and I'll show you those today we've added customizable. Video search. Custom. Themes for your hosted UI we. Have pinning support, we, have allow, you to fine-tune your, your. Career suggestions, and a few other things, that's. A lot to take in so I think you'll be much better if. We actually go to demo. So. Let me just log in here and, hopefully. All of my, logins. I'll go and refresh them really quickly before we switch over because. Sometimes. I, get, logged out in between that's. What was fun and, now I'll just switch over here and we'll. Show. You what's going on okay. What. Am I looking at you're wondering well, this. Is the. Administration. Tool for, custom, search you've. Just signed in it's a blank canvas and it. Allows you to create a, custom. Search in, literally, seconds all you have to do is enter, in your domain I'm, going to pretend that I own all of the content, on. Xbox.com. I'm, a gaming, company I'm. Going to pretend xbox comm as my is my company and I'm. Going, to, to. Use that to demonstrate so all I did was type. In. Audience. Participation, I was just making sure that you guys are paying attention and. And. And and also demonstrate, how hard it is to see my screen with the bright lights so. Ex, ba. Just. Testing, just seeing if you're all still with me it's, getting worse isn't it okay. How. About we do that alright. Ready. Boys just forgetting the name right love it, you. Guys are gonna love it this is great so. It's. That easy and if I come over to the right side it lets me try it out so I can make changes on the left to the configuration, and, I can try it out on the right-hand side so what's, something I might search for I might search for Forza. For example on, Xbox and you're. Probably wondering oh well Bryan if I put in my side how quickly, will. It index. My site that's. The great news we've already indexed, the entire planet, and we continue to do so every day so it will be instantaneous, and that's why it'll work for you right oh right out of the box so, here you can see oh we've got Forza, Motorsport 7.
Yeah That's kind of a new one that they're released and some other ones down there but, you know what I noticed down here I noticed. The. Xbox. 1s Forza, horizon 4. Which is just coming out bundle that's just coming out I'm really not trying to advertise export, stuff here but it's just an interesting result. What, I can do is I can change the ranking of this so you're in control so I'm gonna pin that one to the top I'm clicking, on pin to top here and you'll notice over before I do that we, have an active list here of what's being included. In our search and we've got two other tabs blocked. And pinned and the, pinned ones will allow me to pin, something, to the top so I click pin to the top it came over to the pin tab and, now that particular page for the query Forza, an exact. Match will, allow me to be ranked to the top so now I'm going to rerun that that lets test it again we run Forza and. Sure. Enough immediately. It makes that change and now I'm rewriting to, where this is at the top think, of how you can control people. Moving through your site you have a, particular. Marketing. Campaign going on you know that there's specific queries, coming in to your site and you want to direct people to content. And you. Can you can create an instance, of a searching create multiple instances of a search so, that you can have different search. Boxes on different parts of your site that focus on different things like maybe your you. Have xbox.com but, you want to leave, out a certain part of it like for example I could block, in this instance, let's, see if I can find an example down here so I can show you I think I had one earlier but, you, know things change okay see here's, News xbox, comm Forza, racing world championship let's. Block, news. That Xbox have come so if I click block I can, block news or I can go level below that or I can go a little below that or just the twenty-eighth stuff let's, take the whole thing out. Done. So. I've blocked support I've blocked news great, and, now. When I come back and I do a search for Forza, all, of the news items are gone and everything else basically, got moved up so.
That Easy to manage and create those different things within your search ok great so, so that's that's sort of a very basic view here here's. Here's, one I made earlier. In. In, true Blue Peter fashion, I'm from Ireland so you know have to bring, the local references in the. The, the. One that I've created here I've added Xbox, and minecraft and I've added Microsoft. Store, as well because. I wanted to demonstrate to, you some of the other capabilities that we have including. We. Have added, in version 3 not. Only do we have the. Usual tech, support but we have image support. Look. At those cool cars I really like those pictures we have video support. Including. The. Ability for you to, to. See previews. Of those videos as well so I know you're thinking it's like this is great Bryan I'm in the admin tool but, how do I make this real for my site how do I actually use, this do I need a team. Of developers now, or, do I need to spend hours putting this together well if you want to create. Your own presentation. Or if you want to really really create. That look that that. Template University had yeah we have we, have the ability to give. You an endpoint that. You can use. That. You can get that JSON output so, let me just put a. Key in here and. Let. Me run, a quick and let's do that for is the one again and I, come down here and. I. See the API response and. It, didn't like that key that's okay we'll just go to the host of UI version there. It is down there so. I've. Created a custom configuration. Based on on how I've set this up and. There's. Two ways to, output. This I can either, take. The JSON, output I'll. Go grab the key in a second we'll get that or. I can user what we call our hosted UI so the hosted UI is really really cool, it basically allows, me to, configure. The. Look and feel of the output, on. This tab over here where I can just. Create. Something and I've added a logo in here to save it sometimes and I've I've named it sort of our ultimate. Games but I can, make it a standard, page I can make it a pop-up and. With. Just two lines of script I can add this to my site so. One. Of the additional things that we've added this time around is. The ability to to, configure, this even more work I can change the colors and the look and feel I can, add a logo I, can, have, additional. Colorized. Themes so I can override the CSS, I can change it to my school, colors or my company colors, I'm really. Really nice and then, where. Does that ultimately, end up in terms of being able to use it well I put. Together this, I, put, together this. Page. Great. That's actually showing up in the right size. And resolution, and. What. I've done is. Let. Me tab over here. I've. Taken the output, that we saw back here in custom search I want to show you where you can find this because, it's, really important that you can do this over. In the production area if I go down to a hosted UI after, configuring it I get. A little piece of script that, tells me I can either use that piece of script or this is the actual code. To the actual page that I generated. If. I come over here to my, page I, can. Go ahead and, I can add that in so I made a little here and it's, a really really simple page it's just got a header and a footer and now I'm my framing in this, this. Page here and it's, got my custom configuration, number which is that instance, of that search and you can have as many of these as you want so now, if I come back here and we, refresh that and we have our textbox. Show up now, I can come over here and I can do a search for Forza, and you, can see that that's showing up in line on my page now and it was that easy it was just basically drop, it into my page. I. Can, do images. As well, great. There's this great images and you'll, see all of the great. Filters. That, we have available for example, say I wanted to go with the color I, don't, know I was, looking at this earlier and I really liked color yellow for example I can. Choose the color yellow and now you can see we're filtering just two pictures that have a predominant. Yellow color.
In It which is really really cool and then I can also do videos which. Is one of the things that we added in v3 which we we think is really really cool and our videos if I mouse over them here you'll see we, also support, the, preview, we do all that for you so, in 30. Seconds, of setup and pasting. A little bit of code and page you. Can have an awesome. Experience, that's, tailored, to your business and your needs, it won't be xbox it'll be your it'll be your content, but it'll be just as exciting, for your users which is which is fantastic okay. I'm. Moving back from my demo now because, we have a few other things to cover really quick and and. Then we'll get to our our Q&A. All. Right. So. Some, new products, we've. Got some additional items coming out and. We. Are adding two to, our set of cognitive, services, a local. Business, API. A public. Preview of that so. That you, can do things like this so say I'm getting business, input into my forms I can, use the local API to validate, us did, this business give me the right address. Are, they trying to spoof it in some way or, I just want to be able to use this to do geocoding. Or do, I want to be able to look up addresses, or have the ability to you can submit a search to this including restaurants near me and use. The the lifelong information. Which, is which is really really cool. We're. Announcing, custom. Visual, search so while, I described. To you the ability to use visual, search to take images that, you thought were interesting and find, out information about, them custom. Visual search will. Allow you to take your, corpus, of images, your inventory, of images and do. Additional processing. On top of those to, get information and, annotations. Out of that which is really, really cool, we have that in private preview I'll give you a QR code at the end that will take you to a page if you want to come and sign up for that and. We'll, eventually move into public preview, and nga for that. URL, a preview is moving into private preview this is another one where we can automate items. Into, your mobile. Applications. What does this do well basically for any URL, it, will tell you the title and snippet of that URL it will tell you if it's family friendly or not so is it have any adult content that. Can be really really important, in terms of you, know hey there's people typing, information into, my application so for your chat box for example, Skype. Has this kind of functionality today. Somebody. Types in xbox comm and we. Can we can we can pass back this JSON here what we're saying yes. This is family friendly that's great it's it's content we want and we've, given you the predominant, image and title and snippet so, that you can use that within your mobile applications, as well and. That's. All I had and I'll leave that QR code up there for a second and pass it back to Noel who can who can finish this out, and. Thank, you very much thank you. All. Right wonderful thank, you guys so much so, what we've done you know in in the course of this presentation. Is as I. Mentioned at the beginning is. That there are so many there are many many use cases right. Search is a huge one just giving the ability for your employees, and your customers to find what they need is huge. Being, able to then solve, problems. Like things. That happen when humans get involved, in right things where they shouldn't or just, changing. Your business process, having to accommodate, compliance.
And Regulatory. Requirements. All, of a sudden now how do you do that we wanna the way I kind of think about it is that it's, like we want you to start thinking when you have this problem pop up in your head the first thing you want to think is which, one of those models. Could, I possibly, use to solve this before, you dive into like let's write something ourselves let's, build an you know old-school, business, application, like, first. Think is there, a model that could help me do this faster, better stronger, than. I would do it if I did it myself, and what we have found and, this of course is based, on the data that we are seeing that it is in our best interest, right to try and build on, you. Know what we say like the shoulders of giants right why. Manually. Write the business logic into your application. To handle this if you, have an opportunity, to at least test out what we always say is start. Small test, it out build a POC luckily. Today all of these api's are available, either as restful, calls or through SDKs, so, that building, a POC takes. Very. Very little, time I always. Say it, takes less than a day or less than three days but, it just depends right some people are like wow not you it took me five weeks so. Take. Your time but it's super easy right it's something that we pride, ourselves on, take advantage, of all of the engineers, you. Know these models are not new we've done this for decades, now so, take advantage of the work that's gone into these in. Order to be able to do, things you've never been able to do before that we've all I guess an engineer, I've seen these problems and thought about solving, it and then as, I planned out a solution, said, by the time I finished this project, the problem will have changed, and then we end up not building it at all right, so now, it's different, I have solved problems in I've had executive. Briefings, where I literally, solve problems in the hour that, I have to talk to them which is just very unique and and it's. Just something I've never seen before in my career and, so I think it's really interesting the. Last thing we want to give, you is and. We're gonna open it up for Q&A so we really you, know business process automation is a huge, kind of area. So we wanted to give a lot of time for questions but. This I will leave this up while we do that this, is our opportunity. To. Open the doors to you we'd love your feedback number, one our, backlog. Is driven, by, our customer, feedback what, gets voted, is what, gets put in a backlog and prioritized, and I'm the one like screaming hey, we, act no no no this is what we need and so user voice is our, way of doing that Stack, Overflow is for technical, questions I'm sure those of you who know I have, been there we, do live twitch streams now this is across all cognitive services, so we we, hit them all in a month but. We have a schedule, available, for that we have a Facebook, group so, if you want to join this is for people who are like, building so if you go there you can certainly look around but, if you're building I publish, documents.
Sample. Repos, and most importantly, I test. Your my test market in that group I publish. Repos it do this private, group just, to get some feedback so. If you want early access to some of these samples that are in that 16, box thing that you saw that, would be a great group to join or have a developer, you know join, who. Could help you of course we have all of our samples on github you. Know shout out to all the teams every, model. Has a. Team of engineers that help build the model and make it available to you but, as part of that they create samples, to make it easy for you to get started we have quick starts for every, pre-built. Model so you'll want to take advantage of that and of course you can join me on slack this is an informal, Channel it's literally like, I'm on, the jet bridge headed, to my next location and, I'm just in there like, questions. And we have lots of people I sent, it to every internal team and said hey if you're one of those people like to answer answer people's questions jump, on in so we have representation, from all teams but it's very you know it's informal it's like but it's a chance to get your questions answered, very. Very quickly but. Also be aware that you know we're all people. And we do sleep not a lot but we do so. But I definitely, invite you to that you should be allowed to jump right in so let me invite the rest of everyone, up here but. Thank you so much I hope you enjoyed it we're gonna open it up now for some questions, come. On up guys. Yeah. We're not gonna say that first so if you have a question come, to the mic there, might be presents. Involved only. Because we're recording this so it'd be awesome if we gave people an opportunity, first. Questions, always the hardest so thank you sir good we will, be giving him a nice limited, edition shirt very. Nice you only have a few so don't so I'm, going to. Understand. How, Microsoft's. Wanting, to address the ethical concerns. Around. Giving, such power to, use this I mean the first demo, that was showed which. Was able to pretty much scant credit cards out of any Internet image could, be potentially, used by anyone and. Could. Not, not, such a well-meaning person potentially. How. Is this thank. You how is this addressed, how call you how are you approaching. This issue oh my, god it's such a good question so, in, many I've done a bunch of sessions this week but in many of them I mentioned ethics and. It's an extreme, it's a differentiator, for this company it's why I joined Microsoft actually. So, we have a partnership, in AI or organization. We actually have principles, of AI that, all of our engineering teams, are bound by so these are principles. You can go take a look at privacy security, we, encourage actually all partners that we work with to use them obviously we can't require it but it is a top of mind and it's actually we have ethics. Review boards that monitor, write the activity, the. Demonstrations. That we build so, internally. As we encourage you internally, you have processes. In place to check these things as a company. Like Microsoft that's, pervasive we, also provide, heavy, guidance, to organizations, especially large ones about, how to build things ethically. So. That's how we've started, this, is a evolving. Initiative, but, the one clarification I'd want to make and you guys can certainly correct me is, that, we didn't search the internet right, this was a this, is basically a customer, solution, so it's not like we went to the Internet and like went oh I wonder what I mean maybe I guess maybe you could do that but this, was a customer. Solution, so all, we can do as a company is create. An ethics, board create, a set of principles for all engineers to adhere to and then be super strict about that but, I certainly want to give you guys a chance to chime. In if you have anything more to say about partnership, on AI if, you haven't seen it go take a look at it it shows our tenants, and. It's something I've had work that I've worked on a long time get denied, because. Of those. Policies, so they are in place, but. Thank you that's a great question. The. Bing, search for, business is that still in preview, where. You've got internal, results where, you get insurers oh so there, was an announcement as part of it nice how that's becoming, part of Microsoft search. Which. Is a slightly, different team to mine but. They. Are, anxious. And willing and want to talk to you down. In the modern, workplace area, they have a boot there and. It, is is getting, bigger and bigger in terms of its, scope and yes available, and growing and, you should definitely use it because it's super cool I'm.
Gonna Go to this, side thank you um do you want to give them one sure, you wanna you. Can have a teacher tour we also have these mini, USB key, credit cards. All. Right yeah, this has to do with big, custom, search yes. So it's. Great to be able to customize the, search. Through. The UI but. What about being able to customize it through an API. Yes. You you okay, so you. Can absolutely customize. It through an API by taking the JSON, output and, then, doing your own presentation, layer so, we want I mean, if I want to add a domain to, the search or take away at the mouth okay, that's a good that's a great feature request yeah, yeah you want to be able to update the configuration that's. The fabulous I love that feature request I'm gonna totally take that back to the team we, got I played with it but that's one of the limitations, I'm trying to we, we're. Trying to build a search, that focus. On a community-driven set, of links I see I see so, you want to be able to add to the, to the collection and the set of links something. That we did add in. Our last drove to in v2 was, instant, sharing so that more people could, could, actively, work on a specific project at the same time so we're sort of headed, in that trajectory but. This. Is another great example so thank you for that feedback I'll, definitely take that back to the team and, we'll. We'll, look at API. Editions. Of domain great, and and, I can, run my spell corrector, on it beforehand as well to help me end up putting them in. Great. Brian. Thanks, Brian oh when you know you, did and canned. Searches. I'm. Sorry sagen when you did your demo you did can searches, is there a way you can implement, like. A generic search across your site for information oh, no, these were not can searches you could I could have typed I did, put in searches that I knew would yield beautiful, results, but, you can certainly put any query into that so when you create your custom search instance, you, can hit that that. Box on the right hand side with any crew you want and, and, so could I are there I could have searched for for Gears, of War or any of them and they would have all come back based. On the content so no that's the intent you, want to create a robust, super. Powerful. Commercial-grade. Search, that, is running over all of your content and it'll. Work for every, and any query that's relevant to your content, thank. You yeah definitely if they give it a go I want and give us feedback if, you find anything that you think needs to be improved. You. Expect these capabilities, to be on the US. Government, or DoD regions. That's. A great question I don't know if we can inst. Let's let's talk afterwards, and just. From the standpoint that do, not post that in slack no. It's. Actually we're closer than you think that we. Have is that it's. A. Observation. Process and, we are currently in that process, and so we don't, want to jinx anything. We, are in a path so, we, can inform you kind of informally, and not on YouTube for all to see. But, yeah so let's talk about it but it is in process and we are moving in that direction right, but where we are exactly in that path we can't and the reason I was suggesting we talk about droids is that I'd love to know which areas you're most interested in so that we may be you. Know there. But but. The. The the. The. Active, military sites already use Bing. For first search which is great. Have. You needed good or clients that had successful. Integrations. Which they're on-premise share points and using some of these cognitive. Solutions, especially, clear, they credit card example. Sorry. Can I get you to repeat these. Are these are these, are credit card USB. Keys, by the way they're really nice going your wallet or your purse and, take 16-gauge with you yeah, right and, we have a small RFID. Customers, that have been successful, in integrating a, I experienced, in, their SharePoint on-premises environments, particularly, I'm interested in the credit, card example, with. The image recognition so, we haven't used that in. More of a public way like that like through a SharePoint. We had a document. Library that we use to go against that so. They had them all extracted, from a database to be able to fly through them there's. No reason why you wouldn't be absolutely, that could that could be fight against a SharePoint, document repository. If. Even, also line using Word documents, transform, them transforming. Them to PDF doing. The redaction, storing them as PDF and then doing whatever you wanted to do with the original Word document, so yeah it's very extensible, in there ago yeah. And the other thing that I would mention is that all, of the API is that we showed today that haven't shipped yet I literally.
Have Thousands. And thousands, of businesses, and companies using them today in their own fabulous. And unique ways and we learned from all of you and we appreciate that. For. The redaction. Services, associated, to a cognitive search I'd love to be able to use that within our organization, for purposes of like. About sensitive data and, the demo you're obviously batting a thousand in terms of a success rate but how do companies how do you recommend customers, dial, that in to make sure that it's running optimally, for, them how do they go back and see this. Work percent of the time versus this work seventy five percent of the time is there a machine learning component to that is there a manual, process. Yes. Yes so, in. You, know solution. When we've spoken to people about it often, they're not looking as you say for like a hundred percent they're looking for the balance. Between losing, information. That may not they may not want to be redacted and keeping, sensitive, information so. We showed the cosmos DB thing at their back there where, the the, actual processing, results were restored. And. We've went through the ideas of producing, power bi dashboards, to monitor. As it's going along that the actual amount of documents, that are being produced. Redacted. The amount that I've got the fuzzy, elements, to it so they can determine if they need to add. In, an additional, type. Of form that may have come in that wasn't being scanned it was a different zone in there etc so, we, encouraged. As we were speaking to people about this to monitor as they're going through and adapt, it so there's definitely a learning, period it's not just firing its add documents, and hoping. For the correlation. Match threshold, you can dial up or down to be more sensitive or lessons different. Yeah. There is actually yeah you could do that yeah, we. Haven't, in the example, but thinking about it yeah you absolutely could you, can also have it they, one. In a hundred documents, 150 documents, whatever just flag to a separate directory so, that you could go and manually just check that as well before you commit it.
Exactly. One, of the things that we always encourage, people to do is to make sure that you, remind the business that this is a I and therefore a research. Project. And, so you should, be given quite a bit of leniency around. Like oh this, might, totally, failed, the margins. Might not be decent, and then you you have to have that time where you do small iterations over, and, over and over again and you see that model improve just. Just, a little bit over time because, sometimes when we offer this to the business they're like great produce that and you know I want to ship it in, six, weeks or whatever and we're dealing with AI and, confidence. Intervals and and. The ability and percentages, right so it's a lot of experimentation. So you want to be really transparent, about that and just one more point on there you said yeah within the the demo we were obviously getting we had a sample set that was a hundred, percent accurate we, obviously do get errors one of the things we've seen is if users, filling in some, text on a line and, they put up one at the end of the line the last number in the credit cards are one sometimes. It would look on that as part of the box of. A bounding box so in that case that's why we did, the fuzzy matching and. We're looking to increase the, the. Intelligence. Behind that to highlight what maybe make what may not be important that into a separate bucket for manual processing, I'm. So sorry for the guys like right in the light so I'm gonna start on this side and then we'll go to you and then we'll come back, you. Guys take any kind of like machine, learning into, effect with the search results for instance like if someone. Is in, the custom search picking. The third result every single time after a hundred times does that get, bumped to the second, or the first result so. We take into account. It's. A great question we, take into account, things. Broader, index. Or, global, index when, we when, we manifest, results, for custom search so if, something, is getting more interests, that definitely has an effect on our machine learning there, are many things that do in terms of where, is this being linked, to and, a, thousand. Other things but, we. Do take that into account in. Terms of the overall global index and and not, taking into account at the local level so you're not prioritizing, anything, from that custom, search box. Or that custom, API that, says hey people, that searched in this box or th
2018-10-28 08:26
