Jeff Dean's Keynote: Deep Learning to Solve Challenging Problems

So. My name is Anand Rangarajan I'm the site. Lead for Google Bangalore and I'm, very, excited, to be. The. Program manager with Pankaj Gupta for this con for this workshop I. Just. Wanted to tell you that you know this is the first time we're doing a big outreach, to. The both. The academic and the industrial communities here. About. The work we, have done and the problems with that we are still working on. We. Are very excited that diverse. Set of people from, the industry and academia, are here they're. Going to share their work they're going to share the kind of problems they are working on. So. Bangalore. Google, bangle itself was started in 2004, so it's it was one of the very early, engineering. Sites for Google outside. The US, we've. Always been, fairly small and. Now we are starting to show, some significant, growth. Yes. There, are lots of chairs. Here please. Do come along. Yeah. Through this day we are going to have several. Tracks. Looking. At different aspects of ml. And AI we. Will usually have these tracks. Such that there will be small 10-minute, talks and then there'll be a final ten minute Q&A so, if you can collect your thoughts and then ask them at the end it might be useful. And. There. Is the Wi-Fi. Information TV. Yeah, okay. For. People who want to use, the guest Wi-Fi, there, are restrooms there. Please. Do try, and take, breaks between talks, and. Then during the coffee breaks we will have the PhD, students with their posters in the cafe so if you are curious. About the sorts of problems they're working on you can go, there. Anything. Else Pankaj. Yeah. So for any logistics. Is, Ashwini there and where is Divi. Who. Was right. Around here it is tapped out. So. That is that. Is the guest. Wi-Fi. Access. Point name and the password. Alright. With that we will start the. Keynotes so we have we are very excited because we have people visiting us from us and you also have people dialing, in later. In the day you will also hear some leads in a, Google Bangalore speak about their work so, after.

Thanks. Allen my, name is Pankaj Gupta I'm also an engineering director at Google in Bangalore I lead. The engineering, for. Consumer. Payments. App called taste that's. Part of Google's next billion users initiative. I joined. Google about, eight months ago Google. Acquired my, deep learning startup called Holi labs and in, fact in one of the sessions we are going to hear about at. Least what one part of Holi labs was doing later on in the day so, okay /, - cue notes but. Let. Me introduce you Jeff and. Then, we'll play the video and then, we'll, let you do the slides so I'm. Really, really excited to kick, off this workshop with a keynote by none other than Jeff Dean the. Title of his talk is deep learning to solve challenging, problems. Jeff. Joined, Google in 1999, and is currently a Google senior fellow in Google's, research group where. He co-founded and, now leads the Google brain team. Google's. The deep, learning and AI research team he and his collaborators, are working on systems for speech recognition computer, vision language, understanding, and various other ML tasks, he. Has code design and implemented, many generations of Google's crawling, indexing and, query serving systems and co-designed. Implemented, major, pieces of Google's initial, advertising and, adsense for content systems, he. Is also a co designer and Co implementer, of Google's distributed, computing infrastructure. Including. MapReduce. BigTable. And spanner systems, protocol. Buffers the open, source tensor flow system for Emily and a, variety of internal and external libraries. And developer, tools Jeff. Received. A PhD in computer science from the University of Washington in 1996. Working. With Craig chambers on whole program. Combination, techniques for object-oriented languages, he's, a fellow of the ACM a fellow. Of the American Association for, the Advancement of, Sciences and a winner of the ACM prize in computing, and the Mark Weiser award so. Please. Welcome Jeff. Let. Me play a video that's. A really fun video about tensorflow. We. Wanted to make machine learning at an open-source project so that everyone, outside of Google could use, the same system really. Awesome. We'll, just switch over. Cool. All. Right well so. Thank you very much for having me what. I thought I would do is just, give, a talk about, sort. Of some of the work that our group has been doing on machine learning research and. Some. Of the ways that we think it's gonna impact, some, difficult. Problems in the world so, with that I'm gonna just switch to sharing slides and that I will come back in, facial form at the end of the talk. Let's. See here. Great. There, I am can. You see that. Sorry. Wrong button. Yes. We can see Thanks, great. Okay. So. One. Of the first things is that. The. Interest, in deep learning has, really gone, up significantly over, the last. Six, or seven years. Part. Of us is because we've created a new term for something that is actually fairly old so. Deep learning is sort of a rebranding, of the. Idea of artificial neural networks, but, it's, a bit more than that as I'll discuss. So. There's tremendous interest, in the, machine learning field. There. Has been for quite a while but in the last six, or seven years that. Growth has really. Taken off. This. Is a graph of the. Number of machine, learning related, archived papers that, have been published, per year on a. Paper.

Preprint. Hosting service called archive and you can see that it's, growing, it actually faster, than the. Sort of Moore's Law exponential. Growth rate of. Computational. Performance that, has been with us for quite, a while although that's not slowed down but. This, field is changing extremely. Rapidly and I think and many many people are flocking, to this field and are. Wanting, to do new research and are doing great new things, in this field I think this is a really exciting thing you see lots and lots of young students lots. Of people. In, other disciplines, wanting. To sort of start doing research. In this field lots. Of use across the industry and organization, that's that's tremendous. But. As I said deep, learning is really just this modern. Rebranding. Of, many. Of the ideas in artificial, neural networks which have been around since the, 1970s. And 80s I. Actually, did a thesis. On, parallel training, of neural networks in 1990, when I graduated, from undergraduate. And. The. Idea behind neural networks is actually a powerful. One which is that. You can learn very complicated, functions, through. These. Layers that learn features, and patterns. At. Sort, of progressively, higher and higher levels, of abstraction so. At the very early layers of a neural network the, features, that are learned are fairly primitive but, as you combine, those. Powerful feature recognitions. Ah. Pattern. Recognitions, then. You, get more and more complex combinations. Of these things, so. Many, of the ideas are. Relatively. Old in neural, networks, there's, been a bunch of progress, as, you can see by the previous, graph of the number of research papers which. Is also great. But. Let. Me just go over some, pretty, interesting functions, because when you here oh it can learn functions, you, know sometimes that doesn't, quite resonate, with people about how powerful these think can be so. One. Kind of function that a neural net can easily learn is, given. Enough training data of pictures, and then labels of what. Kind of object is in that picture it. Can learn to take a new picture the raw pixels of an image and then, learn to predict a categorical. Label, for that from, perhaps you. Know thousands, or even hundreds. Of thousands of categories. They've, been used. To powerful effect in improving speech recognition, so you can actually train a neural network end to end going. From raw audio waveform, signals, to. A transcript. Of exactly. What was said in that audio, waveform, how cold is it outside and this. Is in contrast to how speech recognition systems, have been built for many years which is they have lots of, they. Previously have had lots of individual, components, that. Are other kinds of machine learning systems and and manual, features now. We can just learn these things end and with a neural network. These. Systems can learn to translate given, enough training, data of the form you. Know sentence, in one language sentence, another in another and so. You can input a sentence, in English hello how are you and then. Have, the model trained to produce a. Translated. Output bulger coma deli food. Perhaps. More surprisingly, you can they can be trained to emit. Not just a categorical label, given an image but. An entire, English sentence that describes the image which. Actually shows a pretty decent. Level of understanding of, what's going on in these kinds of scenes so. If you give it this image. The output might be a blue, and yellow train traveling down the tracks. So. Why is this all really happening now as. I said many the sort of underlying, algorithmic, ideas are, relatively, old and. So. In, the, 1980s, and 90s. Essentially. Neural networks were, showing. Really interesting, results, for Verto problems, but they needed a very, large amount of computation, much more than we had available at those times and so they couldn't really be scaled to, work on problems that were you. Know real. And substantial, and impactful. Although. They did show really good results on sort of modest, size problems and. So other approaches, that were less computationally. Intensive. Kind, of were. The preferred ones in a lot of machine learning tasks. Through. This time as you have seen by the Green Line but, thanks. To Moore's law we've. Actually gotten. Much more compute, you. Know what I did my undergrad thesis I was excited about bringing to, bear a 64. Processor, machine on training of neural network instead. Of one processor, so that we could get you know maybe a factor of 50, or 60 speed-up it. Turns out what we actually needed was a factor of a million more, computation, not 50 but. Now that we have that neural. Networks are actually a, the. Best solution, for many many problems and that. That gap seems to be increasing, relative to other approaches, as. We add more and more compute which I'll talk about it towards the end of the talk.

So. Just to give you a sense of the improvement, over the last few years in. 2011. The, winner, of the imagenet challenge, which is a challenge hosted by Stanford, University every, year where. You're given, an image and have to give a label of one of a thousand categories, the. Winner of that did not use a neural network and the. Winning error. Rate lowest. Error rate was 26 percent error and we. Know that humans have about a five percent error rate on this task, um. It's actually a reasonably, difficult, task because among. Those thousand, labels are you. Know perhaps 40, different breeds of dogs and you have, to get, the correct breed of dog. By, looking at a photograph which is not, something that humans really excel at, but. Fast forward five years, every. Every, entrant now pretty much uses in real networks and the. Best neural networks are down around three percent error so the winning entry in 2016. Three percent error below. Human, level. Which. Is pretty significant, so basically computer vision has gone from not really working that well to. Working, extremely well in five years and that's, pretty transformative, if you think back. To the time in you, know, biological. Evolution when. Animals of all dies were, sort of at that point in computing today where. We didn't used to be able to see very clearly, now, we can see very clearly and that, opens up a lot of a lot of things that, we can do that require. Vision. Work well. So. For the rest of the talk, I'd. Like to structure, it a little bit around, something. That the US National Academy of Engineering. Put, out in 2008. Which is a list of grand, engineering, challenges, for the 21st century and it's, a pretty good list it's sort of things. That. I think we, as a society if we make progress on these we. Will actually you, know improve the world and live, happier and healthier. Lives. So. I've highlighted a few in red that I'm going to talk about but I actually think machine learning is, going to be a significant. Contributor, to making. Progress in all of these I. Just, don't have time to talk to address, exactly, how but. I will for the red highlighted. Items. So. One. Of them is restore, and improve urban, infrastructure. I've. Been to Bangalore and I've seen the the chaotic. Traffic there and. We. Have our own traffic problems here in the Bay Area but. One, of the things that we are actually quite close to as a society, is actually. Having, working self-driving. Cars a completely. Autonomous cars. And one. Of the things that's really powering, the, fact that we're so close to, to launching these commercially, is, the. Fact that vision, that works right, going, from raw sensor inputs, in these cars where you have lidar, which is like a laser range-finding, depth, sensing. Sensor, plus, a bunch of cameras, plus radar data to. Something, that can actually reconstruct what, is going on around the vehicle in.

A Way that allows, it to plan safely. What. It wants to do to, accomplish. The goals of getting to its destination and, not hitting anything and, behaving. In. A way according to the. Right traffic laws you. Know that requires a pretty high level of understanding but, key to it is that, vision networks and, so. We've. Actually just done some trials, in Arizona, a couple. Of months ago without, any safety drivers, in the vehicle so that's actually you, know a reasonably. Good sign that these things are you know imminently. Going. To be released in fairly. Short order across Laurel which i think is going to be pretty. Pretty. Different, and will, dramatically, change the, urban landscape you know we won't be. Parking. Areas as much we can have cars. Just come pick us up at will when we want to. Be pretty amazing. Advanced, health informatics so our group is actually spending a fair amount of time on how can we use machine learning to improve healthcare. And. I'll. Touch on just a couple of different issues here one is. We've. Been doing a lot of work in medical imaging related, tasks, and, this. Is a. Task. Called. Where. You're trying to diagnose, whether or not a person's. Retinal. Scan has. Signs. Or symptoms of diabetic. Retinopathy which. Is a degenerative. Eye disease, there's. About 400 million people at, risk many of them actually in India and. These. Are graded one two three four or five by. Human, ophthalmologists. And. One. Of the big problems is that if you're at risk for this you should get screened regularly. Every. Year, perhaps even more often and there, just aren't enough ophthalmologists. In the world to, screen all the people that should be screened and. So, we. Built. Up a training set of data, of, this form with human. Ophthalmologists. Giving. Their opinion on, the. Image and then you can train a computer vision model to assign, the grade in an automated way given, the input image you say one two three four five. And. We're, actually now. Significantly. Better than. Human. Ophthalmologists. This, this was, a paper published at, the end of 2016 in JAMA. Which is one of the top medical journals showing. That we were on par slightly, better than, the. Median. Board-certified, ophthalmologist, in, the United States at, doing, this task and we've, actually improved this algorithm. Significantly. Since then so we're now on par with retinal, specialist, rather than general ophthalmologists. We've. Also discovered. Kind of interesting, new findings, in the process of doing this this. Work and so, we've. Actually been able to. Devise. Completely. New. Biomarkers. From. Retinal images that we that, human ophthalmologists. Didn't, even know existed and so we can actually use, these kinds of predictions to, assess someone's, cardiovascular. Risk in. A way that, is roughly. As accurate, as a more invasive technique, where you actually need to draw blood and assess. The, cardiovascular, risk through a series, of blood tests and so this is a. Sign. Of something where, we've. Actually been able to use machine learning to create new kinds of healthcare signals, that, previously. Didn't exist because it's looking at very subtle patterns that. Human ophthalmologist, can't really detect in the eyes, we. Think that's pretty exciting. Another. Kind of, medical. Problem, that we're focused on is predictive, tasks for healthcare so given a patient's medical record data can we actually predict, the future and, deep. Learning methods, are. Actually getting very very good at sequential, prediction tasks, so given. In English sentence can I predict the, French sentence, or given, half. Of a medical record can I predict what are the other things that are going to happen to a patient. And. If we can do that well we'd be able to answer questions, like you. Know will this patient be readmitted, to the hospital in. The next week, or you. Know what are the most likely diagnosis. For this patient right now which, test should I be considering, for this patient which. Patients, are at highest risk for say developing, diabetes in, the next month. And so we, have a collaboration with several United States Organization. US healthcare organizations.

To. Try to assess this and we've actually just published, a paper on archive. That. Essentially. Shows, that we can predict all these different kinds of tasks, using, the same rough underlying, model and. In, particular compared. To the baselines. Techniques. That are used in clinical practice today we can actually predict, things like mortality rate or. Risk of mortality. Roughly. 24, hours earlier so this solid, line at the top of the graph is about, 24, hours, earlier. Warning, of someone's. Risk of mortality been, the traditional. Dotted. Line baseline, here and so, that allows, doctors you, know much more early. Guidance, about which patients are most at risk and they can pay more attention to those patients we think this is gonna be pretty significant. Another. Area is actually. Several, of these Grand Challenges all kind, of depend on better. Understanding, of chemical, properties, of things you. Know engineering better medicines, is all about finding drugs that bind to the right kinds of things solar energy is all about developing you. Know more efficient, materials. For solar panels and many. Of the other ones are also related to chemistry. And. So we decided to tackle a particular. Problem in quantum chemistry which, is given, some molecular configuration. You. Want to predict a bunch of things about that molecular configuration, like. Does, this bind with a different protein what. Are its quantum properties, is it toxic and the. Traditional, way that you this is with a traditional. High performance computing based. Chemical. Simulator that uses something called density, functional theory and is. Fairly, slow so if you run it it takes maybe an hour for, a given configuration to, give you the right answers or the. Simulated. Answers. So. We we. Decided we could actually use this, simulator. This, computationally. Expensive simulator, as a trainer for a neural network and, so. We developed a new kind of. Network. Our all, network architecture, that's good, at dealing. With graphs the kinds of chemical graphs that you see there and what. We found is that the results. On, using. This neural net and then using the neural net to make these predictions are, indistinguishable. And I can receive from the the much, more computationally expensive simulator, so we have something that's about three hundred thousand times faster at, doing, these kinds of computations, and is. You. Know equivalent. Accuracy, and we. Think that's pretty, transformative. You know anytime your tools get three hundred thousand times faster that, just enables you to do very very different things. You. Know you could imagine screening, a hundred million compounds, and taking. The 10,000 that are most interesting and and doing, more investigative. Studies of them. Okay. The last thing I'll talk about is engineering, the tools for scientific discovery, and this will be kind of a whole. Collection of improvements, to varying. Kinds of tools. So. The. First thing is our. Group has been producing, software to, help us with our or. With, our own research, and with deploying, these kinds of machine learning. Systems. Into, Google products and we've. Been doing this kind of work for a long time and tensorflow. Is, a system. That allows us to Express, machine learning research ideas and get results quickly, and. It's, actually our second generation system. And when. We started, working on it we decided we open-source, it so. That, people. Outside of Google can. Use, the. Same tools that we use and we can collaboratively, work together to improve those tools. Therefore. You people, outside have been able to use it for all kinds of interesting things like that introductory, video had. A lot of interesting uses of tensor flow that we never imagined, when, we open sourced it and I think that's one of the beauties of open source software. Is that people can take it and use, it in all kinds of crazy. And and. Great. Ways that you never would have and collectively. Society, benefits from this. So. We wanted to open-source this so that it would be a great, platform for everyone, and. This is kind, of a, growth. Charting, of interest, in tensor flow measured. By github stars which is sort of a people. Can express interest in, different. Repositories on github which is an open source hosting, platform, and this. Shows, the tensorflow. Star. Growth. Stars. Over time compared, to a bunch of other. Open. Source machine learning packages also hosted, on github and, so. You can see people have really taken to tensorflow and that, community, is now working. Actively. Collectively. To improve the, system. One, of the research projects, that we're working on in our group is actually. Attempting, to automate. Machine. Learning so that you don't need as much.

Human. Machine learning expert to. Expertise. To solve a new problem so the current way you usually solve a machine learning problem is. You. Have, some data you have some computational, devices maybe GPU, cards, or maybe CPUs, or other. Things and then, you kind of have an ml expert, a machine learning expert. Take. That and stir it all together and out. Of that you hopefully get, a solution to your problem the machine learning expert runs a bunch of experiments tries, different ways of solving the problem and, hopefully. If. You have enough data and ml. Expert is is good you'll, get a good solution what. We're trying to do is see if we can turn this into a. Automated. Process, where we don't actually need the human machine learning expert to solve new problems, and. Really. If we want to get the systems that are generally, intelligent. We can't have a human in the loop for every new problem that we want to solve and so, this I think is a pretty fundamental. Thing, that we really need to make progress on to really get. Towards, more intelligent, systems that can do millions and millions of different tasks but. Really we want to see if we can use data and a lot more computation, to get good solutions, so. The way this works. One. Of the ideas in the work that we're pursuing is an idea called neural architecture search, so. One of the issues, with. Deep. Learning is, a, human, machine learning expert normally sits down and just makes a bunch of decisions about what kind of. Network. Architecture, they're going to use is this going to have nine layers or, 17. Are. They gonna have you know three by three filters at each layer or four by four or seven by seven how, are they going to be connected. And. So. The idea behind neural, architecture search, is we're gonna have a. Model. Generating, model and we're gonna train that model, using. Machine learning using, reinforcement learning actually and so. The model generating, model can generate, a description, of a network, architecture, and, then. We're gonna train, and. We can generate say ten of those models and we're going to train each of them for a few hours on the problem we actually care about and then. We can use the, loss the accuracy, of each of these generated models as a reinforcement, learning signal, to the model generating model so, that we kind of steer the model generating model away.

From Experiments, that didn't work very well and towards, network. Architectures, where the, results were very good and. If. You run this loop many many times, you know perhaps training, 20,000, models you, end up with models that are quite good. And. So here's an example on the left of a model that the architecture, search process, came up with, for. A image. Recognition task, C 410, which. Has the advantage that it's been very well studied by the machine learning community, so. Everything. Except the last four lines here are sort of human generated. New. Improvements. Of state-of-the-art, results. On the C for 10. Image, recognition. To ask and you can see the error rate dropping over time, and. This at the time we published the neural architecture search, work the, state of the arc was three point seven four percent. Neural. Architecture search, automatically. Got to a model that got three point eight four percent so. That's pretty promising, we. Then scaled that work up to. Image net scale. Which is a much much bigger problem, it's the million images that are sort of full resolution instead, of sixty, thousand images that are very small and this. Graph shows you the. Accuracy, of a bunch of different models for image net and. The. The. X-axis, is the amount of computation that each model requires for, to, give you a prediction, for a given image and so. Generally more, computation, gives you more accuracy. So. You see this general trend, and. Each. One of these dots here represents. Years of effort by sort, of the top machine learning researchers, and computer vision research. Groups. In the world, and. The. Really. Nice thing is when we applied Auto, ml to this. We. Actually can get a range of models with different computational, costs, but, each of those models is better than the corresponding. Human. Generative. Models at that sort of level of computation, and so that's true both at the highest end where you see. You. Know much less computation, and slightly better accuracy than the best state-of-the-art, models at, the time this one and, also. True the low end where you might have a very. Lightweight model that you want to run on a mobile phone and, you. See a pretty significant, jump in accuracy, for basically, the same computational. Cost so. That's pretty exciting and. So. That's kind of an early sign that we can actually build these flexible, systems that with. Enough computation, can solve new. Problems automatically. And. The. Only drawback is we're going to need a lot more computation, and so. That. Actually comes, to, another point which is that neural, networks and all the algorithms that I've been talking about have, two really nice properties. The, first is that, reduce, precision, for all the computations, in a neural network is. Generally, just fine it's, fine to do you know one significant. Digit of precision for, the computations, rather than lots. Of significant, digits and. The other property that they have is that there's a handful of specific, operations, in. These models essentially. Nearly. All the computations, are made up of of dense, linear, algebra operations, things.

Like Matrix multiply is vector, dot products, things. Like that so, if you can build computational. Devices that are specialized, at doing reduced. Precision, linear algebra, then. You. Have a chance of really speeding up the kinds, of computational, complications, you can do here and. So we've been working in this space for a while we've. Developed a, a. Set of devices called tensor processing, units and this is the second generation one, which. Is a device that's designed for all that training and inference. This. Device is. Provides, about 180 teraflops, of computation, which is quite a lot has. 64, gigabytes, a very high-speed memory and it's. Designed to be connect connected, together, into. Larger configurations, that we call pods and, so. Each of these pods is 64. Of those devices and eleven-and-a-half petaflop, of computation. Of. Low, precision computation, but, just as a point of comparison the, number 10 supercomputer, in the world is. About ten, and, a half petaflop, to compute out. Of higher, precision computation. But still this. Is sort of on that scale and. This. Is sort, of a dedicated. Machine learning supercomputer. And. We've actually made these available externally. Through our cloud products, so people can rent, time on these cloud GPU devices, essentially. Get a virtual machine with a cloud keep you attached. We're. Also making. Them available to researchers who are committed. To doing open machine learning research so we have about a thousand of these devices that were making available to researchers who. Have interesting, projects, and, you can sign up at this URL, here by. Sending. In a proposal, and. We're excited to see what people will do with this the, only requirement, is that they'd be willing to to publish the work they do. So. With, that I'd like to just. Highlight that I think deep neural networks and machine, learning are really producing significant, breakthroughs, that are solving, and are going to solve some, of the world's like. Grand Challenges which. I think is tremendously. Exciting, if, you're not thinking, about how to use null Nets to solve your problems you probably should be and, if. You look at our, team. Website gqo, slash brain and also intensive flawed org you'll find a lot more information in, particular, the brain website, has lots and lots of our papers and more information about each of the sub areas that were, working in and, with that thank.

You Very much and, I will now I. Think. You. Know this and, go. Back to not presenting. But. Every time I see them I'm like wow I'm really inspired. I. Think, at. Least some of us or most of us in the room aspire to do research with that kind of exponential curves so. Thanks. A lot we. We. We. Want to do some Q&A. Anyone. Has any questions. Okay. Hi. Jeff Manish, Gupta thanks, for a great talk so, you talked about in, machine. Learning deep learning to on, healthcare problems right predictive. Problem. So. In our experience, what we saw was even when you have very high F scores let's say off the order of, 0.9. 5 and so on. F1d, because, of the class imbalance, you still run into the problem that the, number of false positives is still higher than the number of true positives for, many many predictive. Problems, so, have you been able to kind of overcome, and. Of those kinds of, those. Challenges, I mean get. Accuracy, high enough where. You are actually true positive I mean you're false positives, not, much more than the true positives, yeah. I mean I think. Obviously. If, you, have kind of rare conditions, one. Of the things you can do is enrich, the training set that you have to make sure that you have more balanced coverage of the rarer, conditions, and the more common conditions and then, correct for the sort, of adjustment, you made to the background probability, in, enriching, the training set when you're doing, testing and that can really help a lot and so, that's the general technique we use for that and, also, these. Things allow, you to sort of control the, false positive, versus false, negative, rate, with. Different threshold, and you, know choices. And. One thing I forgot to mention about the diabetic retinopathy worked we've actually been doing, we've. Actually concluded, clinical, trials, in India working, with the Arvind I Hospital. Network and we're. Now actually doing patient. Treatment. Using, this. Model. Actually. We are gonna hear from Erin Wilson in a. Session later today about exactly this work back, signet-ring, yeah, he will give you many more detail.

More. Questions for Jeff. Hi. Jeff thanks, for a very interesting talk is this parcel from IC Bangalore so. Like you know there, is this a well recognized problem, of explained ability, in AI and machine learning I just, want to hear your thoughts on that, yeah. I mean I think it. Is a, definite. Problem for some kinds of, domains. You. Know I think there are some problems where you just want the most accurate thing possible, and you, don't necessarily care, much about interpretability or explain ability but, there are other domains where it's actually very important, to have. The. System be able to give you some insight and intuition about why it's making a certain prediction you know I think healthcare is one particularly, good one it. Where, you really do care about that if you it's, much more actionable, and usable. For a, doctor. If you can say something. More than patient needs heart valve replacement right. It's better if you patient needs heart valve replacement and, it's. Because, I see this like. Brief description, in a medical note from two years ago and this test result is a little elevated and. This. Other commission right, and so. We're doing a fair amount of work in our in our group on interpretability. Of, medical. Images i think you, saw on the, the. Retinal images where we can now use those as a new cardiovascular. Health metric, one. Of the things those models are able to do is to describe what, pieces of the eye and the retinal image they're, actually looking at when making those different kinds of predictions, and. So. I'd. Encourage you to look at a website, called distilled that pub. Which. One of my colleagues Chris, Ola and sham, Carter are actually, publishing. A series of articles there about how do we make models, more interpretable, I may, have some really nice investigations. Of visual models and interpretability for those. Okay. One more. Hello. Yeah hi Jeff I'm, in. Buff from Docs app so. I'm a health tech startup and then we are now building systems, keeping. You know data and science and learning you. Know at the back end of our product so one of the challenges we generally see is that the volume, of data in, medicine, is lot less especially, quality, annotated, data compared. To say another field likes a bit images, imaging. Or, you. Know biet or, you know automated, driving and so on so. How do you overcome these challenges and, how do you work with you. Know people from a different community say the medical practitioners, for example. And, how's. That option, because finally for a tech to be useful adoption. On the other side as well as needed right yeah. Absolutely I mean I think I. Would. Say that healthcare is a you. Know challenging, space to operate in for many reasons first of all the, data volumes, every, individual, healthcare organization, generally has are modest, not enormous. And. So, that, sometimes means, it's. More difficult to get the, accuracy you might want a pretty good problem there's, lots in you mean it states especially there's, lots of regulatory, challenges, for. Deploying machine learning models you, know you need FDA approval. For. Many. Kinds of things, there's. Very. Real and valid privacy, concerns people have about their healthcare data. But. In general we make partnerships, with healthcare organizations, and the ones we've been doing the United States so far we've. Sought. Out healthcare providers that actually have pretty significant, size data, sets. We. Ask them to de-identify the data set so we don't actually get any personally. Identifying, information for. These data sets we just kind of get the medical record without knowing who it is are a. Large, collection of medical records that allows, us to do the research that we need to do to show that these models can have very. High accuracy for, these tasks that we care about and. That I think is the, beginning of a dialogue of how do we actually you, know you, know that generally gets the healthcare organizations. Excited, when. You can show that you can predict something they really care about with. High accuracy and that, then sort. Of loosens, the relationship. A bit and makes it much easier to actually figure out what are the next steps how do we actually take, something like this and actually really, deploy it and you.

Know Maybe get more data maybe you, know label, a bunch more data in. Appropriate ways and so lots. Of things like that I think can lead to to sort of more rapid adoption, and success, in this space. Thanks. A lot Jeff thank, you so much for your time I, think, that's I, know, there are other questions but we'll have to so. Thanks Jeff thanks. Very much enjoy, your day. You.

2018-04-20 15:53

Other news