Improving Doctor-Patient Interaction with ML-Enabled Clinical Note Taking
You. So. It is my great pleasure to introduce Michael. Bruno who is a professor, at the University. Of Toronto, and the, director, of the, Center for computational, medicine. At the university of six children in. Ontario. And. He, is very, much interested in. The. Intersection between machine, learning and medicine, working, on computational. Methods, that. Will allow. Us to provide healthcare, much. More easily, than. We are doing right now. Thank. You and it's really a pleasure to be visiting you guys at Microsoft, where you're doing some of the cutting-edge. Work that's feeding, into the things we're doing and so. As. We're. Slow said I'm. Really. Interested in how to improve. The, the. Clinical. Workflow and the doctor/patient interaction, process, with, smart. Medical, devices and you know even for those of you who are not sort, of in the medical side of things you. Probably know about the pain of medical data and medical, records and this, is should, update the year on the slide but, this is sort of what the medical data looks like today, you. Know doctors claim to be collecting lots and lots and lots of data but, in reality that, data is sitting with. What I think of it like as a broom closet it's, if, you know exactly what you put there when, you put it there and what, you're you know and do how to find it you can go and grab the right Kamil record, but if you sort of like I kind, of look looking, for something, like this the, data is very hard to mine and very hard to to. Work with and, the, electronic health record systems, which are these broom closets, there's, been a lot of work, recently. Talking about this including and even the popular press so you, know New York Times Magazine had how tech can turn doctors, and do clerical, workers talking, about the pain of recording. Data into, the EHR systems, or, great. Article in England Journal of Medicine called, getting rid of stupid, stuff which. Talked about how, many things in the modern, EHR system are really unnecessary and, how you, know entering, data into it is a huge huge pain. And, about. Close. To a year ago now in The New Yorker if you have not this is a really phenomenal article, why doctors hate their computers, talking, about all of the human factors, issues, that, go into making. The. Doctor experience, with. Their, computer, so horrible, so. You. Know you guys are all at the tech company so you may wonder how come we've ended up with this product that the users are absolutely. Hate, because, and it despised with a passion, and the, reason is the.
Electronic. Health records aren't really built for doctors, they're. Built for, the hospital administrators, most, of the structured, data within a hospital right, within your hospital, medical. Record is going to be structured around billing, what, do those people care about well what doctors, did the patients see what. Drugs were they given, prescribed, and you, know we need to keep track of those what, tests were on and who. Do we bill for this and how much. Actually. If you think about this, what's. Actually wrong with the patient is not, one of the things that they really care about it's. It's, a side effect of all of the other things that you do record, but it's not the core thing that. That. The systems were meant to record, and as, a result you know these. Are electronic health records, they were meant for recording. Information they're. Not meant for bringing. Value back to the user and. Doctors. As a result CHR, EHRs, as a necessary, evil and, really. Utilize it minimally, they're there to do their minimum. Amount of work to get there to. Get to be able to build a patient and then I'm done so. It's. So. And the. Second, side of this is that actually doctors still spent a lot of time on the EHR even though they tried to do as little as possible. Doctors. This, is in the ambulatory, setting, time. Spent clinic directly. On patient care is about a quarter of a doctor's time it, makes spend almost, about, 40 percent on each our documentation, or in review and this actually 27 percent does not include. The time spent on the EHR system while with the patient so. The EHR is a huge, fraction of what the doctor does and is. Really. Needs to be optimized, for their use. So. The. Result of this miss. Miss. Develop technology, for doctors is that what, we call patient centric care which is a big buzzword, in the medical space is that, technology. Is actually over, and underutilized, simultaneously. And you end up with cases where the computer, for rather than the patient gets the doctor's attention and. You. Know yeah. And while. The clinical notes are still quite. Sporadic, and. Technology. Is obtrusive. It gets in the way of the doctor-patient interaction. And, so. It's, both there's both too much technology, and it's, not being used, so when you'd really better methods, to capture clinical. Data during the patient visit. And. This, is I often. Do this as a poll when I when they give this presentation or version of this presentation in front of doctors I I think how many of you if you were sort of left alone completely to your devices how many would you want to use paper and, how many of you would want to use a computer and, their. Vote is always. Almost always like you, inherently. Like I want to use paper for. My medical records, unless, my hospital is forcing me to use a computer, so. But. They both have advantages and disadvantages. There. Are advantages, of using paper that's, a familiar, workflow, you can draw you can doodle very quickly it's. Cheap and. You can you know if your paper. Costs, next to nothing and, pens. It, pretty cheap you can throw them out if you don't like them. At. The same time there are disadvantages to, using paper records. There's. Missing information there. Is you. Need extra time to then transfer, to the EHR system, because at the end of the day you do need a digital note based. On current. Regulations, and it's a good thing I'm not saying that's a bad thing. It's inconvenient, to take photos or record, the videos if you wanted to you know capture something in more detail and it's. Impossible to compute with notes, you know doctors writing is as bad as advertised. Computers. Are also, have their advantages it. Can, be faster, than handwriting, you know actually you can type even. If you if you're a bad typist you probably type quite a bit faster than you actually write by hand, it's. Easier to search not, easy but easier at, least on paper it's. More structured and it's. Easier to share the identification, becomes a bit less of an issue. But. There are also significant, disadvantages. With using. Computers, there. Is missing, information there's. Things that sell that you didn't put down it. Can be slow and distracting, when things don't go right you know doctors, are finding, where is the right form for me to put this piece of data or where, in the medical record did they put this previously that I can find it later, it, impedes patient interaction, if you have a computer screen between you and the patient really does not work for a good conversation, and even if it's sideways and some clinicians, are amazingly good at, typing.
And Talking over, their shoulder as. A patient you still lose that, are. They, really paying attention when they're back to their computer screen, and trying, to find a piece of information in, the note and, it's. Still inconvenient, to take photos or videos especially, with modern with, with desktop computers, which is the mainstay, of the hospitals. And. Really, uh when, we, started thinking about well what should this process look like, future. Of note-taking for doctors, we, had to go back to 1945. To. See what they thought would, be the right way to for. Doctors to take notes and this, is a Vannevar, Bush article, called, as we may think where. He coined something which you called MMX which is a device that stores for, an individual, all of their. Memories. And all of their interactions, so. That which can be consulted with expedient speed and flexibility, so. It's. Actually interesting what Vannevar, Bush in 1945. Thought MMX, would look like. It. Basically, was something, that looked like GoPro. Camera, that's. Mounted on somebody's head and. A. Tablet. That. You can use, to. Draw. And. And and pullout, and useful and then she we also be shown information, so. You, know in 1945. He. Did not have these things, available to him but. In. 2019. We're getting pretty darn close to having something like that that you can actually pull off so. We, started thinking about what. Artificial. Intelligence, can bring to the field, of actually not medical, data processing, of where there is a huge, area work but actually clinical, note-taking, you. Know things like handwriting recognition which, has advanced, you know huge, have made huge strides recently. Speech, recognition where. We can listen. To the audio stream. Be talking between, the doctor and the patient and pulling out relevant. Piece, of text, identifying. Who the speaker is, theorization. Near, natural language processing to identify, what, are the key medical terms, that, have been mentioned in the doctor-patient conversation, and. Actually. Giving in real-time clinical, decision support to, give, the doctor, some thoughts, about the data that they have just recorded, so. All, of these have been brought, to build together in a sort. Of test tool called keynote add that was built by Josh one who is interning here now which is why I'm visiting in part and. To. The, idea was to build a. Way. Of better way of taking notes and there you can see this for, clinical, genetics so, this, is just one area of medicine among you. Know hundreds, depending on how finely you want to slice it but. It's. One. Of the things that you. Know we talk about is why is it so hard to do the good health or electronic. Health record system it's because you have to build it for all areas, of Medicine you, know my research group this certainly does not have the capacity to do that and what. So what we've been saying is like let's build something for one group genetics. Is good because we have a lot of interactions, with them it's good because it's very broad it covers all areas of, Medicine but. At the same time there, is the. Appointments, are much longer. So, in, appointment, the doctor could be an hour so. It's, a lot easier to fill in you, know like there's two minutes for a technical issue that we have to deal with or here's a new device that you can try it'll slow you down by a couple of minutes, the doctor is much less sensitive to that than primary. Care setting where it's sort of ten, minutes next patient ten minutes next patient ten minutes next patient, so.
In. This system architecture, it's actually built on surface, and that's what I'm using here to do the demo and I'll be showing you parts of in the pad life and. As, well as, we. Have a HPC. System, basically. Our own cloud where all the speech. Recognition happens we, also combine it with a local. Raspberry pi based computer, that is we used to record video over the encounter, if with. Patient consent obviously obviously all, of this with patient consent and. It. Can, be controlled from the surface, so that the clinician can start and stop the video for example on their, surface that they have in front of them while, and, same with the audio if the, patient. Does not want to once the audio off for a short bit of time that, can be done off of this off. Of the device and. What. We've done this on the surface because we've really found it to be the best in terms of the handwriting recognition and, everything, that we've played around with and that, you know it's been a good device to develop. So. Just, a very quick demo I'm going to show demo, just a small part of it live, and then I'll play a video for the speech parts I'm having some network issues from here connecting, back to our speech, servers in Toronto, but the idea is that it looks like you know paper it, looks like something, like this and. And. I, can just as a you know start writing. And. You. Can see my handwriting is pretty bad but. It's. Identified. That that scribble. That I did there is the word headache and, I can click on that and, it. Is now added, to my. Patients record, as one of the features of that my patient presents with and. If I can actually click it again. Just. Work here you, can actually set it to be negative somewhere, else I'll show you to me later and then, maybe patient, also has fever. See. That, no. It's too bad but. Favorite. Does not come up. There, we go. So. Headache and fever it's, one of the things that the system is doing is if. I write something like. Kidney. Cancer. You. Can see that the term that has popped up at the top is actually renal neoplasm. Which. Is the standard, way a geneticist would talk about it, kidney. Cancer is just a standard but that's the one that sort of been blessed, as the formal name of the term, obviously. The same thing and if. I write CA. I'll. Say hey did you mean cardiac arrest, but. Actually see it could mean. Multiple. Things. Coronary. Artery corneal. Abrasion, calcium, or cancer so. I can easily you know select that no actually you know what I meant cancer and new, plasm will show up and I can delete. Cardiac, arrest is not really relevant to my patient. The. Interesting part is if I you, know I'll show it again here. Far. Ite. You. Can see it's thinking, cardiac arrest but if I right. Breasts see it's, actually recognizes that the breast cancer that's meant so, it's aware of the context, of the text that I'm actually writing.
And It can determine. What the abbreviation, is most likely to mean based. On. Based. Based on that context, it, has also other some we have pretty simple cool features that clinicians, have asked us for for example can. Draw a little Square and say I'd like to take a photo and. Then. And. Then. Can, take photo. And. Insert, it now this, is not part of my note. And. I can draw, on it or I can sort a picture in a similar way or draw, a little, you know pictogram. That makes it usual, so my thought, here is that this is halfway between a computer. And a, and, the. Paper. And gives, you some of the advantages of both, similarly. You can easily swap. It all back into text and, then. Download, it as a, electronic. Medical record into your into, your main EMR system this has many other features like you can input. Whole piece of text, and. Identify. And edit, the text on the fly using a pen just by sort of crossing things out or, inserting. Things and then saying, and then writing out by hand the little piece of text to go into the middle of the medical record which, I won't you know show and the, other part that this has is the audio component and, that is not working live so I'm actually going to switch back and show a little video part. I. Can. Show you I can show you that. So. If you have an existing medical record, you can, cross something out here. At the very top and then, either, highlight and then you. Know write a note about that you, can insert new content, right here so, you just click pull it out write some piece of text and then the handwriting. To text will take it into, into. Actual and, and improve the note we've, used this for clinicians who wanted to basically, preach, art and. Bring, the what they expect the chart to look like into the exam room with them and then just quickly update, it, so that when they leave the room they actually have the note Grady so, clinical practices vary quite a bit between even, within, a specialty, between, individuals, so, for those who wanted to preach art we built that note, that in. So. Yeah and, the. Other part is the audio part and similar to empower MD which you guys possibly, know about or. Have built. There. Are components which allow you for speech analysis, you can turn it on right here so. And. And. Then, you, have a conversation, that. Conversation. Goes through the same text analysis pipeline and we. Pull out medical, terms and then, you have the ability to confirm them set them as negatives as relevant. To, your to your patient as your as, you're doing, this finally. The system of clinical decision support, and you.
Know They can actually show that live. So. If I go to right here I can. See you can see a patient has fever and headache it. Gives you ideas of what else you may want to look for and these, don't really make sense because these are all genetics diagnosis, for a patient who has fever, and headache and shows up with that the genetics clinic not to their you know primary, care doctor, so, don't worry you probably do, not have cutaneous. Hyperthermia. With headaches and nausea that's. Not very likely unless you've. Recently. Undergone, say you know a surgery, and and they had anesthesia and. But. This. But, this gives you a something. Else to think about and it's, short differential, list which, maps back to omean. Which is the rare disease they rare. Disease main rare disease database with diseases. That match and. If something isn't negative you can for example click open ID to. Say no fever and that'll update your differential, or if you add a. Couple. Other features that will. Also. Update your differential. And update, other things that you may want to investigate, so, that is to give this technician, in real time so that they can do, this while they're with their patient the, differential, process is part of the you know thinking about clinical decision support it's super important to get this done in real time as the, doctors with the patient you don't want the decision support once the doctor has left the, patient has gone home yeah. So, what we have found is that for the differential parts, you know our differential. And our decision. Support is much, worse than an experienced clinician at, this point so, experience. Collision gets very little value out of that but. A junior, clinician, actually, gets quite a bit so. The fellows who, are seeing. It's like this, is the first time I'm seeing disease, X I've. Read about it in the textbook but I don't, know about this and this is very common in the rare disease world this.
Is For. Them this is very valuable and it, so, it gives them some it does it's not this enough to be perfect but gives them thinking points to give them I have. I thought about that disease baby even if it's wrong but why, is it coming up on the list hey oh it's like those three symptoms match, so. It's it's actually as a training tool it could be very valuable and. Kids. Is a teaching hospital. Yeah. Some. Of these are obvious you, know but some of them you know, you. Know docent. Actally, I think have asked the patient to take their socks off it's, dodo sandesh likes and actly is one of the two fingers, Jordan and fused together if, you have dos and Ecklie you wouldn't know if you, have shoes on so, it's so. It's a, and. Do it knowing to do this as the patient is still in the room not you know calling back and saying hey by the way or you have huge toes, so. So. I think so this is the shows you sort. Of. The. Hope, can. Turn off this. Stuff this. Shows you sort of the brief overview of how fina pad works. What. I want to do now is talk, a bit more about the, machine learning that. Has made peanut pad possible, and this, is work by a, couple of students somewhat. Just wants help as well want. To talk about named entity, recognition for, ontological concept. So how do we when you know the, patient rights kidney cancer or sorry the clinician rights kidney cancer how do we know that they actually meant. You. Know neoplasm, or small, head is the same thing as microcephaly is. The clinical term, and. This is the work of our A&R Bobby a PhD. Student in, my group so, the goal is to really take on structure text here with assume. That the speech the text part has been done and. Annotated. With structured, terms so, there. Is a little. Paragraph here you. Can give you a few seconds to sort of scan. It, very. Briefly to see if we can no you just finally see there's quite, a few medical, terms that are mentioned in it. You. Know we can. Highlight. A few of them there's a few others, and. Then. How do we pull pull these out so, the. Terms that we actually want to pull out our human phenotype ontology terms. HBO. Is the rare, disease ontology, it's, great for genetics for, those of you who are sort of more, familiar with broader medicine you may have heard about sno-med, CT which. Is the the. Whole clinical. World HBO, is just the genetics you, know types of things you can see in the genetics clinics so, you won't have broken, bones you won't have like a broken, you, know fibia, you will have potentially. To break bones which. Is a genetic, phenotype. And. Then, you know which bone you break is sort of depends on what, part you hit, and. This is hard work of Peter Robinson the monarch consortium, I always make sure to acknowledge them because this is a huge service to the rare disease world, so. If you go back on the annotation, example. Some. Are pretty easy to pick out so biliary atresia there's an HBO term called biliary atresia it's, you just pull it out it's there you. Know you can do it with that ctrl-f for our little grep, others. Are trickier like. Craniofacial, changes, maps, to abnormal, facial shape and. Abnormal. Facial shape if you look at HBO it, has lots of synonyms, it. Has things like facial dismorphism, which. Is I think the standard term no. It's it won something it's commonly. Used, it has unusual, facial appearance has, funny-looking. Face and if, you're hanging around doctors, enough, you will hear. You, know them discussing FL, keys funny, looking kids but. Which. Is certainly, not politically correct but. But. Has doesn't. Appear in notes but certainly I have heard that the I've heard is mentioned by in casual conversation, but. Abnormal. Fascia but craniofacial, changes, is not one of those things so how can we actually identify, that from. The text. So. There. Are lots of challenges with this there are 12,000, terms in HBO so if you think of classic named entity recognition methods, you. Need 12,000. Different classes, training. Data for each of those classes and that's, not going to scale. And. So. Named entity recognition methods. Such as Nero's TM CRFs work, well for a few classes less than 10 when you have training data for those specific classes and.
Here. We have some, labeled data but it's really literally, a few hundred. Abstracts. Of medical, text that somebody has gone through and pulled, out hpo labels, on. The plus-side, medical. Anthologies, have a taxonomy. Of concepts. So what. I did not show you here is that different, phenotypes. Here like coloboma, and globe of neurons they are joined them to a structure, so if you know coloboma, and globe abnormality. Are both, close, to each other semantically, and they're all some, kind of an eye issue, so. And. More. So I'm morphology, issue so, this structure actually can help enable, better and the indentity recognition. So. To, do this we start with word embeddings which I'm probably in this audience don't need to discuss too much so mappings of words, into high dimensional, space so that similar words. Are in. Or. Near by vectors and, further. Combine the word embeddings, into phrase embeddings, where, you, know again pretty standard things you know as of you know two years ago I needed to explain this today I don't think you, know most people are would be aware but basically. When we take you know words we. Have. A convolutional. Layer max. Pooling. And the fully connected layer to build an embedding for a whole phrase, so the, phrase cancer. Of the retina you can. Take the underlined word vectors, and combine, them into an embedding for the whole phrase. So. Pretty standard stuff what. We have done though is because we have not just individual, phrase but, we actually have structures, among the phrases, we, have built a hierarchical. Concept, embedding where, the idea is that instead of embedding, each phrase completely, independently. We. Are embedding. It relative. To its. Relatives. Relative to its ancestors. So. The idea is that the, root term of the hpo is something called phenotypic abnormality, and that has some embedding, possibly. 0 0 0 every vector being 0 and that, I have normality, is something, that's relative. To this so positive, vector that. Adds something, to the vector. For phenotypic, abnormality. Cancer. Is, something that adds something. To there to its parent which is also genetic, abnormality, and then. Retinal abnormality, is something, that adds on top of abnormality. Retinal. Neoplasm. On the other hand is something, that adds to both its parents so it's one of its parents is retinal abnormality, in one, of its parents is cancer. So. The embedding for the term retinal, neoplasm, is going. To be the sum of the what it adds and, all of its parents so it's going to be this. Because, it combines what's, along. Of both, of its parent tracks and the. Idea is that this, can help us learn better embeddings, with less data you, know we can actually learn, about the fact that retinal, neoplasm. Has something to do with eyes and. Has, something to do with cancer even though those terms don't explicitly, occur. In the term retinal. Neoplasm. Right. Because there's, an i parent and there's a cancer parent. And. Then when, we have an actual. Phrase. We, can combine we can compare it with the encoding, that we get from the concept encoder so we encode the phrase separately and compare, it to where the different concepts, of the hpom code using. This hierarchical, structure, to. Actually get a similarity score, say which of these terms is the, most likely one so.
That Years to bring the ontology, into. The machine learning, and. This. Is a sort of a brief visualization, of how much it helps, this. Is the. Same exact. Same algorithm, but in one case we've basically taken, out the hierarchy and said embed everything. The exact same way but without using the using, their each build structure, the, exact same terms you, can see that without, the structure sort. Of like it looks like an impressionist, painting, the, colors sort of all bleed, into each other a little bit while. When. You use the hierarchy, you get much tighter clusters, for each area of the ontology this. Is just the various, systems. You. Can take a look there is this thing called grey thing that sits in the middle and here it's what's. Your words in here right, here so. The. Thing that sits in all in the middle of all of this is. New. Plasma because. You can have a neoplasm of any really body part so, it makes sense that from analogical, from a embedding. Perspective, a duck map to the middle because. Because, of the structure. So. We've compared with other methods so rule based concept recognizer support for HBO and for other biomedical, ontologies. Train. The only training data we used was, 228. Abstract, so basically, nothing and. With. We, fused 40 as, validate, is a set to test the tuning threshold, so the threshold of how close, do you have to beat something, to actually be reported, and the. Other 188, were used as testing and also, clinical reports, from about, 40. Undiagnosed. Disease program patients. These are patients seen. At NIH, for an undiagnosed disorder, where, clinicians, have built notes but also have recorded, hpo terms for every single patient. And. So. I wouldn't. Be talking to you if them our method didn't perform better than the previous methods you, know the data, is not difference is not huge here's an interesting part though so, the, - each is what happens when you drop the hierarchy, and you can see there is actually a quite significant. Difference in f measure if. You are if, you once you if you remove the hierarchy, from from. That from the method the hierarchy helps a lot so bringing, in medical, knowledge has, really improved our ability to do named entity recognition. We've. Also expanded, this to. Basically. Say well what what if I don't get the right exact, right term can I get the directories get a general idea to get a nearby term and they're, what we're measuring here, is to say well, let's. Say the, true patient had these terms annotated, in the ontology but, my annotation, reported. These two terms it's. Probably, not fair I didn't get right I got there's none of them exactly right but, hey you know I got a sibling here and here, I got a parent, of this, term so. How. You know how, good did I actually do so what you can do is measure accuracy. And based, on not just the term but what's, based on what's shared. Including. All of the parent terms HP ontology has is their property so. If something. Is selected, its parents, are also guaranteed, to be true so, if you have a collab oma you, have a normality. Of the eye morphology. And you have abnormality, of the eye and you have some kind of phenotypic abnormality. More. Broadly, so. If, you do this. Numbers. Go up and again, you can see that the method outperforms. Sort, of more standard rule based methods. Same. Thing with clinical, reports and. You can see here our recall, was quite a bit lower than on them abstracts. But. We, wanted to know why this is why did we actually work with or in our precision as well why are we finding things that are not in the, medical record, so. We actually look, back for three cases and did a manual annotation of every false positive, that our method reported there. Were 73, of them, half. Actually. Were judged by clinician to correct them we had more information things that should have been in the medical record should, have been annotated but, they were not added, to the HBO list by the clinician, about. 8%. Was something like slightly more specific like you know we annotated, limb ataxia and, the. Questions, just said ataxia, and there was like yeah, yeah. Ataxia, is sufficient, you don't really care if it's limb or not and. Would. Not. 17%. Were mentioned as negations, so what the doctor said is the patient does not have X. And, our. Method is completely unaware, of this and this is actually a huge problem and something that we need to improve on and, about, 20% of those were true, false positives, things that really were, not present. Usually. Investigations. Or something, like that so something that was a test or something that was just a mentioning, of a body part which, for the system felt was abnormality. Of that body part. So. It's. The, actual system was publicly available you. Can download it under, a BSD license, and you. Can play. Around with it it's, actually used in quite a few different places besides.
Phenotypes, Which I you know pad which I just showed it's used by the tool called Pino tips that also, was developed in my loop in, my group sorry. Some. You. Know used by a paper which is data visualization to. Mine the concepts that could then be visualized, and foundation. 29 which, is a group that's, associated. With Microsoft, and such as part, of is. Is. Using, this text. Mining tools to, mine, rare disease patient. Medical, records. The. Second part of again of how we can use AI to. Help, enable the interaction. Process so I showed you abbreviation. Disambiguation, and this is the work of a master student Martha Martha screechy. So. We go from here things like a fib with our IVR to actually, fibrillation. With. Rapid. Ventricular response and. I. Mean the motivation is kind of obvious you know people want to use abbreviations there it's, fast but. Disambiguating. Is non-trivial so when you say RT did you mean retrograde. Tachycardia. Radiotherapy. Or respiratory, therapy, and there's. A think, there's also, write the. Just word right could be another, example. So. Can. We disambiguate, these abbreviations by looking at the context, and, previous. Work has shown that you can look at not just the local context, but also what other words appear in their document, there's, a global context, as another signal, it makes total sense because even. Though the. Words right next, to a set word may not reveal to you the meaning the, general, subject of what the note is about can, give you a lot more information, so. This is an, example like if you had the phrase patient, was treated with RT in 2018. Well. What's RT could. Be radiation therapy, could, be respiratory. Therapy, probably. Not retrograde tachycardia. You don't get treated with that or not probably not right and that you can get from the local context but which, of these two is right well. If you look at the global context, of the note can you see that the other things that were in note, are si forget to me, chemotherapy. Thoracentesis. Cancer. Lung, sort. Of gives you an idea that radiation. Therapy, is probably more. Likely than respiratory, therapy, so. We. Do this with again, with you know deep learning, combining. Word embeddings. For the individual words in the sentence with the more, global. Word embeddings, weighing, everything by tf-idf sort, of so rare words gift, get. The higher representation. Combining, this into an abbreviation representation. And doing, classification with this network, at. First we did this on. Sort. Of what everybody plays, with their abbreviation, disambiguation, the medical setting with CAHSEE and mimic so, we trained all of our embeddings, on mimic which is a huge data set of of. Medical, records that are, come. Out of MIT and, then. We trained. The actual model to CNN, on on. CAHSEE using, which has about 500, examples for about 70 abbreviations, and. The. Standard. Cross-validation train, test split. And. Ran it and hey guess what it did really well so. All. Methods, actually do pretty darn well even, naivebayes get you 94%, accuracy but. You. Know with our best model which, combined, the local and the global setting, we're, able to get 97%. Accuracy and we were very excited by that and then. We sort, of said that's great let's implement this into, our model. And. And. Into this into. Final pad and very. Quickly realize, that most. Abbreviations, we did not have. So. We have, 67. Abbreviations. And Cassi for whom we have 500, examples of training data we. Looked, at all acronyms, which is a website, and we actually see eventually, us. We. Got, the whole week download the whole thing it has 80,000. Medical abbreviations. About. 20,000. Of those have multiple, possibilities in big you agents so. You really are talking about a, lot more than the 67. For which we have training data and. Our. First idea was can we use reverse substitution. To eliminate, hand labeling so let's, go through mimic every, time we see patient. Was given intravenous, fluid it was. Became. The phrase became patient, was given IVF, because. Intravenous, fluid maps to IVF and created. The training, set in this manner. So. Within. This so this becomes something with the label of intravenous fluid it, was like okay great. The, problem was that the. Patient the phrase intravenous. Fluid appeared. In mimic. 2,500. Times so enough for us to get a decent training set, another. Meaning. Of IVF could be in vitro fertilization. And the phrase in vitro fertilization. Appears in mimic zero. Times. It. Actually is interesting the actual, reason, for it we. Initially thought that just nobody wants to write out in vitro fertilization.
It. Was actually the reason was different the, reason was that the. Word vitro, was. Thought to be a last name so. It maybe has lots of examples, of in, square. Brackets last name fertilization. But. But. You know still, the point still here is that the in, vitro fertilization, as the whole does not appear even once so, the question is how can we generate a training set given, this. So. To, do this we've actually used multiple, other data sets we still relying, on mimic 3 and all all acronyms, we've, also brought, in you MLS which is a large ontology, of all medical. Concepts, that we're actually now going to use to help different, some big u8 the abbreviations, and. The. Actual pipeline works as follows we train fast X and word embeddings on mimic on mimic, we. Collect, training, samples from mimic using, reverse substitution. But. Then we augment, the training, examples. Using. Related. Medical concepts. And finally. Train the CN n for the classification task, so. This. Is sort of how this works when you have we, take all your MLS terms and their. Mimic clinic notes and you. Embed, the ml you are my less terms into. A high dimensional space so, there you will have things like. Realization. Fertilization. Assistant fertilization, pregnancy. In vitro fertilization. All in, one area while. Fluids. Intravenous, fluids, liquids fluids. And electrolytes these. Are all you MLS terms would, be in a very different area of the high dimensional space. So. Then when it comes time to figure. Out what are the, sample, training samples for and Eitri fertilization. We. Take, the. Term India fertilization. Say well hey it's really close to assisted, fertilization, for which, we do have, some. Samples, that we, can pull out from, the reverse substitution. And because, assisted, is not a last name at least according to a dynamic. And. There's, also terms like you. Know toxemia, in pregnancy which is not really related but it would appear in the same types of nodes so the global context. Is going to be quite. Similar or fertilization. Which is again not as close but again would give you some or. Would give you some notes, with the same kind of global context, and, then. We sample from all of these using. Basically, soft necks with a temperature, where. Temperature helps us really hone in this to how much how deeply we want to sample from the node itself if there are any examples, verses notes that are the verses nodes, that are further out. For. Things where there are examples, so intravenous, fluids we still do the same thing so, we actually you know have a fake, distance, of epsilon to, the actual node itself so that we don't sample from it completely but then we have all of the, nodes nearby, and again, the temperature basically, set the temperature to zero you will be sampling from there oops from, the node itself, completely. If you set the temperature to be really, large you're basically, sampling from all of these nodes with equal probability, and. We. Looked at you know how this performs relative to the various, temperatures, that we could set it's actually interesting performance. And mimic, drops. As you decrease temperature because, what really the model is doing it's overfitting, to mimic and the, lower your temperature the more you're actually, getting, things from mimic directly. As they are, well. Performance, on a completely. Orthogonal data set going back to the 70 examples where we have ground, truth the Cassie examples, if she, goes up as the temperature, goes, up and then sort of stable you know for stabilizes, and falls back a little bit so we ended up using Bayesian optimization. With validation loss on mimic to optimize, this but leaving. Out their sort of really small numbers because main liquid that's where it, sort of tends to push them. Up. And. You. Can see that when, you add this, relatives. The model that we have built, right now it, gives, you quite, a big push. In terms of accuracy, on CAHSEE both in terms of micro accuracy, and Mak your macro, accuracy about eight percent higher than you get with the exact same model but on. But. But not using the relative sampling, just from the nodes themselves and. While. Interestingly enough performance on mimic. Drops because, you're in the you know you're you're no longer overfitting, to the mimic data set you're. Actually building something that's much more generalizable. So. As a conclusion so breviary. Effectively, disambiguated, using, local, and global context. And, the. Prior medical knowledge can help overcome some of the biases in our training data but still. A large, gap you know remains we were able to get 97%. Accuracy by training on CAHSEE and testing on CAHSEE we're. Able to get you, know 73, percent accuracy by, not. Using a CAHSEE at all and any of our training and not having any hand labelled instances, still. A lot of work to do but, hopefully, at this as a way forward the cool part about this is now we can do all 80,000.
Radiations. Better. Or worse the. Problem being we don't know how well we're doing because we have no gold. Standard data to even to, test on besides the 70 that, Cassie. Has provided us. So. Finally. I just want to summarize some of the things that I've brought up in my talk and I. Talked. A lot about you, know what going, back to the original point AI enabled, note-taking. The. Cool what, I'm trying to do with this work is to really, use technology to assist the patient interaction, to make sure that technology is a participant. In the, care of the patient and not an impediment to, the care of the patient and to do, this you know the goal is to really capture everything no speech, and eventually video, we're already capturing, video a bit but, we don't really know what to do with it and there are lots of privacy concerns with capturing video so, at. This point you, know the but, at the same time eventually, you, know when, you say you, know does it hurt right here and the patient said yes well unless you had video you don't really know what what. Really happened so I think eventually we may go there as well. You, want to generate recordings, that are easy to browse going, back from the not, just looking have the whole speech but actually pull out information and. Locate. Important, information, automatically, in the future as you're discussing with a patient as a clinician is discussing with a patient a particular. Indication, you may want to pull out previously. Relevant, reports, oh you're, complaining about you know knee pain, oh let's. See I guess, yes yeah you had a you know soccer, injury, for which you had surgery on that knee 20 years ago how, do I know that well hey the, medical, record should be able to identify any, pain like okay let's find out all of the knee related. Notes, that I have previously further, for my patient. And. Provide. The clinician clinical, decision support on the fly beyond. The differential, diagnosis, which is what I showed you for genetics but, you, know what is this what is the right dosage, of a medication, you know on the fly I'm going to give you X given. That the patient is it has this weight and, you're. You may think about this dosage as starting those for this medication kind of thing and. Really. As, we're evolving we need to start thinking about. Electronic, health records, and we need to start thinking about electronic, health systems, so. An. Electronic, health system is an active, participant, in the care of the patient it's, there with the doctor and trying. To walk. The doctor through what could be the things that are a, problem, and read, and then it. Records the clinical data non-invasively. Helps. The clinician navigate previous notes and tests structured, data with computation, in mind and, be. Built with, data sharing in mind which is not something I've talked about but really, the identification. On the fly being, able to share a case with a colleague, on the fly and get, a second opinion quickly, these are all things that really need to be built. Into, the. Electronic. Health system of the future. Finally. I need to acknowledge, the people who did all of the hard work so, dish one led, the development of peena-pod working. With a, team of software. Developers and students, Martis Creta did the abbreviation, disambiguation, and had help from other grad students in my group and ariane, did. Named. Entity recognition work. And. With. Folks. At nih and, machine. Learning folks, at, toronto. Finally, thank you for your attention. And. Very, happy to take any questions. No. So we don't have that right in our, we. Have we we don't we, have, the, global context, of that to know I guess yeah we do have that we could combine it over various notes, for a patient, I've not thought about that it's a very cool idea, and. Because, that's obviously, would have a lot of, information. As well we've, just been doing global context, with at the node level so what are the other words in the note with, the idea thinking that a cardiology, node is going to be very different than a you.
Know Oncology. Note. You. Don't have to at. Least for the just as a follow-up question to that at, least for the data that you do at your hospital, you, don't have to be identified, content of you know, I mean for the data we're dealing with internally, no we don't have to de-identify it, but for, our data we don't have we have zero annotations, you know we don't have anybody who we're paying to go or throw and look at every radiation and see what it actually meant so. Data. You don't have it you, can't. Get one monster to dinner records because. Because. It's identifiable, wants to fund you to your interest. Yes so they changed, the patient, ID for every one of them so when you get that you just, talk you don't get longitudinal. Data. Same. Kind of question for the clinician. But. It seems like a clinician. Will be doing similar, things in, their practice, so. Right. Now no but. Obviously, you know when we're when we're looking at the global context, the, notes with other notes written by the same clinician, are going to have very similar global, contacts, so, it's, going to be taken into account a, bit naturally. Obviously. You know if you the more information. You can build into the model, you know same. Patients, you know same doctor second, visit for the exact same thing you. Know that's additional. Information, that can be used to better mine. The information, and better. Come up with a note you. Know on the empower I'm these are the team side you guys are generating. The note which, is not something that we're doing and they're it's like well okay take the previous note and see what has really changed between the two interactions, that, you know could be very powerful but, for. Us you know for now we're just sort of we're starting from the top and. Mmin. It's you, know the, difficulty with a lot of this work is not, machine. Learning it's, their human factors it's actually getting the clinician time and getting them to use the device you know hey. By the way we'll be waiting, by your clinic, door at 8:00 a.m. tomorrow morning to make sure that you try, using this with all of your patients today otherwise you know it's something goes wrong and you know they just ignore it.
Have. You done usability, studies like yeah, yeah, so we're in the middle of a usability study of finger pad where, we've used, it for about a dozen encounters, where afterwards. We evaluate, both the patient experience, and the doctor experience, where, I think where we think the patient experience is actually a very significant, variable as well so, it's given. That we have relatively small numbers and how I can say, much it's, the significant, but overall the, response has been positive, and. Are you like, what, are your key, performance indicators. On it like are you looking at face time it's. It's. Surveys. It's basically, if, we're doing surveys of the of, the patient after they encounter we're doing with, the doctor we're doing a bit more we're doing interviews, before and after usage of the tool to. Understand, their workflow, and then to actually evaluate how, what, went wrong or what went right for the for the for, the patient is just a survey do you feel like the doctor. Paid more attention to, you using this, tool than using. A computer. And the keyboard where. Do you feel like the doctor was distracted, with you, know with the stool, things like that. We. Catch, them as they're leaving the room. How. Do we the question was how do we, make. Sure the patients actually answer the survey and and, the answer is like you know if you're there as they're leaving the room and you use like can you please do this it'll take you 30 seconds, you. Can get that you can get pretty decent responses, their response time so their response rates. This. One. I'm. Just curious. Those are good to compare, different. For example mimic, versus, Cassie but. If you have a system that is taking notes and for, is to be since, you're being directing a lot the doctors enough what would they expect it's 90. Percent of the number today 198. That's. A great question. They. Want a hundred but, but. But. You. Know and, this is where it, you need to work around the, accuracy. To some extent so one of the things that I showed, really briefly here is that. When. You you, know right, you know are a. And. You can actually click on that and. Say. Well actually no I did not mean that term, I meant something else so, if you make it very easy, in the user interface for, them to correct an error. At. Least our experience, has been that they're receptive. To that but, obviously there's going to be some fraction who is going to throw this out and say you, know this is useless and so the you, know and, arancinis approach, is that there should not be user interface, and that's and.
That's And then no and that's and that's fair enough it's it's. But it's but. But. Your point is it's quite good quite this is valid, you know it's 90 percent good enough well if and the feeling is if it's 90 percent where in the other 10% you make a deadly error. No. Obviously. But, if it's 90 percent where another 10% a term, gets missed in the electronic, health record and really it's still there as are a just. It's, the wrong kind of RA. Maybe. It's dollar able it really, will be in the case-by-case specific, basis. Yes. Yeah, I think it's a question happens. Once, I, choose from 10 or 15 things but, then next time the next time I would wanted to really say yeah. It's always alright. Or not most the time I do this right so fight. My selection, choices, go down to something then is pleasure would say oh this, is actually kind of useful if it gets right every time if I have to repeat. This exercise, every. Single time I'll get relief us so we're actually collecting, those, Corrections. That they, Commission, is doing we're not using, them yet but. That data is getting collected in the system. Oh. Yeah I mean. Yes. I mean and then so it's important to get. That data per user and update. The system with. These and this is their again like the a I need to be, part. Of the UI. So. Along, the lines. Of personalization. Are, you thinking about. The. I. Have, my own abbreviations. And. I, want, to have them be a part of your system but maybe obviously. You don't want to affect the model because their, person. Yeah I mean. Again, I think that's very fair for, a commercial, product we, would certainly you, know that certainly would be the case I think where we are at is we're trying to build a, something. That we can test in a hospital, environment and, you. Know write paper, or a few papers describing. How. Well it worked and then potentially, work with you, know a bigger partner. In terms of getting it to a commercial, in. Getting. Some of the technology, that we have built into a commercial, grade system, we're. I'm. Not you, know planning, that this be sold, you, know to compete with empower and D in the next couple of years it's, it's it's not it's not our goal at all. It's. A great question we just we were just discussing this before about how you. Know it's actually great to take advantage, of the behavior of clinicians who tend to for example restate, everything. That's, you. Know I am. We, have knee pain also, you've knee pain so basically, using the fact that the clinician is going to restate it if it's important. And. Get confirmation, I. Think. That. It's. A good question I've haven't. Thought about it but one. Thing I would you, know say is that pretty. Much every single rule of thumb that I've come up with about how. Clinicians, behave I found lots of counter examples, so. You know talking about behavior, change as a whole it's, it's. It's. Difficult I would say that the rule if I, were to pick on something I would not pick so much on the clinicians, as on. Administrators. Who. Make it the, really difficult to. It's. Very difficult, to bring new technology, into a hospital. Even on a trial basis it's, for. Good reason but still it's that. The number of roadblocks thrown up sometimes, valid sometimes not so much is makes, it makes it makes it really, problematic, so, not, so much collisions. Like. Outside of the hospital environment. A. Lot. Of the work is really loved. Ontological work, in the you know is were really medicine specific, so that's. Going to be hard to validate, outside, of a medical setting. But. The same question. For. Testing cause I'm just wondering can like could this whole thing be mapped to like I don't know. The. Dog, world. Or some negative health. Was like is er sorry. I mean that's could be an interesting because. They have they. Have you, know yeah. Some of the previous tools we built for human genetics were then actually used by vet clinics. And. Actually, dog breeders, who, wanted to track their, dogs and like I had, a pedigree, drawing tool and they went to the track which dogs this headed from which dogs and though the pedigree of like a thousand dogs, and. But. Know that. Medicine may be an interesting one I've actually. Not explored that that much but, that's that's interesting thought. That's, a good interpretation, of what I said because that's not what I said but what. I really was trying to drive that is there like a system, that has, the same types. Of like an. Ontology. And, different.
Types Of ways that you can do the same connectivity, that, you're doing to develop your mapping in your models but. It has nothing to do with this. Idea. Like. Customer. Support as Microsoft. Actually, has it's very disgusted, yesterday alone like it has very similar a lot of similarities, with with. That and. Trying to camp troubleshooting. All. Over. You. Don't have though you don't have the real world like real-time face-to-face interaction. Which. Is its. Phone so you you, have audio, but. But. You know they're like the form factor would be completely different you know you still have an ontology you still have feel to, the fact that you have to write this gun typing, would be probably, much more efficient I. Mean. There are there are examples that I haven't thought about it that much about I'll think about it and even that. Medicine, is actually is is, not a bad example but I. Was. Wondering like if we, hey you know the surgeons, and to draw, us yep, John's, whoopie pie. Oh yeah, I mean obviously. Drawing. Is fundamental. And that's why you, know compute what that's one of the problems with computers if you want to now get it drawing into your electronic. Health record that's. Yeah. So you. Can import an image and draw on top of the image you, know having, and then that's what you know a lot of them do like they have these human. Bodies you, know think for example if you're doing, if. You're doing. Joint. Diseases you'll you'll, have this drawing where the joints are inflated, and sanely it's like the hands are you know like basketballs, but, like that's a little players hands and then the tiny tiny body because there's much of those joints in there and then you can like highlight which of the joints have issues things like that and. So. You basically can import that picture and annotate, on top of that and. This is I think one of the reasons why I think that the device. That. You can actually work with is. Valuable. And. Of our bushes yes. That's. Imply. A little bit with the camera on the surface. Yeah. That's. My. Question is are, your. Action so we're recording, the. Encounter. With a separate, video, already. We. Have permission to put a head-mounted video, on a, doctor, the. Issues we've run into is that, the. Cameras, are too heavy and. The. Batteries don't last long, enough. And. You. Know sort of it's it's they've been technological, issues. At. The end of the day yes you want that video. But. Then the other thing is I don't know the I don't have the first clue as to how to actually. That, data and videos. You. Know if audio brings up privacy concerns, well, videos. Bring up a lot more privacy, concerns, and and and a lot of just comfort, level you know do you really want to be seen you want to see be seen by your doctor or cyborg, you. Know right because that's how you know you will feel if suddenly, somebody walks in with, with.
That Thing on their head and so. It's it's. It's it's, certainly something, that we've thought about but, not something we've actually we, figured out how to act upon in grant, proposals. I do put you, know a doctor. With it where they go for on them but but we haven't we haven't turned, one of those loose on the kids in the hospital. You. Know, III, don't know I'm. Not I'm not like I'm not a video, video. Analysis, person so, you know I don't know how much you can tell by their from the experience of the video, but then if there if the hand is obscuring, then, the doctor won't be able to see themselves potentially. I mean they know where the hand is on the other hand but maybe you can use adjacent. Markers to figure it out but, I think the video analysis part of still is, is, still, very, much sci-fi, it's. You. Know if this audio, analysis, part is you know if you talk to a doctor if you still talk to most doctors today they'll say their voice voice analysis, part of sci-fi but. It's it's, actually getting their the video analysis part I think that this, the you know the technologists will think bit sci-fi at least my mind my view.
2019-10-17 12:45
