You. This, is. You. Know as others have eloquently, mentioned, before you know I'm I'm, here to take credit for the work of many others and this is you. Know my core team which, you. Might notice is, heavily. Weighted in, in in terms of the. Gender. Distribution. Toward, toward, women so and. I understand you know there folks. In this group who. Appreciate. You know the promotion, of women, in data science so, I'm, very proud to work with such a competent, team. Of collaborators. We. Also have a number of advisors. Working. In related, areas I'm. I'm, actually I'm going to kind of go kind of quickly because I, have like an hour's worth of stuff to tell you in about 18 minutes, so. I hope you had coffee. So. Again you. Know there's this idea that there's a complementarity. Between, what. Machines and humans can do and so. You. Know we kind of anticipate, that over time as AI methods improved, that they will progressively. Be able to do more and more of what humans, can do today and we're, seeing that happen and and I think you know Eric's doing some of the most outstanding substantive. Work that I've seen in the area for example in terms of acquiring. Real-world. Knowledge about physics, just through through. Machine. Based learning one. Thing though just, as sort of a thought, experiment to you the. Most interesting, things to me about Eric's talk were. Not what the machines could do but what the humans did so, he. Showed a slide, of. The patient record and he, said you know this particular observation. Having. To do with the intestines and, and and the. Presence of you. Know the drugs that tend to slow, that stuff down. Led. Us to do this other investigational. Study well took a human to, actually notice this thing, recognized. That there was an anomaly there and more, importantly realize that that was an important anomaly there are probably lots of anomalous, spurious, kinds of things but one of them mattered, and. Why. Is it that the human had, to be in the loop to figure that out why couldn't, Shean figure that out what's missing in terms of what machines can do so. And the other example, that I thought was worth mentioning is, you. Know with that wind mapping thing so. We. Have these gems, hidden gems but. It takes a human to realize that they're hidden gems and figure out what to do with them why, human, and not machine so that's. Food for thought. So. Anyway we believe that you know in the future maybe AI can can do some amazing things and solve problems for, us that we can't solve today but but today right now we. Can take the best of what we've got in AI and we. Can take humans, which. Today said but the, most human-like intelligence and, have, them actually work together, and. We wanted to take. This. Approach and apply to a real world problem and. Serendipitously. At the time that we were looking for a problem. I was introduced to a biomedical, researcher at Cornell University named, Chris Shaffer, who. Was. Studying a particular aspect of Alzheimer's, diseases and had, made some important discoveries related. To brain blood flow so we've known since the beginning of Alzheimer's, disease that there's about 30% less, brain. Blood flow in Alzheimer's, patients, than, in healthy brains but. We've never, understood why, and. So. The, important, discovery is that, it, was it's these tiny blood. Vessels in the brain called capillaries, that, were found to, be stalled at a higher prevalence rate, in Alzheimer's. In, mouse models of Alzheimer's disease, than. In the wild-type mice and and. Then it. Was sort of like 2%, in the Alzheimer's mice versus half a percent of capillary, stalls in the wild-type mice which doesn't sound like a big difference but, then when they looked at the downstream, effects the overall reduction in brain blood flow was actually about 30 percent due to those stalls, and. And. The. Other important discovery they made quite. Serendipitously. And accidentally. Was. That. Introducing.
An, Immunosuppressant. It. Was actually they were just using it as a way to sort of to. Tag, some things they were looking at microscopically, but, it had this unintended, effect, of. Releasing. The stalls so that suggested, that there was an immunologic response causing. The stalls which, ultimately was, was. Connected. To the the hallmark. Characteristic of, Alzheimer's disease which is the amyloid oligomers. So. You. Know in in, in further, studies they found that restoring, the the, blood flow to the brain and these mice actually restored, the mouse memories. Reduced. Other cognitive symptoms depression. And. And. So, it looked like a promising, Avenue, for for. Additional, research so. So. There are a number of research questions that need to be asked, that. All. Involve counting. These these stalls in the, mouse brains. The. Problem they had is that in, conducting. A single experiment. They. Would generate so much data that it would take six months to a year to answer a single research question, which is of course a you. Know a glacial, pace when it comes to to research so. This wasn't very, practical. So. And. I'll sort of cut to the chase we, figured out that we could sort of crowdsource, this data analysis, task so we created an online game that anyone can play called stall catchers and, and. By playing the game. You're. Not a subject, in the experiment, you're actually, analyzing. The data. You. Know along with or, on, behalf of the, scientists. So. We have about. Twelve. Thousand. People. Registered, install catchers this, is what it looks like to play the game so we show you a. Virtual, microscope, we. Stole this interface from the start us at home project, and we adapted it with their help and the, idea is that you, look inside the outline, add, an, individual, capillary, and you just have to make a single binary forced-choice. Decision. About whether, the, blood vessel is flowing, or stopped and that's, how you play the game. So. So. We have like I said 12,000, registered users day-to-day you, might see 5200 people playing the game we. Have the usual sort of distribution of kind, of like you know sporadic. Users, periodic. Users and super users, today. We're analyzing. The data about three times faster than in the lab so so. Now we, can answer research questions in, two to three months instead. Of six months to a year and for, this reason the researchers, are actually. Asking questions they otherwise wouldn't have been. Able to ask because of the the the high cost of analyzing, the data so they're actually doing, more research and asking. More, in-depth questions than they otherwise would, have been able to stock. Edgers is the first and only collective, intelligence system, for alzheimer's research data. And. We have some research results. Recently. The. Community finished. Analyzing a data set it was about a hundred and five thousand. Capillaries. From. Something. Like you. Know 30 animals and the. Question was whether or not, regardless. Of whether, the mouse has Alzheimer's, disease or not whether. Having a high-fat diet would increase the incidence of stalls and, and. So. The the preliminary result, this hasn't been vetted by the Cornell, Lab yet but is you. Know and it's very statistically strong, because of the amount, of data is. Is that there's you. Know quite higher stall rate with, a high-fat diet. So. The. Current dataset in stall catchers, is asking, more. About the upstream molecular. Mechanisms, that might be implicated in this stalling. Process. Because. If we can interrupt that safely, then. We could you, know potentially. Pursue. A treatment that does the same thing in humans and, and, restore blood flow to, the brain which hopefully would alleviate the cognitive symptoms, stall. Catchers has also had this ancillary, effect which. The. Wired journalist, thinks may be is is the one of the most powerful effects is that you. Know, it's. A very hopeless, and helpless feeling disease and so it, gives a reprieve to caregivers, and, even patients who play the game and, and, they, can feel like they're making an impact in, the research that could help them within their own lifetime. So, this is you, know sort of the substance, of what I wanted to talk about here. Because. This. Involves. You. Know the generous, compute. Time from. Our Azure grant and and. We've made I think pretty good use of that time so. There. Are different sort of, human-machine, partnerships, that we've had in this project. Originally. So. So, there so I'll just go over the three basic, components so first, of all we have to figure out where. Are all the blood vessel segments and the images and then, draw an outline around them because that's the only way we can show a human, this. Is the one you're supposed to look at this is the one you're supposed to annotate. We. Get really noisy datasets sometimes, we get arterioles, or venules that are huge and we draw these you know weird outlines that really don't mean anything and then, it's.
Wasting People's time because they get served one of these videos and they're trying to annotate it they don't know what it is or what to do with it we have a mechanism that allows them to flag bad movies but it's still wasting their time so, we. You. Know our exploring, machine based approaches, to to doing that and then. Ultimately. We. Have to do a classification we have to count how many vessels are flowing and how many are stalled in order to answer the. Research questions, so. At. The beginning of the project we actually thought we were gonna have to create two different crowdsourcing platforms. One. To find the vessel segments, and one, to do the classification. And. For. Finding those vessel. Segments, we want we had to come up with some way to automatically. Trace the vasculature in in these, 3d, image stacks and so, we were going to, beg. Borrow and steal from the eye wire platform, which is used to trace you, know the retinal neurons and, try to adapt that to to the vasculature, and and. Ultimately. Though. We. Sent, one of our team members to, Princeton, to, study for a week with Sebastian, Seung who created eye wire and learn. About their convolutional, neural network methods, and how they were, matically doing a lot of the segmentation, for eyewire and then taking that last bit that the that the neural networks couldn't do and that's what was. Put, into the eye wire game so, Mohammed, came back and. He. Actually. Yeah. Oh so, I don't. Know why this is this part of this so this is just to say you might recognize this face in the room so, the way we the, it was actually, right at that time that's why it's right here so it was right at that time that Mohammed came back and he said oh I think I can solve this problem using. Convolutional. Neural networks, and then we don't need a whole other game to do it because you know there's sort of this ethical. Obligation. We don't want to waste human time if the machine can do it. So. It was around that time that, that lea who knew about the eyes on Al's project, mentioned it to Bonnie and then, Bonnie contacted. Us and said. Yeah. I'll leave it there so you take the picture, and. Minoo. And then sort of the rest you know bonnie was very supportive and and and, then kind of helped us get underway so we, could and she fast-tracked us which was great because we. Were really in a hurry to do this because we had a user community, that we were building and, doing. It the old-fashioned, way was taking a long time they were hungry for data and this allowed us to. Develop. And test these automated. You. Know these automated methods. So. Then my. Other colleague. Ava, I'm. Gonna try to say her last name I've known her for three years I still can't say her last name but, she's. Done, most of the software. Development and a, lot of the the data, science work including, this, machine learning stuff and she came up with a model to automatically, detect these sort of what, we call bad, vessel. Movies and clean. Out the data set using this so this was another way that we use the compute resources on assure. Was. To automatically. Detect. These different, varieties and also to classify, and taxonomy, and. And keep track of how. Many of these were occurring. So. Yeah. Sometimes we're, sort of whimsical and. And so. You've probably heard of Galaxy Zoo this other citizen, science project, where you go classify, galaxies. So. This is actually, this is not the deep Sky Survey this is from. Our blood, vessel data but. A lot of these things look like galaxies, so we did a galaxy, classification. Of the blood vessels. In, any case. You. Know the so. We've achieved 92%, accuracy in, in, cleaning, up the data. And. Removed. About. 20%. Of the, movie so that's speeding up what, the humans can do by about 20% and and. The. 92%, is good enough we just need to validate that that, there's, no selection bias in that 90%. So. You know the the final aspect is this this classification. Task which is what the Starcatchers game does and. You. Know the basic idea here is of course they tried using machine learning first, and and. The machine learning algorithms, could classify flowing. Versus stalled and get, it right about 80 85 percent of the time so.
And. They were using the best off-the-shelf. Computer, vision algorithms, at the time in any, time something new would come out they would try it so, the, problem is this that, the. Value, in the. Crowd is, not. That individual, members of the crowd are better than the Machine algorithms, it's just that they make different mistakes we're. All different, and there's. Diversity, in our cognitive. Styles and approaches, so, you. Know when you have three you. Know machine learning algorithms. That are all the same algorithm, they're, all going to make the same mistake so it doesn't matter if you try to combine results, from three different, algorithms. On. The other hand you. Could have three humans each of which is underperforming, the. Machine learning algorithm but, in combination, and. In, aggregate you can have something that's very expert, like. They're. There I think one of the talks since. We were here, was. About combining. Different machine algorithms, and that's an interesting approach as well to this kind of problem and one that we're actually we're gonna run a competition and we'll, see what how much more we can do with machine, so. The. Question for us was how many humans do you need to. Combine to, answer about a single, vessel and have expert, like performance, we. Use some very, basic averaging methods at, the beginning and we had to hit these very high sensitivity and specificity numbers. So, we realized that you know as we increase, the number of annotations, it wasn't until we got to 20 annotations, for an individual, vessel that we had expert like performance, that's, a lot of human. Eyes just. One vessel and it didn't feel very efficient, so, you, know we continue to refine our methods and we. Think we're close to the theoretical limit but we were able to through and. I won't go into the details of it but to come up with a more dynamic method. That involves some some weighting, and monitoring, of individual, performance, to, get it down to about seven, people, per. Per, vessel, so. So, ultimately we were able to validate our approach and. And. So. Then they gave us real data to analyze and then something happened we gave them our results, from. The analysis, and then the lab said you know. We. Gave you a data set that hadn't been analyzed but we actually kept, part of it and analyzed it ourselves just. To double-check you there's, not really strong, agreement between what you're giving us and what we came up with so. We sat down with them and we. Realized that we. Actually had better agreement between, our, crowd. And. Their. Individual, experts, particularly, when you looked at only the vessels where there was consensus, between their experts, and our. Crowd and then in the remaining vessels where there was disagreement. We. We. Sort, of talked about each one and ultimately they agreed that. That. They, had been wrong in that the crowd was right so we we felt sort of you know vindicated. About. That and we. Like to brag about that. So. So now we're well underway like I said we have these new data sets in the process and we. Also realized. That, even. Though it takes humans. Using. This approach to achieve, the kind of accuracy, that we need. That. There, might be some very easy. Blood, vessels where. A, machine based system could, with. Almost perfect, accuracy do, the annotation, so, then it was a question of can we come up with a way for the machine to not only classify, blood vessels is flowing or stalled but also tell us how confident, it is about, its result and do that reliably, and so, another, way that we're sort of at innovating. In the machine learning space, and. By we I mean yeba it, is to. You. Know to build. And test models like this on. Measure. And, and. So far you, know in our in our early, efforts. We've been able to classify. Five or ten percent of the vessels with perfect accuracy using a machine based method so that's again five or ten percent fewer. Vessels that humans need to look at so so. The overall impact, of the work we've done with our initial Azure grant over. A six month to, a year period, of time was, to, completely. Remove this other bottleneck, of tracing. The vascular, vasculature. Using automated systems. We've. Removed. 20%, of the movies which were bad and. Where. We. Believe we, haven't put, this into our pipeline yet, but but it looks like we'll be able to at least reduce the human load by around ten percent in terms, of the vessels that need to be annotated.
So, You, know half our work is in terms of engaging. The community and the other half is improving, the efficiency of our platform, and the, combination, of those two things. Addresses. Our bottom line which is analytic throughput, all we care about is getting. To a. Treatment. Target faster, and. And. So, you. Know through our through, our data. Science efforts, you, know we focus on this analytic. Sorry. On the improving. The efficiency, of this, of the platform. So. So. That you know the work we want to do next is is basically to. To. Improve the sensitivity sensitivity. And, specificity of, our auto tracing, so we're not Auto tracing, perfectly right now, which. Means that if we get an image stack that has 1500 vessel maybe we're only using a thousand of those if we can improve that then. We, can squeeze, more data, and. Then, maybe the. The lab scientists, don't have to collect as much imagery, so that can speed up the research further. You. Know as I mentioned before we, need to make sure that that as we remove bad, movies that were not. You. Know sometimes we have false positives and that and that that. Those false positives aren't, pulling out more stalled vessels. And flowing vessels that there's not a selection bias there, and. We really want to as. I mentioned you, know through sort of introducing. More and different kinds of machine based algorithms, possibly, in combination. Start, to increase. The proportion of. Classification. That's being done by machines, so the humans really are doing. Less work and focusing on only the very most challenging, problems. And. And, we're always seeking to improve the, efficiency of our wisdom of crowd methods, so even though we think we're near the theoretical, limit we, think there's room for there's. Always possibly, room for improvement. There, the. Other thing is that we've been approached by by researchers, and other disease research areas who, realize, that our platform could be applied to their, data, so. There's. A researcher. At University of Pittsburgh who's got huge, datasets of sickle cell anemia data the. Difference there is that instead of looking for flowing, or stalled blood vessels they. Want people to pick out what are called torturous, vessels, so instead of being vessels that have a nice kind of flowing shape that sort of all twisted, up that's I think, the hallmark of the the sickle cell. So. I. Guess, the the last thing I'll say is that we've. Set our sights big we do a lot of. Sort. Of community engagement events, sometimes, they're they're synchronized what, we call catcheth ons where everyone. Placed all catchers for the same exact, one-hour. Period, of time all over the world we've done two international, catcheth, ons and and it's been a great way to build, our community and, generate a lot of excitement around what, we're doing, we. Were aiming high. We're. Going up a couple orders of magnitude, here but we have. A lot of reasons to believe we can this raining. For April 13th of, 2019. What. We want to do is Ganga engage, a hundred thousand participants all, at, the same time all within, the same one-hour period, primarily. Through libraries. And schools where we have some inroads. We're. Creating a meet-up capability, that will bring people together in those venues for. That time we. Want to analyze an entire research, data set that would normally take six months or a year, and. Answer a single research question in that, one hour. And. We. Fortunately. Will. Have a lot of help in, doing that. That's. It thank you.
2019-04-25