You. So. Uh as. Many of you are familiar with we're starting this brain. Computer interfaces effort. Inside Microsoft Research it's a cross collaboration, across a couple of teams here and in. Looking, at the related work we came across a Natalia, cuz Mina's a recent. Journal paper was a great overview of the various efforts in this space, yeah. You did your PhD in Kelowna. And. Then in. Yeah. It in RIA and your. Postdoc with patty maze yes and you have a couple overview slides on what the groups doing yeah well that don't know the Patty's, group in media that so, looking, forward to seeing that and then afterwards, we have like. A roundtable discussion. Thank. You so much for this one welcome so, first as Christian, said he, actually asked me to do some kind of, internal. Very fast overview about the group so we will have a quick, grasp but I'm gonna talk about and. Then I will actually move to besides, so. Oh yeah, my name is not patty miles but she's a principal investigator at MIT, Media. Lab and our, group is called fluid interfaces. So, nothing to do with physics, news, fluids, it's, about, technology. So. We. Have a grown in number, of devices on us and they're getting with each month closer and closer to our bodies, but, they take our attention our memory, our cognitive resources, and we just start to understand, what. Effect, they're actually. Making on us and, the. Interest of the group is. Can. We create systems that can sense a user's internal. And external state. And intervene. To seamlessly enhance. Our cognitive abilities what. I mean by that is that right now if I take my phone which is obvious a very very useful device. It, makes, my life easier I don't need to remember a lot of sins but I'm spending my life like this and, can. You do something with it because we're still gonna have those and we have will have more of those actually but. Can we somehow help. The user to, be on task on hand to, be with others, and not be. Dividing his attention, memory. Etc. There's. Really, more. Than 20 projects going on in the group and we, are focusing on decisions, attention, well-being memory, dreams, and learning I'm not gonna go through, all of them because that's not the focus of the group but I'm gonna show you couple of ones so, first, one of the projects for decision-making, and this is a project as it is called ultra, eager and there's a project of now.
A PhD student Arnav Kapoor and he, I'm gonna show you Vida so. It's a variable system. As. It allows, using. EMG. Muscle. Activity to. Pick up. The. Civilization. Of the roads so as you can see he's not pronouncing, any single. Word but he, is subclass on them and this can let the system pick, it up and through, the bone production provide. Him with the feedback in real-time. There. Is two, electrodes, there is also a bone conduction microphone. And. Obviously, a bluetooth, to the Ranger. So. This is one of the projects, this, is actually, a very good example of what we are doing in the team and with my project on BCI when I see it the systems are variable. They, are seamless, obviously you can see it on the user but now we have her actually, Russian that, is much, less visible they, don't require, any, external. Device, to function, and. They're not invasive this, is another example this. Is a project called by essence so it's an open source of factory, display that monitors cordero spiritual information to support mental well-being so. The. Device can be worn in different form factors we have once again even smaller, ones right now and. Once. Again it doesn't require, it's. Wireless and, it can release different types of smell why doing so because a lot of research. Points. Out that when, we smell coffee, or, peppermint. We. Are getting like more awake, we, are getting more attention if like right now I would spread some along Keylong for example, or Peppa means you would pay more attention to me hopefully still do I'm. Going to show you some other kids how to do that as. This is an example of actually how the system, works, it's a hardware and in the software user, can control all the, components, the time, of the release of, the smell etcetera. Finally. This is going to be presented, at chi in one, months from now in Glogau, so, this is a nasa, variable, system, so as you can see all of the projects, are variable, devices, it's, a variable vegetable, assimilation device that, stimulates. The vestibular, system to. Use sensations, of motion, so, as you can imagine. The. System this paper was actually tested, in VR condition, with like Russian mountains but, could be used for an almost driving, in any, situations, of motion, sickness so, it's electrical, stimulation. And. Also right now a bee he's, a master student in the group he's testing it in the projects with NASA in zero-gravity. Because. MIT has a lot of open. Space initiative, so they're from poles they want to bring this. To the. International. Space Station actually.
This. Is this. Is just three and I'm obviously gonna talk about some, of my projects, in actual, talk but we, are interfacing, with sleep. For example masca it's. A sleep. Mask that can detect the REM States, and as, a project, its EDA, it's, called Urmia I detect. Ram, as well and wakes person, up and Prime's the person to talk about something he or she wants to dream about. There. Is fast. Shares that could. Be it's. An Opel modular system, which can be sticked in any VR. Or AR, headset, to collect eg e OG so brain activity, eye movements. Skin. Conductance. And. So, much more so actually, if you do want. To talk about those we, can definitely also, talk, about those a bit later so, it's a very very short of, a brief so, while I'm switching to presentations, if you have some urgent. Questions, please don't, hesitate. Sorry. What. It's. AC. No. It's only, so. I tried, it out it's, actually, just, here on the one, point on the right and on one point on the left is kind of next to the ear, so. Two points and for, that zero, gravity project. I haven't actually tried it but I have seen I think I even have the images it goes so two points, next to the ears and the, third one goes here, in between the eyes so, it's basically like three. Point mini. Head. Cap. Those. Three points on him. So. Now. Up to brain computer interfaces. I call. This community augmentation, 1-0 and I'm going to explain why is this. Name so. As Christian, mentioned already, I did, my master, in Grenoble, in France and artificial, intelligence and then, actually I'm doing BCI. Since, 2010. When they were absolutely, not popular, no one was actually doing them, that. Much especially not, HCI. People, I did. My PhD I, got. My PhD in 2015. Then I moved to another, city in France called ran to, do some ARV, on with, busy eyes and. Since. 2017. I am at Media, Lab and I'm continuing, doing brain computer interfaces. So. Why, actually brain computer interfaces why should we even care. Everything. Started with Big Bang and. Obviously. Has, evolution continued, we. Appeared and though. Dimensioned, without. Creating a lot of tools and. These. Tools are making our life easier, simpler, but.
The Problem is that they're. Not perfect we. Actually. Lose, each single tool we are having here the. Cognitive, demand. Is increasing but actually is very limited. Our country, resources are not getting any better and they're not going to get any better. But. Vc, I. Might be a solution, and I'm gonna in this talk on top of giving you state-of-the-art showing. You what should be done what should not be done, I'll argue, about this point as well so, when, some people hear about this guy's they usually and most of people think about this I am, pretty sure any, of you at least think maybe one of those alters, carbon, matrix you. Name it and they all look very interesting they look like implants. Points. In the neck something. Else well, unfortunately reality, or fortunately still very, basic, you need to have it on your head right now or inside, of your head we. Don't have any other options, right now there, is like one Russian it could be even embedded, in your bed so. I'm. Not covering this presentation, a lot of details on signal processing but we can definitely talk in offline but, for those who might be. Beginners. Be, shy usually, works as falling we, will put. A headset, on. In. The case of the non-invasive systems, on the head of a person we, will do the acquisition, explicit. Implicit, it means that we'll ask a person, to, do something for our so maybe we will not ask a person to do anything we. Will need to go through signal processing I will explain why it's obligatory step, we, will need to do some classification. And then obviously, we. Can do something interesting with the signals member to assign them to some interesting applications that, are of the paramount, importance, for the user so. What. Do we usually use for, brain. Computer interfaces. We. Usually. Use several, different modalities. Visual. Modalities, visual techniques for example and, non-invasive engine, caps. They. Are very, popular, but. It's a nightmare a lot of people say oh when you get.
128. Electrode, setup as you can see here on my head you, will get the best signal possible they. Are good for selfies like this and I would say that's. Kind, of eat, why. Because, on the right that's. What you need to do to put this headset, on me you need to literally put it in. Water. And fool in the solid solution and, then, put. Obviously, some car on me and then. Trust me to wear it on top of this this, huge white wire. It is actual, wired, part, of the headset, it's, connected. It's violet literally, through the machine and. You're. Actually not getting better signal, because imagine you, have 128. Electrodes, dimensionality. Reduction issues. And all other problems and just the fact of installing, is it will take at least 40, minutes. Does. Not rust in 99%, of cases you, will go and opt for the solution in one single case if, you're looking for something for. A phenomenon the brain that, you don't know yet where, it is situated why, would you go for this because, what usually happens, even if you have 128. Electrodes all, 256. You, will still end up analyzing. Some. Local. Sources, so. I would say go, for this one only if, you really want, to investigate in neuro phenomena that you don't know, yet. Where it is exactly or, you try to reimplementation. From. Meg, or from ephemera I'm going to mention those. And. You really, don't know exactly where to put your, electrodes, that might be a version also, everything, that is for source localization, that. Would be also a go-to, solution don't. Go and don't touch your user it will not reverse it you will not obtain a, better signal, or a, better classification. Output. No, way they you, will gain 5-7. Percentage, which is not significant. No. That's even worse so. A lot of people think oh if you are bold is gonna work what. Happens is that when you are bold your. Skin actually grows. Thicker. So it's actually much, worse their. Boldness, is worse than hair the. Worse that can be it's like this kind of dreads, like, traces. Like that's we will need dropped, four completely, different types of electrode I'm gonna talk about this so. What. You have right now is, actually, a huge amount of different, EEG. Headsets, and headbands and they are much better, you, will lose sometimes in quality, but there, might be an option to go depending on the application, and I'm gonna talk about this another. Version, and. This is I would say, the, one that is running right now in, there. Demoing. Stuff. Of applications, that you can do with busy eyes and I'm gonna if you have time I'm gonna show you actual videos, this. Is fMRI, as you. Can see here it was taken, when I was still in France so it's in French but I guess you can still guess what it means, with. This system I. Needed. To do to, make sure that I don't have any metal in my body. Pregnancy. Test just in case to make sure even, if I'm sure and, then, you need to obviously be. Ridiculously. Lying down for, like two hours it's, very noisy setup, in this particular case in my head I also also have an EEG. Compatible. Cap which. Is not easy and the, references. In this case for those who know where how, Eugene usually works actually, on my. Neck. All, night, back so. On top of all of this it's actually very uncomfortable. To lay it down you. Should not be close to phobic it should be pretty healthy, this. Should be only particular sizes. In the sense that fMRI, and even. The cap if you want to do two, modalities. Why. It is good why still people still go for this one obviously if you either you have one in the lab and means that you're on really serious, brain, research, or. You. Really want to do some are hardcore, visual. Decoding, I'm gonna show some meters if you do have time it's, amazing, what you can do with it. Temporal. Specialization. It's another story, and. Finally. That's not the final ones there are more but, this is one that I can see the interest, in example, this, is Meg, so, the.
Idea Here is that on. The, left here, you can see the scanners, it is based, most, of them are based on squid sensors, on, the, right these, are optical sensors. So why it is so interesting and because you see like, I showed it to some of my interns to say it. Looks even creepier. It's, not about creepiness, here it's about the fact that this version is, portable. It's considered to be portable this, is a nice example how, technology. Moves forward result. Of startups, freaking Silicon Valley. Open-water. And, some houses that drew claims is that developing, something that will break fMRI, in cost by 6 to 100. Times and in quality of the signal but, this is where we are right now so there. Where, the most improvement, is made is still in these versions why are you trying to go into. Mobile. Version, or we, are still into a gym, I, want. To finish this very quick overview of technologies, with this image because I think it's very interesting here so, you can see temporal. And spacial resolution. Up. To actually. Local brain imaging, and what. Is interesting here maybe. You. Can read it out it's like a raspberry color, was there ultra-fast, ultrasound. Imaging, I don't, know if you know it's pretty novel it is research in ultra-fast ultrasound, scanners is not normal, at all but, I assisted, April. 2018, to the talk of one. Of the co-authors. Of this technology and it's. Possible right now to. Do it on the neonatal, so, on the babies and he's trying to do it this summer, he. Invited me to come over to do some engine with it to, do it on their adult, human, why. It is interesting. Because. As. You, can see special. And temporally, we, approach. Dramatically. The, implanted. Ages, and this, is non-invasive. Not. Gonna go into details, about this one I can give you the reference but, this is something that you should consider still. Things that are interesting that I'm moving forward right now that, we are six, years ago when I just came first time to the lab it's in Paris actually I was.
Like Well yeah it's interesting, but nothing really six, years later they're. Really about to do it on actual. Adults, and the. Number of frames per. Second, is like 60, or 70 it's like amazing good resolution, but, if you got it I did. All my 10 years of BCI right now in eg and I. Still consider that it is still the place to be in near future at least for these guys but. You just need to have several things in one where you try to design for. And user first. Of all. What. Do, we used to measure and this is very important. Usually. What happens is that people who opt for 128. The most expensive, is the best one or we, only have $200, for a muse, headband, nothing. Against miss heaven but this is kind of extreme situations, you, really, need to understand, what are. You actually doing, with your application. Here. On the top left it's, hybrid, electrodes, released in, September, 2018, by GTECH as they're, based in Austria why, they're hybrid because they take both gel. And dry, at, the same time. Here. And I work with them since, 6, months already it's. On the hand but actually, I'm. Using them for eg. Its griffin based electrodes, you might have heard about those the. Problem is that the, extremely, sensitive, to noise. There. But. You can see why they're so good I don't actually need so I wanted, to put one on me and it will show you how it looks but you would not barely, see it on me it's 80% transparent. And you can do it fully transparent and on top of this this. Is an Alec chart imagine, if you if you have the whole PCB, so if the whole amplifiers. The whole module is, gonna be in these, lines. This. Is actually brought several, with me ours are somewhere in my bag somewhere. These are hydrogel. Electrodes, also pretty normal, pretty recent, I know only for now one band that, is, having. Them very. Good quality as well some, things to keep in mind it, is a pretty classic ones gold. Plated pins and. Once again I just, put one image, I'm using natural. A ninety, two point five percent silver. This. Is also, gonna improve, the signal quality dramatically. And drastically so. This is just a very quick overview that, think. What, you gonna use if, your task is. Person. Who has an, LS. You. Need to think will the, hair be, cleaned, each. On our thirty because, I was doing this for five years it's. Very annoying, and when you cannot tell that it is very annoying that your hair is dried there with the scream you're. Gonna destroy, even, 100 percent accuracy this, iconic can not deliver you 100, they, are happy to take 60 and 70% but. If they don't feel comfortable they're, gonna tell you right away they're. Not gonna use or opt for your application, for your system for what in the world even if you pay them for to do so if, you are trying to do something that is more than, style, something that is about, applications. That will monitor cognitive, states and I'm gonna show those you. Don't, actually need, to opt out for anything of this quality you, might think, about something, of this type or hydrogel. Why. Actually should, you care about what, you are using here, it's just one example I didn't include all of them but you just to show you that there, is comfort there. Is a signal, gain and there, is a shelf, line so. It's. Different for, different applications. They, also have different price manufacturing. Of graphene and a fraction of hydrogel, is 300. Times more cheap. As an AG agcl, so this is very important, on. The other hand for hydrogel, you need to moisturize them at least 40, minutes before you use them that's, another thing it doesn't sound very okay. 40 minutes as okay but if you forgot your. Basically, you're not going to be able to execute it because I need to delay it for 40 minutes just an example of these, little details thinking. Without them beforehand.
Think, Is the person gonna move because they will move in most of the cases if it's not six months before. In the ALS or like some Mir parties they, will move they will create some noise they, will move the hair someone, will move their hair and they don't know all these details but you can think for them. Graphene, tattoos, can stay on touch, for, 24. Hours, and up to five days, with, the deterioration of the signal only by 19%, you. Don't maybe need someone, to wear it for five days for other reasons. But. What if you would like to monitor something 34-7. The. Could be another option so. What. Do we actually measure a lot, of sense really, a lot I'm not gonna go through once again all of those but. I'm gonna show ones the most promising. One the most I, would, say discussed. Once and then, how's that we can actually work towards. Application. So why is the actually user opt, for, those so. One I would. Say number. One. Sighted. Because they. Are kind, of the easiest, from the signal processing perspective, our. Earpiece. So. Earpiece. Look usually, let's, say right now I have a coffee with, milk if. I'm gonna tell you I, want, a coffee with a cat. 400. Milliseconds. Later, approximately. In your brain that will be a peak of activity that will appear because the phrase that I just constructed semantically. Is incorrect if. I'm gonna tell you oh. I'm. Right now in Google and then just relized, know I was at Google yesterday, right now I'm at Microsoft, I. Just, realized that I have, made a mistake and, before I even corrected, myself, there. Is another pic of activity, that will appear the, fact of me realizing. That I just pronounced something that is not correct, error, related, potential, requite. A few of those usually, the name stands for and, of, P it means when, it will occur is it positive or negative, and then, the timing so the name usually 2 or 200 for, 400, it's approximate, timing so. Is it a quite a few of those, we. Published, last year for example it, and we see a paper about, cedar Birds Eye versus, natural roots and. This is one more thing when you pronounce the word and a, person doesn't know what, the word is you can pick this activity, up and using and, 400, on fp1. Our electrode, for example. What. It is interesting because right, now for. These earpiece. You. Don't. Sometimes, need. To do any, training. For the user you. Already. Have, databases. That are good enough to, be initialized. Fresh. Initialize, at least. And then you can either do a calibration instead, of training or you, can give like several trials to the system and that would be enough to have a reasonable. Classification, accuracy. So. A, lot of research right now and a particularly work on motor, earpiece, you can see several, papers I think it's C as this year's even presented, I don't, remember what car, was it was Honduras, someone else who, was actually trying to show this off here Peter can be seen, before, you do the actual movement so before you turn left or right we. Can say I gonna turn, Lou after right between, 40, to 400. Milliseconds, before you were actually performs a movement this, was proven with fMRI was the past planning. Seven. Minutes, before we can traced with 90% accuracy, what passed you're gonna check I will ask you question can you tell me how you go from your from, your home to your work hopefully. Each day, it's kind of the same way you do it so we can trace this in the frame around and we can use the model to actually predict seven, minutes before how you're going to actually execute it for, example if there is I cannot some cleaning. Machines that when I did there you'll usually turn that, little Street left because, you know that you should they do Queen the throat because, you do this for years and years. Another. Very. Interesting the. Next four slides it has a small logo of. Albany. And Neuroscience. Institute so. I shall borrowed them when I was given talks there, because. They are specifically, specializing, in p300. Why it's interesting because it, is one of the most famous. Accessibility. Tools in this child so. They, just recently and I'm gonna go into the idea of p300, is very similar to what I just described, in the case of earpiece, you will provide. Target. And non target, and to, help you use it you will flash the target, and they usually need to focus on target, and ignore the non target, activity so, why it's interested in this particular one because in July 2018, if they, finished, four. Or five years study, of this system, in the wild with ALS, patients there.
Wasn't A total 118. People who opted. For the study, unfortunately. 48 died within. Two years that's one again, LS. This all they say all these issues it's extremely, interesting paper I talked to them it's extremely interesting the whole effort specialized, it seems this right now just, pretty powerful, by actually prove this like someone asked me why do you put this so on the side I think it's very instant because it's a person who typed the whole email using the system, using p300, so. Insides. Are not mind but I think it's very interesting here, to. See. How. People, actually use it for example that sorry, for they call it's in black I'm gonna go like this in white so you can actually fall for example, it's, actually simple phrases for example to say in for the caregiver. Don't, let the dog be don't. Speak. Louder, when. A husband, messaging, her eyes she, does not to rob them so. It's, simple, phrases but, they make a, huge. Difference. To the user so the, paper does state the. Interest is they talking, like, trying. To address all the issues how. Patients, and therapists, talk about BCI's and the trying the third column just says it actually we have solutions. For all of those problems comfortable. Not practical breakfast practical, not comfortable, how to use it without a specialist. So yeah. It's actually very interesting the. Caregiver, was trained one, since three six months the person was coming over for checkup if something is lost in connection, you, will still be able to see this online everything, is actually reported online it's, a huge huge world, cuz the amount of users might not be particularly, impressive, especially for, us used to massive. Amounts of data but, this is all Els, people, from. The Secession of veterans of Sears and I obviously. Could. Be gone, much, further, in the condition this was their trying to get I'm. Going to let you check the paper and I can give you the actual reference another. One, it's called steady suitable, potentials, this. Is um here. I keep Reese it stands for visual. So. Once again just before I play the video because I will not play it in case it's the problem doesn't, even have any epilepsy. Any. Risks. For epilepsy in the room if, not. I'm gonna play the video very it's. A great fun one okay. That's. Exactly, immediately. I have shown you the problem of this system, you, need to ask this before, you're gonna place this video. So. Here what happens you choose it one. Of my experiments some time ago so here you can see what actually, happens, to. Stimulus. Flickering. On different frequencies, you, can plays it was the size so what am i was trying to do here is like to see how, can we reduce the size can we keep it to a small dot somewhere on the screen to pick up and hear that specification. I carry Silver's in real time so this is for ample not train system at all so you have two classes the binary problem, and to.
Decide Where a person is looking on the magenta on the yellow one so, the idea here is that. User. Will. Look at something that is flickering particular frequency could be on the screen it could be a physical object and you, can pick up this activity, there. Are some rules you, can, adjust put it in each single screen. From. Here you can see that's using, the first version, of Holland's. And. Using. The same paradigm so, there are some rules too. Big the cuted for example you can see you, can only take multiples, of the flesh. And rate of the screens that you are using as if. First. That you go they're more, visible and becomes so. You really need to think about the refresh rate of the screen etc, but. In the very end right away. With, no. Training. At all it can give you 75. Percent doesn't, sound a lot, it's. A lot for classic, design. Another. Example which is much. Less. Relevant but. It. Is an interesting example this. System was used to. Control particular in this example, a. Tail. Of the, rat so. A person, was looking at, as, flashing, frequency, and the. Tail of, the of. The red over there was, moving, so, any, personal translator to tell was moving on a person was looking at the rides that L was looking as a ride or, I put it here is because actually I found, at least 20 different papers, who, do this kind of application. Controlling. Someone's, else, part. Of the body, there's, also papers with cockroaches I, didn't put them here, red, still looks a bit better. Energy. Or, motor imagery, so I, guess, you might know that when you. Do. Perform. A movement there, is obviously, some. Activity. Some activations the promoter. And motor cortex s and this is extremely variable started so it's called the synchronization, and I. Would, say motor imagery, is one of the modalities ism studied. The best especially. In particularly in the imagination. Class of this Yaris here's, something, I have done. Okay. Some. Years. Ago even four years ago or something so, if your person, is imagining. Two. Hands, to. Tip of the drone imagine. A left hand movement to. Turn the drawer left and I can you write the woman to turn the drum right and then, both feet. To, landed, and as. You can see. Here. On the video. So. It's in real time which is sold and. The. Person is not moving, and. A, person is varying, and. And. Here, we can see it very well but. The. Elephant's only position on the motor cortex so, is it's just one more example of showing that when you know where you look for a signal you, don't need to go with, 128.
Electrodes Or 64, electrodes or even thirty two electrons, you might be very precise if you know where you go this, is another example of the system this is where emotional imagery BCI, was used to produce, binary. Words and which, was then transmitted, to a receiver, in, the form of the protein 3 HTM, s so, it took like this one person, is looking. And, imagine, in a movement and another person, is decoded. In words. Or in letters, and as a person who actually see, them. It's. Just a two different, example. Just to show how applications, could be twisted because people think oh what the imager is only for movements you. Can actually use it just as the control signal for whatever you want to, be just like a brain switch you. Think about this and this will happen let's say it's like a shortcut. Obvious. The motor imagery is actually. Very well studied if you really want to try it out tons. Of papers all it is there but, it's pretty limited a lot of participants. Are. Considered, to be actually BC illiterate they don't even know how to train themselves to use it we don't know how to train them what is the problem, maybe. Because we don't have a lot of options as well and particularly. There. Are a lot of other. Cortexes. There that also can use imagery, and we. Might be interested in decoding source so. This was an inspiration for. Other. Of, my works but actually, initially, was published in fMRI. Domain. Mostly. So, it is imagination, of objects. Particularly. This one it's not nature scientific. Reports, in last year think. We. Were interested in seeing if you can detect, when. A person, is looking at, one, of those objects. Versus. When he or she is imagining, them this is pretty simple because. As you can see the objects, are having, very. Different visual. Properties, but. Still can they actually pick this up and, here's just an example one, of the user so is a spectral abated common, special part, into. Depth here but on. The top row, you can see the. Activation, for the hammer. On. The low one for, the dasyam and. Literally. Yesterday, in, the Narron, journal. There, was a release of a new paper from, MIT, about. The, specification. Of the TI part. Intro. Temporal, cortex, within, visual cortex, that actually corresponds, to, your. Possibility. Or your ability, to imagine objects, and to detect. Them and to crucify them, what they did first, was with the rats as they, were using. Some kind of drug I don't remember exact name misalign, I think which. Would block. The firing, of the neurons and, when that, happened. Neither, the mice know the markets. Were not able to cognize the object they, were trained to recognize so. It's like very interesting, paper and, i really. Suggest you to check out if you're interested in this one so. Yes. No. Here, yeah here it's only on the visual cortex so. Once. Again it's, just here, sixteen, across as well here's. Another example it's. Also from one so this was like more, and, stand. Media, for it's. Actually did this experiment effect once they feel about Stephen Hawking was out I was, very inspired by this you so, we did this study I, think. It's frontiers. What. Happens, here is that a person, so it's the fully controlled smart house, it's. A bit nothing's about starting, but, a person, here imagines. A door a, microwave. A TV. And. The teakettle five. Objects in total. It's. Only on and off objects, or open clothes on on earth but. We. Obtain 78%. Of classification, accuracy, it does require training, so it says here on the visual cortex, but. It's, not application, where if, you want to control an object, you don't want to go into imagination. Of a, movement, you just want here, it is you're gay physiology. A working. On it you, want to pick a time but you maybe want to turn it on to turn it off etc so. It's another example of. Applications. That, should be true what the, user and we compared versus monitoring imagery and the, pacification, our output.
Was 60%, flow, for, the motor image it was the same types of our, comments. Yes. So. It's this system, will require 25. To 45, minutes of training but, all the participants, that were, disabled. They, actually, perform, at at least one, BCI study in the past not was also, but, the olden knew about the head said the cap that they need to use it etc etc so. This. One this study was within one day only so they were training one day and then, we had obviously, break they always need a break and then we were doing there's experiment. I'm. Gonna come I have the slides for this so it's, completely different activity. In alpha. And I'm gonna I have a couple of slides but actual described in the paper but I was chose us off so, actually. This. Initial, work that. I have done was, inspired by the work from, MSR. As it was done almost well, it's actually 10 years ago now and this is a very nice one because, it's the same idea basically, it, doesn't use any. Visualization. And imagination. It's really observe, ation, so looking at the objects and trying to identify them and to classify them. And. You. Can imagine a lot, of different applications particularly. Particularly. One that is not very, well, studied. Yet but there are already some, papers, of collaborative. Disguise, there is a huge amount of papers particular this one by Wang several. Years ago where. They actually proposed I think the first cooperative framework. That at least I have seen where, the, and simple classifier was used to, have, the, outputs of the single mr. user and then of a group of 20 and the, crucification output. Of a group, improved. By 25%. So bid, from, 60% for. The detection task because they move to 99%, and, this is very powerful if you can improve their classification. By this rate by. Having a group of users performing. The task and not, by having one user perform, in several sessions. This. Is a really, interesting thing I really, surprised actually people don't do more of this work but it's really very interesting, one this. Is a comment a minute of all these interesting things is, that it, all requires training more less and obviously. Was it effective the training of the, user you do need to pay a price not only of the fact that it is not a proper classification, but. Also of the fact that it could be charged hideous, and even if the end user like, in the case of Alice will see the benefit, they, will opt out and they will not come to, the session two and three and I have seen those a lot. And, it will be no. Matter if you're gonna pay them call them stand in front of their house on to the rain it, will be yes. I did this do you, like all of it then it would be the same, so. But. You. Can actually, alleviate this, and you can actually design, something that will help you here's, an example, you. Might know the. Word prime and it's AB Sekulow djegal phenomenon when you present a user was something. Like a stamina is a prime to, modify his or her response to a latest timeless, this is a problem, and we, have a lot of those. Semantic. Visual. Subliminal. For, example I'm gonna tell all Christian. You. Look amazingly, good today, yeah. You could like in 10 hours of sleep I'm gonna like praise you for like 10 minutes I'm gonna tell you how awesome you are I love, all your 3000 papers from Kai I follow, all of them and then. All, of a sudden time, years later ask you occasion, can you just drop, me off for about a hotel place and the probability, that you'll say yes even if you cannot it will be around 99, percent because, I just used something that's called semantic, priming I was praising, you positively, you, we use this in our social. Interactions, as negative, we, said someone oh you're not capable or, sometimes, that used, a lot in schools we currently work with some classrooms, and questions use a lot oh Jessica you're not good in math this, is the strongest, in the person you can actually tell to the kid we're, gonna talk about this a bit late and how this I can also leverage this but, primarily.
The Extreme was extremely. Helpful wine. Because, particular, example, of visual primary when you see something it works usually in the cinema but it could be in the publicity anywhere. You. Will see some sense and you will from a bottle of coke you will, get thirsty and you will end up getting this bottle of cough of course I'm oversimplifying. Here. There should be particular timing when you present it but. This is how it works in general terms, why. It's interesting because. Imagine if you can do Prime in using. BCI, so. Let's say I'm, not gonna tell my user. Anything. About the system not at all I'm gonna just say hey, I want. To do some, brain, activity. Recordings. Can you come over to the lab and, I got 24, users like this and there are all gamers and I said do, you know Dhoom, 3 it's, the first-person shooter game. Endodermis, I think I am, it was my first game doom it was doing I guess. You all know what it is for those who don't tell. Me who don't know what Doom is okay. There are no people exist that's amazing exactly what I'm talking about it's. A dark game and, for. Any level even if you have never seen doom 2 3 4 etc you. Need for the initial level two things a flashlight, because, it's very dark and you, need kind of a weapon at least something, at least to survive the first level, and. So. I'm putting the headset on the users and I'm gonna tell them hey just. Play the game I'm gonna record how Roseau, what is happening in your brain when you see this was its inner but I'm not actually telling them anything, about the game itself and they, start moving and all of a sudden they find themselves in the dark corridor, where it's very dark, and. I, will give them a flashlight they don't have any activation, button. To get it just when I give it it to them and. I'm gonna record. All this as their brain Singh was doing this moment and exactly. One minute later they. Will find themselves again in, the dark, corridor, they, can still continue, the interaction, and not breaking they use experience, just. Pretty dark but because, to move forward but what will you think, about if. Just one minute ago it was the same corridor it was a flashlight what. Do you think about flashlight. How do I get it like what, to do if. You do this if the single match I'm gonna. Give you a flashlight this. System is trained you. Didn't do anything I didn't drop a word of how to do it and you, got it, this. Second, trial. 100%. Of users understood. What. They need to do without explicitly, meet challenge, and what they need to do and these. Were users who were novice and besides they knew, nothing about motor, image or visual image or any imagery whatsoever. It's. Like this yeah. Because the game is, the. First level so obviously. We it was the Takei paper we obviously compared with explicit. Setup. And actually with implicit training, within the game when actually, developed, explicitly. One level of the game but, this immersion. Flow. And VCI, performance, were significantly, better was this one but, I have. Shown you quite a few applications like communication, type in human. Area computing, but, from now on for the rest of this talk I'm going to focus on two on, health related applications, and in cognitive state monitor ones why. I guess. I already made my case about why but it's where, we are right now our. Devices. I. Liked. This. Interesting, mentioned, from the paper of. 1957. By client, who seven. Years later suggested, the first term cyborg. That. Robots, will be our. Mechanical. Slaves well I would say right now in our argue, it's, an opposite and we. Pay the price and the. Price is suicide, rates all over the world, schizophrenia. Mental, into. Says ADHD. Anxiety only. Growing exponentially. All around, the countries not only in. The one that we consider to be those. Who are having issues so the idea would be. Sorry. Yes. This was four papers about frontiers. In. 2015-2016. The, amount of how. Many mentions of ample mental health was. A yeah what's happening, like in the frontiers, in human error science. So. Can even give the, control to the user finally.
So He or she can still, look and, leave. With a mechanical, players around but actually be using. Them, as a continuous. Part because this is so. I don't have any single implant in my body comes it is I'm a cyborg, and we are all actually Cypress to some extent this, is an extension of me and I'm not gonna drop it for any current. Reason, right now I'm gonna drop some applications. But, I'm not gonna drop the device and, they will only go like just closer definitely. But. Can. It be more, seamless can it be better can be more productive can I get most. Of it right now I would argue other. Papers claim we're. Using 3 percent of the brain this. Are very very questionable, a citation, I would argue that we are using 3 percent, of these and. 97. Are still awaiting, we just cannot leverage it what. Can we do and. An idea would be can, him maybe somehow measure, let's say attention, or the cognitive load of the person, and, maybe. In, real time without adding, additional burden, and then, provide some kind of feedback to him or her telling hey you're not on task ahead come, back but. Not using applications. Then don't check them they don't care about the order don't load them it's fancy for two weeks and that's it we actually did a study about those so. Just to make sure that we are on the same page about attention. It's, their holy, war out, there in, psychology. Literature when. You talk about the tensions very important to remediate, leave any of your presentations, mention what it is because. You can get in trouble very fast so, I'm using. This, definition. By allah jensen is a very famous researcher, works, a large on attention, it recently, on besides as well so. We. Can definitely measure, attention, in different ways we. Can use heart, rate we can use galvanic skin response but. If you can also use EEG, and EG, was proven to be the most reliable of all of them you can talk actually about heart rate if you want like wiser or not their, most reliable and, EDA so. How. Can you measure this and this kind of gonna go into line of your question how can we decide, if a person is observing, something versus when he or she's imagining, something so. We can measure. It in the same way I was talking previously, by. Looking. At the ALF activity, so, spontaneous. So just, having something on the head of the user had set a headband, or by, a bulk in it as. I will be presented with some kind of different, stimulus so. While fictive ET why not something else there are several bands as you might know which, I teach are gamma, but. Alpha, is known. To be once again it says kind of as motor, imagery, in paradigms. From this guy but it all started, and actually. Is known to be inhibited, in the task in relevant, regions, let's. Say if I need right now to present to. You my, talk I'm gonna, definitely try, not, to listen to some kind of music if it will try and debate it will be hard but I will do my best the. Different types of attention, really a lot of them it's important always to explain, and to know what you are measuring could, be special temporal. Selective, and the most important, for us internal. And external, so there's, another terminology, trick, that. People need to know when they do these, guys and they want to measure cognitive. State it's vocabulary, it's. Pretty, pretty. Unreliable. Out, there you might think, no, one did this but actually when you start googling, it that's, exactly the term so by internal, attention, what, is hidden it's actually. Imagination processes, so, internal image, internal. Tension, is ever since it is imagination, mental. Imagery includes. Visual, motor. Tactile. Auditory. Etc. External. Attention. Everything that is observation, it means that you are looking in it but, you're not really engaging, with it so. We did our paper. And this is still under you, for Nature, Neuroscience at, this time the, system is pretty simple you, have. Stimulus. That lasts so, first you have a cue. Then, you will have our initial, you will have some kind of time, of the video seven. Seconds and, it, has three modalities epic. Auditory. And visual, and. Then you will have a cue it will be a word one, of those modalities and, then, you will have a break where you would need. To. Imagine. Them and you, don't know beforehand. Which one you need to attend to so you would look at the video get. Turnt all of them because, you don't know which, one will be the one that will be asked, and then. We. Will. Ask you to imagine one of those so. The idea here I can show you in quick rediem. Mmm. So. Person, is varying haptic, gloves they're custom made as you can see lot of cables, out there wearing.
A 32a, jig. Headset, and he, just had. Piano. Sequence, so. The sound is artificial, it's an oval and each. Finger. Is stimulated. By a motor and, then, he, was asked, to imagine one of the modalities, as a haptic or visual. Or tactile. Studies. A lot of different setups a lot of trials, so, why to do this why. For example we didn't use something like jingle bells or some. That is ecologically. Much more valid because if you would use jingle bells you. Would not understand, what's happening, for one single reason let's. Say I'm gonna tell you jingle, bells can you sing in your head jingle, bells right now oh I'm pretty sure all of you can the, prongs it because. It is work. In. Memory because. It is long-term, memory because, on when you verse last, time singing jingle bells it was with all your kids and the year before one, of the kids couldn't make it from MIT on time to Seattle, so one, was me, so you was actually you actually said, you. Have a lot of different activations, when. You're trying to revoke something, that you already learnt and you know, let's. Say I'm gonna tell you blood the, same history oh they all agree how it looks like definitely, when, I'm gonna ask you to imagine it we, will he'll like 135 different representations. Right now and. I'm not sure that we will be able to build something like reliable, as a classifier. On 35, trials, to be very hard because for each of you you're gonna add layers. And layers of something, because, most excuses, just don't know how they need to imagine something they just it's, just hard, it's. Good for people who do, it for years I know it's precise it's very sensitive it's very noisy it's very hard to give instructions, it's very easy to give instructions very hard to make sure that people understand, them because we cannot get into their head so. It's very very. Good in these setups, when. You want to once again maybe test, for something, like the question is not. Directly, application. Wise it's just can we understand, when person, is looking. At something and imagining. It while, we are having all the mortality so kind of as it is in real life in YouTube you have three modalities you usually don't have one. And. The set-top and here's this just an example I'm going on I can share a part, of the paper if you want it's not published, yet but I'm actually giving, you the example of the ones that is published but, it's actually, the same result for both of them. It's. The first of so. It, is immediate, respectable perturbation. So. It's kind of like. How, can experience so it's kind of like a orgy, but much more generalizable, because, you can see all of the bands and everything that has happen in at once you don't need to check that prevent, and what happens here is that in. The left column in blue this, is observationally. Person. Is just attending, to the video and, on. The, right that's, what happens with the person is imagining. Any of modalities, and, is assert its, significance, and, I can tell you if you cannot just see how significant, it is it is 90% so I can say with 90% accuracy, your person is looking, at something or. Thinking. About it like actively, engaging think, you know maybe I need to have this fan to cope with me because. It looks so nice. Jensen. Particularly. Did also election years ago very famous paper tried, even to go further and I see the rental papers try to replicate. This, in locating. The, attention, so. Left-right. Up-down this, is also done with EEG. And they, tried to do this with near, Swiss videos it provided. Worse. Results, here's. An example of activations. So. And. I'm gonna finally show you a system, which. Usually, for. Some reason uses, all these. Setups, that I just shown you but uses a very small. Comfortable. Foam factor and on top of this is, doing something that usually BCI. Is not doing its, iterative. Product. Design, so. The, project is designed, worst. Year at food. Interfaces, and, this is me wearing the first version so, it's called at interview and it's. A band as you can see and also wear a scarf. And. It's, very simple it is that cuz I explained we, will measure if, you engage disengage with the task on hand and, we, will give you a haptic feedback if. You, are not there, are several reasons why it's haptic feedback and get into there, is also reason in white, collar. And not for, example a band which is a brain you have emotional, attachment.
That Is negative, through arts particularly this guy and you actually start ignoring, any kind of vibrations, within one hour search after being using them if you do have this, device so, we can get into these like details later, if you're interested the result of reason why it was a disgusting, and. It, has soaked eg sensing component. Feedback. In this case of haptic obviously. We have some software and we actually have an application we've. Had 24 bands. And we did a multiple, session experiment. With multiple. Users were in the system at the same time, and. We had, actual. Students, from MIT coming, over to the actual, lecture with. The lecturers, it was the same they were getting different types of feedback, some. People will get an actual feedback. About their, brain activity hot. Some people go get in random. Because maybe random rocks as long as random it tells, you hey wake up and some, of course when getting no feedback and. So. Obviously, need to calibrate for this you don't need to train as I said this is not that sensitive. That is not such a complicated phenomena. But you need to do the calibration in, this case we do calibrations. At last 5 minutes tops. It's, actually kind of priming, as well I will I was asking them to do some arithmetic tasks, some, and back visual tasks, and back is when I show you an. Image I'm gonna show you the second one and will not show you the third one and then I asked what. Is the same or not and. In this case is the number of the image that is appearing so, for example and one, will be very simple and five will be very hard it will be maybe potentially impossible, for you to actually recover it here's, an example how it looks like the, calibration session so this is the, same application, we have for miners. Who, have a ddddd, that's why you have some trees. Here ignore the trees no you can still look at them it's a nice animation so. Just follow the number on top of this it is zero and listen. To my voice it is in real time I'm asking. A. User. To. Actually. Do. Mental, calculus, so there is no sheets of papers he or, she does not see any phrases. So it's a mental calculus, purely just, look at the value how it changes, and listen. To what I'm giving as instructions. So. It's a scale of 0, to 100, 100. Being extremely. Engaged zero, being actually. No connection, not engaged in the case of we. Would say dead person, no. Signal. Seventy-eight. 71. 34. 23. And 45. 123. And 8. Forty. Five and eight. 345. And. 78. As, you can see I'm gonna stop on this one once. The numbers had just start sounding, more complicated. Like, three hundred two hundred the, tension, goes immediately higher, without, even proceeding what was the second number maybe I'm gonna say three, hundred and zero which, is going to be very easy maybe I'm gonna send three hundred in 100, or 300 and 500 etc. So you can see this, are really. A difference, it's pretty sensitive so the person stays engaged, person. Reported, that mental calculus, is not the easiest task for him it's very important also to know because, our assumptions, of what is easy. For. Example for me to test in the range of 19. And never goes up and usually on test on myself I think okay, not working whatever when, I did the.
First Pilot with 17 users oh and. Then I just think maybe I will ask them is, it actually okay for them to do this task and then you're like oh actually. Not, it's not as easy even if people have like PhD in computer science it has nothing, to do is it easy or hard task to them you always need to make sure that they, understand, it they're capable to perform, it and actually, then you can set up the, baseline correctly, it's very important, it's, very important, because if a person this is an easy task but, you consider that having, something like 300, and 400 is not easy you really, need to do it correctly that's why, individual. Training, or calibration, is paramount, in most besides right now so, I mentioned that you can still do some, databases. And, instantiation. In the case of like p300, in the case of these kind. Of systems you. Need to go for calibration, it's like you cannot not do it it cannot be generalizable. At least in the beginning like Norway, why. It happens, as well it's also because, right. Now as I had a coffee which is a stimulator, my, activity, will be very different from when I don't have it it depends how many hours, I have Sept it depends, if I'm sleep-deprived, it depends if I've been driving for six hours or not etc etc so, maybe, I go into details, of results, there are papers published, but. In. Yellow you can see the biofeedback in, green. And red you can see the random and no feedback and it. Was significantly better. And, obviously we, have casted, slides and different types like a cabe or an interesting, cannot be always the interesting can be always boring, but for example for the boring ones we actually instructed, the lecturer. Not to just explain at all so it doesn't mean that it would be like formulas, only but, just like don't explain them at all so even if you're engaged in the lecture you like it you want to follow it you admire your professor, here, she's amazing. You will just if he or she is not explaining anything he's, just not gonna follow so we really wanted to tell the system so it's, another example think. About how you can test it if the conditions are realistic. Because we went for realistic, application they. Actually done three, different experiments with, more than 100, unique subjects, in this case I've. Even tried this at the workplace, with. The programming, tasks it's. Very hard actually to do kind, of realistic setups, but we try to do something where we can control, for.
The, Output, of the user so in this case it was like someone, like there are from. Their superiors who can set if the task was actually, executed, or not and. This. Is the interaction I was talking about this, is usually you don't see this in BC eyes and says really the, direction to to, do and to goal with we. Had the system that was functioning, it's, used something that is said, inside we didn't even try to we. Have our own ideas for the algorithm obvious is it be tested but we really. Try to replicate what was in the state of that with, the off shelf device it, provided, good promising results. People. Stopped here, most. 99.9. Percent of the CI research, in the labs stops, there they don't try to do something, that could be released. Make, available or made better in this case batch amines, live. Again not 100%, classification. Accuracy, system. But. It is good enough to, be tried out he, may be in NASA from fector this is an example of the form factor its glasses in this case the, first prototype we have a better, version now since the paper accepted for hydropower be ascend this year, it. Has actually gonna play as vedyam and. You will see it has two modalities so it's another example of how, you can compensate for. Improve. Faction, of this guy you. Don't, design. For user, who lives in ur in. Mars. He. Or she is still here has, all the other disease, modalities. Objects. Devices. Leverage. Them in this particular case, the home factor was glasses. You. Can have two. Positions. Of the glasses that are not used I would say they're. Really not, used on the temple here, no. One is usually putting, anything on the temple here and here. You, have Bluetooth, speakers bone conduction whatever, you want to have here you can have different lenses with different, infrared cameras etc whatever. But you really have two points, that are not used like, we reviewed like 20 of different pairs of glasses they're not used but. You can put electrodes, on them, and on top of this II would have two modalities you will have ng and in. Your dream I movements. And, you. Can compensate. For. The imperfection, of both of them because there is not a single physiological. Modalities that is perfect you name it I'm gonna, tell you when it's gonna fail and why and also, if you name a product, is it I might work with for example I'm gonna even tell why it's gonna sell whatever. It is right now is a market it is not providing, you with 100%, pacification, courtesy, but, you can try to get one potential, by combining it early. Fusion late fusion, I will. Let you watch the, video before I go further. Section. Almonds and anyway. You're. Laughing but I had three users who literally, have fallen asleep, like. You laughing about that careful. Generally. For, surprising. Me. And. Yeah and don't forget you user will try to always, see that. It's not rock not, working and he will always or she will always try to catch this so. It's actually good feedback. This. Was just to show that one. Improved. Design can cover not only. Imperfections. By having another modality but. Can cover several scenarios, it. Doesn't have anything, else as information, and, you can still have several options for example, we just submitted the paper but, you can see that there is still another device that goes with it potential, it's a haptic feedback, why, it is still there we actually did the study 36. Users 30, preferred. Auditory. To haptic, 6, who had EDD, EDG preferred, haptic. Strongly, to auditory, you. Need to have options for the user it might be a minority, but. They will not use it at all if they. Consider, that it was distracting, for, them auditory, feedback was distracting, for, the rest of people uses. A reporter that it was amazing they liked it so. Do. A lot of iterations, it, works it's not gonna work 100%. We're. Trying to put it on one whole person we are doing like this I could feel from but right now the. Third version that I'm working in is actually fully graphene-based for example so, it's another fun factor, it's, gonna be like it's a tool it's fully transparent. It, cannot, read. Visual, imagination but, that's not what the aim is we. Have a set up we have a set of users who want the system who want to try it out for hours, and, if it's what it does and. If it's already validated it, does where's invested. It does worse a try don't forget that you have several categories, you, cannot maybe cover all of them but you can have different sub versions that can cover a lot of them depending what you are aiming for this, is something that is usually not done in BC I like I cannot name a single, product, right now that is actually doing, something in this direction.
I. Have. One, more, just, to actually show, off once, again in this Wayne, and I'm gonna do this very fast but, you actually we can also, like finish because it's exactly one hour so. Yeah it's just like just to show something, else it can go in this direction but it's, really good for now yeah. No. I think it's actually one hour so I think we're gonna move on. Yeah. I mean we can move on with the questions yeah, if there's for example you want to see some like ours applicators, definitely, I can show you of some other system it's it's again in the same vein of it's, completely different application in that case it uses actually multiple electrodes so it's nothing of the form factors that is like flexible, section completely opposite, but, it's an example of how it, can work also. Very well even, if it's evil hook like octopus, after you put it on we provoke the actual ruling and making lines to try it out just, because. You. Make something, on top of it so, yeah but, we can already take if you like want your questions it is like urgent we can do that just. Like one iteration, of completely, opposite system. Yeah. Yeah. So what happened. Actually on my desk right now in the lab and I think body also should once everyone. Right now in tulip wants it I have a lamp, it's. A lamp just, was like, bulb and it's red when, I'm engaged into something it means do, not try to approach me I'm gonna be like very angry crocodile, if you have tried to it, means I'm in the flow of the task the thing is that this, flow state it's really, very short it will never exceed one hour except. If you're a trained sportsman. Like Olympic, sports man I have worked, with this category. Refuses, like Formula One etc, you. If you are not and you're just in civic space it's, never gonna exceed one hour one hour 15, tops if I go on yellow blue eventually. Or green and. The pallone. You but. In the case of like if you're gonna be bothered it. Will actually because, you are trying to internally, rehearse. For example presentation, and someone will come over like your internal combo and say oh could, you please sign this document for me and I'm like I have, this ledge to rehear saying you I don't have time oh just one second, he, or she doesn't understand, the effect but you can actually visually, show them the effort with the lamp yeah, it will buzz because you basically broke it down but, then it will stop because actually there, is obvious a point I mean we have tried different vibrations. Different iterations, it's not going to bars non-stop, but, we can definitely see that you are not engaged, with the task on hand that you were doing so there's a huge change between what you have been doing at. Literally. Time -, one is a time T and if the change is big and it's going to be big if it's a switch between atomic strands that's what I show you on, with those like studied through electrons, we're gonna bars bec
2020-06-13