Good morning. I'm Paweł Woźniak and I am Andrzej Romanowski Welcome to ISS 2021. We super happy to welcome you to Lodz and we are even more happy that so many of you are here and there's a special message for you.
That's actually coming from someone more important. Yes, you just. skip to slides, okay, so we are here and we would like you to see our Rector of the University of the Lodz University of Technology. The place we are here.
So our main host Rector, Professor Krzysztof Jóźwik. Hello. Hello. Hello.
We can hear you. I think it is possible to join the conference. And great, thanks to the dean of the faculty of chemistry, Prof. Iwona [Małgorzata Szynkowska-Jóźwik] You see me? Yes. Now we can see OK.
You see, but I think that the rest of the conference participant is not able to see me. I am not very strongly addicted to the fact that I have to be seen by the audience, but it will be better. So please, if possible share my face to all the participants. If possible, I believe we can see our technical team is working on it.
And yeah, we gave us a sign. Okay. Should we my instead of the face? I think that my voice is absolutely sharing to all the participants. I welcome you very warmly. All of you in Lodz at our University and it's great honours. The to be here with you and to be a host of such Prestige conference.
Thank you. All the decision-makers to agree that Lodz University of Technology as both organizer of such conference. Great. Welcome to participants that are here at The area of Lodz University and for all of them that are directly in front of the computers. The situation is still not very clear and pandemic is in progress. So I think it would it is better to conduct it conference in a hybrid manner.
So once again, I welcome you very warmly. I want to especially welcome all the offers and all people preparing the presentations that were given do during the conference migrate fans to be organizing committee, including fantastic team of the student volunteers. be our host of this brand new beautiful space, in Hammond building. The new one that real power university. And the chairman of the University Council.
It's Dr. Miroslaw Sopek. All of the other members of our academic community and distinguished guests. I welcome you very warmly. I'm pride.
I'm very glad that the for the first time we have the honor to host the major human computer interaction conference in Lodz, at our University. We it means all together as a university focus on both education teaching learning procedures and quality research. And I think we conduct systematic human computer interaction as one of the few in Central Europe. We have both a research group, very strong and a brand-new master's degree in HCI. As a second cycle studies. We all know live in there real Artificial intelligence, but we don't want the technical aspects to take over the human factors in everyday life.
So we hope that HG. I will help us to keep the right balance between the two, the technical aspects and our human being everyday life. I think that the conference is a beautiful time that we are able to have discussion on the scientific achievements of our approach to different aspects of the merit of the conference.
But in the same time, it's time to have contacts to establish new groups. Interesting. In this same area of research. And also friendships. In time of pandemia some problems to have all these acts but aspect together in one place, but I do hope, it will be possible for you to visit the Lodz.
And to be here. I invite you all very warmly to have the opportunity to see Lodz University of Technology and to Lodz City. I think that the program is very dense and beautiful and very important aspects will be mentioned all have been. Because you started yesterday mentioned and I wish you a very thoughtful discussion. And decided discussion.
I Wish You Beautiful Moments. During that conference, especially for when who are here in Lodz enjoy your stay in Lodz enjoy the conference and enjoy our University, all the best for you, and thank you for inviting me to have the possibility to give a short speech. To all the participants of the conference.
Thank you. The Chairman's. Once again, thank you for is for you.
Thank you. Thank you very much, Professor director. Thank you for this. Welcome message. And now we would like to share with you a little bit of information on the background and the results of our conference that is now being started. Thanks very much.
So let me just grab that one. Good. So as as you might know, we have a lot of people here. These are actually the wrong numbers. We had, we had five people register early two days. We have 58 virtual participants in 29 of you here online and I want to thank you once again for supporting our community coming to a conference virtually or in person despite all the hardships that you are facing currently.
We are a hybrid conference. And we are as far as I know, the first six hybrid conference. So, there's a lot of stuff. It will be new for you this year and we hope it will be fun.
And we also hope that you will see how you can have fun. And if I had a hybrid conference on the other hand, of course, we learning how to run a hybrid conference. No one has really done this in our community before.
So we apologize for all the hiccups in advance. As you might also know the ISS program traditionally consists of different parts, so, we had workshops. We had four papers posters, and demos this year. And these four tracks make up the scientific content of ISS. So as majority of you actually may have seen yesterday.
We had a wonderful workshop and the participants debated for over eight hours on different issues in Mixed and Cross Reality, and I they seem to have enjoyed the workshop even laughing. So that was a great start to do ISS. And of course, the core of the conference are the papers and Zoom. As you might know, we now publish our proceedings and proceedings of the ACM and human computer interaction, and that takes a lot of work. And here this year.
We had four people and ensuring that the papers are aware of high quality and they ensuring that the process of selecting the papers was as good as possible. So these were Morten Field from Chalmers and Isenberg from Inria Han-Christian Jetter from Lübeck And PACM Editor Mark Hancock oversees the process of Yes, and I would like to give them a big round of applause right now because they did an amazing job to get us all here and to give you a scoop of what's coming up and talk a bit more about the papers programs is Christian. Yes, thank you. And welcome to the ISS 2021 papers program and I'm very proud that we have selected high quality range of presentations and publications here. So let me talk give you some key facts. So we have 23 accepted papers here at ISS 2021.
They were selected from 77 submission. So we arrived at a acceptance rate is just below 30% and As probably mention that was a lot of hard work to select the best papers from the submissions. And this hard work was done by 40 associate editors and over 140, external reviews.
All of the accepted papers are blood published in the proceedings of the ACM on human-computer interaction. Heck. HCI If you are not used to this abbreviation yet and in the ISS issue of that journal, they are already published in the ACM Digital Library, so whenever you see an announcement of a talk on the conference website This is small ACM.
Digital Library logo. You can click on it and access the full paper right away. So we will have 21 paper presentations during seven sessions in the next 3 days and let me talk a little bit about where these papers come from.
So we receive 19 submissions during the winter around until February and 58 submissions until July in the summer around. And as you can see, the acceptance rate in the summer around is a little higher. Please do not take that as an advice to only submit to the summer round Please use the opportunity to submit your work already, in the winter around, for ISS 2022. So you have the opportunity to revise your work to improve your work and then have an even better paper in the summer around.
When we also look at the geographic distribution of the papers. I think it's really nice to see that ISS really reaches a global audience if you look at this at this. Map, you see there's a wide Geographic. Spread 77 submissions from 23 different countries and four different continents. Of course, this isn't a little narrower if you look at the accepted papers, but still there are 11 different countries and four continents with Japan and Canada in the lead of papers per country. And what I think is also really noteworthy here is that ISS clearly values collaboration.
So we on average, we had 0.8. Also as per paper and on average 1.3 countries per paper. So I think this indicates also that ISS brings people and also countries together, which I think is a really nice thing.
I should not forget to talk about Awards. There will be award. So the members of the ISS during committee and the ISS 2021 papers chairs have selected, two best paper Awards, three honorable mention Awards, and there's also the ISS 10 years impact award. So the best paper and honorable mention award will be announced during the conference dinner on Tuesday evening, and they will also Ten years impact, award being announced today at during tonight's town hall meeting. So please don't miss those interesting events.
Okay, and with that, I'd like to hand over to thank you very much Christian. So as you might recall area in Ellie as an ISS, we've had a tractor, we called application papers. And with the move to Packham, we sort of lost that because there's a different process required for Packham.
So what we try to do is we tried to bring back the interaction with practitioners to our conference and we did get a case study track going And we got one amazing case study that you will see later today and hopefully we will return to having more and more practitioner involvement in ISS. We also have our traditional adjunct program. So demos and posters in this year.
We have seven posters and two demos, and you will be able to experience them in the demo and poster session later today, we've prepared different kinds of possibilities for you to meet and interact with the office. Don't forget to come to the poster reception today. There will be food as well. So I think there's a chance. You might actually enjoy this. And of course, at this point, I would also like to thank all of our chairs, who managed to get us a lovely program in case studies posters and demos.
Okay. So safety your safety is our priority. So please take your time to think about covid safety protocol. This is obligatory to wear protective masks in all the buildings inside the buildings. And please use the disinfection stations to to these. In fact, hence, the entrances.
We are actually running a little bit of out of time. So we will go forever as you already know. This is a hybrid attempt to conference. For the very first time, both for us and probably for you. We will do anything possible to include anyone here on side and on the Zoom, all the sessions will take place on Zoom.
There will be possibility to post your questions on the chat. We have a fantastic team of technical guys. Will try to post the videos of the talks as soon as possible after the sessions and you can also be able to post questions after the sessions as YouTube comments. To this video shows.
The only in person only event not transmitted will be the tomorrow's conference dinner. For both sitting in this room. There is a important thing. All of the seats are equipped with microphones and there is a button to unmute and mute when you would like to ask questions.
So, probably will enjoy this thing after the first session. So if you want to get a question and you want your virtual people to actually hear it, you have to press the button and most importantly you have to press it. Again when You done with your question. Yeah, we also have a nuclear option with super power button here to mute everyone. So, whenever you hear this sound, it will be all. Participants will be in it.
We also have our director said they would like to give you a chance for some social activities. First of all, today will have a poster and demo reception in the afternoon. Tomorrow. We are going to visit another nice area of our city. For the conference dinner in the old factory.
We have a lot of old factories refurbished to new challenges in our city and we also prepared to guided tours today. And tomorrow, they will start at 1:30. So if you would like to join this and I hope you will have time and and like to do it.
Please be punctual. Maybe I will hand it to So another very important aspect of conferencing, both virtually and in person is respecting each other. And as you might know our organization that's organizing this conference Sakai is, and it's matter organization. ACM.
They have a very strict policy against harassment and against not respecting people. So we would just like you to remind you of this. And, and most importantly, remind you that, if in case anything happens in case, you're Is any inappropriate behavior? It's very important that we talk about these things in our community and it's very important that we report these things in this community and married. Our diversity chair is here.
If you want to address any, she's over there. You can. And for those of you attending virtually is probably easier to address by email. Also, if you would like to address any concerns anonymously, you can you can also contact us by email and if you would like to talk to any of our chairs about any issues.
We are always there for you. There's no space for any kind of inappropriate behavior at this conference. The other place where you might get help and this not only applies to do inappropriate behavior by all kinds of concerns, like being lost on eating food or coffee.
I was doing the volunteers and we also got them lovely t-shirt. So if you spot that t-shirt and if we have people in t-shirts, going to have them stand up for a while. I can see Jakob is over there and Michael's over there. So these are the these. These are the people that will give you help.
If you are lost if you sort of if something's wrong. Hopefully not much. They will come and to your rescue essentially and they are and they are, they're always happy to help and always with a smile. Thank you so much. Don't forget that we are on social media.
We putting stuff on the social media, very regularly, social media Church, Sarah and liya at out there, posting, all kinds of stuff. But if you post your own stuff, don't forget to hashtag. Don't forget.
We have a Twitter handle. Don't forget. We're on Facebook. So that also to people who are not here in the room. Can get a glimpse of what we're up to.
And last but not least, we'd like to thank all our sponsors and people who helped us organize this conference. So, it's you Lodz University of Technology, ACM, the local ACM chapter, the deposition information, processing society and Sakai Thank you, Pablo a little bit, you know, and in late, but I'm concerned, because we are eating the time devoted to our magnificent speaker. The opening keynote speaker Professor. Włodzisław Duch I have a long list of his achievements.
I would say try to short them down a little bit, but I need to say a few words. So what the dog is ahead of Neuroinformatics and artificial intelligence group at the University Center of Excellence for Dynamics, mathematical analysis and artificial intelligence at the Nicolaus Copernicus, University in Poland in Turin. Professor Duch is a truly interdisciplinary scientist, with a diverse background.
He graduated in physics with the highest possible honours in 1977. but his Ph.D was associated with quantum chemistry. And again, awarded This Time by the Ministry of Education in 1980, when his fabulation again, was related to physics, but also to Applied Mathematics 1987.
And then eventually, he received his tenured professorship title in theoretical physics and informatics in 1997. Throughout the years, he collaborated and worked with many institutions institutions around the globe. I would just name a few as University of Southern California US.
Max Planck Institute in Germany University of Alberta in Canada and Nanyang, Technological University in Singapore, and many many more. But in parallel, he was always was loyal to his alma mater, but is Nicholas Alice Copernicus University for 8 years. Now. He's been running.
Neurocognitive laboratory where he researches different approaches to neurosciences in order to understand how our brains operate and how they cost different type of behaviors. He authored numerous research papers, but also, it is worth to emphasize his a really enlightening person. So Also published more than 200 popular science articles and he's also a practitioners with record of specialized software development and he is expert in numerous Fields just to name a artificial intelligence has the most but they popular nowadays and as our HCI Community stands at the intersection of among the other fields Computing and cognitive. Sciences, I believe we can draw a lot of inspiration and benefit from meeting this outstanding researcher. We hope to have him on site here in Lodz, our conference venue, but pandemic actually changed it into a virtual meeting, but we admire his with us, so please help me. Welcome Professor Włodzisław Duch what is our virtual stage with around of a big Applause.
Thank you for the floor is yours. Do I have any time left after all this amount? Yes. Yes.
Yes, you can proceed right away. Okay. I need to share my screen and I can do it now. Okay, so I'm very sorry that I'm not not with you today, but you know my last name. Old Slavic language is Duch, which means a ghost, or a spirit.
So it's very appropriate that I appear in the virtual space today. And I'm very happy to hear about this HCI activities in Lodz this is something that we certainly need in Poland and it overlaps with many things that we would like to do looking at how brains work and how we can use this for better HCI, so I want to talk about human enhancement. And the future of the brain computer interfaces. And some of you may remember Moody Blues.
Very old record about 40, 50 years old now on the threshold of a dream and I feel that we are in right now on the threshold of a dream because we have technology that will allow us to interact directly with our brains. There are many Global brain, neural, technology initiatives now and lots of research on human enhancement. Them to musician of some brain processes. And well, the new types of brain-computer interfaces, which are closed loop brain, computer, brain interfaces are counting all this hinges on understanding brain activity, brain processes brain networks.
And so, very important part is, of course, understanding how to read the brain States, the Fingerprints of mental activity and neurotechnology. And of course, We need lots of machine learning Ki and Signal processing to be able to do that. I've been thinking about these things sensor quite a long time, in 96.
I published a very strange paper computational physics of the Mind in computer physics. Communication, the editor road that is very unusual, but they still have published that and recently I've been also writing about pneumatics neural models of conspiracy theories and other things like that. So lots of Things around these topics. Well, this is a mission impossible. How to develop full human potential.
Neurocognitive approach is to understand the brain. That is the diagnostic part to see how brains work, especially those that are damaged or underdeveloped somehow how to influence the development. This is the main topic of your conference. Now, the infant research for example, and how we can Influence infant brain development, increase the efficiency of the therapeutic procedures and all kinds of procedures that influence our well-being. And then sometimes we can consciously control our brain States, especially if we suffer from some disorders. And then, of course, some of this knowledge is going to help us to create artificial brains.
So there are great opportunities, lots of dangers and You can think about what we're trying to do in the description of many of manual kind of man is got the Nobel prize in economics, has written a very nice book thinking fast and slow that there are two systems. One is this fast automatic. Well, emotional, stereotypic rigid associative responsible for perception, lots of processes that go on and our subconscious. And then there is this consciousness Effortful infrequent logical and calculated reasoning. And conscious processes that sometimes you would like to use to control these automatic processes, which are not always optimal. So IEEE has created these brain initiative and lots of societies have joined that human brain project has been the flagship project in European Union.
Since 2013. 1 billion euro is devoted to developing. Development of the human brain project in the u.s.
They Obama brain initiative, has started about the same time and it has the acronym. Brain Brain Research through advancing Innovative Technologies neurotechnology. So lots of things going around these neurotechnologies.
Now that can actually help us to interact directly with the brain and the environment. The mission of IEEE brain is to facilitate cross-disciplinary. Collaboration and coordination to advance research in the development of the neuroscience and neurotechnologies of All Sorts. So many companies have joined in many societies of IEEE including computational intelligence society, which was my Society where I was mostly active. But most of the societies of IEEE has joined and I see em also is contributing to this effort.
Microsoft is also contributing creating all kinds of of new technologies for people with some disabilities. So human enhancement and optimization of brain processes can be well, very simple. For example, we were glasses all kinds of glasses. But some people were actually hearing aids and some people have Cochlear implants, which are directly, transmitting, the sound into their nervous system and some people are using all kinds of rings and maybe watches and other things, but more sophisticated things are coming like artificial retina and things like the deep brain, stimulation, Etc.
So, improving senses eyes ears, touch balance memory and attention skills. This is all enhancing our perception. One thing that we've been working on for some time now is to technologies that can share. Sharpen your brain found that actually some children especially for example, Japanese children or Far East children, where the language does not have phonetic contrast that are very important for languages like polish or English needs some special simulation to improve animatic hearing. So I've created a baby Lab at our place and we started to work on gaze interaction.
Actions of infants that still cannot control their movements to well. But also cannot express themselves speaking and so my friend J. Matulewski working in this lab, and we gonna do now, I have created a guest interaction, markup language and an interpreter. For this language, which is called the decaf, the gaze controller application framework that allows us to make all kinds of applications where children just watch things and initiate certain stimuli which then we can observe, whether they interpret correctly and this gives us kind of feedback and chance to actually create environments that can stimulate children and help them to develop their potential, especially related to fanatic hearing.
So this grew up into a system that is now used for people who are paralyzed various. His abilities spastic people that were what up from coma, for example, and we have applications like, move on. I that allows them to control YouTube and select different things, and we tested different kind of keyboard types, including molecular, keyboards, and two-step keyboards, Etc. Some of it has been presented on the eye-tracking, conferences and cocaine conferences, which are very much. Related to human computer interaction, but this is still one application related just to eye-tracking.
Lots of other applications that you may find now relates to different senses. For example, some people fall because they don't have enough stimulation from their feet or the soles of their feet. And so people have invented this zebra phottix kind of souls that will vibrate and give people more information about the pressure in your feet and so they can control their movements and don't fold it frequently. But in some cases, you have problems which are related to peripheral. Neuropathy is so the nerves are not conducting signals up the body from the souls and so people have invented this walk scenes which just send information from the pressure sensors in your shoes to Well, somewhere above the ankle and that comes to the to the brain and helps them to walk.
Other simple, things are related to, let's say, recognizing your mood by having as few electrodes and the EG to find out how the songs that system has, has selected for you are influencing your mood. This is called melomind. Other things are related to attention where you want to focus. For example, In a complex environment you like to pay attention. Now, paying attention means that your prefrontal cortex.
Has to send information to your Visual and other sensory areas to keep them running and this requires effort and it's tiring and from time to time people just relax a bit. Now, if you just run some currents through these areas there they are actually aroused all the time. And so this direct current stimulation systems and also, stimulation with Magnetic pulses, the TMS are quite useful for this kind of applications. And so they're now journals on brain stimulation where they consider all kinds of situations. Electrical magnetic radio, wave ultrasound focal targets, its pharmacological stimulation, Etc. And there are deep, brain stimulation.
Of course, for people parking some, especially, they are done in our clinics here in. Touring and bit ghosts. And for people who suffer from OCD, the comma persistent pain and many other conditions.
So, now, people experiment with non-invasive approach using ultrasound that helps to achieve similar goals, but that does not require implanting electrodes very deep in the brain. We've been discussing that with neuroscientists. Scientists, unfortunately, these projects, if you want to make a bigger scale project are very costly and so we still haven't started anything, but about 1% of people and some animals. Like, my cat, for example, suffer from epilepsy.
And sometimes epilepsy is not really reacting to pharmacological treatment. And so people have developed neurostimulation systems where you have electrodes in the region, where the epileptic attacks, start, and lead to seizures and then you have some electronics that try to pick up the signals. Okay. It's starting. So, let's inhibit that. And this system, the RNs system is now commercial and can actually stop epileptic seizures before they start.
So that's another kind of human machine interaction where electronics interaction and then they're really crazy projects like that burger from my old University of Southern California, who started to company called Colonel. They've tested on rats monkeys and now people memory implants. So a part of deep part of the brain called ca3 area in hippocampus is, cut out and put and some Electronics is replacing the transformation of signals coming in and coming out and in recent papers they've claimed that the Improvement of memory is at the level of 35%. And so DARPA has also initiated all kinds of new neuromodulation and brain reading project.
So lots of these things related to neuromodulation show that we can now try to influence at least in case of serious problems, like OCD parking some Tremors in all kinds of chronic pain, but also depression and epilepsy etc. We can influence the brain and try to optimize how the brain is working. Okay. Technology is developing and one part of it, which is the oldest part. I've written a paper on that in popular Journal. 40 years ago is called neurofeedback, of course.
So we need to show people through the senses, stimulating, their senses. What happens in their brain, so, we need to read that. And we can read that in many different ways and then present that in a form that we can experience consciousness through our senses and neurofeedback may actually repair your network. If you just put people in the scanner, unfortunately, this scanner is rather costly device about two million dollars and then, of course, using it, it's also quite costly. Then you need a supercomputer to run the real time.
Reconstruction of what happens to the brain. When you have to extract the signals from the measurements. And then calculate how much information flow or how strongly synchronized different brain areas are.
And you can find out which brain pathways are not sending enough information, create some kind of index and present that in a way that people can recognize consciously. And that is a kind of neurofeedback. That is really Helpful and works there too. Kind of neurofeedback that we would like to have right now. Then your feedback that people have with EG is basically based on the power back, Alpha keta and wave power. But this is an old technique, not very effective.
And in the meantime, people working on, fmri techniques have come up with what they call. The dexnav. This is the work of co-author group in Kyoto and And in this case, they look at the targeted activity, brain pattern of healthy people, compare that with people who they would like, to change this activity. And then give the index, which is based on similarity of how far are we from more optimal brain activity patterns, and this is decoding the local activity or regions, and the other one. Is that you like to have a stronger connectivity between different subnetworks in different parts of the brain.
And so the connectivity neurofeedback is something which is also very useful. And unfortunately, it works quite well for treatment of some disease. Like, autism depression, schizophrenia, Etc. But it requires scanner.
It's kind of neurofeedback based on functional, magnetic resonance. So people thought about this idea and called them closed-loop systems. In closed-loop systems, we have to record what the brain is doing. The codes that somehow and code stimulation and stimulate the brain. And this can have lots of different applications, of course, like controlling the artificial limbs Etc, but it can also help neurofeedback treating different brain disorders. Lots of things that we can now use, mostly currents.
So the transcranial direct current stimulation systems, which are relatively inexpensive, and then transcranial magnetic stimulation systems, which are a bit more complex and require lots of power in these coins. And also, people are using the vagus nerve stimulation, which is just in the neck, peripheral stimulation and try to use Ultrasound, laser, Etc. Lots of things that we have at our disposal.
Unfortunately, all of this requires close collaboration with neurologists who just kind of do these kind of experiments on their own So, the big question is why this works. Neuromodulation and feedback - and nobody really knows that. But there are lots of results showing that neurorehabilitation is possible. The mechanism is unknown, but maybe it's just changing activation of threshold of neurons which is sensitization that will change the way brain network work and maybe it's just that neuromodulation leads to changes in cardiovascular coupling. So there is more blood coming to certain brain regions And so they work more intensively. This can be tested.
For example, we've been talking with Professor Bernhard Sabel in Magdeburg about the using his Transorbital alternating current micro stimulation device. They are used to restore some of the vision - low vision. He has written a book about that. So far We haven't got money for this project, but I think it's extremely important to understand the processes behind all this because then we can devise a better way of using neuromodulation and neuofeedback. The neurofeedback mechanism seems to be rather complex and there Is just one paper that I found, which has a model of what can be happening. That there is no proof that it's really good model at the moment.
So lots of things to be done. So brain-computer interfaces now: Well, we can capture some signals like EEG signals, extract some features and try to use that to control wheelchairs to improve our performance to restore certain functions, to enhance driving experiences by for example warning that we are drowsy, supplement or do the research supplements and some activities and functions, Etc. And it could be evasive, if we have a brain implant, like in case of deep brain stimulation. It could be partially invasive like in case of electrocorticograph and it could be completely non-invasive, which is usually EG or other techniques like the monitoring of infrared Etc.
So unfortunately all this is rather complex. That is the signal processing is complex and then lots of machine learning. And finally we read the brain states and interpret them as well. Some kind of decisions like we go left, we go right, move.
We move our fingers, etc What people try to do with this with this closed loop is to read the brain states of experts. And then activate similar brain states of novices and the US Army has been using system, which is called Intific Neuro-ETS for a while. So, the expert knows how to behave in a complex environment, like Urban fights, and the novice has to learn. And so people try to transfer learning or knowledge of skill from experts to novice and these type of devices may have like 100 channels And lots of things are going to happen in this area, as you will see in a moment.
So for doing things like that, you need a nice cap, which has EEG sensors and some of this direct current stimulation sensors. Which are right here. And it could be used to, well, duplicate states in other brains treatment of depression, neuroplasticity changes that help to alleviate pain. psychosomatic disorders Etc. And then it can well actually have lots of electrodes. Now, 256 electrodes for example to change to induce these changes in the brain.
So consumer EEG that simple EEG that you can buy is now quite big. But these EEGs have few channels and are not the devices used for research. There are more like for gaming. And the combination of virtual reality with EEG is also coming. Lots of equipment on the market where you have EEG or maybe nears, the near infrared systems and coupled with some visualization in 3D in virtual reality. This is quite interesting and we have a group of students actually that got together and there was a big hackathon in the spring school.
Organized by g.tech in Austria, which is big company producing all kinds of EEG equipment. They got the first place for kind of a closed-loop system. Simulation stimulation that is based on neural beats on the auditory activity, stimulation. And then reading EEG, and then changing the stimulation and then students also have organized in May a neuro-hackathon in our place. With people from all over the world, creating teams and then working on some applications that they have to do in 24 hours.
Okay, so partially invasive interfaces are really great because electrocorticograph and we'll read the cortical activity directly from cortex, but unfortunately, there are not too many volunteers would like their brains to be open. And if you have that, you can actually treat the epilepsy OCD obsessive-compulsive disorders or lots of phobias and other things like that. It's quite effective. Lots of work is being done.
But unfortunately, only in case of people who have either epilepsy which cannot be treated. So in the usual pharmacological way, or I have some kind of brain tumor where they will put also this kind of a mesh Etc. So this allows people to control themselves by running. also currents.
So you have a handheld device and you can actually change this date of your brain in a conscious way. And then, of course, there are those new projects like DARPA projects in neural engineering system design or electrical prescriptions as they call them. Which should enable our digital modulation of peripheral nerves to restore a healthy passionate patterns of signals in neural circuits etc. There is the Elon Musk with his neural link or he makes a lot of press and noise but is still rather far from real applications with humans. And there are really incredible projects with neural dust, very small devices which are like grains of sand or microscopic wireless sensors that can be implanted in the brain and then send the signals.
So we don't need the wires but you have to have them implanted in the brain Now having that, what will that give you? Well. Lots of things related to this targeted neuroplasticity train. This is what DARPA tries to do and that will enhance our human computer interaction very much. But if you have the nano wires, just experiments on monkeys showed that if you have 205 neurons monitored with this very thin nanowires and you have the spikes of these neurons where they just read the, the activity of selected neurons you can decode from the spikes almost exactly what the monkey sees, which is quite incredible. 205 neurons are sufficient to collect information in the form of Spike distribution, that can be used with machine learning to recreate the faces that monkey sees, and it's almost perfect recreation and you may find neurons, which will react only to the formation of these faces along certain well components or deformation of types.
It's not the principal component analysis. It's not the ICA, the independent component analysis. It's a bit different, but it's still similar. So, 205 neurons. It's incredible. And then, you can also implant these kind of nanowires in the motor cortex and inject to the motor cortex of monkey information how to behave.
So lots of these things are coming. And if you think this is still very impractical because you cannot do it with humans. Well, here is a mobile deep brain recording that still that also includes distance stimulation platform and you can carry it around. It's still 4 kilo, but this came out last year so you can have virtual reality.
You can have brain reading, you can have brain stimulation everything in one place. Now how that will look 10 years from now you can only imagine So to do something useful with this we have to understand some of the brain processes which go on quite deeply. And this is what we've been concentrating on. The way you do it is that first you have to look at what our structural connections, which parts of the brain are connected by the axons What are the functional connections that is, which Prides parts of the brain work together in synchronous way somehow. You look in the scanner at different patches of the Brain, and look how the signal changes there.
And look how the signals is correlated. Look at the correlation Matrix. You simplify that because there is lots of noise and create kind of graphs that show you what connects to what and how these sub networks are activated, depending on the task. This is what people have been doing and with the structural connections are functional Dynamic connections.
They find that there are big differences between people with, for example, autism spectrum disorder and the control people, this is the control. This is the Autism see Lots of very weak connections and some other diseases, like Tuberous sclerosis, have even less strong connections and look very bad. And so we know that analyzing connectomes analyzing these connections.
We can find out that certain things are wrong. We would like to increase the information flow, where it's very weak. And so, people in Japan have shown that if you have people, with let's say, autism spectrum disorder and typical people.
We just look at this correlation Matrix. You look at where the information flow is weaker or synchronization between, which parts of the brain is weaker in case of ASD and you can combine this information and have an index which allowed them to have 85% of accuracy predicting which patient is neurotypical and which has ASD and they just took the same system to the US. Two very different population. Different scanners, Etc. The prediction is slightly lower but it's still 75%.
So we can have indices which are based on the information flow in the brain that can objectively tell you how strongly this activity of the brain is connected to The ASD, schizophrenia, major depression and other things like that. So, the idea is that the functional connectivity can actually be used as a kind of biologically meaningful index that will create a certain space in which you can say. Okay. I mean, these diseases may be really continuous.
It's not that this is sharply. Schizophrenia. This is strictly depression or ASD Etc. We are somewhere in the space But we have to learn how to characterize the space in which our cognition can be placed. We can be in the space, which is roughly the normal, but somehow bit depressive somehow bit autistics somehow a bit obsessive compulsatory or even schizophrenic. diagnosis and at the same time, we learn how to interact with this.
And so the National Institute of Mental Health has created this research domain Matrix, which shows different large-scale systems related to cognition controlling things like attention, perception, language, memory, etc. to all kinds of affect - negative valence related to loss, frustration or positive to reward, habituation, expectancy etc, arousal regulations and social processes regulation. And so they have a huge Matrix where they want to put all the information about the genes, molecules, drugs, cells and behavior.
And then also this networks. And so lots of work has been now done to show that this kind of closed loop systems, actually can improve learning in some, in some ways. There is a paper on externally induced frontoparietal synchronization that will modulate Network Dynamics and hence, working memory performance, Etc. And there are all kinds of things related to Natural brain information interfaces, which would recommend relevant human brain the information by picking up human brain signals. The Army is also using things like that. If you just noticed something your brain might notice, but it may not reach your Consciousness.
And so you have to tell yourself that "Ok my brain has noticed something" lots of other papers that I don't have time to talk about but basically trying to synchronize better for the frontoparietal areas in the top of the head and the front of the head will increase the working memory capacity and will increase your ability to solve problems. This has been demonstrated by synchronizing these processes with alternating currents directly. So this is a very nice direction. For learning, though, to understand that better We have to understand what kind of a sub-networks do, what kind of things. And people came up with this scheme now that there are many sub networks, like sub networks, that will internal stimuli. The auditory, visual and other sensory stuff, but also things related to ventral attention, which is just Kind of automatic, if something happens, that draws my attention.
But also the dorsal attention, when I want to pay attention to something So it's voluntary. Lots of things related to this and then it seems like information flow between these different sub networks is controlled by the fronoparietals flexible half. For example, if you don't do anything, you have activity in these two parts basically. And this is called the default mode network. Your thoughts are about yourself your situation, relation with other people, work that you have to do etc But if you are focused on doing something, the central executive network is is turned on and inhibits the default mode network, and that, this is something you have to pay attention to you have the activity in this yellow regions here.
We thought that we cannot do it with the EEG because this is quite different type of signals that fMRI gives us. But in 2015, our friend Bodurka died a few months ago He was a director of the center in Tulsa. Oklahoma has shown that actually, you can die. Making a clever processing of EEG.
You can extract similar activity in the brain regions. As you can do with fMRI and then several other people followed and it seems like we know now how to get it's still very difficult, but we know how to get this activity characterizing what happens in the brain. And to do that you have to well go from sensors from the electrodes that measure the signal to source. We have to recreate the pathways of signals and how it's changed. It's mixed when it goes through all kinds of tissues in the brain. The skull Etc. Again, lots of mathematical signal processing systems.
Loreta is perhaps the most famous one. We've recently also probably start our own toolbox called SUPFUNSIM which has some at least theoretically has some very good properties and we want to show that it is really better than others of course and we work on all kinds of approaches to brain fingerprinting as it's called. And this is we try to read information about what happens in the brain from EEG from fMRI, and there are several approaches to this analysis Microstates, task dependent modes in fMRI what is called contextual connectivity, Etc. So unfortunately, these networks are very dynamic. They change in time and
Depending on what you do. So for example, if you have to count for 100 by one, so 99-98, Etc. So you see that, then there is lots of these local activity in the default mode network because you can still think about yourself and it's gorgeous. Just counting goes on without much effort. remember what was two things back and then React to them. And then remember the next thing coming at the same time.
Then you see that the default mode network locally is not active because it's a difficult task. This has been shown by Karolina Finc who is in our group and has made This was a part of her PhD. And when you look at the same situation, and look how strongly things connect between the frontal and parietal.
And then Other temporal and occipital, brain areas. You see that in case of hard task, there are big activations and lots of information flows all over the brain. So this is a dynamical change depending on the cognitive load. And another thing is that if you train your memory, for example, this was based on six weeks, training with harder and harder working memory task, you find that the control people have not changed. Almost at all. This yellow part, is this frontoparietal synchronization network. But those people who had to use their brain,
in hard situations where the frontal and parietal area have to synchronize. Arer here, much more red, which means that there is more brain areas, which became parts of modules synchronized together. And so we have done quite a lot of analysis with fMRI And it has been just published last December in Nature Communications Another approach is just to focus on brain region and look at the This is called spectral fingerprinting and quite recently a paper appeared Unfortunately, we have not been the first to look at that, where they could actually distinguish two subtypes of major depression and two subtypes of PTSD. It looks like there is a subtype which is quite different and reacts in a different way to TMS, to this transcranial Magnetic Stimulation procedures, Etc.
So, we work on the packages that will allow us to analyze what happens in the brain. You can see here, the recurrent patterns, plots and then based on the short term for your signals and then the information flow here. So just to finish: neuroscience has now strong influence on artificial intelligence and HCI.
And then AI also is very important for Neuroscience. Hassabis, who is the head of this deep brain inspired Artificial Intelligence paper in the journal Neuron. Yoshua Bengio has written the Consciousness Prior paper.
So some people talk about all these different things from the point of view of models and understanding what brains are doing. came up with is now providing insights for development of working memory, gating maintenance Etc. So the language that neuroscientists use comes from neural models, and then AI systems have been inspired by understanding of neural models.
Visual attention, complementary Learning Systems, models of working memory and the general turing machine, synaptic consolidation etc So people are creating systems which have to behave like the intuitive physics knowledge that we have. Allows us to behave still understanding and supervised learning - one shotgunner is generalization, imagination of realistic 3D environments in deep neural networks Etc. Monitoring the development of children and infants. This is what we've tried to do. Perception, working memory, curiosity, unfolding full potential of children, and then precise diagnostic of subtypes of mental disorders is within our reach.
And that includes all kinds of subtypes of schizophrenia Etc, which is a big problem because people have never been able - by just looking at the symptoms - to diagnose that more precisely and enabling early diagnosis, for autism is very important. Another development of problems. So we work on some of these things and especially the non-pharmacological approaches to various form of pain management. These are going to be very important and this is done through some neuromodulation. This is already done by neurosurgeons.
They will put in your spine a piece of electronics that will actually inhibit the well spikes or the signals coming from the nerves through the lower part of the body. This is a big Market. It grows very quickly and the estimations are that it can reach 27 billion dollars in about five years time. And closed-loop Neurofeedback for neurorehabilitation is going to be very important targeting neuroplasticity and brain-computer interfaces that will be based on new forms of neurofeedback. And also, there are a few people working on disorders of Consciousness, trying to help people to get out of coma in the hospice unit that we have in Torun. Their applications in education, their Technologies for General optimization of brain processes, well-being, wisdom, entertainment, Etc.
So we've published quite a lot of things in the past few years, in journals, like Nature Communication, Nature Protocols, Human Brain Mapping, Brain Imaging and Behavior, Neuroinformatics and Sensor and other patterns in other journals of the sort. This has been a part of the big project that we have, which is called "In search of the sources of brain's cognitive activity" and which has been running with people from nencki institute of experimental biology for a really neurobiologist. Experts in institute of physiology and pathology of hearing which are even more so in doing this cochlear implants.
And our center for modern interdisciplinary technologies. Thank you for synchronizing your neurons. There is lots of other things that I would like to tell you. But there is not much time to go on so thank you.
Thank you very much *applause* I think you can hear the great applause here on site and probably a lot of virtual bravos to you. Thank you for this inspiring talk. I don't think there is lot of people in our community that would like to experiment on the open brain, but we will surely like to use your results into our research. We have time for discussion so first of all I would like to ask participants on site for questions and on the other hand we will also copy and read the questions from the zoom Any questions here? If not then I will start. So I have maybe some more philosophical questions.
So you told us that neuroscience is very important for AI and AI is very important for neuroscience. And our decoding of our understanding of our brains function. But I would like to follow that the idea that our actors actually hosted in the beginning.
So we are living in the applications of artificial intelligence coming into our daily lives. How do you think can we use your results and our HCI community methods to be sure that this AI technical applications would comply with our human rights, human expectations and not to overwhelm our human feelings? What can you say about this? The first thing is always understanding the processes behind what we do. And if you understand them.. for example, this very recent paper that I've mentioned that can distinguish different sub-types of ADHD, the traumatic post disorder, and also major depression etc if we understand how they differ from the point of view of what happens in the brain we can actually have better intervention But your question is quite general of course and it relates to well how can we be sure that this is not going to be used in a harmful way. Of course, we can not be sure. I mean I don't know what the army is doing. One reason I'm very much interested in this is trying to follow what is happening in the world because we see lots of danger. For example, this idea that have all kinds of conspiracy theories that we can be controlled remotely.
Unfortunately, we can not yet be controlled remotely. Unless we have all these nanowires in our brain and you know receivers and other complex equipment that we'll very easily notice It's not something that you can inject into people in any way. But it's something that can be used for more effective brain washing.
And it's not very complex technique because electric shocks have been used really to change the way people think. Because they can induce new pathways in the brain So lots of danger is that to understand them better we have to watch what happens in the field. But I don't see how we can guarantee that the army is not going to use that in a bad way. And especially if we see what
DARPA is doing. DARPA is a very powerful organization. And of course they say "ok we want to train our soldiers faster. We want to you know give them better cognition so that they do better analytics and other things like that". This is what they tell us. But what really happens we don't know. So I'm very much worried about potential military applications of some of this. I see very good things coming for medicine.
And the idea that you may get sooner or later a cap with electrodes plus a smartphone application that will allow you to stop addictions, to stop certain compulsive behaviours etc is very nice. And we can probably treat lots of people but at the same time you don't know what other applications people will invent. Right? So there is no simple solution I think in this area. As all kinds of technologies it may have very bad consequences that we have to watch. As we know, especially in AI now, there is lots of activity to regulate certain applications. Because you can not regulate AI. And it's like regulating mathematics basically.
AI is a bunch of algorithms and Machine Learning is just algorithms So you can regulate a set of applications based on these technologies and this is certainly before we get any commercial device This is certainly something that has to be done. The European Union is very interested and there have been some documents already issued. We've been asked also to express our opinion on the ethics of AI. And all these big projects that we have now in Europe in artificial intelligence they are basically related to the ethics of AI. Ok. Ok. It doesn't sound really very optimistic but it means that we need to put a lot of effort into ethical part of our research So I think this could be one idea but should stay with us I'm taking a look into internet questions but I think we had some problems so I don't have any questions from the internet for the moment So that is the last chance for you guys on-site to ask a question.
There is one question. Please use your power to.. Pawel Wozniak - Chalmers University of Technology: So somewhere in the middle of your talk you talked about there are those phenomena but we still don't we know that this works but we don't know how it works. And funnily enough, I saw the same when I was working with my colleagues in psychiatry where they said the same thing. Like we have therapies
we know that therapies work. But we have no idea how. And we're not even close to how. So given that you've spent your career working with brains: do you think we can even ever understand those systems? Cause you know this is an HCI conference We like the deterministic stuff. Stuff that we can analyze. Do you think that this is gonna happen within our lives? Is it 100 years? Is it 1000 years until we understand it right? I hope it's not 100 years because there is a big process lately in first modeling these processes and then measuring these processes and verifying our ideas But it's true that in medicine that it's so complex that we don't know how things work. I mean all the pathways on the molecular level are very difficult to trace - even for simple drags but still people can now understand how viruses work Assimilate that. Assimilate the parts that you can actually disengage somehow.
In the case of the brain, my feelings are mixed in the sense that I see that when you look deeper you see that it's a very complex hundred billion neurons connected together with hundred thousand billion connections. It's incredibly complex, but when you look at that in a very rough way you see lots of experiments showing how different brain regions activate in different tasks situations so we kind of understand that the involvement of make the lie in the few responses There is a key. It's not that I make the law is creating fears. This is created by the whole brain network activity but it is a key element this is what we can do and now the problem is that is cannot be too much reduced to the interaction of simple elements. it has to be holistic. Now holistic means that at some point, we will see okay there all these major brains diseases and it's kind of a continuous space.
I talk a lot with people are in epilepsy in one of the big hospitals in Cincinnati US. They claim that it's only half of different types of epilepsy that they try to diagnose and put into a specific subcategories but it's so fiction because some people just want their names to be attached to things and they say okay. This is subtype I've invented and it's called with my name but the fact is this is the continuous situation in which different parts of the brain to different degrees different types of neurons regions etc. may actually create a specific kind of epilepsy which will have very different manifestations and so I envisioned this situation in which we will just look at these holistic processed and say okay. This division is just very rough and it's also connected to
how we use the language to describe things which are continuous basically. continuous processed in the brain that psychologists try to categorize. if you remember until 1971 psychologists just believe in memory and then suddenly tooling came and said oh yes but the episodic memory the autobiographical memory and the memory for facts which is the semantic memory are two different things and then they came up with this idea as procedural memory is still the one thing the recognition memory is quite a different thing right so there are lots of brain processes which contribute to memory of something and now we have 10 different forms of memories so we try to use the language which is more and more specific and is attached to is connected to a certain brain processes. can we do that with the diseases and understanding of general behavior only to some degree I'm afraid so we have to rely on simulations and systems that will simulate with brains are doing then we can create better human computer interaction knowing how the system will respond and this is what people do in molecular biology where they have now simulations of microbes very detailed of all the metabolic processes, genetic processes etc. and then they can simulate what will happen when a certain drug comes and changes that this is the future I guess.
Thank you very much once again so last round of applause for Vodak Thank you very much once again for joining us actually in Poland, we traditionally are quite flexible but I know that the next session chair is from Germany we need to conclude this opening ceremony now and once again thank you for joining us virtually I don't know if we have time for a coffee break We will do 11:35 back in here so we'll have it 20 minutes we have nice food for you guys, all right So thank you very much so we have a short break and we are back at 11:35 11:35 back in here please thank you so much Thank you
2021-11-30