Enriching the World with Artefacts and Algorithms (PAIR UX Symposium 2018)

And. She's. Innovation, director at the Microsoft, Research in Cambridge and also, appears on some, BBC, documentaries. Big, likes and, yeah. Welcome. Hi. Hi hi, everybody so, we're so close to lunch so make this snappy and quick. Hello I'm hi Anne I'm director, of innovation at Microsoft. Research in Cambridge, in the UK the, original, Cambridge if. There's any doubts. And, I, call myself well, I'm a designer, an. Engineer so I started. Out in computer science and I moved. To being a designer. So. I worked at a, consultancy. Firm called IDEO for about seven years and in the last five years I've been. In Microsoft, running, innovation teams and Xbox and in. Research, and I. Essentially. Call myself a maker of things because I think it's a really nice, fusion. Of design, and engineering practice, to treat technology, as a design material. And. To work with that from day one so. I, think, a quote that really encapsulates my. Philosophy. About design is from, Eileen Gray who's a. Self-taught. Architect and furniture, designer she said to create one, must first question, everything you, know this is a really broad statement, but. To me it means just, you. Know whether you're designing an interface, for a I. Phone, app or an Android app or, a product. You're, essentially baking, into that design, how, you think, people should go. About their day should, live, should have a lamp in their house the, quality. Of that lamp and I think to me with. Every single active design I think that it's also, has. The potential to be an action to change the world so. I'm a big fan of knife, fights in the design process so meet. Me in the dark alley and we'll debate, over who who's, collective, narrative whose narrative, of the future will win because. I think we you. Know within, design within the service design, interaction. Design we've been talking about these narratives, that we all share things like the sharing economy, things. Like sustainability. Things, like open source and. And these are I think within our culture in Europe and in North America these, are sort, of narratives. Of kind, of a sign or a science, fiction or a design future, that we would like to see, come to pass, and, but. You know just hiking back a little bit on my childhood, that was not always the case I was born in. China. In. The aftermath of the Cultural Revolution and, in, my early years this is not a picture of me I. Did. Not look like a boy. In. My, early years you know I went. To primary school in China, and at, the time in the early 80s, it was very over indexed, on, the. Communist party and party philosophy. And. So I, was in a system that was very much focused on, stripping. Individuals. Of the personal, agency of, opinions. That. Differed from from, the party it. Was kind, of like an institutionalized. Fake news. System. Where. Every. Piece of information was, filtered through this kind of philosophy whether, or not it was true or not had to be sort of bent. To. Its, will and, then transition, that -. When. I moved to Australia I moved to Australia the year that Star. Trek The Next Generation aired, on TV. And. It. Was, super. Kind of it was a revelation, to me to go from this, kind of system where individualism. Was frowned upon to, a system where, you. Could, this. World where you might be empowered. To pursue, your. Own interest and to achieve what you, might be greater. In the world and. Particularly I was a big fan of Star, Trek, and. Particularly enabled. Empowered, by technology where. Technology, would, serve. To. Enable. Your personal agency to allow, you the freedom to explore what you were interested in, so. I sort. Of went into. Technology, as a, as a. Practice. And always. Thinking about sort of using technology, as. This kind of design material. To. Change the world and in, recent years I've been involved in a BBC documentary series called big life fix which is bringing together seven.

Engineers. From around. The UK engineers. And designers from around the UK to, invent cutting edge technology, for people in need which is really funny because the audience, on BBC. Two as. We're told are kind of like granny's in the Midlands. So, it's a real juxtaposition. So, one of the projects that came. Out of the show. Was. Around. Sort. Of supporting, a, young lady called Emma and. Emma when, she was twenty eight years old she was diagnosed, with Parkinson's disease which. Is actually a very rare event and as a result, of her. Disease. She. Had acquired an, action, tremor in her, hand and she was being, a graphic designer and a creative director she was no longer able to, pursue. Her job. Because. She had, a she had her, hand shook when she was trying, to draw wireframes, or write her name and so. I worked with her to think about well how can we, create. Something that might be able to help. Her overcome the, symptom how can we fix this problem, it's, through a design, and an engineer, England so I just quickly show you that video. Alright, I'm gonna try and replicate this, here. Roger. A great start I. Tend. To kind of just avoid doing sketching, or writing now because it's just it's not really worth it if you get something like that. Anything. You could do that would just make my hand do, what I wanted to do and, sign. My name would be an incredible, thing. How. Do we even just begin to help her overcome this, this. Particular symptom, of her tremors and helping her be able to regain. Her, writing, ability her drawing ability, you. Know I don't think we're ever going to get. That back 100%. You, know my challenge, is is I. Mean, it's immense. Someone's. Made a spin, it actually counteracts, the tremors. You get from Parkinson's, so the spoon actually vibrates, in opposition, to, how your hair might be shaking. And it's, therefore it is steady. I'm. Making. A very, rough prototype, and. What. This board does is, I can connect, into it through these wires these, are tiny coin-cell, motors, so, these motors, will vibrate. Hello. I am Alison. It's. Affecting, something I don't quite know what's happening something is going on for them what. This is doing is its short circly, whatever. Feedback loop there is between the, brain and the hand that's causing the tremors. I'm. On something, right, now I'm onto something. Oh oh. Jesus. Christ Oh. God. It. Makes me forget that I have a Trevor. I've. Actually just written my name for, like the first home ages. What. Was called the Emma. My. Name. Thanks. So much thank, you um. Yeah. So I think I mean one of the takeaways that I have from from, doing, this work. And. There are many I want one of them is that I think you. Can't quite tell if this is a design project or an engineering, project because. Designer Lyon couldn't, solve this problem but at the same time I think I. Mean. The clip didn't show up but my, undertaking. Was a design, process and it was the design process that led me to the insight to start prototyping with, this kind of intervention. And. So, an, engineering. Process on his own could not have achieve this either, and, I. Think the other. The. Other takeaway which I will, talk, about in a bit is is just, that. There. Are so much potential for for. Technology. In all as forms, and in. Its physical form factors. To still. Have relevance, and and and make, a huge impact that that we just we. Haven't explored yet and I know today we've been talking about algorithms. We've talked about smart systems, but, you know my, focus and passion for, I guess the last 10 years has been about hardware. And devices, and I, think it's through opening, up new form, factors, new.

Inputs. New outputs, that were able to, create. A new stage for where algorithms. Can, come into play beyond. Mobile. Phones and and you. Know desktop metaphor. So, we, are with. Microsoft. Research team we're continuing, on some, of the research into this technology. The. And. Some of those challenges, that we are now encountering. Are you, know we really do think that smarter. Algorithms, can make the effect. More. Amplified, but. There's, this transition or, there's this question between signal. Processing and machine learning so where where, do we where, as a signal processing end and where is it better to pick up machine, learning, as a, tool to do this and we're not quite sure without, going, through the hard work of, each, piece of technology, to test it out. And. And so I mean I think through the some, of the discussions, that we, have around machine learning there's this kind of idea, that machine learning is the solution for everything I think, we need to really question that and understand. Sort of where where the boundaries of machine learning end and where, we can, best. Leverage other kinds, of, algorithms. Of rule based algorithms, still. So. We're continuing, to develop, some of the. Study. Apps. And, tech. To to, capture, this data for more people. I. Think one of the really. Interesting things that emma says herself is that you know she. Is dealing with so much in her life and so, much from this disease and. Wearing. This watch has been able to give her back a little, bit of that. Dignity, that. I think the disease is taken away it's not I mean, it's not a cure and it's not fixed everything, but. If, the technology can give her back a little bit of her, agency and her dignity its I think in that itself, is a really worthy. Value. For us to pursue and, at the same time when we think about. Algorithms. In the loop or AI in the loop you, know the. By, wearing the watch emma is able to regain, some of the clarity, and her drawer in her writing, but it's not perfect this. Is not an augmentation, to, allow, her to write like a robot, to. Give her perfectly, straight lines there is still a little, bit of quiver. In her, writing, and she. Enjoys that because, this, is that. The disease is also a fuse, with parts, of her personality so. She. Enjoys, the imperfection. And, that the the technology, through not, working, perfectly, is actually, giving, her back some of that personality, that, she misses. So. Let me transition, into another project that I did recently. Last. Year working. With a young, girl who when, she was 8 years old she had a car accident with. Her family and she suffered, some traumatic brain injury and she's. Now 12 and. The time when she had her, accident she, it was quite devastating, and she could no longer you. Know she couldn't walk she couldn't talk but over time she was able to rehabilitate and now she's up and about and running around on the playground and going to school but, she's also sustained. Sort, of more long-term. Issues. With, her memory. And so. I went, in to see if we could create technologies.

To Help her regain, some, of that memory to help her capture some of that memory so, through the design process working. With her I was able to kind of start to map out what the challenges, were memory, is kind of like trying, to define what memory is kind of like sand, running through your fingers you know everybody, said she had tried she had short-term. Memory loss but actually she didn't have that much short-term memory loss what she did have was this inability, to. To. Remember, events. Or episodic, memory in the long term so if she'd go on a picnic with her family she could remember for, you, know 30 minutes but she wouldn't be able to remember remember, at the next weekend, if, she took a class she'd be able to remember things in the class but she wouldn't forget, it the next time, the class came around, so. I really said about sort of trying to think well how can how, can we create, sort, of technology, that augments. Her memory, augments. Her brain and in some. Ways not. Through, everything, not through everything that she's doing but at specific, moments and these might be different, pieces of technology, so thinking. About how can we build. Her knowledge to complete school how can we help her. Relive. Her emotional, moments, with her family, and I would say that the results, are you. Know they're interesting, and they do work for her but at the same time the physical form factors I still find clunky. Because I had to use you. Know tablets, and off-the-shelf, equipment, and. In. Order to kind of augment, some, of her, daily. Routines, do. You remember us arriving. I remember. Like, waking up but. Then I like don't, remember, what, I did. We. Went for family wedding we just literally remember waking up in hospital, they, kind of just said she's, not gonna make it everything. Was was baby steps she literally had to learn how to eat again, how, to talk, how to walk it, was almost like having a newborn baby I. Think. We. Need to fix --is one. To help her with. Her, classroom. So. That she's not falling behind the rest of the class and one. To help her with her family, memories, so that she can just remember, her childhood. The. Food is killing, me. It's. A tablet that sits. Next to you so as the teacher is giving the lesson you'll, see the text of what they're saying appear, on the screen if Miss, Powell says something. Um I. Could just play it back on this yes, so, if you try it. So. Just, to be clear it. Works. So. I mean, I think. Part. Of that solution as you saw was some. Cognitive services where. We, turned the teachers, speech, into, text and she's able to rewind that in the class to see what happened five minutes ago, and. I think you know these today, and also other AI conferences, we talk glowingly, about AI and what it's able to what's been it's been able to achieve. I think, when the rubber hits the road and we're trying to really make solutions, that work for people there's still a long way to go. The, this, technology, I mean I think it was a great proof of concept but at the same time you know how did we capture the voice of the teacher I had to put a lapel mic on it because we couldn't isolate her voice in the classroom, she, had a Birmingham, accent I, you. Know I had to Train specific, models just on her on her accent, and, then the. Kids in the class had, names. That weren't you know, anglo-saxon. Names and that was difficult to translate so so I think there's still a long way to go and and also, in terms of form of Technology. Form factor the fact that she has to have this tablet next to her in class is not ideal and, then finally, I'm just gonna quickly share, some, of the work that my team and I've been doing at Microsoft Research for the last couple of years where we've kind of really. Looked at well how can we open up the space of, physical. Digital exploration. How. Can we kind of create. New, theaters new.

Spaces With within which people can interact with computers, and and. Kind of create these new kinds of experiences, so this, is a we've just we've. Won a best paper award for KY 2018. So please come see us if you're in Montreal and this. Project really started because we were you, know really obsessed with expressiveness. Through tangible user interfaces, and frustrated. With the, history. Of two, eyes and. How. They were, fixed. In place they required a lot of setup there was occlusion. Issues, and. We really wanted to set out to create kind of a more. Fluid. T UI system, and I may be old-fashioned. But I freaking. Love to you eyes man all. Right so. So. And then this is the the the project that we created. So. That's. The the technical, preview and we've worked with the Microsoft, edge ml team to do onboard. So. The machine learning in terms of detecting the different gestures hover gestures, and at the same time we're thinking more. Upstream about what are some of the applications that, can. Open. Up the expressiveness. Open. Up new learning, and play opportunities. And. Here. We see, much. More space for new. Kinds of algorithms, that can enrich these, play, and learning experiences. See. A. Our. Car. And. So. We've, actually engineered. The entire, platforms, from scratch because. Within, the team working in this multidisciplinary, team we really don't see, a divide between the design vision and the, engineering, possibility. Of this technology. So. You know I work with a multidisciplinary. Team of hardware, engineers and designers, product, designers, our computer scientists, and. If you are in Montreal, come see us we'll be giving a hands-on demos as well thank you. So. We're running a little bit behind but, if there are a couple of questions. Otherwise. If everybody's, hungry. Thank. You for the talk it was very interesting and what, interested, me about. The. Problem. Solving for a man was. There. Was a step of like you had the little diagram of how memory is divided, and then jump to the solution, and I. Was, wondering if you could maybe tell us a little bit more, how. You came up with the actual design, idea that will relay, text, into sorry. Speech into text and, then highlight it in colors and and also, the one for the family memories because that was the very interesting, part about it oh. I. Think I'll talk about the family memories because that was very, interesting, to me as well. So, Ayman. She. Can't remember memories. With her with her family, and. There. Is some. Research around memory, loss and using photos, to jog someone's memory and, in her case it didn't work she could look at photos and she wouldn't remember. But. We did a design exercise, where we went out for the day and we had her take the photos and, she. She. Was punting, on a on, a boat. In Cambridge, and she thought the the punter, guy was really attractive, so, I he, looked like somebody from one direction or something um and, so she was giggling the entire time she you. Know she was so delighted, and then afterwards, we went to a cafe, and she. Was sitting there she was looking at the photos and when it came to the picture she'd taken of the punter she didn't remember him at all she was like I don't I don't know this guy so she'd lost that kind of delight from from her memories. But, then while, she was doing that her mum sitting, next to her just, said Oh Amon do you remember this is the guy he looks like Niall from one direction and, then, this sort of smile came over her face and she suddenly she, remembered the. This. This event is through kind of her mum, retelling. That story of how she felt, and so, just, for, me through that I'm seeing, that interaction. Allowed. Me to kind of create this app where I mean it's a very simple app where her family can upload photos and record, personal memories. For her and just, through this combination, of sort of personal stories and images. She's able to recall some of these memories which is really amazing. And. How. Long does that recall, last like does she have to kind of almost remember. Like create, that memory every time she looks at it or is there some kind of retention, of the information well. I think for me this was a kind. Of short. Kind of project, to create, something so, there hasn't been a more, longer-term study, and I think every person is very individualized.

This Kind of trauma. As well I. You. Know I don't, know it's challenging because I talk to neuroscientists. Who do, research into memory and you, know they talked, about how, people. Like, Armand may never be able to experience, a memory the way that we experience in memory so they may be able to remember the details but they may not be able to remember the emotional, quality of. That details and I think, I. Feel. That through watching her she, is reliving, some of that emotional, component. And perhaps. Through. Regular. Retrieval. She's able to, maybe. Build, those, connections in. The long term it's, very cool thank you very much thanks very much, mr.. Arnaud more question, then we can break for lunch, lunch is over, there and we will be here at the quarter past three quarter, to three sorry quarter, to three to forty five sharp to. Go on with the program.

2018-07-12 21:01

Other news