So I think we're ready to, get started I guess I just want to welcome everybody to this. Session on cars computing. And the future, of work, and this. Is a collaboration between Microsoft. And our, u-dub team which actually, includes more than the u-dub it includes Harvard, Wellesley. University, of New Hampshire as well as the University of wisconsin-madison. And we'll be introducing everybody, in a second, I'm. Going to actually have Edie give, some logistics, first and then me and Eric will get started on the program excellent. Well thank you all very much for coming today we're extremely excited to have you here we, have a great program with, lots of new research slides, video, everything, but, the most important, part of today is going to be the discussion and collaboration that, we have so. We're very excited to have you working, on the papers on your tables we're gonna be brainstorming, together and we're fortunate to have some, good friends from urban wild studios who are gonna do graphic note-taking, throughout, the day, so. This is going to be very dynamic we, will ask because we're recording it and we have a couple collaborators, on the phone if you have questions, we'll bring you a mic. And, I think it's gonna be a wonderful day you know where the coffee is so please caffeinate, accordingly, and, I think we'll just get started okay, great thank you very much ed okay. So. Just. Want to give everybody just overview. Of what the agenda is so we're gonna start off in the morning with just welcomes, and introductions so. We'll we'll do all introductions, about everybody in just a second, we're, gonna have an overview of four, topics. Basically. On ongoing projects, this, is going to include Shamsi and Eric, Andrew. Boren. And Raphael o who is on the phone as well, as Satish, and then, what, we'd ask you to do while they're talking so the talks are only going to be 10 minutes and I'm gonna be a major, downer. By basically, clocking, people but, what we really want to do is make sure we open it up for discussions, for everybody so we're gonna give you ten minutes for discussions, and what we'd like you to do is as you're asking questions I'm thinking about ideas you, have post-its, and notepads, on the table. As well, as pens and what we like to do is just write out your ideas, and then as you write up your idea you'll see that there's actually white papers along the board go, ahead and pick it up and stick it on there and then John Lee is gonna be our runner, for the morning and he's gonna go around and take a look at what those ideas are and then try to consolidate, it so, that we have a 20-minute break during. That 20-minute break we're gonna actually go through and try to get you guys into, groups for, right after the break so, we're gonna put you into four groups and basically. Discuss, some of the topics that we went, over and then right after that we will have you. Know follow up and then lunch and then the afternoon we're gonna do it again but instead of talking about ongoing projects we're gonna talk more about just theoretically. Like where, should the future work be in cars and what we should be going so we'll have four additional talks in the afternoon, and then, again. We're gonna ask you to do the same exercise then, we'll have a break and then, we're gonna again have the discussions, and then and then kind of wrap up and tell you what our next steps are so, hopefully that would be something that you'll enjoy doing, so. What we'd like to do is just kind of get to know who everybody is so. We wanted to be able to do introductions so because, many of us are academics, and we tend to talk really long I'm trying, to keep you to like something really short which is I'm gonna ask you this number one say. What your name is what. Where, you're from and then, in the future what you hope to do while. You're drier traveling in the cars so, for example I'll get started hi, my, name is Linda Boyle I'm from the University of Washington and in the future I hope to be able to not be motion, sickness while I work on a lot of stuff in my car. Eric. Would you like to go next and then we'll go around the room. Known. As microphones which you won't. Get the remote you won't get the recording without the mic. Okay. Then I would. Just be very fair my. Name is Vera Horvath, I am, from Kirkland, Washington in.
The Future I hope to be, able to minimize, distractions. And drive more safely on, all, fronts. And. Then. We'll start passing around the mic and, be expedient, mm-hmm. Good morning my name is adorned home from Microsoft, Research and in the future I hope to talk to my digital assistant, while I'm in the car. Hi. I'm Andrew Coonan from the University of New Hampshire, and in the future I'd like to be able to listen. To audiobooks and, annotate, them while I'm in the car. Hi, I'm Oh reach share from Wellesley College in, the future I would like to, effectively. Rehearse my PowerPoint, presentation, as I'm going to this workshop. Hi. I'm, Duncan Brumby. I'm from University, College London and. In the future who's. Gonna say go for a nap while driving. Hi. I'm Sean said well I'm from Microsoft research in the future I hope to be able to capture, my thoughts while, travelling in my car and. I also suffer from motion sickness so I. Am. Wendy Cooper I'm with MSR, and in, the future I'd like to have an intelligent conversation with. My car in, my car. I'm. Hot on salmon, I'm with the University of Maryland, in the future I like this safely, goes off in my car just as, I do on the train and on the plane. Hi. I'm Lewis Lam from Brazil, federal. University, of Rio. Grande and so it's in the south of the country in. The future I hope to feel, safer, when, I sleep while, traveling, my time. Hello. Everyone my name is Jacob check a risky I'm with the University of Alabama actually, don't like driving I bike everywhere, almost everywhere so I hope to be able to continue doing that in the future. Hi. My name is Nicola. Banished I'm from Ann Arbor Michigan in. The future I hope to spend more time with my family while traveling in the car which we do a lot. Hi. My name is Jesse, yang I'm from the University, of Michigan Ann Arbor as well in. The future I hope to sing car okay in my car without, being asked, to take, over. My. Name is John Baum I and with Microsoft's. Business development. Team in the, future I hope not to drive while traveling, in my car. Hi. My name is Meryl I'm the first T of Washington, and Emma star is an intern and future. I want to finish writing my tight papers on the car. My. Name is Carlo I'm from Harvard, Medical School, in the future hope, if. I fall into sleep I feel. Safe. Oh we're having like huh. Hi. Good morning I'm Satish I'm from MSR Bangalore, in India and, if any of you been in India you know how driving, there is so in the future I'm. Like him I'm a biker too and I hope people driving, in cars you know pay more respect to Vipers. I'm. Alice fern I'm originally. From the University, of Arizona College of Medicine but. Now in. Microsoft. Office the. Future I hope to capture. My thoughts while minimally, working, with it well travel in the car. Good. Morning my name is making for foreigner from Microsoft research in the future I hope to be less. Startled. By drivers. Who push me off the road or cut off or scare, me in other many, ways while traveling, in my car. My. Name is Jim ping Coleman I'm from Microsoft researching, outreach, and. In the future I'd like to do anything, but, worry about traffic or driving, much. Like a train or a plane sleep, learn do it anything but worry about the, stress of traffic. Hi. I'm Jen Tian from University of Virginia so, in the future I also, hope to do everything the car but, without worry, about my privacy, and safety in, the future. I'm. John Lee from the University, of Wisconsin, and I hope to practice, mindfulness, as I Drive. That. Seems to be the recurring theme, I'm, Travis Wilson I'm from Microsoft speech and language team and I'm gonna actually second, the very boring the very feeling thing I would like to sleep while I'm traveling tonight there. My. Name is Yvonne Tasha I am from Microsoft, Research and, I hope to be well in informatin, until tonight while I'm driving. Down. Norman, from University, California San, Diego. There, are two futures the near future I hope.
To Stop the car companies, and some of the people in, this room who have made very. Dangerous, suggestions, about what drivers should be doing while also driving. Because. I want to be able to feel, safe and, not, just from my own car but from other cars that might hit me in, the very far future I, hope, for driverless cars where there are no controls whatsoever, and so we can do whatever we feel like doing. Unfortunately. I'll be dead by the time that happens. I'm. Marc save Rex I'm from the National Science Foundation and in the Catholic University of America in Washington DC in. The future I hope to now. Repetitively, do, a whole variety of these tasks that people are mentioning because I won't have to attend to the, key aspects, of the driving. My. Name is Audrey Schmidt I'm from the University of Munich in Germany and, I, hope in the future I try, less. And. I travel less in the car so, and. While, I'm in the car I one, of these days. My. Name is Jill I'm from, the Chicago, branch of urban wild studio and I'd like to be able to look, around at the scenery while I'm driving in my car. Hi. My. Name is Leah I am from, the Portland branch of urban wild studio and I would like to feel more. Refreshed rather than drained after a long car ride. Okay. And last but not least we have Raphael online, Ella. Will. Be happy, relaxed, and productive, and not be taken by road rage in the TOC yeah very good excellent. Okay. Great thank you very much everybody. So this is what our workshop goals are we're, hoping, that we're gonna be able to share some really amazing, ideas and, look, for challenges, and opportunities, where we can, actually then come together it's try to find, some solutions, so that we can work together and so, our goal is to basically identify what these potential, collaborations, are and that, maybe look for opportunities. Where we can work on proposals, projects. Research, grants maybe even, autoships, papers. And so, what we're hoping to do later on is we're gonna have a matchmaking, exercise, so that we can kind of match you to like the right kind, of research topics as well as the people that, that. Can kind of move forward with some of the topics or areas, that you'd be interested in working on and with that I'm going to move the mic. Over to Eric, so that he can discuss, oops. Discuss. Some of the future. Of work topics, it. Just says a little bit of background, but, maybe three years ago we hosted National Academy of Engineering, regional. Meeting in, at, Microsoft, and that's where I met Linda, who was speaking about what. Happens to people in their abilities to to. Drive after, they have experienced, long periods of automated, cruise control. Dynamic. Whose control and, my. Talk was on human AI collaboration. And mix of designs, for mixes of initiative, we, just thought it'd kicked, off some discussions, given to interests in, particular around my early experiences. With Tesla, auto pilot I'm an avid user of it and I, find out that I almost experiment, too much and if you want to read about. Some. Of my experiences this. Search. On Google. Or being my last name and Tesla, and read about my accident in Wired magazine and some. Comments about that which is very unfortunate but. I, think. Today this and this is just up we just put up a six, bullets here of some topics but this shouldn't be constraining, it should be more like framing, and the starting point just some. Thoughts from the organizers you. Want to think about what it means to support work in mobile settings and. In. Advance, I'd say if. Possible, at all Fitness, importance comment, of full. Automation what. Does it mean to do work or to think or to to. Even. Listen to email, for example in a car or, have meetings for example and this, brings up the topic very quickly of human. Cognition, and. The. Availability, of cognition. To do different things and. If. You believe in multiple resource. Theory of mind what happens when you attend to something. While, trying to drive and. Surprising. Situations come up for example, it. Might seem quite safe but when you look at tens of millions of miles of driving even, slight changes, to the probabilities, of rolling. The dice leads. To large-scale. Life-changing. Events, and crashes and disabling. Accidents. That affect people for life and changing. Statistics, of that based based on designs for how audio. Feeds work or people checking their device or. Even looking at how. To control the Eastman dashboard. In the car a very interesting and it gets into we're. Here later from Shaam see it's understanding. Psychology in, cognitive cognitive. Psychology, and new studies that are adapted to these situations that. Give, us insights, and designs. And the interaction of design with. What, we understand about human cognition, on.
Safety, Risks here I want. To break this out and, just highlight this because we've. All all of us in this room have been horrified. Looking. To our right and seeing, somebody on the highway or on surface. Streets looking, at a device and then. We. Realize. That's. Really us because. We do the same thing and it says it so the situation, is unbearable yet, we're all on the slippery slope it seems and, that slope is. Has. Been found for. Several studies, to. Be one of the factors in the bump going up after, seeing. Highway. Fatalities in, the u.s. dipped for a number of years now. Start, swinging up that's been what's. Been called out is yes. Safety. Is getting, better with, the cars and the machine itself, and survivability, of cars but. We're seeing increased. Number, of accidents non-fatal, and fatal because of the. Obvious, distribution. Of people who are looking at the device for some segments during, their driving time, and. So the question with this is is okay so but. Something, we can do about designs. Short. Of legislating, in most, places there's, legislation but short of enforcing, laws about, not. Looking at devices while traveling, what. Can we do if we understand, the desires, and, and goals, and what people are you can go in the roads today to make it better, to. Reduce that, upswing, in safety. Challenges, what a directions and its design, and limiting. What can be done and where for example one. Of the comment here but but this is John. Crumb MSR. And I two. Years ago published three years ago the Tripoli I published a paper where. We did we took a large-scale. Data, set from Minneapolis, Minnesota of. Accidents, by, time of day and a week and what properties and where the Sun was in the air I mean, in the sky and Zenith and azimuth and. We. Basically. Showed. How you can run a risks a risk sensitive, planner to. Provide. Safe, safest, routes to travel on versus. The shortest. Or. The fastest, and the idea. Of being able to understand, cognition, and work possibilities. And designs, enough. To, fold that with certain. Kinds of roads like taking the highway, with less turns and, less. The best no pedestrians, for. Certain kinds of work and and folding. These new kinds of technologies. For, for example safety men safety, maximization, on roads with, needs of cognitive, needs they also this is one example of of that. At. Microsoft were actually looking at this idea of card. Rides, as. Productive, times for meetings, and. It's not just us. Large, large motor, companies, car companies are thinking about designing. Cars. Even with given by somebody else as meeting spaces, we've. Had some interesting. Reflections. About the prospect, of someday. When you go to Outlook or exchange. And. You're going somewhere, looking. At pending, meeting, requirements and, even pop, you know jumping. On sahteen Adela's car right to the airport this is a seat they're available for a meeting and you can take your own car back or be, driven back but how can we take all that time and you smart. Scheduling, to get people together for, conversations. One. Of many directions there and. Then thinking about on. The, design side and what cars can do how. Do we look at the roadmap, for automation as Don. Norman said and. There, could be a breakthrough and by so, it might not be dead, went. You know when we have full automation, but but anybody but I'm gonna say it is tough and I can tell you right now even when I trust, my tests of the most.
After. Some. Few thousands, of miles there's. A major surprise even. What I thought I knew to, be relatively, deterministic. Behavior given. Road properties. Exactly. Exactly and again, back to that Wired magazine article. But. So, so. The question is if we have a roadmap on automation could. We even give this. Community, give guidance to people. Developing. Increasingly. Powerful. Versions, of. Automated. Semi, automated driving to. Understand. Given these kinds of needs we have and courage that I'm going to be able to work how can we do that more safely first so as a simple example you. Know I'd like to know if, I'm doing anything in the car on. Autopilot, or not that, I'm coming up to a safety. High-risk, safety point, like. A like pedestrian, crosswalk, and. I've had some near misses even, there in autopilot it just happened to be looking the wrong way and trusting my car and the. Car doesn't know about people or or. Where crosswalks, but that's known by planet you know by maps and, yeah. Even understanding what people have had trouble with hot, spots and finally, what, does all this mean for policies, and legislation to. Come up with the right sort of methods. For. Example my turn out that back. To someone, said they were horrified seeing some of these people in cars on automation. Maybe a car my car lights up and says I'm a town I mean this car is given by an attention attention. Deficit. Constrained. Driver, stay, away. Pedestrians. Can see this crackling on the road near their crosswalk. A big, pink light flashing, get out of the way cuz I'm not sure this person's attending, to the road you, can imagine other kinds of policies mixed with design so I'll stop there just a few ideas and we'll be hearing more about this later yeah thank you very much and I think I have I'm, good all, right so just kind of really quickly and they were in a get to next speaker which, are yeah next speaker which is I'm. Hoping that we can create like a matrix such as this, where we put together what, collaboration. Opportunities there. Are we. Just we just made up some topics for now but we're hoping that as the day unfolds that. Will cement, what certain topics might actually be, to. Basically look at ways that you know we can act we can say like you know we want to do some collaborations, we might have some data collections, we, may also even have resources, that are available there may be students, that may morning work at MSR, there may be some additional data or data collection opportunities. We're, gonna hear later about a survey that that we've been collecting information on that we would love to get feedback there, may be some part-time appointment, and with. That I'm I'm going to, great. Oh. Turn. Around that's a great question so Don Norman just asked what we're gonna do it notes so. So, what's what we're gonna do is we're gonna have each engine put, it up on the white. John. John's, gonna go through Johnson, so what's gonna happen is we're gonna put it up there just like what was your numeral sorry I know you were mark what Mark's gonna do is see what he did goldstar. Mark and Don, Goldstar, so you're just gonna put it up there and I'm glad that we have people that are initiating, already and then John's gonna walk around and look at them so, that he can go ahead and start moving them around so, with that I'm gonna go have Chauncey come up to give her a talk and Eric did you want to just do a quick introduction on shamsky's talk. Well. We put up this. Time originally, we were gonna do this together cuz II. And I work, so closely, on, projects. That kind of booted up our. Deep, dive into the road simulator and. That. Led to a set of studies and then later on with interns and Thompson, continued for, a number of years and has been working on to this day and the basic idea is. What. Do we understand, about, results. From cognitive psychology, when. It comes to memory attention, judgment. The. Ability to do mechanical tasks. In. Divided. Attention, and. Dual, task or triple task scenarios, there's, been a rich literature in this space, we. And so a starting point is going to the literature but then we'll actually want to put people in simulators designs, studies.
That. Specifically, address various, kinds, of what, you might think. To be aspirational. Work, tasks, and cars to. Understand, limitations. And where that might be, availability. That wouldn't be so threatening, I, I. Had some experiences, again driving. And visualizing. Where. I thought is it possible, that that we know enough about the mind that, when you visualize. Thinking. Visually that, sucked sucks away attention from your visual attention on that and the actual live scene and. It, turns out that those, questions framed with your deep dives and I. Thought. The results were surprising and. And maybe. Get into them later in, some of our early studies. But. It really is shocking. And surprising and, interesting to, see what really happens, in the safety safe, rooms of simulators, when. People are trying to do two things even. Would be conversation, or a recall task or even. More intensive, tasks, and. Some. See turn it over to you thank you okay. So she has ten minutes and I wouldn't like to see some of you start picking up some of those post-its, and start, working on them and if you need a seat we have plenty of seats on this table too so, go. Ahead chamsy, okay can people hear me. I. Am Sean Sabo I am. A researcher, in Microsoft Research AI and as. Eric mentioned I, have been working with Eric in this field for a while I started. As a researcher, back in 2008, and Eric. Said that well we have this driving simulator and, would. You like to work on some attention stuff, in the car and that's. Amazing, because I worked on attention management, in the desktop and so it was just a cool way of moving domains. So. Yesterday when I was putting, these slides together I went back to some of the old sites that we had and this. Was because, I felt that we needed to start this discussion this, workshop around the topic of safety when, I was listening to listening, to people give their one, one. Sentence. Desire, from what they wanted to do in their car safety. Was. Predominant. There, were six. People who said that they want to safely nap but, I think that everyone, else said, that I want to be safe in the car and do, X, Y & Z. The. Fact, of the matter is that driving is no longer a single attention, task and hasn't been a single attention, test for quite a while we, even. If we're not doing something in the car there are other people in the car we are talking to them many, of those people in the car are actually not, really attention, aware people if you, haven't encountered any of them my kids are a good example that they have no sense of when is a good moment to interrupt, when a person is driving, but. There's also these. Devices, and these pictures are dated because I pulled this slide from, 2010. But. What I did update this morning are, well. Yesterday, is that there. Hasn't been a lot of changes, in terms of the legislation, and this is the June 2019. Distracted, driving laws, there. Are no cell phone use. At all for novice, drivers, in 39. States and this, number has been the same since 2014. Because, because I change the side I know that no. Cell phone use at all for bus drivers in, 20 states we have 50 so I mean there are 30 other states that are still falling behind, handhelds. Are banned for all drivers in 19, states I think that, change from 14 to 19 in the past five years note. That it's handheld, bang you, can still talk you can still use Bluetooth and text. Messaging is banned for all drivers and that is in 48 states that was 44, that four, more states joined in. No. State, bans, all use. Of cell phone for adult drivers when driving now keep that in mind that means that we are still legislation, is allowing, us to do, things in the car via, a cell phone via bluetooth, and. Car. Companies are picking up on this if, you, think about, the. Inveigle assistance, there now coming into the cars and the goal is not to make us more distracted, I hope but it is with the desire that well these vehicle, assistance, will help you be do, things in the car now, what I haven't heard a lot is about the safety how are they going to make you also be safe in the car, so. There are some unique things about the car environment, right and I'm going to just go to like manual semi-auto, and then autonomous vehicles because I think one of the lot, of the things that we're going to discuss today are going to eventually apply, the most enormous vehicles, but, what we have today as the majority is the manual car in, a manual car which is basically the cars that regular, people like I drive it's driving. Is your primary task it's a continuous, attention task and sometimes. We interleave, these other non driving tests it could be fiddling, with your in vehicle control, it could be talking. To someone over the phone we, have Bluetooth. It could be talking to other passengers, in the car now. When you have semi autonomous vehicles, that is when the, driving sometimes, becomes your secondary task because now it allows you some cycles, to get involved, in other things and I'm thinking mostly about the level 3 and level 4 cars.
Which Are mostly, driving themselves but, they, do require you to take over control at, times and that means that you have to be aware of the road situation you can't be totally totally. Zoned out and, then. There are no no's vehicles which is where the car will basically become, your chauffeur driven car and you'll be going from, point A to point B so it. Is still, not your office it's still a limited attention environment. It's a moving vehicle so even if we think about doing tasks in that environment things, like motion sickness things, like being in a clustered environment all. Of that comes into play so which means that we have to think about designing Testament. In a different way so. I. Want, to quickly go over this because I have Bill aboard on this a bit but the reason why non-driving has do it in a car can be unsafe because, of the improper attentional, distribution, people, often underestimate that. The the cognitive resources, that are required when you're driving and you're adding an additional, task on top of this, driving. Is a continuous, attention test you typically, don't want your nan driving tasks to be also continuous attention because then you're competing and then, at some point you might just deplete, your cognitive resources, we. Also choose improper, moments to engage in secondary tasks, so if you are in a busy road with lots of traffic around you it's not a good idea to start something new which is not driving and also, not understanding, when to shift focus back so. All of these we saw as opportunities. To think about how can you let drivers be more safer in the car, so. And in particular keeping, in mind that non driving tasks are going to happen a conversation. Even via bluetooth that is going to still continue so, how can we make sure that we maintain the driving safety but, also allow people to continue. Or complete some of these non driving tasks remember. Safety is our key, point. Here we, are just looking for opportunities, where we can, allow people to do these tasks without compromising. Safety so, we look at the questions around when what and help I'll. Come back to the assistance, later as. We go through but as. Eric mentioned our premises, here the studies that I'm going to touch upon are going, to be all simulator, studies we're not doing anything on the road in, most cases hands-free, phone conversation, is the secondary test that we were looking at in these studies and there, was only one one. Study where we it was a semi of nama scenario and we looked at typing and video captioning, as that and there, the goal was to see how easily people were able to switch back to driving. Okay. So and I. Probably wouldn't have enough time but I will be happy to bring, this up during the discussion section but there were four studies and the first one which Eric had hinted that we, wanted to know when, is a good time for, someone, to initiate or engage in a phone conversation and, also, what does the conversation, look like the. Second one we wanted to see that okay so once, we have kind of like an idea what when should be even, think about engaging in a secondary task how. Can the car help the person who is engaged in one, more task on top of driving to, be alert and, vigilant. On the road we. Also wanted to see that okay you're having a phone conversation there's, someone on the other end, can. We do something or share some information with the other person, on the other end so that they can also help the driver be safe, and. The final one is that we wanted to look at the role of audio, alerts, in a semi or no scenario where the, audio alert can help. People, get. Out of their secondary, tasks when the car is still in autonomous, mode and then start gaining awareness about the driving, scenario. So. Very. Quickly we wanted. To let, me go back because I do know that we don't have time so how is driving, performance effect. By a simultaneous phone, conversation. And we, were looking at three types of conversations. Or three, types of content, one was when. A person, is asked, to give. Step-by-step. Road direction to someone else when. They're driving the other one was they were asked to recall some information, and the, third one is that they just listened to a news, headline and we. Also wanted, to see. When. Or what, types of roads, was. This being impacted, by so people who are driving on super simple roads there's no other traffic there than, they were driving, on roads where there were a lot of traffic and third is that there is something unexpected, that happened on the road so people have to be able to correct. For that. So. Again. I'm gonna quickly drive to this but. What, we found was interesting so, Eric, and I had this hypothesis, is that clearly, the, direction giving tasks is going to be the most difficult one because that's the most I mean, involved one is that you are asking someone to give me directions from.
Your House to the grocery, and I'm driving and so there should be conflict, in what. I'm doing and what what I'm doing in terms of driving and what the task. Asked. To do turned. Out that people were actually breaking, that task into, smaller chunks so they would say one sentence, they would pause they, would make sure they're driving properly and they would say I'd say the second sentence and they would just basically microcast. This whole whole, conversation, in contrast. With the other test, which was recalling information these were information, like when did you last put the gas in your car or, when did you go to the grocery last people, had much more difficulty in those tests and what we we, realized, that whenever, you needed sustained, attention for a task that was much more difficult rather. Than test where you can actually tank, it and break it down so. The. Next direction was mediation and we wanted to see that okay so if we have if. A system that knows about the driving complexity, during during. Driving can it effectively, intervene and it's basically can the system it. Alert. The user at, a point so that they can focus back on driving if they're, having a phone conversation during, during driving. Okay. So there are three things that we had looked at so we had looked at providing a very general message about okay pay attention and then the other one is that there is a roadblock coming, up you need to start paying attention the. Second one is that we would put the call on hold or we would. Call. Continue, and see how people would just basically taper off by themselves and the. Timing, of this intervention so would we enter people, and right in the middle or should, we wait until they reach some kind of a break point let them finish that sentence. And. What. We found is that interventions. That had more explicit. Messages, that told people exactly what to expect so those were more effective, which makes a lot of sense but we were worried about the part that load that it might put on people when they're trying to process that this. Was traded, off by a slowing down in the conversation, so people would actually slow down in their conversation, when. They would get these alerts. Not. So surprisingly, drivers, preferred the intervention, because it directly impacted, them the colors on the other end they really. Didn't have any awareness or they didn't really care all that much though they were drivers themselves and, it basically means that we need to educate people on, the danger of driving and conversing so. I know that I don't have time but, I'm going to quickly talk.
About This one because the, last result where the caller's don't have really, they didn't seem to think that the intervention, was useful because the call was put on hold this, is where we thought about okay let's share some information with the drivers with. The colors I'm. Sorry and see, how they regulate the conversation, and we found that the callers actually, if you shared growth sounds, with the callers during the conversation, they would actually regulate the conversation, they would speak in shorter sentences, and they would take on the load of conversation, on themselves more I. Am. Gonna skip this part because I know that I'm over time I'm gonna just quickly. Leave. You with this rather dense slide and. Let's. Leave it up. I kept, it there because I thought that it would be good for the discussion if people are able to read it but I think that we'll just stop, here I mean I hope you don't mind we're gonna stop here just because I would like to get some feedback from the audience and you, know we will kind, of yeah so that is the goal yeah, so please go ahead and start writing and putting up those and John will walk around and try give mission I see that Don has his hands up so. Yes. We're, now in the discussion, phase so, I'm gonna I'm, gonna go ahead and do discussions, and then about a minute or two before I have you go put up your slide before now yeah go go Don Oh. Microphone, we need the microphone yeah. Don. Norman what. I'm curious about is the timing so, in, the first, study you talked about in the cellphone and there's, a warning. As. You. Know at one hundred at sixty miles an hour in one second, you go ninety feet so. Did. You actually what I'd like to see is a simulator, study where, some. Unexpected, event happens. How. Much time does it take people, to be able to start their response, and, or. We could do this even in where you have, an expected, response and you alert them and then I still want to know how, many seconds, it takes them before they could actually respond, have, you done that study so. We haven't done that study per se but the way that we set up the alerts so I mean we had told people that you're gonna drive in 50 miles an hour so, that's where we had constrainted, that you can drive hundred miles the. Other one is that we provided, the alerts, within. With enough time so that if they listen, to it and they they, hit. The brake pillow they should be able to stop in time what, we saw was a lot of kind of like slamming, down on the brakes because. One. Is that it's a simulator study so they are really not worried about really, killing people but no one also wants to kill someone in a simulator but, there was a lot of slamming down we, didn't see a lot of kind of like slowly, pacing down but one thing we did see is that people, in general they would drive, below. The speed limit because. They knew that they were engaging, in the secondary, task and so they would almost kind of like correct themselves ahead of time and they. Would drive. Slower so that they would be able to react but. I think that that that's a great point about kind of like figuring on what is the safe stopping distance, and all, of that but I believe there have been other stuff that has looked at that yeah. So. I'm a little curious about the intervention the intervention, that you described. And that was you would put the. Call, on hold correct, have, you noticed that people would perhaps, react. Differently. To being, placed, on hold where all of their cognitive attention would go towards what, just happened and completely. Leave the roadway was there any adverse effect. So. It. Was interesting, because we thought that the call on hold would be I mean one is that it is daring it did give you a message. At the beginning the call is being put on hold and so this is something that both, the, color and the driver could hear so, for the driver it made a bit more sense because they were in that environment they could see what was coming up and so they could easily kind, of like understand. What what, the reasoning was for, the color on the other end they, had no clue and so they would suddenly abruptly, be cut off and then, the driver would start speaking again and then they and so for them it was actually, not a super comfortable, experience. But. Yes, when we didn't put the call on hold we did find that halfway through the conversation the driver would stop and say that wait I need to take care of this and so, I mean, maybe there are better ways, of smoothing, people into that whole situation but. At that point we just wanted to see that okay let's see what happens we just turn up the coat well entirely and one thing before I go to the next person is, you know as you're thinking of the ideas that you know I'm hoping that shams ease talk, generate, some ideas for maybe some other, projects, or other things that you can do in the driving simulator please write them down and post it up okay, there was another question on this yeah.
So. This. Is this related to two Don's coming away it's it's switch cost and, in part it, seems, like in I mean there's always the question about the environment, in which they're set so they actually are set to. Be preparing themselves to switch between tasks. Right and part of the challenge for actual, drivers, is that they don't do that right, so they're not engaged in the phone conversation with the question, in mind okay I'm, monitoring, too, tasks, on an ongoing basis. So there they may be in a different cognitive, state, with. A task that has characterized. That way, the. Most relevant and this is the study that I did skip, over is let's. See so. This, was a study where we're in a semi autonomous scenario, and so we were looking at what happens, when we provide alerts to people way, ahead of time right now I believe tesla, gives you an alert Eric I not in turn is your house okay, so. So, what we wanted to see is that let's add another, alert, say like 30 seconds before and there was literature that shows that it was about 22 seconds and just to be on the safe side that 30 seconds before we will start alerting, you that's there's something coming up on the road and these were for all credited. Things. That are gonna happen this is not for situations, where someone suddenly runs onto the road so we kind of constrained us that way but, we wanted to see that we start from 30 seconds and we tested, with two different types of alerts one was kind of like a bursty. Alert where before. 30 seconds, and then at 15 seconds and then at 10 seconds and then you get your Tesla style alert then. The other one was kind, of like you get this increasing, pulse so it starts it, starts, pinging you or starts like this this, pulse at 30 second then it gradually increases, in, in its. Pulse rate and in its intensity and we found that people were, reacting better, to the to the the, increasing. Frequency. And, they. Were able to actually leave their tests that they were doing at a reasonable, State because otherwise they're just dropping their phone and there's immediately, switching over here. They were able to smooth out of their test and at the same time via themselves, prepared for taking over so, maybe, in a semi of Nova's vehicle it makes sense to not just have one alert, but multiple. Answer. Your question. So. -, if you can have so one thing is just practically, whether or not you can get the, relevant. Information but, the other thing is just the cognitive state one of the challenges, for the drivers is that they're, not in a state where they are saying oh okay, let me attend see when the alert comes then see when I got a response so it's, a complicated, thing to do in a simulator, but, having, them in a state which is typical, of what the drivers state, is. Other than one in which they are experimental, set to, respond differentially. To those that's, that's a hard task but it's so.
That's One of the reasons why similar to generate, generalizing. From similar interest studies to the real world is always difficult and of course we have constraints around how, much we can actually test, out especially, at Microsoft, I mean we we haven't been able to move beyond the simulators, I, think. For me one of the questions is I think we have very different, driving, scenarios, people. Go in for hours on the motorway versus. The city and I think what we have done in research over the last years, is very, often not making very clear distinctions. Noting, some of those automation, tasks are really easy on the on the motorway and you can and I think also, looking at the timings, we have we have very often these running, it ten minutes running it 15 minutes or nothing this is typically, not giving you the problematic, stages. So, usually, it's after if, you look at statistics. It's after hours of driving, and nothing so my question, is if we do, sort of these academic, exercises, we publish, how. Much they, really translate. Into sort of the transformation, of how, we really. Can use the car so my question is really should, we not rather really look at what, are the things which are difficult and, basically. There's, a lot of hand over things like this which, are tricky and and my question is maybe. More useful to think what are the things we can automate, be, sort of the motorway with additional infrastructure, and the things we can not automate, and not try to mix, them at least for the near future. Okay. Great, I'm. Gonna ask Angie to go ahead and get set up for the next one is there any other comments, if. Albert's. Question I'm. Sorry so. I think that's really a very very important point and I mean I as an interruption or a tension researcher, I always worry about kind, of like trying to generalize too much from simulator. Studies but I think that this is a great, discussion point for this group here is thinking about how can we design these studies or how can we move them move, the knowledge forward if we're also it's. Not viable to think that many of these things that we can do out or test out on the road so. What, is kind of like can we determine ahead of time okay these are the things that we are not going to be able to automate and these, are the things where we see some opportunities, and in. My later talk today we'll talk a little bit about that but. Yes I think that one of the things that I do want to make kind of like very clear is that our goal first and foremost is the grabbing safety. Right. I. Think. I think it's a fair point to discuss how we can design these kinds of studies I know for sure that there are some studies that look at mind-wandering that happens when people started to get boring, during when they're driving. And so in those cases if you actually strategically. Place some tasks. That makes, people more alert I mean that's the reason why we listen to music or listen to news when they're driving. For a long time and no one else is in the car who's who can keep me alert but, I think that that's also a fair discussion point because. I really, don't have an answer for that either well. Thank you Shamsi, thank, you very much so please put up your ideas just some couple of things well you know we talked about switch costs, you know the switching of the cost associated. With attending to the road to attending to other things so, this idea of task switching might actually generate some ideas actually. Capturing, the appropriate, cognitive, state as well, as the value, of capturing information in a simulator, versus the on-road and what is the difficulty about looking at you know trying to get some information on safety in that respect and then, also looking at what we can and cannot automate, so maybe that might help generate some ideas and I bring it to my next speaker and Joo Koon so. You have 10 minutes go, ahead. Yes. Your your oh good. Thank. You Linda. Okay. So um. What. I wanted to. Talk. About today is an. Overview. Of where we stand I was really curious if if I could figure out what the state of the art is and. I know we, know from Eric what the state of the art is for him and then just you know what exactly do we have on the roads today and.
What Is the outlook, sorry I wanted to start with this really nice. Figure, from, paper. By Stephens of his colleagues. From. This year's CAI where, they, compare what. Today's, travel. To work and back looks like to, perhaps, what it might look like say 25 years from today if we have decent automation, in place for vehicles so what you see at the top of the figure is. The distribution of time for today right so you have breakfast, at home then you travel to work and you do your work 9:00 to 5:00 travel. Home relaxed dinner and so forth and. What. We, might be able to do if we, have better vehicles. That allow us some automations, and freedom in the vehicles this is the vision right so perhaps I don't know are we gonna have breakfast in the car I'm not convinced, but you know you might be able to eat you. Might be able to do some work in the vehicle, and then, both, going. To work and perhaps traveling back you, might be able to relax this is going to come up later in. One, of our in, our work, as well and I. Wanted to point, out a couple things look. At the travel time for example right the travel time is extended, because, you feel so comfortable in your new vehicle that's, automated that you're willing to travel further from, from, work right so you're willing to move further away notice. The fact that work has extended, right. So we have more time for work this is by the way a tricky, point this came up in, various, conversations throughout, this this faculty summit is that a good thing where is that a bad thing is this a way that we're gonna try to make work, creep into people's lives more, and more and more and take advantage of them or is this you know someone. Said I would like to finish my kite paper that's our perspective right give me some more time so I can write back hi paper and so that's a valid that's a concern however. Overall. Do. You notice that, the vision is that the total time including. Travel and work is shorter right. So the this person in this scenario. Leaves. At 8:30 comes, back at 5:30 but, leaves at 8:30 in his back, having, relaxed, by 5:00 p.m. right so even though they worked more they actually could the, total time spent is less so, we, can definitely discuss how exactly this works out and what exactly those distributions. Are but there is some interesting hopes, and interesting.i potentials. Oh this. Is the state of the art though right it's at least from my perspective so, this is me sitting, on a bus. With. Or it and writing a workshop. Proposal, and so, really what what my what. I perceived. To be the state of the art is that really I'm gonna take my laptop then it's gonna be on my lap and and, I'm going to work oh I, did find one interesting, what.
I Think is the state of the art so this is an uber commercial, and maybe. I can play it, yeah. There goes so. This. Is a commercial, for a, upper. You, know higher end service sorry and this, is what, they're saying look you can sit in the back of the car is going to be very comfortable and what exactly are you going to be doing calling. There's. Going to be an iPad what, do you call it a tablet right in a computer, and these are the things that you're going to be working on it's. Comfortable, and nevertheless. Status, sort of the state of the act of what we can expect. To be doing I, want. To say that John. Lee pointed out to me done and one of these frames. There, were two people sitting next to each other it. Was supposed, to be showing. Two different cars but that's also an interesting question of I. Think, privacy came up in one of the comments when, we went around the. Room you know if you do travel with other people even if it's comfortable and, you're working how does how does privacy, play into this and, here's. Another, promotional. Picture, for, uber so Martha Stewart promotes uber. And this is you, know the vision from Martha Stewart's perspective, and I really like this, photo for a couple of different reasons. One. Is that it shows, her doing things that you're not doing right now in the car and actually cannot do in the car so you know Martha Stewart is preparing, some, sort of a very, fancy looking. Some. Sweets right there for your fee she looking and she's just this is what she does anyway she, makes us feel kind of bad about how we can't do this so easily but. True the truth is you were not gonna do this in the car it's gonna be bouncy and bumpy and you, know it's, not really but again, think of the let's. Think about what we could be doing in the future that's what I think is it interesting about this there, is also a printer, and that actually. Is starting to be something you know you could sort of see something like that so what. I'm what i like about this photo is that it encourages me to think about things that it we're not currently doing not just like that commercial, before where you're you know here you are on the laptop but what other interesting, things might there be that, could be possible to do and I think that's we're thinking about and then worth pointed out to me that Martha's looking at the window it's probably, a feature, that, was supposed to be artistic but or, its point is that a lot, of us motion, sing right, and the only way we can handle working in the car is to frequently, look out the window so again as we think about automated, vehicles this is something that we should consider I.
Really. Like this paper to Boston, plinging who, was Albert's, PhD, student and a. Couple. Of colleagues. Wanted. To see what people are interested in doing in the automated vehicles but the part that I'm showing here is what, people are currently say. They are doing in automated or in manually. Driven vehicles, so this is a three hundred participants who, completed a survey and when. They drive they, call and text. In. A pretty large numbers. From. The paper I can't quite tell I'll have to ask question what it means the text in this context, do they mean I'm stopped, at the red light and I'm texting or do they mean something else do they mean voice, interfaces. But nevertheless these are the types of tasks, that they're undertaking right now and. Passengers. Do all kinds of other things as well. Including. Email including, whatever, they call office tasks, right so. I think it's really interesting because this does capture something, about what people are doing right now and this is going to be something that we'll talk more about from, our own. View. Of this on, through. This this. Nsf-funded. Project. The, next. Mobile office so the five of us and you can you can talked about talk to us about this project throughout. The day, but. The. Vision. That we have is, something. Like this I was just, talking to Yvonne about the our. Need for the, next version of hololens, but you know what. If you, were that driver, and your. Car can take over for, some, amount, of time maybe it's five minutes ten minutes whatever it is you, know how could we support. You in, doing. Work, beyond. Having. A laptop on your on your lap and we. Think that as, far as the human machine interaction is concerned, augmented, reality speech. Interaction, and tangible interfaces, might. Be the way to go so I'm showing augmented, reality here, this is one of Word students, really, nice work. Here showing, this and then, some, sort of a speech interaction, right and some tangible interface some something, physical, but probably small we don't necessarily, think that a laptop, is something that's all that great for this for, multiple reasons one, is that you have to look down but the, other ISM really. Curious what everybody thinks about how. Long you know these handovers, are gonna be interesting cars will eventually, say look it's your turn and it's actually probably going to be often right, and, the. Handover will have to happen and that's.
Probably Going to be short one, major, manufacturer. Last. Year was promoting, a vehicle that isn't. Quite doing this right now but they said fully, taking, ends off to the wheel we're, gonna take over driving and the, time they give you to come back is ten seconds and I don't think you can stow your laptop in 10 seconds safely right so so. I think this is interesting to think about, and. I think it's really important we talked about safety we it's, critical, this hand over all happen and then, whatever is there to support your work at the very least has to get out of the way and ideally. Actually, can support you in driving, so if you do have augmented, reality that, can be something that that, can help with when. Navigation, for example or. Helping. You with with finding, you know street. Signs and so forth I. Wanted. To quickly, put up a couple of things that I think would be interesting. These, transitions, this is work with shawnzy, and actually, led by Christian Johnson who. Was a Duncan. Squeezie student. And his. Colleague Stella. Stella. Donker and. The. Idea. Here is the following the. Transitions. Between the, non-driving task and the driving tasks are not just simply we're. Doing one and then we're doing the other and we're coming back but rather there are stages so. What the stages are we can argue about that the, details, of that right but, you will be going through some. Stages in terms of handing, back and forth and importantly. There's gonna be interleaving, meaning, that while. You're doing the or non-driving task and you're starting to do your driving task you might come back to the non driving task and go back and forth and this is a safety concern right, this is something that has to be designed properly so. That for example the actual physical, transfer of control, doesn't. Happen you, know after. Suspending. The non driving task right so you might not have suspended, the nine driving, task you might still be engaged in that non driving task even after you're driving this diagram and, then. I have. Two more slides on that so it's really quick now, I, knew. It was not gonna run over in time because you know she won't want me.
One. Paper that I thought was really very interesting is this, paper. By Magnuson and Pope, and they. Talked about it's a review of biomechanics, in and how. In working postures and one really interesting thing about this to me was. That they found drivers. With lower back pain right. And especially if the drive is long and there's lots of vibration, as well as musculoskeletal, problems, neck, shoulder arm especially, if your arms are say not supported, and, what we're talking about thing back to that first slide extended. Drives because, you're so happy that your car can drive itself but, there is this concern that since you're driving longer now, you're introducing, issues that could hurt, you physically, right, so, I think that, this. Is one of those things that we really need to be. Concerned about as far as we as, our, design so, to. Wrap up. Today. We have simple tasks right I think the state of the art we can we can argue is that so, the vast majority of us the best we can hope for is that we'll be on our laptop or some, sort of a pad and. We have high hopes for tomorrow but, as we do that, as we look at that tomorrow. We should really look at motion sickness we. Should look at privacy and but how do you become partners. In privacy as John I think really nice to put it and then. Safety I think, it's really interesting to look at this interleaving, between tasks, and then, economics, we must, worry about that so great thank, you Andrew so, we're gonna go ahead and go into the discussion portion I'm, gonna ask our speaker, to repeat the question, once. You hear it and I, will now open it up for comments, a discussion. Jim, Dinkelman for Microsoft Research are you aware of research. That that, that, approaches. A problem. By. Sense, and the cognitive, load. That, each of our senses, require, so. For instance listening, is very easy and, very low cognitive when we've been doing that for decades in the car we've been multitasking. Listening we've, started to talk and that's a little bit more of a cognitive load once, I get into visual and touch actions. In the, car are, you aware and aware of any research that there, is that is that relevant thinking, along that dimension I think, you're you're hitting it right on the head I think, a lot of us have been, thinking. About that and I think Shawn's in fact just mentioned that it. In. Her, in her presentation, as well so as far as cognitive. Load and the mall and and Eric actually mentioned this as well right so in terms of multiple. Resource theory I think a lot of us do, you think that this is a this. Is exactly how you should look at it and and hope to. I. Have, to do a correction this is not exactly, a you should look at it there's been a huge amount of research on this and actually Alan Baddeley published, a paper like a, century. Ago in, which he had people driving, and they, listened to the radio station. Playing. And say, the news broadcast or they're listening to a radio station with, people playing football, look. This is an England so, it's soccer and. Listening. To the soccer game was. Really. Wrecked the driving performance because. It's, a visual task right they're listening to a visual thing so they're listening auditorily, so, the point is there's a lot of literature and it's a bit more complex, and, one sensory, modality, versus another I. Would. Like to add something to Don's point, which was interesting, because when we had designed our initial study about giving. The road directions, the first one that I had to describe. Where. People were trying to either recall, information, or give Road directions, and Eric and I hypothesized, that the road Direction path would be much more it, would conflict much more for driving because that is also visual, I mean at least when I give Road directions, to someone I visually. Think, about the roads and, it. Is complex, because I believe, just the fact that people are breaking it down made it somehow cognitively. More manageable, where, when they were trying to recall information maybe, they're also trying to visualize that when did I last take gas, in my car well, I have to visualize that when did I last go to Costco or wherever or so. I think that it is not as simple it's just having the speech or listening, as soon, as you start visualizing. It and as, you mentioned soccer is a visual thing to think about I mean Eric has examples about, visualizing. An airport. What. What is the thing called the. Display, board. Sorry. Of. The actual schedule, screen until, someone shouted stop as, I said visiting, Irvine at the giving it talk down at Irvine UC Irvine and, all the grad students shouted, Spoleto.
At Dinner and. It's. Like the this, visual. UI, had evaporated, and, I see through the windshield again, pretty. Surprised, all. The professors, in front of us stopped their car quickly and with it taking them out so, so, the cut chomp she's coming it, even. Beyond football versus, you have to be cautious done, because, the the, football versus, the news reports, and the music is, I think unfortunately. Maybe. There's some marginals, there some. Over but it's too high. Level of me more details the. Fact that we found and, it's just one study we did that. Memory tasks, recalling when you did an oil change he. Wouldn't lose their try the road and being, trouble with. A memory task we don't know what how much what, combination, of visual. Reminder. Meant remembering, what it was getting in the way of their being able to navigate and be aware and sense with their with that, situation. But. Um it. Suggests, the theory we came up with it is in the paper. Getting. Into this into. The detailed, structure, of how cognition, works when it comes to giving directions versus, remembering, oil changes, Wow. Could you really put. Stuff on the on the stack and push, and pop the stack about directions, with memory when, you need to attend and still. Remain, alert and you can't do that with this gigantic. Call. We were waiting for the answer to come back about the oil change so I really. Want to suggest. To this community, of folks interested in this that. Would be great to get into. The right level of detail and, then. To abstract from that versus start with high-level. So. So I really, okay. Don't go ahead I just, wanted to pick up you're talking about basically. Current day activities, and, and. You talked about like bastian survey and so what. Do I can't help notice I owe you, CL the offices in central London but, we live like, 50 miles away towards, Cambridge and. So. Commuting, on a train going, into the office and just, you can't have a noticed look when you get onto a busy commuter train just the. Patterns, of people's natural behavior, right and I think one of these things is to start understanding the way people work and want to work and, so. Picking up on what we saw a Gloria Mark's work on the opening. Down Wednesday, the. Types of work that people do when it's very clear in the morning people, are getting their head down and doing focused work there's, a lot of activity, you can hear it with the fingers tapping people. Preparing slides writing, emails in, the evening, coming home there's. No work there, are no laptops out if. There is a laptop out if somebody watching a movie or play a game on their phone there's, very little activities, it's kind of funny just to understand, I think one of things without understanding more. About the types of activities people, doing when how that fits into their daily schedule, is, I'm, getting kicked off but I totally. Agree and one. Thing that we're gonna hear soon about is what is it that people would like to do and I think that goes along with and then.
I Think it's really important what what. Everybody's. Saying right including Derrick is what is it that tech can provide and, I think that there's this push and pull of you. Know let's not design things that people don't want but also let's
2019-08-25