Thank you for joining us this afternoon for a colloquium on assistive technologies which was organized by the Robotics Graduate Student Council and Michigan Robotics. My name is Andrea Sipos, I'm a rising second year PhD student acting as the colloquium chair for RGSC this year. It will be my pleasure to introduce you to our speakers this afternoon. We ask that you hold any questions until the end of the presentations we'll have a Q and A panel then, with all of our presenters up on stage. We further ask for your patience in advance because
this is the first event that we're holding in our new building and this event is being recorded and should be available for our remote colleagues at a later date. To lead us off i am excited to welcome two special guests who travel to Ann Arbor from Minneapolis, Minnesota, Shane and Hannah Burcaw, married in 2020 are disability activists from the viral youtube channel called squirmy and grubs i first learned of the couple through a video they posted last year entitled this robot would change Shane's life because my youtube algorithm knows i'm a sucker for robot videos i watched a lot more of their videos after that first one because they bring a unique humor and charisma to the important topics of disability and accessibility in addition to the youtube content that Shane and Hannah make. Shane and his cousin Sarah co-founded a non-profit organization called laughing at my nightmare in 2012. laughing at my nightmare has a two-part mission
to educate children on kindness positivity and empathy and to provide free adaptive and medical equipment like adaptive bikes custom power chairs and adaptive strollers to people living with muscular dystrophy Shane and Hannah are here today to talk about their experiences with assistive technology and to help bridge the gap between researchers developing these technologies and their end users without further ado please join me in welcoming Shane and Hannah Burcaw. how's everyone doing just gonna get situated here i think it would have been funny if we uh run around behind the stage to uh ostensibly come up on the stage but we just left through the doors back there and you're all like, what happened? Already demoing some assistive technology here. Hi I'm Shane and this is my wife and probably the most important piece of assistive technology in my life Hannah hello thank you for that intro we're really excited to be here today um assistive technology plays a huge role in my life and our lives um going to talk about that it's going to be real tangible uh i want to talk about kind of why it's important to me um but then really just dive into what devices i use some terrible stories about them uh because it's a lot more fun to talk about failure than success um yeah we'll just uh take it from there and then we'll be in the q a at the end so save all your questions for them so i'm going to bring my notes up here on my first piece of assistive technology which is my iphone um but we'll talk about that in a bit uh just like i think we should go back and talk a little bit about my diagnosis so you have a groundwork for why i use wheelchair and assistive technology i was born with a neuromuscular condition called spinal muscular atrophy which basically makes all the muscles in my body really crappy and so i've never walked i got my first wheelchair when i was two years old um and i was told or i am told by my parents that my favorite thing to do when i was learning for the first time how to drive my chair was to crash into strangers walls obstacles at top speed i still enjoy that but i'm a little bit more reserved nowadays but i i the overall point that i want to make today is that assistive technology while it obviously allows me to have function that i wouldn't otherwise have it allows me to do everyday activities it more than function it provides a a satisfying and fulfilling life experience so like well yeah my wheelchair gets me from point a to point b more so it allows me to thrive and because of all of the assistive technology i use i've been able to become a an author and a prominent youtuber which i never thought i would say i'm a a husband and a traveler of the world all kinds of things that are more than just going from point a to point b a really good example of that is one of the stunts that uh i use my wheelchair for as a child do you want to talk about the basketball yes sure i will of course yeah um so i was not there for this story shane was like 10 years old um your worst yeah shane's worst age 10 years old and it shows what his childhood was like so when he was 10 his younger brother andrew was six i guess six or seven um and shane was a terrible influence on him andrew really wanted to dunk the basketball they had a basketball hoop in their driveway andrew wanted to dunk it and couldn't obviously he was six years old so shane realized that he could use his wheelchair to help andrew so he told him to go into their garage and get a rope andrew did that poor little andrew and uh brought the rope back outside and shane was like okay now tie the rope around your waist and poor little andrew was like okay ties the rope around his waist can you please stop referring to him i was poor little andrew i think it adds something poor little andrew did that and then shane was like okay throw the other end of the rope through the basketball hoop and andrew did that and then tied it to shane's wheelchair shane was like okay hold the basketball and i'm gonna pull you up so shane begins to reverse and andrew starts rising into the air and then begins screaming in pain because the rope is like cutting into his stomach he wasn't wearing a shirt which in hindsight that was that was a mistake i don't think the shirt would not have helped so instead of stopping and being like well that was really dangerous i'm sorry andrew shin was like okay get some pillows so andrew goes inside gets pillows puts them between the rope and his stomach and shane tries again it worked and as i began to reverse my chair my brother began to levitate into the air with this ingenious pulley system that i had invented um he's lifted into the air one foot two feet three feet four feet he's got the ball in his hand he's ready to dunk and when he was near the top maybe a foot away from being able to dunk and i don't know why this happened i think it has to do with science all of you might understand it better than me uh he got a lot heavier and my chair was no longer able to lift him that final foot that he had to go so i am in full reverse my wheels are spinning there is smoke coming out of the back of my chair and nothing about that made me think you know what maybe i shouldn't be doing this andrew has the nerve to yell at me to try harder like all right i'll put my back into it i'll reverse him smoke everywhere at that moment our dad comes outside doing this horrifying combination of laughing hysterically and screaming at us uh he didn't like what we were doing um so i lowered andrew back down to the ground in utter failure well there's more so you know shane's dad you know tells him that that was really bad don't do that again you're done playing for the day come inside i think he also complimented us i think he was like guys this is amazing but i can't not punish you for this he doesn't remember that part of the story i don't think your dad would say that he did that but anyway he's like go inside shane starts to go inside and finds out that his chair can only turn left it no longer drives in a straight line that's bad and a little further inspecting about our dad found that all that smoke was the deer motors in my chair shredding themselves um i had to replacement cost of eight thousand dollars so i was in trouble and this is obviously a kind of silly example but i think it shows how yeah throughout my whole life especially when i was a kid um i've been using assistive technology specifically my wheelchair to have fun to get in trouble to hurt my brother all kinds of things and it is it does more than get me from point a to point b it allows me to live a full life all right that was the basketball story uh next let's talk about my one step ramp which is a very rudimentary piece of assistive technology but at least in terms of travel it has really changed my life can you talk a little bit about it yeah yeah and i think you'd be surprised how long it took us to find a portable lightweight one-step ramp that could like fold up and we have one that it like hangs in a bag on the back of shane's chair and it folds in half but it also is strong enough to hold his chair and we've brought it on all of our trips so that when we come across a curb that doesn't have a curb cut or a store that we really want to go into with one step we can still access that store so i think that is probably the most used piece of you know technology that we have i think people don't realize how much of the world is still inaccessible to wheelchairs like a a five inch step exists in so many places um throughout my whole life i just haven't gone into those places and then like two years ago we were going to uh was it poland or london um and we were thinking about kind of how often we might encounter that obstacle along the way and we didn't want that to ruin kind of the adventure and the the joy of going into where we wanted so hannah was like why don't we bring a ramp and i was like oh no like they don't make those like it won't fit in her bag or on my chair and hannah was like yeah i'll find one and she did like on amazon in about five minutes of looking um so that has been really helpful for me as well um kind of a cute acute piece of assistive technology is the phone holder that i i used in bed when hannah and i were long distance you want to talk a little bit about how we met and then get into that yeah so shannon and i met five and a half years ago i was a freshman in college in minnesota and shane had just graduated from college he was living in pennsylvania and we were long distance for two years so on and shane was living with his parents and uh on many nights we would want to be facetiming really really late into the night which would be difficult if shane was going to ask like his dad to bring his laptop you know into his room set it up and then come back at like three in the morning and take it down you know so uh shane got a phone holder like a long what would you do you probably know what i'm talking about so yeah amount but it has a long arm on it bendable yeah so shane got one of those and used that every single night for the two years that we were long distance and it let us you know communicate in the way that we wanted to and it didn't you know it would have been difficult if shane hadn't had that and we weren't able to talk after his dad went to bed yeah and that's another one that it sounds like a no-brainer but you're like yeah obviously why wouldn't you put your phone on your mouth um but for a long part of my life i hadn't considered even the possibility that i might be able to use my phone in bed so like in high school when cell phones came about and all my friends were texting hours into the night on our flip phones i often had to you know step out of the conversation around 9 or 10 when i went to bed and that sucked i couldn't be a part of the social jumping zones of my friend group because i thought there's no way i can lay in bed and hold my phone like my friends then i met hannah and i was like there must be a way that i can lay in bed and hold my phone um so simple solutions allow for very far-reaching possibilities uh another huge probably the item that i use the most after my wheelchair is an app for my iphone that turns my phone into a bluetooth keyboard for my laptop i'm a writer and i began a blog in 2011 uh that kind of kicked off my career as a writer and blew up in a way that i never imagined and right around that time i lost the ability to physically type on my laptop terrible timing as my vlog is taking off and i looked around i used like voice dictation and had a horrible time with that it didn't understand me very well even after hours of training it um so i would spend more time editing and i did writing and then i found this app and it allows me to type with my lightning fast thumbs right on my phone and the words appear on my laptop so i've written reboots that way thousands of emails every week uh blog posts articles um it's allowed me to have a career that i love and that you know makes me feel fulfilled all right this one is a bit of a letdown is that photo still up there oh no can you talk about my jacob yeah yeah this is a huge letdown so we we recently flew on an airplane we were on a flight and for that flight we took off all of the parts of shane's chair that could break because they they often break on flights um and one of those things that we took off was jane's robotic arm his jayco arm it's this like huge arm that sits here and it you know does a ton of stuff and we forgot to put it back on and we realized that as we arrived here so it would have been great if shane had his robotic arm and was like holding the microphone we didn't bring it but normally he has a really cool robotic arm that sits here you have to trust us it would have been so relevant like there has never been a more relevant time to show off my robot arm i'm so sorry um but can you talk a little bit about like the journey to getting it and that yeah when did you get it do you remember was it like six months ago yeah so the robot arm is extremely expensive and fifty thousand dollars and uh shane knew about it because some people that we know had gotten it through insurance and like it was just sort of like on our radar but we assumed that shane's insurance would never cover it um you know they've not they've denied his wheelchair before they've denied like other really basic things so we're like there's no way it's not worth it um but more and more people we saw in our you know like network of people yeah getting it and so we were like maybe we should actually try to to get this and so um shane's occupational therapist like wrote this you know big thing for insurance and submitted it i think there was a letter from you in that first one and and a month later we got that it was denied and we were like okay um but the company selling the arm is like we've never gotten it through without being denied once so like keep going like appeal it it might actually go through eventually and this piece of technology seemed like it would be revolutionary for me i physically need assistance from my human assistive device here with like almost every aspect of daily life like i need help getting a drink from the fridge itching my forehead uh putting my shoes on all kinds of things everything that you do in a day other than like the work i do on my phone and my computer this robot arm connects to my wheelchair and allows me to use my joystick of my chair to manipulate it in a variety of ways so i had the opportunity to open doors and pick up my phone if i ever drop it get food for myself uh all kinds of things and it was denied and the reason was that it was not medically necessary which is kind of their catch-all phrase that they use when they don't want to pay for anything um somebody appealed it or you're successful uh hannah wrote a letter explaining a variety of ways that without it i could die essentially like if there was a fire and i couldn't open my own door or if my head fell and i was choking and couldn't lift my own head up all the real things probably won't happen but insurance doesn't need to know that uh so it was improved and i doubt that jake alarm it has been revolutionary for me um to be able to go and get myself a beer without having to ask hannah i don't know if hannah likes that aspect of it um but uh it's been really useful and it did end up saving my life not even today um doing tell that or vision telling that yeah i didn't realize you were going to tell this story so i guess we're telling it um i don't know how relevant this is but i guess since it saved shane's life he wants to talk about it anyway um in in december of this past year uh shane was coming out of my parents house we were visiting we were there for dinner and their house has about five steps to get inside and they have a big wooden ramp that uh they built for shane to access the house so he's coming down the ramp and it had been snowing or like it was snowing it was actively snowing uh and every single time we go down this ramp i always say to shane do you want me to hold your chair like you know walk behind you holding it and every single time shane says no i'm good um so we the same thing happened i was like you want me to hold your chair he was like no i'm good i let go and within like a quarter of a second he had begun to like spin on the ramp and was hurtling toward the edge yeah it was one of those moments where like you hear the phrase like your life passed before your eyes i had like flashes of like oh i'm about to die like this is about to be how i die so i'm fishtailing on the icy ramp and slamming into the edge which does not hold my cheerios over the edge of the ramp when i'm i don't know four feet in the air and tips off of the left side so that i fall in between the edge of the house and the ramp my bat right wheel is the only wheel that stays on the ramp and my jaco robotic arm on the left side catches on the house keeping me from falling all the way forward so i'm dangling literally down going straight down with the arm against the house and my wheel on the ramp had i fallen further i would have been crushed like it's funny now but i would have died um i'm fragile as hell and uh would not have fared well in that that incident so i mean what was that like from your perspective other than awful it's a terrible question it was horrible and it all happened so quickly it really felt like i was watching a movie like i was just standing there and then within like two seconds it was like two seconds from when i let go and then shane was like upside down so it was just like i could not believe it was real and then i like ran down the ramp i just didn't know what to do like that was you know the main problem because it was like shane's chair was too heavy i was like pulling on it but then he was like no don't because what if it like what if you dislodge it and then it falls all the way it was horrible and then i decided to like take shane's straps off to try to like get him out of the chair and then he was like no i'm gonna fall it was just it was horrible it was a mess yeah it was like a nightmare scene basically um so hannah's mom heard the promotion and ran out and somehow i mean my chair weighs 500 pounds somehow she hannah and her mom ripped it back up oh no wait you got me out of it first yeah she held she held it in place and i took you out and then once you were out we were able to like just let it down so i lived well louder i lived um surprised and uh you know i i never kind of expected that the jaytown would end up saving my life but like the next day we realized had that not been there i would have gone all the way and i'd be i wouldn't be here right now so thank you jay taylor uh yeah that we didn't burn uh maybe they have one here we'll hold her up for it after um the last uh piece that has been pretty big in my life is again a simple solution but it is the uh powered mount that my joystick is on right now again i'm awful we forgot to charge it in the hotel room last night so i have a switch here that is supposed to activate my joystick to swing down in front of me where i drive with it it's not working right now sorry um never hire a sedan or terrible but i for most of my life had a manual release on my joystick that then had to manually be moved out of the way for transfers or if i'm pulling up to a table to work or to use my phone on my lap where i have it now and that was limiting because i was not able to move my joystick on my own so if i had it flung out like this and i've got my computer and then i want to move to escape the house if there's a fire i can't do that on my own i have to call for help um i know they met this company that was like oh no we made power mounts like we didn't install that and my life's changed forever and so now with the touch of a light button over here i can swing it down move it out of the way um and it's been really helpful do you remember the first night i got it yes uh yeah so before shane had it obviously if he was sitting at his computer in you know our living room and i was in the bedroom reading a book i knew that that's where shane was and you know that's where he would stay and that's how it had always been so the that sounds like weirdly like i don't know so uh that first night i had like kind of forgotten that he had gotten this you know adaptive technology and i was just reading a book in the bedroom like nice and calm and all of a sudden i did like i guess i didn't hear shane approaching like i think he was doing it extra quietly just for this to happen he was in the room and i see him out of the corner of my eye and he screams he literally snuck up on me to scare me i screamed it was horrifying and then for the rest of that week he would just like i would just be sitting in a room and he would just like come in do a lap and then leave he was obsessed with using this joystick to just like roam around the house at his you know at his whim it was really annoying at first but it has been really useful these are all the hard-hitting important uses for all of my adaptive technology we tell insurance that like you're going to escape a fire but really shane just does laps around the house endlessly i need this device to bother my life uh i don't know if that would be approved but um i hope you see like there are many more kind of devices and things i use in my life um but i hope that you kind of see that they allow me to live like really live a fulfilling and full life they don't just like do the one thing that they're intended to do um and so that's been a really cool part of my life i was very lucky to be born in a time when all of this technology is available and as we look forward hannah and i are you know in the early stages of you know thinking about having a family and we know that there will be more and new ways that the devices i have now will be helpful like holding our kids with my jayco arm in the air when they're bad um we've seen devices that like allow you to have a stroller attached to a wheelchair things like that are excited to know that they are available and will continue to benefit me in all the stages of my life so um not to be like overly you know not to watch politics here but the work that all of you are doing is so important because you know while it might feel that you're you know just in a lab like doing research or um you know working on a device remember that there are people out there who will be extremely benefited by it whose lives will be more fulfilled you're improving access and function and you know missing life's good so thank you for everything that you do thank you for having us and uh yeah cool that was an awesome presentation and i thank them for being just so candid with us because i think it's really important for us to hear as researchers the perspectives that we may not get every day when we're just looking at the numbers and the wires we need the human bit and i think that's something that shane and hannah have provided for us and we really thank them for it uh so now we're going to shift into some short form research presentations from some of our colleagues in michigan robotics uh next up to the stage is going to be michael gonzalez who is a co-organizer of this event he's a fifth year phd in the rehabilitation biomechanics lab most of us know him for his love of board games at rgsc game nights but today he's going to be presenting his work on the impact of sensation on the control of robotic prosthetics thank you andrea and thank you shane and hannah again for for that talk i i really love getting those perspectives because that really connects us to to the work that we do um and this clicker is there we go all right um so the work that i do and and that i have been uh spending the last uh four years studying is prosthetics so prosthetics are neat because they are robots that someone is actively wearing so they're not only serving a function but they're also serving a form they are designed to replicate the function and the the aesthetic of a missing limb but there are many people who choose not to use prostheses and this is something that really interests me that there are individuals out there that have things they want to do and we have some assistive technologies that are designed for their use but they end up not using them and it's it's a curious thing to me of understanding why that might be so one person that really helped me uh sort of understand this and was part of the reason why i was interested in prosthetics in the first place was hugh hair hugh hair does uh really amazing work at mit but he started out as an individual with no engineering interest whatsoever he just loved rock climbing uh and and mountaineering and uh one day he got lost in a snowstorm and he lost both of his legs below the knee to frostbite and when he sort of recovered a post-amputation he was really upset with sort of the state and quality of lower limb prosthetics and he felt like he couldn't walk the way that he wanted to and he couldn't get out and do rock climbing in the way that he enjoyed previously and he had this really amazing quote that that i enjoy and that sort of centers how i think about prosthetics which is i didn't view my body as broken i realize that a human being could never be broken technology is broken technology is inadequate and i like this because it frames it as what we're doing is not trying to replace a missing limb or replace all of the you know functionality that we might think is quote unquote missing what we're trying to do is find the needs of individuals and make sure that they can do those things through the technology that we're working on so what does it mean to be adequate for a piece of assistive technology well of course with many things the answer is it depends for lower limb prosthetics maybe adequate means it gives the individual stability they can very confidently walk without worrying about any sort of of wobbles or falls maybe it means symmetry so making sure that the forces on both sides are equivalent so they don't get overuse injuries or maybe it means energy cost it uh the the prosthetic is replacing some of the lost musculature that the individual doesn't have anymore or it just makes it so that it's not difficult for them to walk at all for upper limb where i spend most of my time and energy dexterity is a really big piece of what is adequate and a piece of assistive technology so you not only want a hand that can make different grasps depending on the function but something that has a level of specificity in terms of the force that you're using you want to be able to pick up an egg without breaking it and also without crushing it and something that modern technology is allowing us to do more of is create prostheses that give users some amount of sensation but above all else adequate means people actually like using the technology that it is not frustrating that it doesn't break on them or that it does what it's supposed to do 99 of the time that you ask it to do that thing thankfully we have an amazing prosthetist that works for with us for the comfort side of things because for prosthesis this is also something where an individual has to wear it they have to put weight on it in the case of a lower limb prosthesis so comfort is a big deal thankfully i don't have to deal with that part what i am really curious about is sensation and understanding how sensation plays a role in both function and also the experience of using a prosthesis there are endless questions when it comes to exploring sensation what is it what does it mean to feel something what is our brain doing when we touch an object and get information from it how do we use that sensation for function so you know grabbing an object i want to know when i come into contact with it i want to know how hard or soft it is are there different types of sensation that are more useful than others so if i need to design a device do i want it to beep at you when you have grabbed an object do i want it to vibrate do i want it to be able to communicate how hard or soft that object is and something that i've been looking into more recently is how do we generate artificial sensation in that missing limb so a lot of people with an acquired amputation they still have nerves that used to run to where their hand was and we can actually leverage the fact that those connections still exist the brain still has some space in it that's organized to process that information so we might be able to actually make them feel something in their prosthesis and then pardon the pun but how does sensation make people feel so there's an emotional component to it as well you want someone to enjoy using that device because what do we use our our hands for we we touch and interact with objects with loved ones so there is that emotional component as well so as an engineer obviously quantitative measures are a big important piece for us so one of the questions that i started with is how do we measure how do we quantify sensation usually in our field it's described qualitatively so hey i do or don't feel something when you put something in my prosthesis or you know in anecdotal uh evidence a pro a prosthetist might say oh this type of prosthetic gives me more sensation than that one the two main types of prostheses for the upper limb are body powered so these are typically a harness you've got a cable running down to a hook and when i pull my my scapula or my shoulder back and forth it can open or close that hook and we also have myoelectric devices where an individual is typically flexing some of the muscles or extending some of the muscles that they still have in their limb and this is translating to an opening or closing of the device but it's hard to tell how much one might make you feel more than another so you might have force that is transmitted through that cable it makes sense that they would be able to feel some of the force on their device but how do we put numbers to that so i had the privilege of working uh on a project where we're studying how to actually do this we had a methodology where individuals were interacting with these paddles the paddles could be programmed to represent different objects of different sizes and stiffnesses and then we could actually give them specific types of information so okay when you grab this virtual object you're going to see a color change or you're going to feel some vibration or you're going to feel some resistive force or you're not going to feel anything at all and we can sort of study how they respond how they're able to match these targets and while our results weren't necessarily surprising we actually got to confirm all of these anecdotal pieces of evidence so here we've got you know anatomical so you know a natural hand a body powered and a myoelectric prosthesis and across every category yes in fact the human hand is the most accurate good glad we confirmed that uh we can also confirm that myoelectric prostheses were the least accurate across all of these different perspectives across all these different types of feedback but you know we can also see interesting things that we maybe didn't expect so for visual feedback didn't matter what type of hand the person was using they could all see when the color switched from black to white when they grabbed the object so these sort of disparities that we still see in these results mean all right there's other things going on there's you know maybe a control issue that is there and that's for a talk in another time anyways i so after i got done going okay engineering brain i want to quantify these things i got curious about this uh this question of how do we actually give people sensation what does that look like how do we create artificial sensation in these people that are missing their their hand so we started this project working with uh one of our participants who is implanted with uh what is called an rpni so a little bit of background so these nerves that once innervated that hand these people still have those sometimes they scar over sometimes they don't and again we still have space in our brain that is looking for signals that used to run to the hand so here at the university of michigan in our lab and with the cindy chestex lab we can use regenerative peripheral nerve interfaces which are these biological constructs that are these muscle grafts wrapped around a nerve that used to run to the hand we can collect signals from these and so we can understand what the person wanted to do with their prosthesis so if they're thinking all right i'm trying to flex my thumb this can maybe pick that up and then we can pass that along to a prosthesis and tell the prosthesis flex your thumb but we can also send signals to these nerves and the brain can pick up okay this used to be something that ran to the pad of the thumb it's getting some signal now that must mean that something is touching my thumb even though that thumb does not exist anymore so this is just a quick video of some work with one of our participants so here we're stimulating um in her median nerve which stimulates this half of the hand and her ulnar nerve which stimulates that half of the hand so we can get her to feel something here we can switch the channel that we're stimulating to in these wires that are embedded in her arm so now we can get stimulation over here and then we were curious what would happen if we stimulated both areas like the same we just did some science so it's it's hard to see on this video but essentially you're feeling two different locations right you're feeling a middle ground so what we had happened was instead of when we stimulated channel 8 on her median rpni she felt stimulation here when we stimulated channel 12 her ulnar rpni we got stimulation here what i was hoping for was all right we simulate both we get a sensation here and a sensation here cool we can use that for neat things we can tell what side of the hand maybe an object is held in or we can use that information to to give her some sense of how her hand is oriented in space instead we got actually a synthesis we got a stimulation that she felt right in the middle of her palm and while right now we're still exploratory we're still figuring out all right what do we do with this information because it is important to sort of think all right how is this actually functional for her what is neat about this is there's no nerve that we're hooked up to that says hey i'm getting stimulation from the center of the palm this was a new thing it means that the brain was actually synthesizing these signals in an interesting way that we didn't expect and that's really cool that means that we can dig into that and explore that means that we can dig into that and explore hey can we make these sensations move around can we do interesting things that might have some function for our participant so just to start sort of coming to a close uh i want to walk away what my research really focuses around is the fact that sensation serves more than just these functional purposes so we know that it can improve uh functional performance in these tasks we know it closes this control loop as engineers that is what the control loop is is all right we we have some goal we are sending out some signals we get some feedback and we correct and we sort of iterate on that process until what we built does what we want it to do but sensation also is shown to improve ownership and embodiment over a prosthesis so this is how much an individual actually feels that that device is part of themselves it improves their satisfaction how much they enjoy coming in we have a new participant now who if sensation is not part of the conversation he has checked out he's like hey you promised me that i'm going to feel some stuff with a hand that i haven't had in a couple years but most importantly it improves how often these people are actually interested in using that device so there's not too many studies out right now that have this sort of technology as something participants can take home but in the two studies that i have been able to find when their prosthesis is giving them sensation they're putting it on more days out of the study collection time they're putting it on for longer on those days because they want an arm that gives them that sensation back so i'll i'll end with this slide so our participant you know we have her in um she is a fantastic participant that has been coming in for the better part of three years now and we've done all these experiments with control and now with sensation but we make sure to ask her every once in a while what do you want out of this device what do you want to do when you go home that you can't do right now um and last time she she told us she she told us about how all she wants is to make some meatballs she just wants to make a decent sized meatball right now she's got her one hand and she's she squeezed them as much as she can and then they go in the pan and with her body power prosthesis you can see in here she can maybe hold the pan steady and then uh use her left hand which was not her dominant hand before her amputation in order to stir things around and move things in the pot she wants a prosthesis maybe not necessarily to do the other half of the meat i don't know i don't know how much how waterproof or meatball proof it would be but um she wants something that she can handle a utensil with she wants a prosthesis that she can tell when she has a solid grip on something she wants a prosthesis that she enjoys using more than the one that she has and if we aren't answering questions about how to get her to that place then we're asking the wrong questions thank you last night when he was writing his slides i saw just one empty slide that said one plus one equals and then nothing else so i'm glad that you brought it back full circle our next researcher is the newly minted doctor brad sonde from the autonomous robotic manipulation lab brad defended his thesis belief representations for planning with contact feedback last friday and graciously agreed to speak with us again today before he leaves for california at the end of the month please welcome brad to the stage this is audrey's garden audrey is my friend and relative audrey is in a wheelchair and cannot walk and has limited use of her arms she can plan out exactly where she wants everything to go she can order the seeds but she has to rely on others to actually do the physical planting i would view it as a huge success of my career if i could build a robot that could allow her to independently do the gardening when i think about what technology would allow this and what i've asked her you know i want it to be a robot but the answer is not a robot at least not first what audrey really wants is to have her arms back and that that you know it makes tons of sense but it's actually taken a long time for me and to internalize as a roboticist is that i cannot bring the ideal solution um because i i can't do surgery um but you know it it's a starting point for what are the potential solutions the next realization is that it doesn't necessarily need to be complicated the best tool that she has now is a five dollar gripper that allows her to extend her reach these are a far cry from actually having your hands back and being able to do full manipulation but it's very simple and if it breaks you can just order another one as we move more toward robotics you can have robot arms and the most common type are teleoperation these allow much wider range of activities but still can be hard you know slow and pretty dumb what i want the solution to be and what i've what i've worked for is autonomous systems systems that are able to perceive the world and understand their environment now these will not be the only solution and often these are far more complex and expensive than than the easier solutions but uh i think in order to in order to give certain functionality back we need systems that are less dumb the hardware you know we can always want better hardware but the hardware is mostly there i want it to be cheaper more reliable but here's a system that i've built or i am tele operating a robot in virtual reality so this is me i have a vive headset on i'm controlling the arms by grabbing the grippers on the side and i'm viewing exactly what the robot sees through the camera so i've obviously set this up to be possible i don't think i could do this in a regular kitchen but i've set this up and you can see things are not going exactly according to plan i'm constantly correcting right i wanted to just slide in there and get this pancake um and it's it's not working i need to keep fiddling around until uh you know move things around and uh this is a a bit scary i don't know if okay so that that that actually works pretty well um i was able to flip that pancake you know never having done it before this is this this is the first run for other tasks i need to be kind of creative and slow how do i actually open the syrup with these pretty bulky you know not not that dexterous arms and then how do i actually make the syrup come out uh well i can just uh squeeze the bottle and get a little dab of syrup so all this shows that you know uh you know the the hardware you know it's it's definitely not human level but it is uh you know the the hardware does much better than my algorithms can do suppose we give our robot a pretty simple test suppose we want to put this drink in the fridge that's probably what's going to happen right this happens all the time is the robot just you know collides with something that we don't expect fortunately now we have pretty pretty sophisticated pretty good robots that can sense this collision and actually stop before it like knocks over the fridge or destroys the can or destroys itself but now we need some intelligence because if if we don't if we don't do anything more then a person needs to come and reset the robot and figure out what the robot need uh you know how to how to do the task differently in order to succeed a lot of my research is focused on what do we actually learn from this contact so one thing we learned as we learned this this patch of red well we probably can't enter that patch of red uh based on the joint torque feedback we know that there's some region that there's probably a collision all this blue area we've learned is free the robot's already gone through it so we know that's safe to move as we bump around and apply more we learn more information about where contact could be in the world and then we can construct plans that go through more of the blue and avoid more of the red until we can eventually uh achieve our goal and put our drink in the fridge this has a pretty uh a pretty dumb prior the robot doesn't know anything about the world so we can maybe um we can maybe improve it a bit if we know what the shapes are in the world so here uh we have this table and on the on the right or on your left you can see the robot bumping around it knows that there's a table in the world that knows that there's the surface it doesn't know exactly where it is so it's constructing the same contact patches as before but it also it has a belief of where the table is as the robot moves around it can update this belief and at first you see has a very very wide understanding it doesn't really know where the table is as it bumps around this belief can collapse and you can you can then do more and more autonomous tasks because you have a better and better understanding of the world we can now take this one step further and instead of knowing exactly what the what the shape in the world looks like what if we just know kind of some vague things you know we we know a little something about shapes in this case learned from a neural network so here we see the side of a mug and we know that a mug has a handle but we don't know exactly where it is at first after the robot makes contact we want to be able to update these shapes so that any shape you produce has to have a mug handle or at least something that explains that contact i can explain that now with a uh a method like just completing boxes so we see the front of a box there's noise so it's not a perfect surface there's some true box and then as we um we can envision a bunch of different boxes that match that visual information but we don't necessarily know the depth as the robot moves through the world we sweep out from some free space and most of the boxes this won't really affect because it's not really intersecting but for some we can define a loss function and create new shapes that have to obey these constraints similarly once we make contact we learn this contact patch and there has to be some some part of the box needs to be in that contact patch in order to explain that contact we can up we can using our network we can create new shapes now that both agree with the visual information and this new contact information as we update these shapes maybe we'll create some that don't obey this contact information and we throw those out because those those don't agree with our observations that the robots made the last crucial part is that we need to make sure these shapes stay consistent with the visual information if we only use the contact information we might create shapes that yeah they might obey the contact information but they're just they're not consistent with what we actually see applying this on the real robot we need to combine it with more more intelligence here we need to uh see the scene but then be able to segment out the the different objects and from just the segmentation you you can see how how bad the pure camera information is it's missing patches it's kind of creating points that don't exactly correspond to the box but we can still create create these shapes that maybe you know some were kind of close and many are fairly different than the real shape but once the robot actually makes contact we create this contact patch and update our shapes to then uh to then explain this contact these don't necessarily look exactly like the uh the the box in the real world but they have very similar occupancy and this allows the robot to reason about possible ways it could move in the future so that hopefully it will uh you know to allow it to accomplish tasks based on the information that's learned from contact we can do the same thing with a pitcher so this uh robot has seen the pictures before so it's able to have a pretty good first guess but you know this the the the picture that it completes is different than the true picture that it in the world and it learns that when it makes this contact future updates future samples now need to explain this contact patch when i think about the space of robotic problems and the problems that we're trying to solve often uh i think many of us including myself like to write papers that push the robots into the brilliant stage if you think of like deep blue or go these these are these are algorithms that achieve superhuman performance i think though there's a lot of room just to make robots less dumb i mean it's it's really robots make so many mistakes that once you look at the it's it's they're embarrassing to show because it seems so obvious that the robot should be able to overcome it part of this i think is an engineering challenge but part of it i think is really real research it's not just iteration we need we need new ideas about how do we overcome these embarrassingly easy traps so as i look forward uh i cannot report success yet i do not have a robot that can that can really aid someone in gardening yet and i by no means want to uh imply that i i have uh you know have have the answers for how we should build assistive robots um i look forward to discussion from uh from everyone in order to help help come with ideas to to better build these robots all right thanks brad and congratulations again we'll be sad to see you go uh our final speaker for today is professor ben kuipers professor kuipers has been at the university of michigan since 2009 and leads the intelligent robotics lab he is a founding member of the robotics institute and his research center's trust between a robotic agent and its human counterparts please welcome dr kuipers i got it done thank you very much i really enjoyed your talk this morning so this was great um let's see here how do i get this to go forward ah there we go um i'd like to tell you about some work that we've been doing for a while on a robotic wheelchair and here it is with the lovely peter gaskell driving um so about oh 20 some odd years ago i was teaching a robotics class and we were making little lego robots and i i figured you know we've got to have a guest lecturer who can tell us what the real world is like and so what we got was a friend of a friend who was paraplegic and in a powered wheelchair who used a power wheelchair and we said listen we're roboticists we work with these machines how can we help what can we do that actually makes a difference and what he said was every time i rent an apartment i lose my damage deposit because we ding the walls we ding the furniture we bump into all sorts of things and it's really terrible make a wheelchair that won't do that and so i thought whoa maybe we can and that sort of blew up into a lot more stuff so um vulcan is an intelligent robot and it builds a cognitive map it explores an environment and it learns where it can go and where it can't go the first question we want to think about is who's in charge where's the autonomy who has the autonomy people talk about autonomous robots who wants an autonomous robot we want a person to be autonomous what you said this morning or earlier about your wheelchair is that it gave you autonomy it wasn't that the robot that the wheelchair is autonomous so but the person can delegate certain chores to the robot and so here are two somewhat different models of delegation for a robot that has a um a cognitive map you can say take me to the front door and what i'm doing is i'm delegating a couple of things one is safe navigation but the other is making the plan and carrying it out or i can say take the next right and then a left after that and now what i've done is i haven't delegated how to get there i've maintained the control over where i'm going but i've still delegated the um the safe navigation now if you're driving with a joystick which you do all the time then you're not delegating the safe navigation either you're saying i'm going to do the navigation now as we'll see in a minute not everybody can handle a joystick um and so they may need to delegate that and be able to work at some of these levels now i want to show you a picture here now okay here is our wheelchair and it's it's going through a classroom building at class change time this is showing off what it sees it's showing it's seeing the pedestrians it's planning various routes in a moment i think it'll get to a point where it can't go anywhere and so it just stops see everything's red and it waits for something to open up and it keeps going now this is vulcan and it's demonstrating that it can move in a situation where it has incomplete knowledge and the world is dynamic and it's not completely known because pedestrians who knows what pedestrians are going to do they're going to do something crazy mostly now we have a theory about how we do this and i'm not going to go through it although i could spend an hour on that easily but basically we distinguish between small scale space which is space i can see and i can explore it by moving my gaze versus large scale space which is space that i have to travel around in and i have to knit together the different things that i see in order to make a map people mostly think of mapping as metrical that you build a model that actually is a scale model of the real world but in fact most of the time what you actually want is a topological map you want a network that says what can i do here well i can go down those stairs or i can go there and turn right those are my choices if i go here it's going to be a bad thing so it'd be worse for you so um and we can put we can have metrical and topological maps of small-scale space and large scale space and we put all of those things together and so we call this the spatial semantic hierarchy and it is ever so wonderful but it has taken many years to get as far as we have and there's a lot more progress to happen now one of the cool things about these different representations is that they let you say different things so in small scale metrical mapping i can say go over there and then use the knowledge to figure out how if i want to say turn right or turn left i'm using the local topological map but if i want to say go to the front door or the kitchen or the doctor's office i can use this large scale map now in fact people use paper maps all the time because in your head you mostly don't have large-scale metrical knowledge some people do some a lot of people don't so who is vulcan for well i would love it to be for people with disabilities and it's intended to be for people with disabilities but it isn't ready it's a research platform and so we do research on cognitive mapping on perception on control on on avoiding collisions and all sorts of things but what we're hoping for now clearly it's people with mobility disabilities that need mobility and that's why you have a wheelchair but if you've also got perceptual disabilities i once watched a young lady who was both blind and needed a power wheelchair the bravery of crossing a street is amazing and so you would want more capabilities for a wheelchair to be able to see various hazards and avoid them communication disabilities now you communicate with the joystick if it turns out you have substantial tremor then you can't use a joystick and so you've got a problem cognitive disabilities plenty more to be said about that too so in the long run there are possible applications for this kind of intelligent robotic technology to help a whole variety of different kinds of people um now so this sounds like a great thing is it all completely a great thing or is there drawbacks here now one issue is de-skilling if it turns out when you work hard to practice a skill that keeps it going if it turns out that there's a way to sit back and let the machine do the work then you can lose the skill this is a problem with autopilots on jets as well as wheelchairs um one of the scenarios for the for the intelligent wheelchair is for elderly people who might be stuck basically bed bound and if they could have an intelligent wheelchair they could go visit their friends and being mobile and being able to autonomously visit your friend by saying take me to jane's house or jane's room then that makes a big difference but if it's de-skilling where does that trade off i don't know you may know more about that than i do but it's a big deal um there's all sorts of technical and ethical problems that come up when you think about ai and robotics and that's a thing that i want to spend some more time on what if the robot now we're talking here about intelligent robots so it's perceiving the world it's building a model of what it thinks is out there and it's deciding what it thinks is the best thing to do what if it's wrong what if it hurts somebody what if it hurts somebody out there what if it hurts the person in the wheelchair there's there's a real issue here that weighs about 350 pounds if it falls over and you're in it it's a big problem um autonomy i've made a big deal out of the fact that the autonomy belongs to the human who can under some circumstances delegate aspects of that to the robot what if the human wants to do something dangerous even really dangerous should the robot stop it can it stop it the only model that i know of right now is that there is a philosophy in the mobility and orientation community for blind people seeing eye dogs a seeing eye dog can prevent its person from crossing the street it gets in the way but it is not authorized and it doesn't have a way to communicate that it's okay to cross the street that's the human's decision so there are some of these questions that come up as you're you're sitting there saying well what's my robot supposed to do and then as you think about it you say this is actually a deep question and when you look around you find that there's people in the mobility and orientation community who have actually thought about that deep problem we don't necessarily have people who thought about all of them but there's a whole bunch of interesting things so here are some research problems there's an ai problem what is space how do you observe it how do you learn the structure of space there's a robotics problem this is not just an ai this is a physical object that perceives the world and acts in it how does it do that how well does it have to perceive the world how well does it have to act there's a human robot interaction problem we've got a robot now robots don't perceive the world anywhere close to as well as you think people typically perceive the world a lot better um here we have two agents the human and the robot and each of them has perceptions each of them has knowledge each of them has some portion of the autonomy how do they work together how do they communicate with each other this is tough this is not an easy problem by any means and the assistive technology problem what do people with disabilities want and need we can sit back in our chair and say oh i'll bet they need this but a far better approach is to ask them and find out so i think this is one of the most well we've written a bunch of things you should read them um this is a fantastically interesting research question you guys should work on it it's been lots of fun for me thanks thank you professor kuipers for leading us right into our question section with some nice open-ended questions for everyone um so at this time i'd like to invite all of our speakers sort of back up to the stage for our panel yeah so my wheelchair 100 feels like a natural part of me um so much so that like when i am getting a new wheelchair that process happens about every eight nine years and when i sit in the demo chairs or my new chair i feel like i'm wearing the wrong pair of shoes like i i can't sit upright comfortably and my driving uh declines significantly in terms of my maneuverability and um that is because i become like deeply attuned to the little aspects of my chair and how it handles and you know my drift set provides a very kind of delicate level of maneuverability so a few millimeters has a very drastic effect on what i'm doing and i get used to that and any slight change is very noticeable um the json arm the robot arm that i've been using not so much it hasn't it hasn't felt natural yet and i think that is kind of because there is that lack of sensation uh that was being talked about there's not a whole lot of feedback that i get other than visual um for how hard i'm grabbing or you know the area that the hand is moving in space um so that one still feels kind of like i'm operating a robot rather than moving something on my own thank you well uh okay so i'd i'd like i'd like to
2021-08-07