Robotics Colloquium: Assistive Technologies

Thank you for joining us this afternoon for  a colloquium on assistive technologies which   was organized by the Robotics Graduate  Student Council and Michigan Robotics.   My name is Andrea Sipos, I'm a rising second  year PhD student acting as the colloquium chair   for RGSC this year. It will be my pleasure to  introduce you to our speakers this afternoon.   We ask that you hold any questions until the end  of the presentations we'll have a Q and A panel   then, with all of our presenters up on stage. We  further ask for your patience in advance because  

this is the first event that we're holding in our  new building and this event is being recorded and   should be available for our remote colleagues at a  later date. To lead us off i am excited to welcome   two special guests who travel to Ann Arbor from  Minneapolis, Minnesota, Shane and Hannah Burcaw,   married in 2020 are disability activists from the  viral youtube channel called squirmy and grubs   i first learned of the couple through a video  they posted last year entitled this robot   would change Shane's life because my youtube  algorithm knows i'm a sucker for robot videos   i watched a lot more of their videos after that  first one because they bring a unique humor and   charisma to the important topics of disability  and accessibility in addition to the youtube   content that Shane and Hannah make. Shane and his  cousin Sarah co-founded a non-profit organization   called laughing at my nightmare in 2012.  laughing at my nightmare has a two-part mission  

to educate children on kindness positivity and  empathy and to provide free adaptive and medical   equipment like adaptive bikes custom power  chairs and adaptive strollers to people living   with muscular dystrophy Shane and Hannah are  here today to talk about their experiences with   assistive technology and to help bridge the gap  between researchers developing these technologies   and their end users without further ado please  join me in welcoming Shane and Hannah Burcaw. how's everyone doing just gonna get  situated here i think it would have   been funny if we uh run around behind the  stage to uh ostensibly come up on the stage   but we just left through the doors back  there and you're all like, what happened?   Already demoing some assistive technology here.  Hi I'm Shane and this is my wife and probably the   most important piece of assistive technology in my  life Hannah hello thank you for that intro we're   really excited to be here today um assistive  technology plays a huge role in my life and our   lives um going to talk about that it's going to  be real tangible uh i want to talk about kind of   why it's important to me um but then  really just dive into what devices i use   some terrible stories about them uh because it's a  lot more fun to talk about failure than success um   yeah we'll just uh take it from there and  then we'll be in the q a at the end so save   all your questions for them so i'm going to  bring my notes up here on my first piece of   assistive technology which is my iphone um but  we'll talk about that in a bit uh just like i   think we should go back and talk a little bit  about my diagnosis so you have a groundwork for   why i use wheelchair and assistive  technology i was born with a neuromuscular   condition called spinal muscular atrophy which  basically makes all the muscles in my body   really crappy and so i've never walked i got  my first wheelchair when i was two years old   um and i was told or i am told by my parents  that my favorite thing to do when i was learning   for the first time how to drive my chair was to  crash into strangers walls obstacles at top speed   i still enjoy that but i'm a  little bit more reserved nowadays but i i the overall point that i  want to make today is that assistive   technology while it obviously allows me  to have function that i wouldn't otherwise   have it allows me to do everyday activities it  more than function it provides a a satisfying   and fulfilling life experience so like well yeah  my wheelchair gets me from point a to point b   more so it allows me to thrive and because  of all of the assistive technology i use i've   been able to become a an author and a prominent  youtuber which i never thought i would say i'm a   a husband and a traveler of the world all  kinds of things that are more than just   going from point a to point b a really good  example of that is one of the stunts that uh   i use my wheelchair for as a child do  you want to talk about the basketball   yes sure i will of course yeah um so i was not  there for this story shane was like 10 years old   um your worst yeah shane's worst age 10 years old  and it shows what his childhood was like so when   he was 10 his younger brother andrew was six i  guess six or seven um and shane was a terrible   influence on him andrew really wanted to dunk the  basketball they had a basketball hoop in their   driveway andrew wanted to dunk it and couldn't  obviously he was six years old so shane realized   that he could use his wheelchair to help andrew so  he told him to go into their garage and get a rope   andrew did that poor little andrew and uh brought  the rope back outside and shane was like okay now   tie the rope around your waist and poor little  andrew was like okay ties the rope around his   waist can you please stop referring to him i was  poor little andrew i think it adds something poor   little andrew did that and then shane was like  okay throw the other end of the rope through   the basketball hoop and andrew did that and then  tied it to shane's wheelchair shane was like okay   hold the basketball and i'm gonna pull you up so  shane begins to reverse and andrew starts rising   into the air and then begins screaming in pain  because the rope is like cutting into his stomach   he wasn't wearing a shirt which in hindsight  that was that was a mistake i don't think the   shirt would not have helped so instead  of stopping and being like well that was   really dangerous i'm sorry andrew shin was  like okay get some pillows so andrew goes   inside gets pillows puts them between the  rope and his stomach and shane tries again   it worked and as i began to reverse my chair  my brother began to levitate into the air   with this ingenious pulley system that i  had invented um he's lifted into the air   one foot two feet three feet four feet he's  got the ball in his hand he's ready to dunk   and when he was near the top maybe a foot away  from being able to dunk and i don't know why   this happened i think it has to do with science  all of you might understand it better than me   uh he got a lot heavier and my chair was no longer  able to lift him that final foot that he had to   go so i am in full reverse my wheels are spinning  there is smoke coming out of the back of my chair   and nothing about that made me think you know  what maybe i shouldn't be doing this andrew   has the nerve to yell at me to try harder like all  right i'll put my back into it i'll reverse him   smoke everywhere at that moment our dad comes  outside doing this horrifying combination   of laughing hysterically and screaming at us uh  he didn't like what we were doing um so i lowered   andrew back down to the ground in utter failure  well there's more so you know shane's dad you   know tells him that that was really bad don't  do that again you're done playing for the day   come inside i think he also complimented  us i think he was like guys this is amazing   but i can't not punish you for this he doesn't  remember that part of the story i don't think   your dad would say that he did that but anyway  he's like go inside shane starts to go inside   and finds out that his chair can only turn left  it no longer drives in a straight line that's bad and a little further inspecting about our dad  found that all that smoke was the deer motors   in my chair shredding themselves um i had to  replacement cost of eight thousand dollars   so i was in trouble and this is obviously a  kind of silly example but i think it shows how   yeah throughout my whole life especially when i  was a kid um i've been using assistive technology   specifically my wheelchair to have fun to get in  trouble to hurt my brother all kinds of things   and it is it does more than get me from point  a to point b it allows me to live a full life   all right that was the basketball story uh next  let's talk about my one step ramp which is a very   rudimentary piece of assistive technology but at  least in terms of travel it has really changed my   life can you talk a little bit about it yeah yeah  and i think you'd be surprised how long it took   us to find a portable lightweight one-step ramp  that could like fold up and we have one that it   like hangs in a bag on the back of shane's chair  and it folds in half but it also is strong enough   to hold his chair and we've brought it on all  of our trips so that when we come across a curb   that doesn't have a curb cut or a store that we  really want to go into with one step we can still   access that store so i think that is probably the  most used piece of you know technology that we   have i think people don't realize how  much of the world is still inaccessible   to wheelchairs like a a five inch step exists in  so many places um throughout my whole life i just   haven't gone into those places and then like  two years ago we were going to uh was it poland   or london um and we were thinking about kind of  how often we might encounter that obstacle along   the way and we didn't want that to ruin kind of  the adventure and the the joy of going into where   we wanted so hannah was like why don't we bring  a ramp and i was like oh no like they don't make   those like it won't fit in her bag or on my chair  and hannah was like yeah i'll find one and she did   like on amazon in about five minutes of looking um  so that has been really helpful for me as well um   kind of a cute acute piece of assistive technology  is the phone holder that i i used in bed when   hannah and i were long distance you want to talk  a little bit about how we met and then get into   that yeah so shannon and i met five and a half  years ago i was a freshman in college in minnesota   and shane had just graduated from college he was  living in pennsylvania and we were long distance   for two years so on and shane was living with  his parents and uh on many nights we would want   to be facetiming really really late into the night  which would be difficult if shane was going to ask   like his dad to bring his laptop you know into his  room set it up and then come back at like three in   the morning and take it down you know so uh shane  got a phone holder like a long what would you do you probably know what i'm talking about so yeah  amount but it has a long arm on it bendable yeah   so shane got one of those and used that every  single night for the two years that we were long   distance and it let us you know communicate in the  way that we wanted to and it didn't you know it   would have been difficult if shane hadn't had that  and we weren't able to talk after his dad went to   bed yeah and that's another one that it sounds  like a no-brainer but you're like yeah obviously   why wouldn't you put your phone on your mouth um  but for a long part of my life i hadn't considered   even the possibility that i might be able to  use my phone in bed so like in high school when   cell phones came about and all my friends were  texting hours into the night on our flip phones   i often had to you know step  out of the conversation around   9 or 10 when i went to bed and that sucked i  couldn't be a part of the social jumping zones of   my friend group because i thought there's no way  i can lay in bed and hold my phone like my friends   then i met hannah and i was like there must be  a way that i can lay in bed and hold my phone um   so simple solutions allow for  very far-reaching possibilities uh another huge probably the item that i use the most after my wheelchair is an app for my  iphone that turns my phone into a bluetooth   keyboard for my laptop i'm a writer and i began  a blog in 2011 uh that kind of kicked off my   career as a writer and blew up in a way that  i never imagined and right around that time i   lost the ability to physically type on my laptop  terrible timing as my vlog is taking off and   i looked around i used like voice dictation and  had a horrible time with that it didn't understand   me very well even after hours of training it  um so i would spend more time editing and i did   writing and then i found this app and it  allows me to type with my lightning fast thumbs   right on my phone and the words appear on  my laptop so i've written reboots that way   thousands of emails every week uh blog  posts articles um it's allowed me to   have a career that i love and that you know  makes me feel fulfilled all right this one is   a bit of a letdown is that photo still up  there oh no can you talk about my jacob yeah   yeah this is a huge letdown so we we recently flew  on an airplane we were on a flight and for that   flight we took off all of the parts of shane's  chair that could break because they they often   break on flights um and one of those things that  we took off was jane's robotic arm his jayco arm   it's this like huge arm that sits here and it you  know does a ton of stuff and we forgot to put it   back on and we realized that as we arrived here so  it would have been great if shane had his robotic   arm and was like holding the microphone we didn't  bring it but normally he has a really cool robotic   arm that sits here you have to trust us it would  have been so relevant like there has never been a   more relevant time to show off my robot arm i'm so  sorry um but can you talk a little bit about like   the journey to getting it and that yeah when  did you get it do you remember was it like   six months ago yeah so the robot arm is extremely  expensive and fifty thousand dollars and uh shane   knew about it because some people that we know  had gotten it through insurance and like it was   just sort of like on our radar but we assumed that  shane's insurance would never cover it um you know   they've not they've denied his wheelchair before  they've denied like other really basic things so   we're like there's no way it's not worth it um but  more and more people we saw in our you know like   network of people yeah getting it and so we  were like maybe we should actually try to   to get this and so um shane's occupational  therapist like wrote this you know big thing   for insurance and submitted it i think there  was a letter from you in that first one and   and a month later we got that it was denied  and we were like okay um but the company   selling the arm is like we've never gotten  it through without being denied once so like   keep going like appeal it it might actually go  through eventually and this piece of technology   seemed like it would be revolutionary for me i  physically need assistance from my human assistive   device here with like almost every aspect of  daily life like i need help getting a drink   from the fridge itching my forehead uh putting my  shoes on all kinds of things everything that you   do in a day other than like the work i do on my  phone and my computer this robot arm connects   to my wheelchair and allows me to use my joystick  of my chair to manipulate it in a variety of ways   so i had the opportunity to open doors and pick up  my phone if i ever drop it get food for myself uh   all kinds of things and it was denied and the  reason was that it was not medically necessary   which is kind of their catch-all phrase that they  use when they don't want to pay for anything um   somebody appealed it or you're successful uh  hannah wrote a letter explaining a variety of ways   that without it i could die essentially like if  there was a fire and i couldn't open my own door   or if my head fell and i was choking and couldn't  lift my own head up all the real things probably   won't happen but insurance doesn't need to know  that uh so it was improved and i doubt that jake   alarm it has been revolutionary for me um to be  able to go and get myself a beer without having   to ask hannah i don't know if hannah likes that  aspect of it um but uh it's been really useful   and it did end up saving my life not even  today um doing tell that or vision telling that   yeah i didn't realize you were going to tell this  story so i guess we're telling it um i don't know   how relevant this is but i guess since it saved  shane's life he wants to talk about it anyway um in in december of this past year uh shane  was coming out of my parents house we were   visiting we were there for dinner and their  house has about five steps to get inside and   they have a big wooden ramp that uh they built  for shane to access the house so he's coming   down the ramp and it had been snowing or  like it was snowing it was actively snowing   uh and every single time we go down this ramp  i always say to shane do you want me to hold   your chair like you know walk behind you holding  it and every single time shane says no i'm good   um so we the same thing happened i was like you  want me to hold your chair he was like no i'm good   i let go and within like a quarter of a second  he had begun to like spin on the ramp and was   hurtling toward the edge yeah it was one of those  moments where like you hear the phrase like your   life passed before your eyes i had like flashes of  like oh i'm about to die like this is about to be   how i die so i'm fishtailing on the icy ramp  and slamming into the edge which does not hold   my cheerios over the edge of the ramp  when i'm i don't know four feet in the air   and tips off of the left side so that i fall in  between the edge of the house and the ramp my   bat right wheel is the only wheel that stays on  the ramp and my jaco robotic arm on the left side   catches on the house keeping me from  falling all the way forward so i'm dangling   literally down going straight down with the arm  against the house and my wheel on the ramp had i   fallen further i would have been crushed like it's  funny now but i would have died um i'm fragile as   hell and uh would not have fared well in that that  incident so i mean what was that like from your   perspective other than awful it's a terrible  question it was horrible and it all happened so   quickly it really felt like i was watching a movie  like i was just standing there and then within   like two seconds it was like two seconds from when  i let go and then shane was like upside down so it   was just like i could not believe it was real and  then i like ran down the ramp i just didn't know   what to do like that was you know the main problem  because it was like shane's chair was too heavy   i was like pulling on it but then he was  like no don't because what if it like   what if you dislodge it and then it falls all the  way it was horrible and then i decided to like   take shane's straps off to try to like get him  out of the chair and then he was like no i'm gonna   fall it was just it was horrible it was a mess  yeah it was like a nightmare scene basically um so   hannah's mom heard the promotion and ran out and  somehow i mean my chair weighs 500 pounds somehow   she hannah and her mom ripped it back up oh no  wait you got me out of it first yeah she held she   held it in place and i took you out and then once  you were out we were able to like just let it down   so i lived well louder i lived um surprised and  uh you know i i never kind of expected that the   jaytown would end up saving my life but like  the next day we realized had that not been   there i would have gone all the way and i'd be i  wouldn't be here right now so thank you jay taylor   uh yeah that we didn't burn uh maybe they have one  here we'll hold her up for it after um the last uh   piece that has been pretty big in my life is again  a simple solution but it is the uh powered mount   that my joystick is on right now again i'm awful  we forgot to charge it in the hotel room last   night so i have a switch here that is supposed  to activate my joystick to swing down in front   of me where i drive with it it's not working right  now sorry um never hire a sedan or terrible but   i for most of my life had a manual release on  my joystick that then had to manually be moved   out of the way for transfers or if i'm pulling  up to a table to work or to use my phone on my   lap where i have it now and that was limiting  because i was not able to move my joystick   on my own so if i had it flung out like this and  i've got my computer and then i want to move to   escape the house if there's a fire i can't do  that on my own i have to call for help um i know   they met this company that was like oh no we made  power mounts like we didn't install that and my   life's changed forever and so now with the touch  of a light button over here i can swing it down   move it out of the way um and it's been really  helpful do you remember the first night i got it   yes uh yeah so before shane had it obviously if he  was sitting at his computer in you know our living   room and i was in the bedroom reading a book i  knew that that's where shane was and you know   that's where he would stay and that's how it had  always been so the that sounds like weirdly like   i don't know so uh that first night i had like  kind of forgotten that he had gotten this you know   adaptive technology and i was just reading  a book in the bedroom like nice and calm   and all of a sudden i did like i guess i didn't  hear shane approaching like i think he was doing   it extra quietly just for this to happen he was in  the room and i see him out of the corner of my eye   and he screams he literally snuck up on me to  scare me i screamed it was horrifying and then for   the rest of that week he would just like i would  just be sitting in a room and he would just like   come in do a lap and then leave he was obsessed  with using this joystick to just like roam around   the house at his you know at his whim it was  really annoying at first but it has been really   useful these are all the hard-hitting important  uses for all of my adaptive technology we tell   insurance that like you're going to escape a fire  but really shane just does laps around the house   endlessly i need this device to bother my life  uh i don't know if that would be approved but um   i hope you see like there are many more  kind of devices and things i use in my life   um but i hope that you kind of see that they  allow me to live like really live a fulfilling   and full life they don't just like do the one  thing that they're intended to do um and so   that's been a really cool part of my life i was  very lucky to be born in a time when all of this   technology is available and as we look forward  hannah and i are you know in the early stages of   you know thinking about having a family and we  know that there will be more and new ways that   the devices i have now will be helpful like  holding our kids with my jayco arm in the air when   they're bad um we've seen devices that like allow  you to have a stroller attached to a wheelchair   things like that are excited to know that they  are available and will continue to benefit me   in all the stages of my life so um not to be like  overly you know not to watch politics here but   the work that all of you are doing is so important  because you know while it might feel that you're   you know just in a lab like doing research or  um you know working on a device remember that   there are people out there who will be extremely  benefited by it whose lives will be more fulfilled   you're improving access and function  and you know missing life's good so   thank you for everything that you do  thank you for having us and uh yeah cool that was an awesome presentation and i thank  them for being just so candid with us because   i think it's really important for us to hear  as researchers the perspectives that we may   not get every day when we're just looking at the  numbers and the wires we need the human bit and i   think that's something that shane and hannah have  provided for us and we really thank them for it uh   so now we're going to shift into some short  form research presentations from some of our   colleagues in michigan robotics uh next up to  the stage is going to be michael gonzalez who   is a co-organizer of this event he's a fifth  year phd in the rehabilitation biomechanics lab   most of us know him for his love of board  games at rgsc game nights but today he's   going to be presenting his work on the impact of  sensation on the control of robotic prosthetics thank you andrea and thank you shane and hannah  again for for that talk i i really love getting   those perspectives because that really connects us  to to the work that we do um and this clicker is there we go all right um so the work that i do  and and that i have been uh spending the last   uh four years studying is prosthetics so  prosthetics are neat because they are robots   that someone is actively wearing so they're not  only serving a function but they're also serving   a form they are designed to replicate the function  and the the aesthetic of a missing limb but there   are many people who choose not to use prostheses  and this is something that really interests me   that there are individuals out there that have  things they want to do and we have some assistive   technologies that are designed for their use but  they end up not using them and it's it's a curious   thing to me of understanding why that might be  so one person that really helped me uh sort of   understand this and was part of the reason why i  was interested in prosthetics in the first place   was hugh hair hugh hair does uh really amazing  work at mit but he started out as an individual   with no engineering interest whatsoever he just  loved rock climbing uh and and mountaineering   and uh one day he got lost in a snowstorm and he  lost both of his legs below the knee to frostbite   and when he sort of recovered a post-amputation  he was really upset with sort of the state and   quality of lower limb prosthetics and he felt like  he couldn't walk the way that he wanted to and he   couldn't get out and do rock climbing in the way  that he enjoyed previously and he had this really   amazing quote that that i enjoy and that sort of  centers how i think about prosthetics which is   i didn't view my body as broken i realize  that a human being could never be broken   technology is broken technology is inadequate and  i like this because it frames it as what we're   doing is not trying to replace a missing limb or  replace all of the you know functionality that we   might think is quote unquote missing what we're  trying to do is find the needs of individuals   and make sure that they can do those things  through the technology that we're working on so   what does it mean to be adequate for a piece  of assistive technology well of course with   many things the answer is it depends for  lower limb prosthetics maybe adequate means   it gives the individual stability they can very  confidently walk without worrying about any sort   of of wobbles or falls maybe it means symmetry  so making sure that the forces on both sides   are equivalent so they don't get overuse  injuries or maybe it means energy cost it   uh the the prosthetic is replacing some of  the lost musculature that the individual   doesn't have anymore or it just makes it so  that it's not difficult for them to walk at   all for upper limb where i spend most of my  time and energy dexterity is a really big   piece of what is adequate and a piece of assistive  technology so you not only want a hand that can   make different grasps depending on the function  but something that has a level of specificity   in terms of the force that you're using you want  to be able to pick up an egg without breaking it   and also without crushing it and something  that modern technology is allowing us to   do more of is create prostheses that  give users some amount of sensation   but above all else adequate means people  actually like using the technology that   it is not frustrating that it doesn't break on  them or that it does what it's supposed to do   99 of the time that you ask it to do that thing  thankfully we have an amazing prosthetist that   works for with us for the comfort side of  things because for prosthesis this is also   something where an individual has to wear it  they have to put weight on it in the case of a   lower limb prosthesis so comfort is a big deal  thankfully i don't have to deal with that part   what i am really curious about is sensation and  understanding how sensation plays a role in both   function and also the experience of using a  prosthesis there are endless questions when   it comes to exploring sensation what is it what  does it mean to feel something what is our brain   doing when we touch an object and get information  from it how do we use that sensation for function   so you know grabbing an object i want to know  when i come into contact with it i want to know   how hard or soft it is are there different types  of sensation that are more useful than others   so if i need to design a device do i want it  to beep at you when you have grabbed an object   do i want it to vibrate do i want it to be able  to communicate how hard or soft that object is   and something that i've been looking  into more recently is how do we generate   artificial sensation in that missing limb so a lot  of people with an acquired amputation they still   have nerves that used to run to where their hand  was and we can actually leverage the fact that   those connections still exist the brain still has  some space in it that's organized to process that   information so we might be able to actually  make them feel something in their prosthesis   and then pardon the pun but how does sensation  make people feel so there's an emotional component   to it as well you want someone to enjoy using that  device because what do we use our our hands for   we we touch and interact with objects with loved  ones so there is that emotional component as well   so as an engineer obviously quantitative  measures are a big important piece for us   so one of the questions that i started with is how  do we measure how do we quantify sensation usually   in our field it's described qualitatively so hey i  do or don't feel something when you put something   in my prosthesis or you know in anecdotal uh  evidence a pro a prosthetist might say oh this   type of prosthetic gives me more sensation than  that one the two main types of prostheses for the   upper limb are body powered so these are typically  a harness you've got a cable running down to a   hook and when i pull my my scapula or my shoulder  back and forth it can open or close that hook   and we also have myoelectric devices where an  individual is typically flexing some of the   muscles or extending some of the muscles that they  still have in their limb and this is translating   to an opening or closing of the device but it's  hard to tell how much one might make you feel   more than another so you might have force that  is transmitted through that cable it makes sense   that they would be able to feel some of the force  on their device but how do we put numbers to that   so i had the privilege of working uh on a project  where we're studying how to actually do this   we had a methodology where individuals were  interacting with these paddles the paddles   could be programmed to represent different  objects of different sizes and stiffnesses   and then we could actually give them specific  types of information so okay when you grab   this virtual object you're going to see a color  change or you're going to feel some vibration   or you're going to feel some resistive force  or you're not going to feel anything at all   and we can sort of study how they respond  how they're able to match these targets   and while our results weren't necessarily  surprising we actually got to confirm all of   these anecdotal pieces of evidence so here we've  got you know anatomical so you know a natural hand   a body powered and a myoelectric prosthesis and  across every category yes in fact the human hand   is the most accurate good glad we confirmed  that uh we can also confirm that myoelectric   prostheses were the least accurate across all  of these different perspectives across all these   different types of feedback but you know we  can also see interesting things that we maybe   didn't expect so for visual feedback didn't matter  what type of hand the person was using they could   all see when the color switched from black to  white when they grabbed the object so these sort   of disparities that we still see in these results  mean all right there's other things going on   there's you know maybe a control issue that  is there and that's for a talk in another time   anyways i so after i got done going okay  engineering brain i want to quantify these   things i got curious about this uh  this question of how do we actually   give people sensation what does that look  like how do we create artificial sensation   in these people that are missing their their  hand so we started this project working with uh   one of our participants who is implanted  with uh what is called an rpni so   a little bit of background so these nerves that  once innervated that hand these people still have   those sometimes they scar over sometimes they  don't and again we still have space in our brain   that is looking for signals that used to run to  the hand so here at the university of michigan   in our lab and with the cindy chestex lab we can  use regenerative peripheral nerve interfaces which   are these biological constructs that are these  muscle grafts wrapped around a nerve that used   to run to the hand we can collect signals from  these and so we can understand what the person   wanted to do with their prosthesis so if  they're thinking all right i'm trying to flex   my thumb this can maybe pick that up and then we  can pass that along to a prosthesis and tell the   prosthesis flex your thumb but we can also send  signals to these nerves and the brain can pick up   okay this used to be something that ran to the  pad of the thumb it's getting some signal now that   must mean that something is touching my thumb  even though that thumb does not exist anymore   so this is just a quick video of some  work with one of our participants   so here we're stimulating um in her  median nerve which stimulates this   half of the hand and her ulnar nerve  which stimulates that half of the hand so we can get her to feel something here   we can switch the channel that we're stimulating  to in these wires that are embedded in her arm so now we can get stimulation over here  and then we were curious what would happen   if we stimulated both areas like the same we just  did some science so it's it's hard to see on this   video but essentially you're feeling two different  locations right you're feeling a middle ground so what we had happened was instead of when we  stimulated channel 8 on her median rpni she felt   stimulation here when we stimulated channel 12  her ulnar rpni we got stimulation here what i   was hoping for was all right we simulate both  we get a sensation here and a sensation here   cool we can use that for neat things we can tell  what side of the hand maybe an object is held in   or we can use that information to to give her some  sense of how her hand is oriented in space instead   we got actually a synthesis we got a stimulation  that she felt right in the middle of her palm and   while right now we're still exploratory we're  still figuring out all right what do we do with   this information because it is important to sort  of think all right how is this actually functional   for her what is neat about this is there's  no nerve that we're hooked up to that says   hey i'm getting stimulation from the center of  the palm this was a new thing it means that the   brain was actually synthesizing these signals  in an interesting way that we didn't expect   and that's really cool that means  that we can dig into that and explore   that means that we can dig into that and explore  hey can we make these sensations move around   can we do interesting things that might have  some function for our participant so just to   start sort of coming to a close uh i want to walk  away what my research really focuses around is the   fact that sensation serves more than just these  functional purposes so we know that it can improve   uh functional performance in these tasks we  know it closes this control loop as engineers   that is what the control loop is is all  right we we have some goal we are sending   out some signals we get some feedback and we  correct and we sort of iterate on that process   until what we built does what we want it to do but  sensation also is shown to improve ownership and   embodiment over a prosthesis so this is how much  an individual actually feels that that device is   part of themselves it improves their  satisfaction how much they enjoy coming   in we have a new participant now who if sensation  is not part of the conversation he has checked out   he's like hey you promised me that i'm going to  feel some stuff with a hand that i haven't had   in a couple years but most importantly it improves  how often these people are actually interested in   using that device so there's not too many studies  out right now that have this sort of technology   as something participants can take home but in  the two studies that i have been able to find   when their prosthesis is giving them sensation  they're putting it on more days out of the study   collection time they're putting it on for longer  on those days because they want an arm that gives   them that sensation back so i'll i'll end with  this slide so our participant you know we have   her in um she is a fantastic participant  that has been coming in for the better   part of three years now and we've done all these  experiments with control and now with sensation   but we make sure to ask her every once in a while  what do you want out of this device what do you   want to do when you go home that you can't do  right now um and last time she she told us she she   told us about how all she wants is to make some  meatballs she just wants to make a decent sized   meatball right now she's got her one hand and  she's she squeezed them as much as she can and   then they go in the pan and with her body power  prosthesis you can see in here she can maybe hold   the pan steady and then uh use her left hand which  was not her dominant hand before her amputation   in order to stir things around and move  things in the pot she wants a prosthesis   maybe not necessarily to do the other half of  the meat i don't know i don't know how much   how waterproof or meatball proof it would be  but um she wants something that she can handle   a utensil with she wants a prosthesis that she can  tell when she has a solid grip on something she   wants a prosthesis that she enjoys using more than  the one that she has and if we aren't answering   questions about how to get her to that place  then we're asking the wrong questions thank you last night when he was writing his slides i  saw just one empty slide that said one plus   one equals and then nothing else so i'm  glad that you brought it back full circle   our next researcher is the newly minted  doctor brad sonde from the autonomous   robotic manipulation lab brad defended his  thesis belief representations for planning   with contact feedback last friday and graciously  agreed to speak with us again today before he   leaves for california at the end of the  month please welcome brad to the stage this is audrey's garden audrey is my friend and  relative audrey is in a wheelchair and cannot   walk and has limited use of her arms she can  plan out exactly where she wants everything to   go she can order the seeds but she has to rely  on others to actually do the physical planting   i would view it as a huge success of my career  if i could build a robot that could allow her   to independently do the gardening when i think  about what technology would allow this and what   i've asked her you know i want it to be a robot  but the answer is not a robot at least not first   what audrey really wants is to have her arms back  and that that you know it makes tons of sense   but it's actually taken a long time for  me and to internalize as a roboticist   is that i cannot bring the ideal solution um  because i i can't do surgery um but you know   it it's a starting point for  what are the potential solutions   the next realization is that it doesn't  necessarily need to be complicated   the best tool that she has now is a five dollar  gripper that allows her to extend her reach   these are a far cry from actually having  your hands back and being able to do   full manipulation but it's very simple and if  it breaks you can just order another one as   we move more toward robotics you can have robot  arms and the most common type are teleoperation   these allow much wider range of activities but  still can be hard you know slow and pretty dumb what i want the solution to be and what i've what  i've worked for is autonomous systems systems that   are able to perceive the world and understand  their environment now these will not be the only   solution and often these are far more complex  and expensive than than the easier solutions   but uh i think in order to in order to give  certain functionality back we need systems   that are less dumb the hardware you know we can  always want better hardware but the hardware is   mostly there i want it to be cheaper more  reliable but here's a system that i've   built or i am tele operating a robot in virtual  reality so this is me i have a vive headset on   i'm controlling the arms by grabbing the grippers  on the side and i'm viewing exactly what the robot   sees through the camera so i've obviously set this  up to be possible i don't think i could do this in   a regular kitchen but i've set this up and you can  see things are not going exactly according to plan   i'm constantly correcting right i wanted to just  slide in there and get this pancake um and it's   it's not working i need to keep fiddling around  until uh you know move things around and uh this   is a a bit scary i don't know if okay so that  that that actually works pretty well um i was   able to flip that pancake you know never having  done it before this is this this is the first run   for other tasks i need to be kind of creative and  slow how do i actually open the syrup with these   pretty bulky you know not not that dexterous  arms and then how do i actually make the syrup   come out uh well i can just uh squeeze  the bottle and get a little dab of syrup   so all this shows that you know uh you know the  the hardware you know it's it's definitely not   human level but it is uh you know the the hardware  does much better than my algorithms can do   suppose we give our robot a pretty  simple test suppose we want to put this   drink in the fridge that's probably what's  going to happen right this happens all the   time is the robot just you know collides with  something that we don't expect fortunately   now we have pretty pretty sophisticated pretty  good robots that can sense this collision and   actually stop before it like knocks over the  fridge or destroys the can or destroys itself but now we need some intelligence  because if if we don't if we don't   do anything more then a person needs to  come and reset the robot and figure out   what the robot need uh you know how to how to  do the task differently in order to succeed   a lot of my research is focused on what  do we actually learn from this contact   so one thing we learned as we learned this this  patch of red well we probably can't enter that   patch of red uh based on the joint torque  feedback we know that there's some region   that there's probably a collision all this blue  area we've learned is free the robot's already   gone through it so we know that's safe to move  as we bump around and apply more we learn more   information about where contact could be in the  world and then we can construct plans that go   through more of the blue and avoid more of the  red until we can eventually uh achieve our goal and put our drink in the fridge this has a pretty uh a pretty dumb prior the  robot doesn't know anything about the world so   we can maybe um we can maybe improve it a bit  if we know what the shapes are in the world   so here uh we have this table and on the on  the right or on your left you can see the   robot bumping around it knows that there's a table  in the world that knows that there's the surface   it doesn't know exactly where it  is so it's constructing the same   contact patches as before but it also it has a  belief of where the table is as the robot moves   around it can update this belief and at first you  see has a very very wide understanding it doesn't   really know where the table is as it bumps around  this belief can collapse and you can you can then   do more and more autonomous tasks because you have  a better and better understanding of the world   we can now take this one step further and instead  of knowing exactly what the what the shape in the   world looks like what if we just know kind of  some vague things you know we we know a little   something about shapes in this case learned from  a neural network so here we see the side of a mug   and we know that a mug has a handle but we don't  know exactly where it is at first after the robot   makes contact we want to be able to update these  shapes so that any shape you produce has to have   a mug handle or at least something that explains  that contact i can explain that now with a uh   a method like just completing boxes so we see  the front of a box there's noise so it's not a   perfect surface there's some true box and then as  we um we can envision a bunch of different boxes   that match that visual information but we don't  necessarily know the depth as the robot moves   through the world we sweep out from some free  space and most of the boxes this won't really   affect because it's not really intersecting but  for some we can define a loss function and create   new shapes that have to obey these constraints  similarly once we make contact we learn this   contact patch and there has to be some some part  of the box needs to be in that contact patch   in order to explain that contact we can up  we can using our network we can create new   shapes now that both agree with the visual  information and this new contact information   as we update these shapes maybe we'll create  some that don't obey this contact information   and we throw those out because those those don't  agree with our observations that the robots made the last crucial part is that we need to make  sure these shapes stay consistent with the visual   information if we only use the contact information  we might create shapes that yeah they might   obey the contact information but they're just  they're not consistent with what we actually see   applying this on the real robot we need to combine  it with more more intelligence here we need to uh   see the scene but then be able to segment  out the the different objects and from just   the segmentation you you can see how how bad the  pure camera information is it's missing patches   it's kind of creating points that don't exactly  correspond to the box but we can still create   create these shapes that maybe you know some were  kind of close and many are fairly different than   the real shape but once the robot actually  makes contact we create this contact patch   and update our shapes to then uh to then explain  this contact these don't necessarily look exactly   like the uh the the box in the real world but  they have very similar occupancy and this allows   the robot to reason about possible ways it could  move in the future so that hopefully it will uh   you know to allow it to accomplish tasks based  on the information that's learned from contact   we can do the same thing with a pitcher so this uh  robot has seen the pictures before so it's able to   have a pretty good first guess but you know this  the the the picture that it completes is different   than the true picture that it in the world  and it learns that when it makes this contact future updates future samples now  need to explain this contact patch when i think about the space of robotic problems  and the problems that we're trying to solve often   uh i think many of us including myself like  to write papers that push the robots into   the brilliant stage if you think of like deep  blue or go these these are these are algorithms   that achieve superhuman performance  i think though there's a lot of room   just to make robots less  dumb i mean it's it's really   robots make so many mistakes that  once you look at the it's it's they're   embarrassing to show because it seems so obvious  that the robot should be able to overcome it part   of this i think is an engineering challenge but  part of it i think is really real research it's   not just iteration we need we need new ideas about  how do we overcome these embarrassingly easy traps   so as i look forward uh i cannot report success  yet i do not have a robot that can that can   really aid someone in gardening yet and i by  no means want to uh imply that i i have uh you   know have have the answers for how we should build  assistive robots um i look forward to discussion   from uh from everyone in order to help help  come with ideas to to better build these robots all right thanks brad and congratulations  again we'll be sad to see you go   uh our final speaker for today is professor  ben kuipers professor kuipers has been at   the university of michigan since 2009  and leads the intelligent robotics lab   he is a founding member of the robotics  institute and his research center's trust   between a robotic agent and its human  counterparts please welcome dr kuipers i got it done thank you very much i really enjoyed your  talk this morning so this was great um   let's see here how do i get this to go forward  ah there we go um i'd like to tell you about   some work that we've been doing for a while on a  robotic wheelchair and here it is with the lovely   peter gaskell driving um so about oh 20 some odd  years ago i was teaching a robotics class and we   were making little lego robots and i i figured you  know we've got to have a guest lecturer who can   tell us what the real world is like and so what  we got was a friend of a friend who was paraplegic   and in a powered wheelchair who used a power  wheelchair and we said listen we're roboticists   we work with these machines how can we help what  can we do that actually makes a difference and   what he said was every time i rent an apartment i  lose my damage deposit because we ding the walls   we ding the furniture we bump into all sorts of  things and it's really terrible make a wheelchair   that won't do that and so i thought whoa maybe we  can and that sort of blew up into a lot more stuff   so um vulcan is an intelligent robot and it builds  a cognitive map it explores an environment and it   learns where it can go and where it can't go the  first question we want to think about is who's in   charge where's the autonomy who has the autonomy  people talk about autonomous robots who wants an   autonomous robot we want a person to be autonomous  what you said this morning or earlier about your   wheelchair is that it gave you autonomy it wasn't  that the robot that the wheelchair is autonomous   so but the person can delegate certain  chores to the robot and so here are two   somewhat different models of delegation for a  robot that has a um a cognitive map you can say   take me to the front door and what i'm doing is  i'm delegating a couple of things one is safe   navigation but the other is making the plan and  carrying it out or i can say take the next right   and then a left after that and now what i've  done is i haven't delegated how to get there i've   maintained the control over where i'm going but  i've still delegated the um the safe navigation   now if you're driving with a joystick  which you do all the time then you're   not delegating the safe navigation either  you're saying i'm going to do the navigation   now as we'll see in a minute not everybody  can handle a joystick um and so they may need   to delegate that and be able to work at some of  these levels now i want to show you a picture here   now okay here is our wheelchair and it's  it's going through a classroom building at   class change time this is showing off what it  sees it's showing it's seeing the pedestrians   it's planning various routes in a moment i think  it'll get to a point where it can't go anywhere   and so it just stops see everything's red and it  waits for something to open up and it keeps going now this is vulcan and it's demonstrating  that it can move in a situation where it   has incomplete knowledge and the world is dynamic  and it's not completely known because pedestrians   who knows what pedestrians are going to do  they're going to do something crazy mostly now we have a theory about how we do  this and i'm not going to go through   it although i could spend an hour on that  easily but basically we distinguish between   small scale space which is space i can see and  i can explore it by moving my gaze versus large   scale space which is space that i have to  travel around in and i have to knit together   the different things that i see in order to make  a map people mostly think of mapping as metrical   that you build a model that actually is a scale  model of the real world but in fact most of the   time what you actually want is a topological map  you want a network that says what can i do here   well i can go down those stairs or i can go there  and turn right those are my choices if i go here   it's going to be a bad thing so it'd be worse for  you so um and we can put we can have metrical and   topological maps of small-scale space and  large scale space and we put all of those   things together and so we call this the spatial  semantic hierarchy and it is ever so wonderful   but it has taken many years to get as far as we  have and there's a lot more progress to happen now one of the cool things about these  different representations is that they let   you say different things so in small scale  metrical mapping i can say go over there   and then use the knowledge to figure out how if i want to say turn right or turn left i'm  using the local topological map but if i want to   say go to the front door or the kitchen or the  doctor's office i can use this large scale map   now in fact people use paper maps all the time  because in your head you mostly don't have   large-scale metrical knowledge some  people do some a lot of people don't so who is vulcan for well i would love it to be  for people with disabilities and it's intended to   be for people with disabilities but it isn't ready  it's a research platform and so we do research on   cognitive mapping on perception on control on  on avoiding collisions and all sorts of things   but what we're hoping for now clearly it's  people with mobility disabilities that   need mobility and that's why you have a wheelchair  but if you've also got perceptual disabilities   i once watched a young lady who was  both blind and needed a power wheelchair the bravery of crossing a street is amazing  and so you would want more capabilities for a wheelchair to be able to see various hazards  and avoid them communication disabilities now you   communicate with the joystick if it turns out  you have substantial tremor then you can't use   a joystick and so you've got a problem cognitive  disabilities plenty more to be said about that too so in the long run there are possible applications  for this kind of intelligent robotic technology to   help a whole variety of different kinds of people  um now so this sounds like a great thing is it   all completely a great thing or is there  drawbacks here now one issue is de-skilling if it turns out when you work hard to  practice a skill that keeps it going   if it turns out that there's a way to sit back and  let the machine do the work then you can lose the   skill this is a problem with autopilots on jets  as well as wheelchairs um one of the scenarios for   the for the intelligent wheelchair is for elderly  people who might be stuck basically bed bound   and if they could have an intelligent wheelchair  they could go visit their friends and being mobile   and being able to autonomously visit your friend  by saying take me to jane's house or jane's room   then that makes a big difference but if it's  de-skilling where does that trade off i don't know   you may know more about that than i do but  it's a big deal um there's all sorts of   technical and ethical problems that come up when  you think about ai and robotics and that's a thing   that i want to spend some more time on what if  the robot now we're talking here about intelligent   robots so it's perceiving the world it's building  a model of what it thinks is out there and it's   deciding what it thinks is the best thing to  do what if it's wrong what if it hurts somebody   what if it hurts somebody out there what if it  hurts the person in the wheelchair there's there's   a real issue here that weighs about 350 pounds if  it falls over and you're in it it's a big problem um autonomy i've made a big deal out of the  fact that the autonomy belongs to the human   who can under some circumstances delegate  aspects of that to the robot what if the   human wants to do something dangerous even  really dangerous should the robot stop it   can it stop it the only model that i know of right  now is that there is a philosophy in the mobility   and orientation community for blind people  seeing eye dogs a seeing eye dog can prevent   its person from crossing the street it gets in  the way but it is not authorized and it doesn't   have a way to communicate that it's okay to  cross the street that's the human's decision so there are some of these questions that come up  as you're you're sitting there saying well what's   my robot supposed to do and then as you think  about it you say this is actually a deep question   and when you look around you find that  there's people in the mobility and orientation   community who have actually thought about  that deep problem we don't necessarily have   people who thought about all of them but  there's a whole bunch of interesting things so here are some research problems  there's an ai problem what is space   how do you observe it how do you learn the  structure of space there's a robotics problem   this is not just an ai this is a physical  object that perceives the world and acts in   it how does it do that how well does it have to  perceive the world how well does it have to act   there's a human robot interaction problem  we've got a robot now robots don't perceive   the world anywhere close to as well as you think  people typically perceive the world a lot better   um here we have two agents the human and  the robot and each of them has perceptions   each of them has knowledge each of them has some  portion of the autonomy how do they work together   how do they communicate with each other this is  tough this is not an easy problem by any means   and the assistive technology problem what  do people with disabilities want and need   we can sit back in our chair and  say oh i'll bet they need this   but a far better approach is to ask them  and find out so i think this is one of the   most well we've written a bunch of things you  should read them um this is a fantastically   interesting research question you guys should  work on it it's been lots of fun for me thanks thank you professor kuipers for leading  us right into our question section with   some nice open-ended questions for  everyone um so at this time i'd like   to invite all of our speakers sort  of back up to the stage for our panel yeah so my wheelchair 100 feels like a natural  part of me um so much so that like when i   am getting a new wheelchair that process happens   about every eight nine years and when i  sit in the demo chairs or my new chair   i feel like i'm wearing the wrong pair of shoes  like i i can't sit upright comfortably and my   driving uh declines significantly in terms of my  maneuverability and um that is because i become   like deeply attuned to the little aspects of my  chair and how it handles and you know my drift   set provides a very kind of delicate level of  maneuverability so a few millimeters has a very   drastic effect on what i'm doing and i get used to  that and any slight change is very noticeable um   the json arm the robot arm that i've been using  not so much it hasn't it hasn't felt natural yet   and i think that is kind of because there is that  lack of sensation uh that was being talked about   there's not a whole lot of feedback that i get  other than visual um for how hard i'm grabbing   or you know the area that the hand is moving in  space um so that one still feels kind of like i'm   operating a robot rather than moving something  on my own thank you well uh okay so i'd   i'd like i'd like to

2021-08-07 16:35

Other news