um yeah so samantha um yeah hope i see you thank you so much i will just uh get the slide sharing going okay i'm hoping everyone can see this slide so thank you so much tammy for the intro all right so at the end of this 30 minutes um hopefully you'll know a few more things about uh how we can augment human memory through cognition aware and ai based systems hopefully can take away some new perspectives and design insights on supporting cognitive abilities and experiences so memory is the cognitive ability that we use every day and however our capabilities will eventually decline and our adaptive nature of memory in fact uh could result in a few memory troubles uh by by 2030 like one in six of us will be 60 years or above and with this there will be new requirements and opportunities in my work i focus on a type of memory called perspective memory and we see that the majority of our memory troubles are related to prospective memory forgetting to do to do intended actions in the future so this could be forgetting to take a medication uh bringing items or forgetting to bring items when you're leaving the house and it it affects our ability to perform many daily tasks and it can affect our sense of independence so in this talk i will go through four directions that uh that i took to to augment perspective memory so one of them yeah so the first one enhancing memory through memory training and and and then the next one encouraging receptivity to memory training third one encouraging receptivity to memory assistance and lastly understanding opportunities for memory assistance so just to give you a visual a more visual overview uh so my vision is to kind of use this biosignals like a heart rate variability skin conductance eye movements to infer a few of our cognitive effective context and so such as you know where you come or maybe our intention is to recall something and we use this to inform when and how to give memory training and memory assistance and hopefully all these can be combined together to to foster this next generation of memory augmentation technologies and so the first area i looked at was in enhancing memory through memory training and we use this technique called the when then technique or some people know it as implementation intentions and it's known to strengthen perspective memory right so we use this um we use this technique that involves two steps and so the first step is uh to formulate and verbalize this thing called when then sentence so here's an example uh when i leave home after breakfast tomorrow then i will bring the documents uh so this helps you to form stronger associations between uh the cues they look out for so when i leave home after breakfast tomorrow and then the task that you intend to do so bringing the documents the second step was is to visualize yourself actually performing this uh this action and seeing the cues and performing your tasks and so in a very initial step of my phd um i i uh we developed like sort of an app that sort of facilitates digitally mediates this technique so just a just kind of a video screen screenshot of this uh it's a very basic you know text typing and you know we try and formulate the sentence ask them to formulate the sentence and then uh take some time to visualize uh this intention and just with this very basic in a sense implementation and facilitation of the technique where we can see initial studies show that users are more on time and completing a few of the memory tasks that we gave and even in terms of like the feedback that we had from a few of the users here for example this lady felt that it was actually helping her remember and she was using it to adapting these techniques with other techniques in her life to to remember things and and so another example would be uh this uh this user who um who who felt that it coached him to form stronger associations uh for remembering what he had to do uh so with this we kind of wanted to build more into this idea and we wanted to see if we can encourage uh the receptivity through this memory training and here we added the idea of using cognitive effective sensing so the idea is that okay maybe we can detect low cognitive load or maybe a calm state via these biosignals physiological signals so we use the e4 wristband device with biosignal sensors and okay and we developed a system called prompto and then for our case we try to infer this this state right through uh physiological signals of uh uh electrodemo activity eda and heart rate variability and uh this where we sort of detected this the system detected this gives you a prompt to start a memory training session and over here uh we just see that it guides you in the same when then technique implementation intentions but in the more chat based interface so let's look at the video on it when you're relaxed and potentially more receptive you'll receive a prompt to practice the memory technique starting session tell me a task that you intend to do check my emails all right in what situation are you thinking about doing this later this afternoon good in what location would you need to remember this at home proctor guides you to form a random sentence when i arrive home then i will open the computer to check my emails and asks you to confirm when you visualize yourself performing your tasks at a given situation and location right um so we conducted a receptivity study on uh with older adults um and we prompted them at the low cognitive load and high cognitive load so we gave them uh four prompts per condition and uh we just kind of mapped out their responses to the prompts uh so this responses could be either start practice uh start the memory training practice to to maybe ask it to try later so it's like snoozing and uh or another could be they ignored whole thing altogether and just to break this uh complex diagram down uh more so people were started more practice so they they were more significantly more practice sessions in low quantity load compared to high cognitive load and another aspect that we found was that they were also significantly more responsive to the problems so they responded with try start practice or they responded with try later during high cognitive load compare during low cognitive load compared to high cognitive flow so uh we also did a field deployment uh of tomto over seven days uh the main idea is to identify some of the challenges to uh deploying this in a while and also uh so we prompted them about at most two two pumps a day and then uh prompts were at least five hours apart and we mainly looked at uh prompting that at a low cognitive load and here's some his overview of some of their responses in the deployment we we see that the response rate wasn't as high as you know the receptivity study which is done in the lab but uh we still do see uh that there were they were more likely to respond to prompts and start practice sessions and we also see that most users appreciated the concept and found prompto helpful in the third direction i looked at encouraging receptivity to memory systems so the idea behind this is that okay now can we investigate how can we do so by by sort of using cognitive influence and curing so i'll give you more explanation about this so after the video rook is in kin voices where we explored the voices of friends and family in voice interfaces [Music] from our surveys and interviews the voices of friends and family were perceived as having higher co-presence social presence and telepresence they were more persuasive credible and charismatic and more likable safe and easier compared to common virtual assistant voices as an example application we design a technology pro kin voice it is a voice interface that uses ai voice cloning tools to synthesize reminders in the voices of a friend or relative users tested kim voice in their homes for two weeks the western entrance was built between 1275 and 1281 alexa ask family voice to pay the reminder this is for your reminder at 255's is that my family we derive implications for voice interfaces and design to rethink our notions of voice realism using kim voices for personalized tasks mixing the use of synthetic and real voices and building on the attributes of king voices for more details please refer to our paper [Music] so just kind of looking back at the sort of the way that people perceived these skin voices uh we kind of compared that with existing uh google assistant voices so we compared voices of friends and family with that google assistant voices and we see that they found that these skin voices had voices and family had you know more connection co-presence a social presence so they're more engaging and telepresence feel feel to them and then they also felt a bit more persuasive credible charismatic they're also more likeable and felt safer more comforting and then the last the last thing that we thought was quite interesting was that it actually felt more eerie than uh google system voices and one of our one of our discussions or one of the things that we think could be happening here is because people don't expect voice interfaces to have uh voices of your friends and family so that might be a possible explanation and in terms of our technology group we we use ai's synthesized voice reminders so we use this real-time voice cloning tool to actually uh you know replicate your sample you know of of uh your family member's voice and uh we kind of tested it out and over here we we gave four users uh so in three houses uh over two weeks so here uh we have two sisters in different households and we have a couple living together so more details and you know after like co-design and everything were in the paper so you can check it out um but i i really want to i guess this is a great avenue to also discuss a lot of these um ideas and also you know some of the questions and some of the design considerations that we have is like you know should we should we limit this uh the realism um a voice realism in a sense so maybe we need to balance this naturalness and non-naturalness uh this has been quite an argument or quite a quite a debate uh amongst voice designers um over the past few years so if it's too real right it's it's very prone to impersonation attacks and it's too if it's too unfamiliar you know that this people don't recognize the voice and there's no there's not much positive benefits another thing that uh we want to look at is another thing that we can consider is matching uh the role uh the role of the the kin uh with the content of the interactions uh so for example if my uh yeah for the sisters example uh you know one of the the elder sister uh would always encourage the the younger sister to to exercise uh she will call you know she will call her up uh on her own uh maybe every every uh you know three times a week um but let's say if we can uh match if this voice interfaces actually does match uh this role that uh the big sister has already played right this this makes it much better and uh and much more personalized so we do want to see a bit more of customization aspect to that and another thing that we we saw was that okay can we actually mix the use of of real uh voices so maybe when uh you record your message uh in a voice right and can we also use it with this ar generated voices so real voices you know you had to took the time to record it so it's more meaningful and but maybe synthetic voices it's good because it you can generate dynamic dynamic content and you can generate anything you want so maybe it would be uh good for us some cases and the last thing that we saw was that okay can we build on the attributes of all these things that we we showed before right so the the comforting the the uh kelly presence social presence of it uh can we use it for you know for very beneficial um in a sense applications that really benefit from from it so maybe therapists companions uh confidence uh notifications to shared use you know and so so many other uh applications so these were things that we we looked at the very last um aspect that that we investigated was we wanted to understand the photo opportunities for memory assistance and so in this case we looked at eye movement patterns and as well as the other biosignal so we also use the skin conductance and electro dermal activity and heart rate variability and we wanted to see the difference in patterns between okay seeing a familiar image versus an unfamiliar image this project is still ongoing uh but the whole idea behind it is that okay can we modulate the amount and the type of memory assistance that's given so if i see an unfamiliar image i can detect it through the system can detect it and then it will give you more memory assistance so if it's a familiar image then maybe it will give you less information so there's uh something that we are still i'm still exploring uh hopefully you know um still writing up in a sense because i we are still analyzing the data but now um we kind of kind of made a system that uh it's more of a wearable heads-up format and i'm combining all these things that i have uh you know learned from the cognitive contact sensing and also uh giving implicit uh influence and queueing uh yeah so this is something that's still ongoing uh and will be finished soon um and overall see all these systems uh are small steps towards this combined version of um giving sort of a memory next generation of this memory augmentation uh and i would really love to to you know hear some of your comments here and some of your ideas here as well so just to wrap up a lot of these um things the some of the future opportunities we see is that maybe since uh memory or you know a lot of these cognitive abilities are um in that sense uh yeah their cognitive abilities right uh so can we use a more in a sense cognitive approach can we combine cognitive understanding of the user or an understanding of the user with a more cognitive influence and cueing and the other thing that uh we could sort of consider was is that okay in terms of like the human computer synergy i think uh if i if i recall i think uh there was uh i've gotten the name but um there was a paper uh quite long ago uh i will search it up later but uh he he sort of uh pitched the idea of a human copyright synergy in terms of memory and part of it is actually giving uh sort of working with our internal memory so if we can combine memory training and memory systems maybe uh that's also a very interesting opportunity to explore uh we'd love to discuss more and so in this decade of healthy aging i think this it was very recently dubbed by the world health organization i hope that we can really continue to further augment memory and cognition uh i just realized that it's i did this really quite fast so i'm i will be prepared to you know go back to a few slides to answer some of the questions um so i'll leave you with my contact you can feel free to reach out and for now i am actually ready to take some of the questions yeah i'll head it over to tammy
2022-01-29