panel 3 emerging technologies haptics and screen based approaches image a black and white drawing of a finger tapping a touch screen and a person wearing opaque eye shades and headphones text emerging technologies panel moderator joshua miley panelists darren dufresne sheila omodrain aaron rodriguez matthew schifrin ed summers hello everybody and welcome back from lunch for those of you joining just now um thank you for coming so um we are kicking off our afternoon session and in case you are just joining us today um just wanted to go over some uh access reminders um so we have a few access notes you can turn on captions by clicking the cc button but for more personalizable captions you can also visit tinyurl.com forward slash lawnmower captions for better access we have turned off the chat today however we encourage you to use the q a box as if it's a chat room so in the q a box you can pose questions for our panelists you can also share resources and reactions and connect with other attendees additionally we have a facebook group started um to allow for additional conversations that can continue after today's symposium is over um if you'd like to check the reminder link we sent um you can find the link to join that group now um our um handy student assistant nathan has also posted the links in the chat window to the facebook group and to the attendee directly uh directory and then finally we have a hashtag for this conversation for anybody who is live tweeting or communicating on social media that hashtag is hashtag comics a11y and now we have our next session we have some panelists and moderator and because this is a larger session a larger panel um panelists i'm going to ask that you keep your videos off or if you like i can turn them off for you and um dr joshua miley since you are monitoring the panel um i invite you to keep your video on um for the remaining panel so um you guys this is the emerging technologies haptics and screen based approaches to accessible comics my name is ting soo and for those who would like a description i am an asian american female i have short chin length hair and as i mentioned this morning i tend to smile a lot especially when i get nervous which is when i'm speaking publicly um so um i am really honored and have the pleasure of introducing this panel um as a teacher of blind and low vision students um i have seen how it is a privilege for you know typically cited individuals to say no to technology but technology can play such a key role for our blind and low vision students and adults in ensuring equity and leadership opportunities in the classroom in the workplace and just in the community um so so far in the symposium we've talked about audio approaches tactile approaches and i'm sure some of you have um are really looking forward to look at what what are the multimodal opportunities for engagement with information um so with that i would like to introduce this panel's moderator dr joshua mealy josh mealy is a friend colleague also known as a mad scientist and technology innovator um for those of you who were on this morning josh is also the inventor of you describe so josh take it away hey ting dr sue thank you so much um i didn't have a chance to do a mic check or a video check so how am i how am i looking um everything everything okay with my video tang yep everything looks good all right so um this is going to be super cool and what we're going to try to do today is have more of a conversation about sort of possibilities for advanced technologies and comics access obviously um comics are just one of many media that may benefit from uh some of the technologies that are emerging some of the technologies that our panelists are playing with and and um evaluating and prototyping and i think it should be remembered that technology is not just something that you build out of uh out of hardware or software but technology uh i i consider the alphabet to be a technology so um systems and uh and sort of frameworks and ways of thinking about things and ways of uh ways of organizing and displaying information i think are also uh within the realm of sort of advanced or at least um unconsidered technology so we have a great panel um we are uh i'm gonna um ask each of our awesome panelists to introduce themselves try to be try to be brief and in the introduction introducing yourself maybe just say um real quick what the technology is that you um or maybe maybe one or more technologies that you're hoping to uh bring into the conversation today i'm going to start with um uh we have a a team who has developed a thing called um vizzling it's a it's an app and so um aaron rodriguez and uh darren defrayne i'm gonna ask aaron to go first aaron is a doctoral student at um uh well you can introduce yourself uh aaron why don't you start us off and maybe just give give it a minute and then we'll switch over to darren sounds good thank you so okay there you go thank you so much josh um so quick visual description i am a bald white man uh 30 years old in a in a very messy office so the the the technology that uh darren and i are talking about today is vizzling which seeks to make comics accessible by translating text and image which we feel like is the building block of comics into sound and touch uh and we do this um by using uh a tablet basically um we're using ipads and android based systems so that that's really where if we're talking about technology it can be systems uh we're looking at uh translation as a technology great and is that um is that is there an automated component to that or is that a hand uh do uh is the the translation um done done by humans uh the translation is entirely done by humans at this point but we hope in the future that some of it can be uh can use ai to do some of that translating cool i suspect that will be part of our conversation today um so thanks and welcome erin um i love that uh aaron and darren um it's like you know talk about comics it's a little bit superhero-ish so um darren defrane um associate professor um uh please uh introduce yourself i'm not sure from the robin or the batman in this scenario uh i'm darren dufresne i am a white male somewhat older than aaron with yellow glasses in a likewise messy office in wichita kansas where i am the director of the writing program and professor of english and i'll just add yeah we're our app relies on actually combines several things that combines visual linguistics with the haptics that aaron mentioned and coding so that we'll also hopefully have all of these materials fully searchable by the time that we're done with it fascinating yeah i'm interested in hearing more about the the catalog and the way the way you're doing that but for now um let's move on to um sheila omorin um if i'm pronouncing your name right and um sheila is uh the lead on a number of cool projects including uh what has been what has been sort of colloquially dubbed the holy braille a uh a tool for uh tactile tactile graphics and braille representation sheila you want to tell us about yourself sure yes i don't know in terms of age how i compare with these other people uh oh i need to start with it sorry um there we go you're right yes so um i have curly hair that's sort of blondish uh i'm sitting in a dining room in ireland in the sort of latest evening but uh yes here for a couple of weeks so the project i've been working on as as uh josh says the holy braille and what we're trying to do is to use pneumatic techniques so air or fluid to move dots up and down so that we can prep we can basically pack lots and lots of dots together to make a full-page tactile array and i'm visually impaired myself so working on this project has been a real privilege for the what is it six or seven years we've been doing it now so yes that's what we've been up to thank you sheila and um and i'll just mention that that um the idea of that sort of dot array is to be able to display sort of a um uh tactile graphics and braille in such a way that um you know sort of like a dot matrix uh you know a highly pixelated uh image for for for sighted people's reference right yes want to add anything to that okay um uh and okay great um and you're also you want to say you're a professor of something somewhere right oh yeah i hold a joint position um in the school of music where i teach in the performing arts technology program and in the school of information where i tend to do some more sort of hci work although my music role is really hci for digital musical instruments so it's not as weird as it sounds cool um super psyched to have you on this panel um matthew schifrin um you have developed a um an accessible lego instructions uh system for uh for blind people to use uh you wanna tell us about that and introduce yourself oh there you are okay i am now unmuted uh so i'm a blind guy in his early 20s and i've created a system of text-based building instructions that lets blind people build lego sets on their own and if you guys would like to check that out you can visit legoftheblind.com but the project that i'll be discussing today is project daredevil which is a motion simulating helmet that uses gyroscopes to affect your vestibular system and the vestibular system is what controls your sense of balance and as a result wearers of this helmet will be able to experience different motions like falling or flying things that superheroes experience in comic books that's super awesome so uh a a virtual reality uh mode that actually goes beyond hearing and touch and and even sort of haptics um so that's very cool and i'm looking forward to hearing more about that and finally last but definitely not least we have um mr ed summers from he's the director of accessibility at sas a big data big company um and uh you are the co-author of reach for the stars an accessible ebook on uh astronomy um ed you wanna introduce yourself and talk about what what comics what comic technology you uh you've been thinking about yeah hey there josh can you hear me hey ed yeah i can't okay tink i don't i'm not sure think we're supposed to have our videos on or off i think i turned mine on i don't know who cares hey um before i introduce myself i do want to say thank you to um to you josh ting amazon and san francisco state for the leadership in this area and and sponsoring this event thanks yeah um at summers i'm i am blind as well i'm middle aged uh white male and i'm sitting in a very bland looking uh home office right now that is kind of in the process of being remodeled so there's not much behind me except for blank walls i uh yep i'm director of accessibility at sas the market leader in analytics the big data analytics company and what we focus on from an accessibility perspective is making data and data visualization in particular accessible to everyone of course and and particularly people with visual impairments because that data visualization part is a you know particularly difficult problem and a huge barrier so yep reach for the stars was i think probably one of the first um born accessible textbooks of any type but this was particularly for astronomy it was uh sported multimodal all kinds of multimodal features and more recently we published a free tool called sas graphics accelerator which enables people of all abilities to view data visualizations using a wide set of modalities which includes sonification awesome and sonification i think is um you know an interesting uh potential uh tool for uh for uh representing or you you know for comics access it'd be interesting to uh talk about that there's a couple of terms that i want to make sure we define before we uh before we go on too much um i think we'll be using a couple of different uh different terms that i want to make sure everybody's familiar with so um uh and and so that i'm not the one doing the talking um matthew i i wonder if you could just tell us what um uh tell us what haptics is and um and what um how that compares to well tell us what haptics is first of all haptics is a process by which people can receive information for example your phone vibrating that's haptic feedback it's feedback that you can touch or feel and in some cases it's done using motors like in your phone or one might argue that reading braille is a haptic process because the pins are moving under your fingers as your fingers move across them um awesome thank you and um and sort of the you know i think uh haptics you know there are things like haptic joysticks that have forced feedback and um you know things like that so um i think haptics is just uh a fairly fairly common at this point technology but i think it's one of the one of the things that will come up a couple times probably sheila i wonder if you can um tell us what the difference is between haptics and tactiles so um yes so typically when we think of tactiles we think of using our fingers to feel small features on a surface whereas haptics is a bit more general it has to do with basically the kind of forces that we might interact with as we for instance pick up a cup we might just move it a little bit to see if it has any liquid in it or if you pick up something and you're trying to balance it those kind of i suppose more hidden forces are also more aligned with what with what we would think of as more general haptics so we then can think about different kinds of displays such as like tactile displays which are like the braille displays that people use um whereas haptic displays are more of the kind that we're thinking about with the the helmet display where you're you're communicating with another part of the system that feels the world but not in just through the fingertips thanks so much and then ed since you mentioned um sonification why don't you uh give us a quick quick overview of what uh what what is sonification anyway yeah so so i consider sonification to be you know the art and science of representing data using sound and i think it's i consider it to be kind of a a subclass in this broader category of auditory display um great okay so for those of you who are saying enough of that let's get to the comics um i uh i'm you know so i think that my point really is that you know all of these technologies we're looking for ways that um that comics and visual information can be communicated to a blind audience and really i think that we can all recognize that there's a combination of these these kinds of technologies that are you know that are going to be used for stimulating or interacting with non-visual sensory modalities so that's why um that's why all of these things are really really critical but it doesn't have to be terribly um fancy i mean i grew up um as a blind kid you know my friends and family would read me comics and they would you know they would do it in a very descriptive fashion we had a great uh panel earlier today on uh on description and how description works but um you know that's uh you know when you're when you're sitting at the breakfast table with somebody reading you the funnies that's um that's you know that's not exactly on demand that's not exactly independent and so i think by uh bringing in technology a big hope is to foster that independence and um and sort of uh ability of somebody to um to make the choices that they that they want around the the way this information gets delivered so um i'm gonna return to um to darren and aaron and um ask you guys maybe to tag team a little bit on talking about um the way somebody how does somebody use your um uh your vizzling app to read a comic like what describe the if you will describe the um the experience for us um so i actually have a uh i made a video of the the prototype so josh would it be okay if we showed that right now i did you give it did you give it to ting i did yeah it's it's right yeah seconds long and i have a a brief description of the video for everybody uh and i'll read that description right now that's it okay i don't think that's me um so so why don't you go ahead and read the description and then ting can roll the video sounds good so um this video is 56 seconds long it's filmed from the point of view of the user in three different scenes the video depicts the three reading modes in the vizzling app each scene begins with a white rectangle with the words vizzling in black capital letters and the name of the reading mode in a white rectangle at the bottom of the screen in each scene a samsung s6 light tablet rests in the left hand of the user who is resting her arm on the table she is using her right index finger to navigate the program the first scene depicts the global narrative mode this mode outputs audio of the general page level narrative this mode is triggered by continuously touching the play button in the bottom left corner of the screen the second scene depicts the narrative grammar mode this mode outputs audio of panel specific events this mode is triggered by dragging your finger across the page when the reader is dragging the finger in the correct order a single vibration will pulse when the user's finger goes over a gutter if the reader is dragging their finger in incorrect order a double vibration will pulse the third and final scene depicts the free exploration mode this mode outputs audio of specific objects at specific coordinates this mode is triggered by touching individual coordinates on the screen and not dragging the finger when the user touches an object on the screen the app will announce what the object is the description is longer than the video thank you global narrative mode page one beneath the glowing word vizzling we mean aaron narrative grammar mode panel to panel panel one in the upper left corner panel two comic figures are addressing the reader on panel two upper right aaron is addressing the reader while holding his tablet in panel 4 middle right a warning sign panel 2 upper right aaron is addressing panel 4 middle right panel 3 middle left we see the panel neatly divided into 4 sections with panel four middle right free exploration mode object coordinates darin in a circle cupboard door speech bubble cupboard drawer um so okay great thank you uh thanks ting thanks aaron um uh maybe um real quick tell us about how you expect like what what kinds of what how do you expect people to use it like we you showed us the mechanics um from uh from a user's perspective from a from a person who wants to read a comic um what what would you expect them to to how would you expect them to proceed um so i think darren and i were both teachers and we both had students in our class who were blind or visually impaired and we recognized that the needs of the reader are different in every purpose and so we wanted to create a product that was sensitive to those needs so based on you know various situations if you're reading the comments for pleasure you might not care about all the small details however if you're studying in the classroom and you have a very a page that you are analyzing with everyone in class this translation allows you to touch certain coordinates and understand how the page layout how the page is laid out thanks so much darren do you want to add anything to this yeah we also were concerned about um the spectrum um that uh uh people's vision lie on um anywhere from from no vision obviously to to full vision and so um someone in my age has a lot of trouble reading smaller text and so when they're fixed free exploration mode that allows you to just touch that and get a little bit of feedback or if you're trying to navigate like aaron described the miz and page then that allows you to kind of explore that on the page and get a sense of what the layout of the page is intended to be great um and uh so i guess um i wanted to i wanted to use this as a as an example of sort of a a technology that's being you know that you guys are currently developing to provide access to access to comics for blind people by the way um i actually don't know the answer to this do either or both of you uh self-identify as blind or visually impaired we do not know okay cool just just curious um so i want to just throw this out to the the group and i think i'm gonna um uh maybe i'll i'll ask ed to lead it off and um uh in the answers but i'd like each of you to sort of um uh sort of fill in the gaps that are left by the ones before you and hopefully we'll get a big picture but what does accessibility mean in the context of comics i mean in other words um uh what are the requirements for a comic to be accessible like what what are the things we expect and and i just want to sort of build that up so that um as we talk a little bit more about the technologies we can see what questions those technologies are answering so ed just real quick what do you see as being um what what are what are some of the requirements for a comic to be to be called accessible for blind people yeah sure well so i think of accessibility in general as the absence of barriers and and it's the absence of barriers to you know great many things um you know either the information in the comic uh you know like like literally what's the storyline who are the characters what are the characters intent what is the conflict how does it get resolved you know all those kinds of those those things that you enjoy about comics but i would also say that um another important part of that are the um you know the social dynamics of things of all things but including comics right so the fact that you can talk about the latest issue with your with your buds or you know your friends online or whatever it may be and kind of have that that shared social experience and you know and plug into that so i'll leave it at that and just to keep it short and flip it represent next person like fill the details yeah hey matthew why don't you why don't you weigh in on this what is what is what are the requirements for accessibility of a comic book what would what were or any comic what would what would a um what would a fully accessible comic for blind people need to have in order to be called accessible for me a fully accessible comic would need to have not only the kind of essence of the story and the characters but also an understanding of the environments that they're in and i've often gotten this information through comic book scripts but because these scripts are very hard to come by and they change from kind of the initial script to what the author eventually releases i think that knowing kind of what vizzling is doing with describing the various aspects of the images that a sighted person sees i think it's important not only to understand who does what but why and how and where they do it as well great um sheila what do you think um i guess oh gosh sorry uh i just lost audio there for a little bit do you want me to come back to you yeah can you just uh repeat that last yeah i mean the question is the question is what are the requirements for a comic to be accessible what do we mean when we say absolutely accessible comic well i think um i was thinking about this uh earlier on too i think first of all the narrative has to be clear i think somebody brought this up but also i think you have to have the opportunity to discover the narrative as well it's it's like a part of the enjoyment i think of a comic is connecting the dots yourself um is sort of seeing how the frames are there and then sort of getting the thrill of getting the joke does that make sense absolutely yeah and i think being able to do that on unassisted i think is a really important part um and um darren what what what's your what's your take on on sort of the requirements of an accessible comic so as a sighted person when i open up a comic book one of the things that strikes me right away is that there's a lot of competition for my attention and i think for a comic to really be fully accessible it should allow the user to have access to all those different cultural visual visual layout and narrative elements that are important to a comic make a comic fundamentally different than an alphabetic text great um and uh finally uh aaron any any any holes you see left by your colleagues um i think at the bare minimum accessibility means allowing the the reader to ask questions about the text and being able to find the answers to those questions um themselves yeah okay so so i think um we've we've laid out sort of what the um what does accessibility mean in the context of comics and i think i'm interested in that i didn't hear anybody mention i guess um darren sort of touched on it but i mean i uh i wonder also i wanted to add um that uh comics part of the beauty of comics is the visual aesthetic that each artist brings to the page and um and sort of the the ability to communicate that aesthetic i think has to be has to be part of it um when it's when it's important and i i think that one of the you know one of the key requirements would probably be um the ability to highlight what's remarkable about a particular comic like you don't really want to be talking about the comics aesthetic all the time when that artist or that comic is not particularly invested in the aesthetic but you you know but if they are invested in something else you want to be able to focus on that so sort of knowing what the what the crucial uh communication elements are is is really important i i also um one of the other things that i have always really been fascinated by is the um the visual language of comics the um the the vocabulary that's used for communicating speed and smell and uh and um emotion uh there's just there's a great deal that goes into it and um i think that there's a lot to be thought about in terms of um how important is it to communicate the raw part of the of the visual aspects of that uh of that language and how important is it to uh to simply communicate what that language is trying to say so i'm not gonna draw any conclusions about it today but um but i will just kind of mention those as things that i wanna um i wanna get into so um i'm gonna ask uh sheila i'm gonna ask you a question about um so some of the folks listening who were watching um who may or may not be familiar with uh with blindness or uh tactile representations why can't we just turn the comics into uh sort of touchable images and be done with it wow well it's it's a you know it's something i think that anybody coming to creating tactiles for the first time you know it's a it's a very valid thing to think why can't you just make a tactile version of the picture and i know that a number of people on the panel including josh have looked at this in great detail but there's a lot to do with what are things that our tactile system is aware of um and how those differ from the things that our visual system is aware of and it turns out that the tactile system is less tolerant of things being overlapped or very close together or sort of cluttered and that one thing for instance which makes tactile images legible is leaving plenty of what we call white space or empty space around components of the image so that those components can be uniquely perceived in their own right and so that involves thinking about the tactile image in a very different way to the way that you think about the visual image and that's that's just one thing but i also wanted to sort of note that um in thinking about communicating things like emotion in and other parts of comics it's easy to think that we have to rely totally on tactiles but we also have audio and of course we all know from our childhoods listening and watching cartoons that there's a very well developed um sonic language for communicating things like the speed that something's happening at in a cartoon for instance and there's probably a lot of language that we can borrow from those um kind of that some that audio kind of lang you know developed that audio language that was developed from for cartoons that might also help us in comics so it's not just about touch but it's how can we make these things work together i think thank you um and i'll i'll just also throw in that um you know there are things like perspective that uh make um you know while a blind person can learn to interpret perspective from a line drawing uh attack to line drawing it certainly doesn't come naturally uh yeah um and it's it's a any any sort of visual perspective and a tactile representation is um just adds huge complexity and is extraordinarily difficult to interpret so um the uh it's it's really um i i really like your point about um that we're really looking at a combination of technologies so um so with that matthew why don't you tell us about how the um the helmet that you're that you're working on and the the sort of um the uh the balance the balance haptics uh um are uh how do you how do you see those as being used in um in comic comics accessibility the goal of these haptics is really to convey the nuance of motion when it comes to characters and some people are going to say oh such-and-such artist they're such a great artist because they draw figures in motion so well and our goal is to take those dynamic motions those leaps those flips the swings off of buildings and to translate them into motions that you or i can safely feel and experience while sitting in a chair super cool how's that going it's going well we're doing safety testing it's my friends at the mit media lab and i and and we're just making sure that it's safe to wear then it'll it'll be going so so when you put on this helmet and you sort of um you know execute the swing from building to building uh script does it feel like you're swinging from building the building and um and what other kinds of motions can you induce we can induce motions like falling for example kind of accelerative and decelerated functions um and falls and jumps and kind of rapid switches of directions if you're driving in a car there's a car chase and the car suddenly breaks or turns or spins then those motions can also be induced can you make it feel like you're turning left or right that's super cool i i wonder i i feel like that's going to have a lot more uh a lot of other applications as well i'm really interested in that technology and learning more about it um have you have you applied it to comics yet have you like written a demo script or anything and if you have what what have you used as your demo script we're working on that now uh so we have an audio version of the first issue of mark wade's daredevil comic and that is the kind of thing that we are adapting motions to and we're working on that now that's really cool um so i'm going to open up i'm going to ask another question and i'm going to i want to ask um uh aaron and darren to sort of talk about this a little bit maybe to start and then see if other people have comments about it but as i mentioned at the beginning i see technology as sort of having a very broad scope and um when comics you know were in their infancy in the 30s and 40s there was another uh there was another industry in its infancy and it was radio drama and i think that radio drama and comics really grew up together um i would call radio drama um one of the earliest or probably you know the earliest form of accessible comics because they really were um they were part of the same genre often and um and i wonder what is the difference between sort of uh i mean and and also um chad allen um you know um who was one of our earlier speakers spoke you know gave some examples of his uh his accessible comic that he's developed and it it strikes me that it's very much like a radio drama um so why can't we just what what's why can't we just call it you know say that accessible comics or radio dramas and that's that's that's that we're we're there already i think that removes jo i think it's a really good question i think that removes a little bit of the agency for the reader um one of the things that you do in just reading a straight alphabetic text often is you're recursive in your reading you'll you'll you'll hit on something and then wait what what set that up but i want to go back to that and so one of the things that we want to try to mimic is the ability of the reader to move back and forth and to explore different parts of the text if you're if the text is being read to you like a radio script you're on a train and you're not you're not controlling that training at all doesn't mean it's not a great experience but it's a different experience than actually engaging the text in this way so i think it sounds to me like you're saying those are a form of accessible comics but we can go further is is is what i'm getting from that yeah yeah i i i mean that's our our hope um is that we can we're not going to like actually uh make something with what we're working on that is exactly the same reading experience as a fully sighted person is going to have but we want to offer as much of that experience and the ability to engage with the comic in the way that um that's accessible for everyone i don't know if aaron wants to add on to that yeah i was going to say there's a spatiality to reading comments that you can't get in a radio drama which is occurs in a temporal setting um we're touching on some issues around um around choice and sort of flexibility and i think ed this is something that's really uh near and dear to your heart you wanna you wanna talk about sort of how this um what sort of um how choice and the the um the customizability of the experience plays into this yeah and i think this was following up as well on on sheila's point earlier about multiple multiple modalities um there's actually you know an intersection of of issues here um you know as we think about uh you know the modality of audition of hearing and and and vision right for those because there's plenty of people who are low vision versus touch and other modalities you know not everybody processes information in the same way and some people you know have even though maybe blind they haven't may have multiple disabilities so so for example a very poor tactile acuity or a loss of use of their hands or or hearing impairments as well so so i think you know it is crucially important to be able to design technologies that enable kind of the right modality for the individual user to the greatest degree possible and with that said you know to your right to your question uh josh take um music right you know you can you can listen to a piece of music or you can look at the score on paper or a braille representation of the score for example and those are really fundamentally different experiences i mean in the end you might be able to get the same information but the way that you experience the information is i think for most people um you know it was very different and then you know the last thing that that i think comes up here is is is to i think there's a huge benefit and lots of benefits for people with disabilities um if they have the independent access for the whole workflow you know the whole ecosystem of the comic right and just get back to your original question your very first question uh josh about what is accessibility and you know there's there's the accessibility of the act of reading a comment the kind of thing you would bring into a usability lab you know or a lab for an empirical study but then there's also that broader context of well how do i you know how do i choose comics and how do i kind of download them and get them on my device and how do i browse them and you know and like it's there's a broader um set of activities that go beyond just reading the actual comic and if we design accessible access technologies that put other third parties between the author of the comic and the consumer of the comment the reader with a disability so for example josh your your breakfast table you know kind of reading the having somebody read you those comics that's the third part that's kind of in your way and it prevents you from just unlimited access to information that satisfies your personal curiosities and usually those comics that i was reading at the breakfast table were the comics that the reader wanted to read right so not necessarily the ones that i wanted to read although you know i mean i mean you know i would you know take requests and stuff but if i wanted to read a whole you know a whole uh uh graphic novel i was sol right um so uh right so yeah and you're you're i was gonna say yeah you i'm just gonna i'm sorry josh yeah i was gonna say that you also might have to wait until the next day of breakfast in order to get started on that graphic novel because the reader might be busy exactly exactly sheila your thoughts um about um uh kind of do you have you have anything to add to what we've been discussing yeah there's a couple of things i mean one thing to come back to the point about multi modalities that is that um you know in the general population some people are spatial thinkers and other people are not and that's equally true of the visually impaired population they're a sub group of blind people who very much prefer to be thinking in a spatial modality and others who particularly don't like thinking like that so you know trying to cater to that but the other thing i was going to bring up is that blind people and people of low vision should be able to author their own comics too i've often wished i've had ideas and i've often said i wish i could get somebody to draw that comic uh strip you know because i think it would work really well and i don't have a means of doing it so creating authoring tools that make sense for visually impaired people but can also communicate their comedic ideas using you know that um the uh form of of comics i think is equally important and then finally the last thing is that we think of comics as satirical things but i know at least two or three people now who've published textbooks on things like signal processing and in comic form and it's frustrating because i can't access those books yeah and i think you know um sort of going back to matthew's you know i think the the whole idea of um graphical information you know um like like we talked you know i i mentioned his lego project i mean these are these are sort of instructional a lot of the time instructional uh things come in the form of comics with with you know if you will narrative and graphics and they work together to tell you how to do something whether it's a a textbook or an ikea uh manual and and um uh the idea of access to the you know the fundamentals of that information are you know that's one of the use cases for sort of comics access it's not just about uh satire it's not just about um about um you know graphic novels it's there's there's uh the the the types of genres that are going to be uh utilizing this format i think i don't need to tell this audience or only expanding um so um talking sheila about your um you know you mentioned um [Music] um oh shoot it's gone you mentioned something that i wanted to to bring back um social thinking or uh well i guess um no it was it was just i i can't remember maybe a little bit of return to us um so uh i wanna um i think this is gonna be something that sort of intersects with what a bunch of us are have been talking about and i know you and i have done a lot of talking about um structured data and how to um how we can um for all sorts of visual purposes whether it's uh bar graphs or uh scatter plots or um uh block whisker diagrams or comics how can we structure the data that's that goes into a visualization how can we create um create a a sort of scalable and and deeply sort of expandable mechanism that can be used by uh that can be used by something like vizzling or uh or um matthew's uh helmet or uh to render to render tactile simplified tactile graphics of comics or to say the cupboard doors in the upper left corner of the frame um what and and possibly even you know to use this format as a as an authoring tool to use it as uh as a representation tool what ed you've you've done a lot of deep thinking about this kind of stuff what what do you what are you imagining in that in that area the area of sort of um comics comics xml if you will there you go yeah yeah and i think what you're pointing at josh is there is a systems thinking approach to to to accessible comics or anything for that matter um in that you know if there are different different kind of components of the system agents of the system they're really going to tackle this thing at scale we need a way for publishers to publish you know representations of comics that can be that can be converted into multiple modalities um you know and it would be just beautiful just beautiful if you could produce we could define a language let's say a schema a grammar that can be used to formally formally describe a comic and then i could take that and turn it into sonification and haptics using an xbox controller which is what we're working on now matt could take that and plug that into his helmet sheila can drive her her tactile display with it and darren and aaron can can run um their application with it right and and then what that affords obviously is is getting at all these multiple modalities and different users uh with different abilities or other disabilities may choose to use any one or more of these devices or modalities because it meets their needs you know that's that's that's where i think that's how we that's one of the ways we could solve this problem at scale and uh sheila i remembered what i what i wanted to mention it was about authoring and it's that i i just wanted to sort of give you a plus one thumbs up on the desire to author uh comics a number of years ago i had this uh you know sort of semi-auto autobiographical idea of a of a tragic superhero uh comic called uh blind man and sighted girl that um that i never made because um it just wasn't uh i couldn't think of any ways to author it but um but it's uh it's absolutely something that we should be thinking about if comics are uh uh you know a mode of communication and more and more accepted how long is it before um well i'm sure you know you know how is a blind kid or student going to take a comics class um where part of the expectation is to author uh author comics you know is is is you know there are there are real um real needs it's not just sort of my uh my goofy whim um but there there probably are now and will definitely be in the future needs for blind and visually impaired people to actually author uh comics can i come back to just um something i love it yeah so thinking of the xml issue and how we might develop a language for building structures that could be rendered in multiple ways i think we've you know if those of us in computer music have been around this block before with synthesized music where we can have a score that's completely automated but the one thing that we have to be careful not to lose is that that element that josh was talking about earlier which is this idea of style or um you know the format of a comic that says this is a far side versus this yeah the aesthetic and so you know in computer music when you just automate a score and have the computer play it you lose that um artist's input and i'm thinking that if we were thinking about an xml version that would be like i suppose midi for for computer music we have to find i think a way of making sure that that gets captured as well and i i think um you know my uh it's really interesting because um from a blind perspective the aesthetic um you know it goes back to description so so often things go back to description but i um i have a very difficult time imagining how one would convert a um and a visual aesthetic and have it be meaningful for a blind uh reader other than through description um because even a tactile or an audio uh representation doesn't necessarily communicate the the the the visceral uh aesthetic that you get when you look at the way you know the way uh various comic um uh artists draw and write and and do their um render their their scenes and characters um matthew what do you think about um about this idea of a um a comics xml is that is that uh is that something your group is thinking about and if so what have you thought in terms of because a lot of xml um a lot of sort of renderings you know sheila was mentioning the the aesthetic um and i think that it's not just what's in the scene but where are things going what are they doing and your your technology has a lot to do with motion capture and um and not not the things but what the things are doing um any thoughts about how any thoughts about and comics xml and motion i think the comics xml could definitely be used theoretically to drive the helmet to tell basically in our current iteration everything is first person you are the main character and because we are combining it with a 3d sound radio drama theoretically if you wanted to switch characters or switch perspectives the soundscape could shift as could the kind of perspective from which you're viewing events and if you would like to stand still as a character swings past you and not feel their emotion but hear what that would sound like then that is definitely an option uh kind of this more more third person standpoint definitely could be achieved through sound topics going back quickly to what you were saying about authoring your own comics i think that it's very much i write musicals and so like when you write musicals you have the composer and the lyricist and often there are two separate people and it's a very collaborative process like comic book writer and artist and if you wanted to make your comic it's i think it would just be a matter of finding finding an artist writing your own scripts and it's very much about trust and i think if you were if you wanted to kind of find an artist and engage with them and kind of make sure that you could convey trust them enough so that they can give their own artistic spin to it convey their own aesthetic but you would be responsible for the story and use a comic book script to convey that story to the artist it could definitely be done and i think that that's the approach uh um georgina kliege who is one of my uh one of my favorite people um i believe she was in one of the panels earlier uh that's the approach i think she took when she uh in her writing of of of comics and um but getting back to ed's point and i i totally matthew i that's that is in fact how we would i think how we would do it today um but getting back to ed's um earlier point about choice and independence um you know uh is there a way and i think it's legit for us to consider whether or not technology can afford us independence in the authoring process uh now where we're um and a collaboration you know obviously collaboration um would forever be an option but um but what if you don't want to collaborate what if you're what if you're a power hungry control freak and can't stand the idea of giving someone else uh the the artistic license with your ideas so um so but i i hear what you're saying um and i want to just um quickly go to um darren and aaron um the batman and robin of comics accessibility uh and see if you guys have any thoughts about a comics xml what how would you um how would your app consume such a such a technology or what would you have to what would you have to sort of uh caution us about something like that or any any thoughts about that at all yeah i'm glad you asked actually it was the comics xml that inspired our project so i think in 2008 john walsh who's a professor at the university of indiana developed something called the comic book markup language which is uh an offshoot of the text and coding initiative which has been used for a long time in literary studies whether that's to mark up a poem in xml or markup illuminated manuscript which has you know very closely related to comics um so what we're we're trying to do is make that less of a uh i guess mathematical hands-off approach but more of a sensitive editorial approach to using xml so kind of like when um when there is alt text on websites we want to create alt text for each for each panel that captures that aesthetic beauty of each panel that what you're talking about which is a translator problem and you can kind of think about it as a phrases poetry written about great works of art um you can capture that same aesthetic feeling that you get in reading a poem um that you can in in a painting or so people argue uh and so we want to combine that with the the cbml in order to kind of have the best of both worlds ever cake and even too so that's our approach at the moment yeah i'm so glad you brought up at frasius i mean i consider um almost all of uh descriptively oriented accessibility to be a form of a crisis i i really um i so thrilled to hear that and and also my apologies for uh not being aware of that comic uh xml uh history that's the the punishment i get for not um not knowing everything uh is the um is the comic xml that you referenced is it in use anywhere is it um uh and does it uh is it is it something that only goes to the sort of granularity of the panel or does it go deeper does it go into the panel uh in terms of sort of object-wise uh just for for descriptive purposes yeah so in in short it can it can be as descriptive as the editor or team of people working on coding the comic want it to be so you have an editorial board and you decide the specifics of what you're looking at when you code the comic um to answer your question about is is this a working issue um not really because there hasn't really been a reason to spend so much time using cbml for comics besides creating a database it's time intensive but what we feel like this is the perfect reason to code comments so then everybody can use this you know machine readable and human readable text so whether that's using it for our app or getting a program to write out specific translations of comics um and then making braille versions of those that could be used for cbml so what we'd love to do is get a working group of people um all working together to code a bunch of comments as fast as we can a comic hackathon exactly yeah accessible accessible comic hackathon um hey ting um how are we uh do we have a lot of questions i'm i wanna we've got about 15 minutes left for questions i want to um i'm going to close this section out pretty soon but i i don't i'm i'm just wondering uh how do we have a lot of questions or not too many um not too many there's been a little bit of discussion about um the styling like css the xml framework and exploring that there was one question from ruth uh ruth tate that i noticed hang on before we before we dive into the questions i just wanted to um i i i just wanted to get a sense of how many questions we had i'm sorry so we'll we'll we'll start in on questions in a minute but um uh um darren i i wanted to ask if you had any sort of closing thoughts about the the comics xml um i think aaron covered most of it but i do think it's the kind of thing that if we um can can gear up for that it's it's a tremendous amount of work obviously there's there's 100 you know 100 plus years of comics that would need to be coded um it could be the kind of thing that that could work in a number of different platforms and with another number of different applications yeah um i want to um go through uh and see if um if folks would like to sort of offer some closing remarks before we go to questions um i'll go uh let's uh let's ask uh ed first um any any closing thoughts before we go to questions yeah um well first off thanks again to you josh ding amazon san francisco state it's thank you for your leadership your insights and i don't have any closing thoughts but i do want to take the liberty of asking the first question okay well i'll come back to you then um um uh matthew any closing thoughts kind of going off of what ed said thank you so much to everyone for organizing this event making it happen and it's just a very interesting time now i think in comics and technologically speaking and there are a lot of mediums that we can employ to make comics more engaging energizing and kind of more more for us and i think that it's just a great time to be in the space and creating and making it more accessible for us uh darren yeah um again i would just uh like to say thank you josh to you and everyone else um and especially to nick who reached out to us originally and um i think helped us feel a little less alone in what we were working on and and helped us find our people so um thanks to everyone involved with this thank you aaron yeah thank you so much again josh everybody who organized this um i think the one ending thought i i have is that i think we all have part of the answers in our fields that we spend most of our time in and i think it's really important for us to come and have these conversations in order for us to learn what we're all doing and what we're all experts in in order to use all those powers to find the solution amen um sheila any any closing remarks sorry um yeah i just had one thought and it's maybe throwing the cat amongst the pigeons at the late stage in the conversation but first of all thank you but then to think you know i think we do have a lot to learn from the world of animation if you think of the way for instance keyframe animation is done at the moment where the keyframes are defined and then the the in betweens are often algorithm algorithmically specified so how does an and get from a to b one could imagine that a keyframe could be just something that's not just visual but haptic or um auditory as well and that then the in betweens the algorithms that generate the in-betweens might be specific to sound or haptics or vision or whatever so that that could then open up a whole potential space for both author authoring and rendering existing materials in multimodal ways that's wonderful you really you really masked it though with the the imagery of the cat among the pigeons so i i had to pull myself away from that um thank you thank you sheila for that um i think you're right um and i uh um i wanna um i also wanna thank um i did very little to organize things i just sort of came in um but i really wanna thank um uh ting and emily and um and nick for organizing the whole uh this whole thing and really pulling together a community to discuss this and um and i want to thank all of the panelists who um have brought such great perspectives and are working on such important technologies to um to not just help comics accessibility but to um improve uh access to information by blind and visually impaired people in general and in so many different domains so you guys you guys all rock um and um and so let's uh ed i'll give you the privilege of the first question and then ting will we'll go to questions um ed okay josh this one's for you um my request is that you give us i'm not a panelist wait a second yeah okay go ahead my request is that you give us the 30-second elevator pitch for blind guy and sighted girl um it was uh it was basically a um a blind guy who thinks that he's all that and wants to um you know is is really uh really um you know very blustery and important about how independent he is but really he completely depends on sighted girl to do all the things that he needs done um and sighted girl uh does them but um it's sort of a it's a wooster and jeeves kind of thing but but sort of um superimposed on a on a blind uh a blindsided couple um so it's it's um it's it's sort of like that um but thanks for uh thanks for thanks for asking um and uh and with that thank you very much ed we'll um we'll go to ting um ting you want to throw us some questions sure well i mean naturally the follow-up question to ed's question is would this be an autobiographical sort of pitch i did mention that it was slightly autobiographical slightly or totally um anyways so there's um there were a couple of technical questions that came through on the chat um there was one particular question that was upvoted so asked by two participants so i'll just start with that one in case we run out of time so a couple of our participants asked what would be the difference between the cbml comics based markup language i'm guessing and a layered epub so for instance with smil all text image descriptions yeah i think ed is probably best qualified to to start us off on answering that yeah um having you know written reach for the stars or co author reached the stars with with some wonderful colleagues here at sas and nasa um you know that is kind of what we did is we we created an instance uh we created an epub which is kind of like a finished product of a rendering of a book um and what we're what we're looking at with like like a markup language is much more kind of foundational and versatile in the sense that it's a a fundamental description that can be used to render or generate an infinite number of renderings or presentations of the information and i'm trying to think of a good analogy don't know josh or others if any other computer scientists are out there um you know maybe like for a piece of music you know you can have the recording of somebody playing a particular piece of music but the score itself or maybe the music xml you know for sheila the the digital representation of that um is can be rendered and used by human beings by different bands different interpretations you know that's really the difference for me i think probably you know some of the differences are that um you know you can you can use things like svg and um and smile um and epub etc to sort of uh encapsulate and render a whole bunch of different things but you know as with um as with so many markup languages um you wind up spinning off um sort of specialized uh sub um uh codex or whatever the right word is um because there's there's baggage in the original parent um encoding that you might not really need um that might slow you down in your um authoring or or interpretation so i think the idea is not so much that these other that you know we don't yet have a language that can do it i think the i the what i'm imagining when i talk about it is a um a format that's actually optimized for um for comic accessibility and that uh that allows you to you know group things so that you can have objects with lots of child objects and lots of descriptive information associated with every level of those um those groupings um along with you know directional information if they're moving and stuff like that any other um thoughts from the um well actually let's let's let's actually move on let's let's take another question we've spent a lot of time on the xml and it's my fault no that's right and actually joshua with only two minutes left um i wonder if instead of um taking another technical question that we might not have enough time for if you would like to share some closing thoughts um before we conclude this panel um thanks king um i think my um my the what i'd like people to take away from this is that um this is one special case of information accessibility that we need to um that we need to address we we will never run out of accessibility problems to solve this is one that is really important because of the way our reading habits are changing we've we've spent a lot of time trying to get access to um to text information for blind people and we're almost almost there really um but uh but now we're shifting into a world where video is much more important and we're still dealing with um we don't have anything like uh the you know the chaffey amendment for audio description um similarly uh you know captions there's there's no there's no rule that says um uh i'm allowed to audio describe any material that i want um and so that's a a sort of political block to accessibility there and as other forms come along like as comics become more and more mainstream and important for not just not just in enjoyment not just entertainment but um education and employment as well we need to figure out how to how to make comics accessible and um the longer we wait on this the longer this is the the problem we had with audio description people thought oh audio descriptions not that important it's just for giving people access to you know to tv shows well now um everything is a tv show and without access to um without good access to audio description or other other ways of making video accessible to blind people we are leaving us them behind um in the information we're able to consume and therefore the opportunities we can have the same thing is true of comics we need to we need to jump start this process or um and the longer we wait uh the more people will lose out on opportunities economic social uh etc so um this is not just an academic question this is a really important question that impacts people's lives even though many um many folks may not quite see it that way yet um and i want to once again thank you ting for organizing this conversation and emily and nick and san francisco state and thank you to um uh amazon the amazon people with disabilities affinity group for sponsoring um and uh thank you to the panelists um and of course thank you for being part of our thoughtful and engaged audience okay thank you emerging tech panel and josh for moderating and all our panelists for supporting this event i know you left many of us with a lot of thought and hopefully inspirations for what to design and build next um so we are now off to our next 15
2021-09-16