Melody Ha: Storytelling in immersive technologies at IxDA Sydney October 2018
We'll introduce melody. Huh. Two-stage. Melody, has produced, and worked with a wide variety of mediums from VR to, TV. Seas to webseries animation. And even, projections, on the Opera House for vivid. Her, background in 3d animation and, post-production. Coupled. With her experience as an actor and writer, have, given her the skills needed to weave, creative, and engaging stories. With, a technical, and logistic, eye, currently, she works as a producer at the pulse and I'll let her finish introducing herself. Hi. Everyone. As, she, mentioned oh yes and hello this is Julia she. Will be participating as part of the upfront. Think, sorry I'm struggling really hard with putting this thing in my pocket awkward, okay, hello so just to quickly explain what upfront, is it's, this really amazing initiative as, as. You mentioned by, Lauren cardi that really. Tackles the lack of diversity on stage by. Inviting speakers like myself to share this space it. Helps, others including, myself struggle. With stage fright you probably can't tell you probably can I have crippling. Social anxiety. Luckily. I had some alcohol with you can tell I'm actually blushing for real. My. My, secret. And. What it does it helps you feel empowered to be able to be part of this space and hopefully feel. Like you can actually stand, here in front of all these staring faces to do it themselves, so. Before. I kick off to me please join me in giving her a lovely warm welcome. Alright. So. There. You go so just to give you a bit of a backstory about me you've. Already got a pretty comprehensive one, as it is but I'm, a producer, at experienced the pulse, experience. The pulse is this company that specializes. In non-traditional content. So we've. Done the projections on the Opera House we, do, experiential. Content at car shows and, mainly. We do virtual reality stuff so, training. And education, in VR and most. Importantly to me storytelling. An original content, so. Out. Of curiosity how, many people here have actually done. VR before. Oh. Okay. Create it and experienced, it. Okay. Cool, so, for the benefit of like. That looks like half of the audience I might just quickly explain that, two different types of the other are out there at the moment, there, is, mobile. VR which is probably the most common, experience one which is where you've got a phone and a headset you're. A lot to position and you can kind of move, your head around, and up and down but you're stuck and it's kind of like a vomit rotten you do it and you want to vomit. It still has benefits, but it's it's. Not quite as advanced. As the other one which is tethered to a PC that one, is your, oculus, your vive your psvr, it's a lot more fun because you can actually move around in the space you can pick things up it's. A lot more immersive. What. Was really interesting about making VR is I didn't, start off as a VR, producer, I started in film I started as an actor and making. This leap over to, VR has made me I think give. Me a really rudimentary understanding of, interaction, design like it's this, whole talk as a result of my really, steep learning curve, trying to figure out what the hell it is so. Some. Of this is repeating things you already know again I'm really sorry in advance but hopefully, you can join me on this journey. Okay. So. What. I want to do tonight is, give. You guys some questions that I want you to take with you and ask yourself if you ever decide to make VR content on your own, so. Out, of curiosity for. Those, people that have experience for your content before what, stuck with you the most was it the prettiest, thing or was it the thing that had the most meaning. If. Anyone wants to speak as well, the. Interaction, what about the interaction was interesting, to you. And. You, could take you know your partner with you and you could both wear headsets, and you could go hey come over here look at this space could you see you. Can see each other in it yeah avatars. Yeah. I. Think. Yes I. Would. Say quality of vision and, well, the sound design was really important, in terms of the immersion awesome, I'm, gonna come back to those things and that's that's really important, that. Yes. I I. Recall. I was at the antenna Film Festival, and it was a documentary. Ask. Immersive. Experience, and it. Was so engaging that I felt, as those feeling, and smelling the, water in the tropical plants so this in serial aspects, of VR, and, how potentially it could be really. Beneficial and, medical and, particularly. And, well. Rien gauging with the different parts of humanity cool. That's. Really interesting that was just purely for my own curiosity, so sorry, took a bit of your time. So. Quickly. I know that, the title of my talk is specifically. Storytelling. But in hindsight that's kind of the wrong word because a lot. Of what I'm gonna talk about today is about empathy but. As you see it storytelling, is nothing without empathy in connection, right one can't really exist without the other it's, like I could.
Tell You about how. I fell off I, felt a translation. The other day yeah, whatever, or, I could. Tell you about how I stopped. It and I fell between the train, and the platform, and I, cracked two ribs and. Yeah. It's legit I can show you the scar if you want. And. It. Was like really traumatizing, I didn't realize that I had actually like cracked two ribs and, like needed to get five stitches but, it's what's, really hilarious Susan you're kind of listening to me now all right you didn't care that I fell at the train station but now I'm that idiot that fell between the train gap and. That's. The thing without some, kind of connection whether it's empathy, I'm hoping somebody some, of you empathize with me instead of just judging me silently. That. Laughs tells me you were mostly judging me. Without. An emotional investment whether it's waiting for the punch line or. Empathizing. With someone you don't really care so. In. VR, empathizing. With your audience and their experience, is so fundamental, to making something that's good. Especially. Because. Making. VR and being in VR feels, almost. Like you're completely real learning English, start it's it's familiar but, it. Also doesn't really make sense because it's kind of. Really. Different. If that makes sense like it's it's really confronting, because you're strapping the screen to your face and you, kind of forget how to human, cuz it's not really what you think the real world is and I. Think that's really interesting and you have to really keep that in mind when you're designing stuff, for that space, you. Know when you're when. You're creating empathy. In traditional film we tell stories through careful. Scripting, and performance, we've got techniques like music, and specifically. Framing the shot but, storytelling. In VR is same. Same but different your. You've, got this whole other dimension that you have to deal with right you still. Can manipulate the music you can manipulate, the sound and the color but the. Person is literally, in this space with you and. You're. Actually able to really add this whole other dimension to what, you're seeing and it's. Really isolating, cuz you're by yourself you're. Really lucky cuz you managed to experience, something where you could see someone else in Avatar but most of the time it's. Just this one solo experience and. Again. If it's your first ever time like, it's it's kind of confronting, because you can't, reach. Out to someone next to you and laugh about it you can't ask. For help because you don't really know what to do you're literally, in there by yourself and you really need, empathy when you're trying to design an experience in that case. So. A. Few. Things that you need to think about when you're making 360, content, is you. Have to consider it in a different way to a traditional, sixteen. By nine straight, format in. Mobile. Content your car your camera is kind of locked and it's this really strange feeling where. You're. Either forced. To feel like you're in the first-person perspective, where this whole story is unfolding because, of you or if.
It's All from a third-person perspective you, automatically, feel like a voyeur, because. You, can actually choose what you're having a look at and, it was really strange like I don't know what I am like what is my role here and that's exactly what it is in the you. Search, for context, clues to figure out who you are and why, you're in this space and when. You're designing an experience, for this you need to take this into account otherwise, you're gonna have people that are, completely, lost, don't know what's going on and it's gonna make your experience not, enjoyable, and again, you're not going to have a reaction or a feeling out of it and that's, that should be the most important thing, so. In. Experiences where it's like a mobile based 3d. Sorry. It experiences where it's in a mobile, headset, where it's a really locked position, sometimes. The best way to deal with that is actually to lean into it so. For example a film that we recently made called 30 minutes of danger, what. We did was we, we. Made that restriction of not being able to move part of the experience so you're actually poisoned, and as. You, get more and more poisoned as the film goes on you, you. Have basically. No control over your body but we were able to bring points of immersion into it by adapting. Real-time technology, so what we did was, there's. A point in the film when you get slapped it's. Really fun and you. Get slapped in the face and we changed. The film so that you, physically have to react to it to continue to watch the story or as. You get more and more poisoned you. Start, seeing your vision start to cloud over and encroach and we, design it so that if you shook your head you cleared your vision and these are all things that tie back to thing to, actions that we understand. As a person you know when you see. A cartoon of someone getting knocked over and they see Tweety Bird flying over their head what are they doing they shake their head when they want to clear their vision, that's. The thing you also have to try to connect, these. Actions, and these interactions, to real-world. Examples. Another. Thing, sorry tell me if I'm boring you guys another. Thing is in room scale you would. Feel less like a voyeur, because. You can actually walk around in the space and physically, look for these context clues, and, you need to account for this as well when you're designing because. Just. The way that people approach. Let. Me backtrack when you're, presented, with a film you kind of already know how to deal. With it you know that you're going to watch it and certain, things are going to make you feel things but. A lot, of people who are trying VR either for the first or second time. Probably. For the first five to ten minutes they're just kind of freaking out about how great it is how. It's like. So mercy look I can see a tree that's purple, oh my god you know like that's that's more what they care about as opposed to the story or what you're trying to build to them and, that's just something that you really need to consider as well because it's just really disorienting, it's like it's. Kind. Of like a lucid, dream. But. The really cool thing about making, VR is you can literally, put yourself in someone else's shoes when you're building it so. You. Know when you're making film you kind of have to try to try, to put your mindset into like what if I was Schiele uh from, down the road and I was 38 years old with two kids I have no idea what that's like but I have to pretend for a hot second that I am so. I can understand, whether or not she sees this film and something that's legitimately, good to her but. To. A certain degree you're still not going to be able to pretend to be Shelia down the street with three kids or two kids but.
You're, Still able to experience by, putting this headset on when you build it whether, or not it makes sense to you if, that makes sense. So. What. Makes you care about something that's happening in front of you. Like. If you watched. A clip of a complete stranger. That. You had, no context, about you, had never, heard about start. Talking about something. Really intimate, and deep like their, sexual assault. Sorry. It's getting a bit deep here with. That how. Would that make you feel. Like. Uncomfortable. But. Would it really affect, you like would you actually genuinely respond to that on an emotional level. And. Then. What if someone. Stood right here in front of you like where, I am and I was telling. You about, the. Time that I was. Harassed. At the bus stop. Now. How does that make you feel I. Wasn't. Harassing a bus stop it's okay I'm an actor. But, how did that make you feel for a second right I was right in front of you and maybe. I should have made you a bit more uncomfortable but, I feel, like that would have made you feel something a lot more then. This. Person that you've never met telling you about something that may or may not have happened to her and that's. What's really amazing, about VR is that you can make. Yourself, feel like you're physically in front of someone. You can feel. Like you're literally, sitting, in front of someone and experience, them talking. To you and connecting with you directly and. That's. So. Amazing, when, you're creating a product or a pace that needs to really touch the audience, specifically. In VR you need to consider these things how, does your real world affect you on an emotional and human level and how. Can you translate that into your experience, because. You know when you're designing for like a web or for the screen you you. Used to have to find a parallel to that but now you can literally translate. It completely, almost almost, completely. So. Empathy sorry, VR has been referred to as an empathy machine and there's this video that I'm gonna play for you guys called the machine to be another and it's, a really simple concept you've, got these two people, wearing a headset you've got a camera strapped to the headset and they're. Moving in, sync. Would. You be an immigrant, would, you change your gender. Would. You doctors yourself, with, physical, disabilities. We've. Decided to hack neuroscience. Experiments to, experience, what it's like to be in the body of person. Two. Years we've, been investigating how people, interact, with the existence of someone else, but. Kentucky. Baby was moving into to do Baylor espera haters. They. Machine to be another combines, neuroscience protocols. With art performances. To, trick the brains perception, of one's, own body. Instead. Of seeing themselves in, the body of a digital avatar like, in most of the neuroscience studies, the. Machine allows, users, to see themselves in the body of a reoccurs, the. Protocol, of interaction, combines, visual, l.joe. Physical. And motor stimuli. The, system, is coded in Arduino pure, data and open frameworks, and uses a low budget Harvard. Composed. By video games has mounted displays, first-person. Stereo cameras, are, doing and servo, engines. Exhibitions. Performances. And academic, congresses, of, anthropology. And embodying, action we, developed, several, applications. Of the machine to be another address nations. Like mutual, respect gender. Identity, generational. Conflicts, immigration. And physical, disability, bias by. Extension, and neuro, rehabilitation. More. Than an art installation the machine to be another is a creative commons initiative. That can be replicated by anyone, interested. In understanding the, world through the eyes of the other. Of. The international, press and also, of researchers, from different, views from, queer theory to, psychology. And conflict, resolution and, we are planning to go much further in this long term research, that. Aims to raise the awareness that, were more than individuals.
We're, Part of a big system a, big, collective, called humanity, and have. You ever imagined, how, would this rope be like if. We could understand, better just. Because, it's quite long but I like how you guys laughed at them looking at your genitals thing but let's be real if we had the chance to be in someone else's body that is totally what you would do. Sorry, if there are any under 18 people in the audience, but. What was really fascinating about this is all it took was a, headset. So seeing, a different point of vision. Tricking. Your brain into thinking that what you were feeling yourself is, reflecting. Is exactly. What the other person is feeling so I like, when. You looked at the headset that wasn't your body but they were doing the exact same thing but then you're also feeling, exactly, what they're doing, it's. Crazy. But it's actually, that's. Actually all it takes to make you feel, like you've been transported somewhere all it takes is a couple of your senses. To really. Make you feel like I'm no longer in this space right here and. It's. Super, cool. So. The, way that the empathy machine works is that it can either wall. Steven to the perspective of another person, like, like, you just saw it, can, put. You directly in front of someone and confront, you with their emotion like I was talking about previously, or it, could just shove you right in, the middle of somewhere where you, otherwise wouldn't be able to exist, not exist experience. Like. Do you feel like you would be more inclined to donate to disaster, relief funds if you, literally, stood in the middle of what happened like in. Puerto, Rico when everything, went down do you feel like you would give more money and be more giving that way. But. When you see images on the screen and this kind of like 2d. Thing you don't really feel anything right like we've all seen those UNICEF ads we don't really care because. Yeah. And. That's. That's. The empathy machine if you can transport someone somewhere. To, live, in the shoes of someone else for a day or for a few minutes sometimes. That's kind of all it takes to make you feel for another person. We. Can also apply this in so many different ways we can apply this in training and in education, we're.
Currently Building this product where safety managers, are working with heavy machinery every day and, they. Get really complacent. That they're. Actually working around things that can kill them every, single second of the day and, what. We're doing is we're building this world, where you can safely show. People the consequences, of being. Complacent. And hopefully. That's something that will then jolt them back to being conscious, that they need to take care total. Care of their staff and themselves because. It's just so easy when you're next to this giant thing that weighs about 30 tons every day you don't you, don't think of it as a killer machine but it's. Really scary of being next to those things it's horrifying. So. This goes back to considering, what. Perspective do you want to tell your story from, what. Are you trying to convey how. Do you use this new new tool of force, curse of active that brings. Different types of empathy to serve your purpose, because. To tell a really effective story you need to generate genuine. Empathy and there. Are a few ways you can do that. So. Let's. Be, really really, deep for a second I had. A lot of fun putting these slides together can you tell. How. Do you know what's real. That's. A genuine question how do you know you. Are in this room right now listening. To me rant about stuff, how. Do you know this isn't part of a simulation and you're actually part of a really bad game of Sims. You okay, that's getting too deep right now that was supposed to be so it rhetorical, we. Can debate I mean, was that will debate this afterwards over a drink cuz I love solipsism as much as the next person yes. Whoever, stood over there you can you can't talk to me right now but you can technically touch me if you came up here on the stage you, can touch, the chair underneath you you can smell. Alcohol. Hopefully, not me, you. Can you, can hear, me you can see me you can you, know you all those five senses that's how you know you're physically. Somewhere right. The. Really cool thing about VR, is that they, can only manipulate three, of those five senses its. Sight, its, sound and its touch and as. I showed you just then that's kind of all it takes for you to feel like you're somewhere else and that's. A very dangerous tool to have so. I want to show you guys two examples, two, of my favorite examples of effective AR one. Hopefully. The volumes a bit down because it's going to be a lot of screaming. Does. Anyone play this before. It's. Fun right. Oh. Yeah. Oh. My. God oh my god. This. Is legitimately, how I reacted, when I did this by the way. So. You don't necessarily need, the plank to do that like when I was when, I played it for the first time I was literally I knew I had literally make this stuff for a living so I knew it was fake and I, didn't have the plank and I was literally doing this. Because. I was so scared that I would fall to, nowhere, but. Isn't it interesting all you needed the graphics aren't even that amazing they're like PS to quality they're not that amazing right but. You've. Got this sight you can feel the steps you've, got this sound and you've got for some of them the wobbling ball underneath them and instantly, they had this visceral reaction, they, were there, now.
Here's Another version which is a bit less funny it's. Called a thousand cuts. You. Have now become Mike, Sterling, look. At yourself in the mirror. That. Was it. Short. And sweet right, the. Reason I showed that to you was because that. Also. Kind. Of concern what. Even. Though it's got low fidelity graphics. It's again put more like ps1, level graphics for that it. Uses those same three senses of touch, of sound of. Visuals. To then transport you into the body of another person, in this case it takes, you through several stages of this man's life, so he's african-american and, he experiences, prejudices. That some of us may never have experienced before and, at. Every, stage so you see him as a baby you, see him at five years you see him I, think. 15. And then you see him at 30. And then 50 but at the beginning of each stage it starts, with you sitting in front of a mirror and, when, you're sitting in front of the mirror you see him, not yourself and. You've got these these. Controllers. And you. Move your arms and that's his arms and I'm moving and after. A few minutes you realize oh. This is me and everything. That happens to him when, he gets yelled at by his teacher for something that his, classmate, his Caucasian, classmate did that he's. Getting in trouble for when, he gets approached and attacked by police when he's facing, prejudices, when looking for jobs that. Feels like it's happening to you and all, that took, was. Three points of sensory, contact. Alright, so, how do you build your own empathy machine. You. Don't have full control. That's. Really scary I'm a control, freak and. The. Thing is with again, traditional, flat formats you can kind. Of filter down to a tee exactly, what you want that person to see because, it's all framed a certain way it's all like colored a certain way and, you can do that to a degree but at the end of the day you can't control what JimBob, is looking at if there's this flashing crazy, thing over here that you need him to be looking at he could still decide to look over here because screw what you Caravelle and. You. Need to consider that when you're making a does when you're making something like this because you, just it's up to you to try to mitigate, that as much as possible there's. Always going to be one guy that just is gonna ignore all social cues, but. You, just need to do your best to make sure that you're driving them to this shiny, flashing thing over here without overtly, having arrows saying look you. Onboarding. Takes a lot longer than you think, or. Maybe you guys are aware that onboarding, takes a really long time but it took me way too long to understand, that again. Going back to the idea that VR. Is really isolating and even fusing, and how when you put a headset on you're not a human anymore people, can't even walk when they put a headset on for the first time like if I literally got, you guys to put a headset on and you, saw, exactly.
What You're seeing now you. Still probably be doing that weird dance that I was doing before with you like this. And. That's, because, it. Just doesn't feel right because again you're isolated, and you just don't feel like you're in the real world, so. Onboarding, you really need to think about how onboarding, works for the person cuz a lot of the times you can't actually see what they're seeing if they're doing my BA they are you're. Just kind of going look, to the left I think look to the other left I don't know if there's a red thing over there I swear you can't see what my hand is doing Shh. Ah and. Again. You have to consider that when you're creating the onboarding process of your, product and. Really. Try to integrate storytelling. Into the onboarding process as well because it's already disorienting. Enough that, they're in this place and you're. Probably staring at them if, you, kind of ease them into it and give. Them time to. Kind. Of settle into the new environment and figure out where they are and then, let that time, lead, into, onboarding. That's hiding the story then you've, got yourself a really really good experience or first-timers, and that's kind of the people that we want to make experiences, for because. It's kind of scary to me that probably. Over half of you guys haven't done BR before but, this is my lifeblood and it's really scary to say that it still hasn't come into the mainstream which. I'm gonna talk about in a second, you. Also want to have things react. Like. They would in a real world like, if I was to knock these things over they. Would fall right, how. We would, it be I did, this and it literally face through everything. What. Do you think I was Kitty Pryde or something like wouldn't that be really weird, do. You want to make sure that whatever, the interactions, that you build into it relate back to what, the real world does as much as you can. Also. Just a note reading, is really hard in VR so if you decide to place text in you're trying to do onboarding and stuff be really conscious of your font choice of where it's placed in there's dead space and the, size.
So. Off the people that have done the are before how many of you guys felt super, vomiting just. You. Okay. Mom there you go be honest I've also felt really vomity doing be out before yeah so, does. Anyone know why that is. Was. It really shaky was it like a roller coaster experience, what was it. You. Do VR and you go into shopping. Yes. Was. It like it was at 360 was a live-action, 360, film or was it. But. They were moving you forward right. Yeah. Hmm. So. The. Reason behind a lot of the reasons we get vomiting is because our, brain doesn't really like the fact that they're seeing the world move around it and you're not physically moving because, that doesn't compute if. Everything. Around me started moving right now I would obviously, hurl because that does not make any sense to me so why does that why, does that make sense in VR. Right. So. When you're building locomotion, in VR, which is moving around in space you. Either allow them to just physically, walk around or you. Use teleportation, which, is basically you click and you kind of make, this arc and then you appear there it's. Still a bit disorienting, but it's a lot better than people vomiting, up in your headset, that's. Not nice, and, last. But not least the, most obvious thing please. Test like crazy, because. What, might be really good to one person, might be really really. Vomity or difficult, for another and the. More you test the better your products gonna be but, I feel like I don't really have to tell you guys that. Why. They are. Sorry. I just really love Simpsons. Does. Anyone know about that steam tends sane, yeah, so someone literally made that in VR, they. Had nerd that had all the time in the world they made in India but the question why, I'm asking why VR is if, you're gonna make a VR product you have to ask that question, because if. You're making the Alpha VR sake you're part, of the problem not the solution, the. Reason why I was, saying I'm scared that only half of you have done VR s because, it's. Not fully. In the mainstream yet and. I believe it's because of a few things not, just cost but. Because a lot of your first experiences, would be are probably, really shite because. Of the vomiting thing or because people are making the OO for the sake of VR they're, not really thinking. About why, they're doing it and if. You think about that because. If you've got this concept where you could probably do it a million other ways that doesn't need to be in VR you're. Doing it wrong no. Offense. But. If you've got this concept that makes, total sense to be in this more. Than three dimensional space that you can move around in then you can look around in and have your user have complete control then. You're totally doing it right because. I think it. Is your responsibility if you decide to make virtual reality content to, make the best stuff that you possibly can, because. If you don't you're. Perpetuating the problem and, if, you decide to make the choice to come into the VR space you're.
Just Making it worse for yourself, so. As the. Incredibles are representing. It's your responsibility to, take, all, these things into account when you're making quality, via content, because, that's the only way we, can grow the industry. So. The only takeaways that I've got for you guys is, make. People feel good and feel. Stuff to tell good stories and, please for the love of God make VR that's good, that's. It.
2019-02-21 13:37
