Toronto VRC Meetup - ‘The Choice VR': Volumetric Documentary Filmmaking

All. Right. Also. Especially big thanks, to our speakers today, because. They muscle, works I. Really. Don't find the speakers please, drop that I have such a funny speakers, thank you he shares like that so I really. Appreciate. We're. Going to talk like you're sharing with us today and. I'll. Just pass the mic on and let them let, them do the thing that's really important in this group which is share our stories about detective. Experiences. And content, that, we're creating virtual reality. Hello. I'm, John Ashe police, car. And. We're toronto-based. Vr filmmakers, which you. Can imagine everybody else here as frown face - and we recently gave this talk at GCE, a couple weeks ago and. Then Stefan, was like a. Ghost. At. The same time so a whole girl having a good time at GGC so far here, the price is really good to get in. So. We're, glad you came and we're here to talk about our, beer documentary, project that we're currently making twice. We are and our stereoscopic. Volumetric, we, do capture technique that we developed in order to make it possible and. This. Light behind us is our company, so. So. The choice is a documentary. About reproductive, rights and, I'm so happy to see so many men here, in the room it's pretty awesome and so. And, what. Are you into separately you meet people who had abortions and, they share their experiences, feelings, and how they came to their decision it. Is a documentary, which means that, we have some news oh it's. And. We feel that some aspect. Is important, and this was our starting point, our goal is to share this, feeling with our audience, and, the experience itself the near experience, starts with our empty void the, craving device is that you are inside, body, and mind for the fictional person who, discovers, that she is pregnant and and, is not sure what to do nice also disclaimer I keep. Using the female. Version but of course not, only women have an abortion is just a side note for now so. This woman who looks for some clues and meets other people who share their experiences, with her and the important, aspect is that it will be your story as a viewer and you, are the crucial part of the whole experience and, people that you will need are great people people, sharing their own personal, stories. Vieira. Is not the easiest menu it has many limitations, and issues. Make it challenging, but, do we think it has many really, incredible, opportunities. I may disrupt I made this project to use the strengths of it and avoids. The immersive, technology, is really good at so I want, you not just to hear those stories but to meet those people like, you met them in real life so.

What Is documentary, you can think about the, motion picture as the continuum, between cinema, and journalism. Documentary. Is somewhere in the middle, cinema. Is creating, a fictional world and invites. Their lives to look through a window into, it how, the window, is framed affects. How they feel and understand, what they are saying when. The news and journalism use the motion motion picture the fundament, is that we learn information about, the real world and they, don't hide the cameras or the studio the microphones, there is no magic just reality, as close as, it can be represented, of course sometimes they use our belief that they only tell the truth and massage. It or fabricated. But then suggest about journalism. And then. Documentary, what is the documentary. 3:7. Some misconception. What it and what the documentary, actually is some. People believe that it's, kind of visual or Wikipedia so. It should tell you oh that's known about some phenomenon, take, it from all possible angles perspective. And explain it to you but, documentary, is not Wikipedia. And it's not scientific. Paper, documentary. Is showing some aspect of the world through, the lens of the filmmaker, so, there is a point of view there, is only some part of the whole image and there, is probably some interpretation. Maybe even assumptions, and when. I share this description, some people ask is it not neutral, and very much so documentary, is not know each other but, they are reward that is very important, here is that documentary. Has to be honest. If, you look through your lens you have to both identify lens. For yourself but, also for your audience I think that's the baseline no matter what the documentary, is about, so. The question is what is my lens. I see that abortion debate is very much focused on studies, stereotypes. And entitled, assumptions, but when you actually talk, to people who have personal, experience, with it you realize that there is no two people alike and no two likes, alike and no two stories, like, it's. Often easy for you if you didn't have this experience yourself to say that I would do this or I will do that and easy. It's easy to judge decision that someone else made but, when you meet them those people and hear more and more details, and more and more aspects, they fought through suddenly. You realize that there is always so much more in every single story that you would be able to think of Aaron and this, is what I think is missing from the whole debate so, not statistics, not numbers not experts opinions bad people their, stories, their, emotions, sometimes, they have without fear. Or relief. We. Are still in production production. And I, think, I've spoken to over 100, women so far not all of them want, to be in VR experience, many, times they just called to me strike me my only just to share their stories to share some, burden, and some relief and. I, think I haven't heard a single person, that said, that it was an easy choice for them it was maybe obvious, one to make he's, the one that was right, for them and almost all of them would be the same words that it was right that they would do it again but. There is also much more emotions, that that. We think, that. We decide, openly and this, one is very challenging, because the. Anti-choice, movement uses. That so what they hear that someone has the doubt or fear that. This is for. Them to prove that abortion, itself, is damaging, for the person, to either close it and, this is huge misconception because, when we take every, big decision in our lives we have many doubts many fears the, outcome may not be as good as we imagined, but doesn't mean decision yourself was wrong so. If we would allow for this discussion, just this discussion, about this aspect I hope that many stereotypes and, false assumptions, would finally go away and, so. When you meet this women in the choices you see the process they all went how, carefully, they make every decision how, responsible, they were in it and that's among the main, point to, see into. A low for it to, allow them to make it for themselves. And. The, people that I talk to trust, me with their stories and I would love for everyone, to have this opportunity, as well but, it can be exhausting, for them to share it over and over publicly, and also for the public to listen especially, when, you, maybe are not ready yet to hear it but, it is important, like with every other stereotype, to meeting, a person face, to face and to listen so.

Hearing, At the stories is changing, me a lot and I want the audience to experience the same thing and, we, are is very personal, menu and, it's, not like the motion picture there is no window, between you and the subject and I want to use this aspect. There. Is a lot of talk, about, presence. In PR but we usually define it as a principle presence of the. Viewer being in the neutral world when, I record, this interviews, I give those people a real space where, they can talk freely they. Can open themselves about all the emotions, that they want to share and I'm here I'm there to, listen and to record it so, when the you were enters, to speak to a world that I think, they now feel and bidirectional, presence with the woman women telling, their stories, the, viewer feels like they are they. Are there with them so, the viewer takes my place in, mounting, with no, judgment without comment, nothing. They can do or say can interrupt these from the subjects as they tell their stories, so, it means that to participate, is to listen and SS. Story goes you now not only physically, took my spot but, also the tracks and feelings that they put towards me and now yours is a new way and now, how do you achieve that. So. In order to maximize. Feeling a bidirectional presence. We wanted to film our interviews Polly magically, so, what it is small I'm educated, for those you don't know it. Is the concept of recording a video that also includes. Spatial, data so a motion picture is shaped. Early. On we started to look at what technologies, existed to allow us to do that. First. We looked at light stage and photogrammetry is a possible approach but it is much, too heavy and expensive and, also challenging to make the, subjects themselves feel comfortable for example you. Know you might have heard Microsoft's. Mixed reality stage, which is I believe the top, one there and, the, going price for, ten minutes of completed footage is $100,000, so. Documentary. Films, feature, films typically have a budget of $100,000. So that's not going to work with our needs. But even besides that using. The. All. I step on. Yeah. Steal, them from some touch controllers, this one I do at home uh yeah, so even, though it costs a hundred thousand dollars for your 10 minute you know blade, render experience, the results still feel very waxy and dead I and, they, don't really feel like living people it's like you're watching somebody, who's made out of clay and it's also a lot of extra work just to record the back of people's heads, like for examples and if any of you right now really need to see what the back of us look like at this given moment so.

We Also looked into performance. Capture is another option which has most, of the same issues but also, introduces. Ethical, issues of being an abstraction, of our subjects and it, raises the question what's, a genuine emotion and what's a technical, interpretation. And it's, important for us that the viewer believes that there's no filter or processing, between them and who is speaking. There's. Also on a very affordable tool set called depth kit which has been used extensively for, a lot of really great VR, experiences, but. It's more of kind of like an impressionistic. Or. Is it method I'm. And I'm there's actually a lot of really great depth kit filmmakers, in Toronto and there's a lot of really cool things you can do with depth kit but it's much more of like a hologram, aesthetic, and it. Makes, people feel like they're ghosts and it's, really hard to believe that you're sitting, in front of a real person if you're so caught up in the, process. Of how. They're represented, this. Thing is a quick. Oh and maybe this. So. We decided what to, focus on what's important for our needs our subjects. Are not moving and they're seated which relaxes. The subjects, as you've seen in any documentary. People, are seated but also it relaxes, you the viewer if you're, in a seated position as well we. Don't need the full geometry, from all directions we. Only really need the point of view of the observer because, naturally, people want to stay in the eyeline, of somebody they feel like they're having a conversational. Connection with and. Will so we need to record for long periods of time in, unstructured. Ways and not shooting in short bursts. Of what the system can provide we. Need to give them time to feel comfortable with the process and, it can take many hours to really get somebody to open up like on our film. We did an interview that was four hours long and actually we found out last or two weeks ago did you see that what it cost us what twenty-four million dollars so. Yeah, that's not a viable solution you don't have it we don't get that kind of money, but. Most important of all besides, all those other things even beyond cost is, their faces need to feel a lot, unbelievable. And a. Lot of volumetric, video, processes. Don't create, that illusion. But what does is 3d. Stereo and our backgrounds, is in 3d cinema, we're both. Filmmakers. And camera people that used to work back when 3d was a thing everybody remembers avatar and. It's a process known as stereo graffiti so. What is stereo, field we, see the world and VR most of us any wave we have both arise through. Binocular, vision and it, gives us our ability to perceive depth we know distance, just by looking at something based on how the. Separation, of parallax between, one object appears in both our eyes helps, us understand, if it's closer further, away and stereo. Fee is the reproduction, of this effect by, creating two discreet images and then displaying each to each individual, eye that's why VR headset has two eyepieces. But. There are less understood, aspect, of our, vision is something called retinal rivalry, or in computer vision is known as binocular, rivalry and it, isn't just the parallax, between, objects, that tells us about what we're looking at through our both eyes, by. The oval but, it's the difference. That each of our eyes sees an example. I've been using since avatar to, explain why, just putting parallax into an image isn't a good way of making stereo and it was more of analogy about why converted, movies aren't. So good and now knowing this next, time you watch you, know Avengers. Part. 20 and, in 3d you wonder why it will swap this is why, so. There's a stereoscopic, image. Of the lemon and you can see between as it puts between the left and right news the tips and marks on the surface move, left and right in our differently space because it's round so you, can see how much depth there would be based.

On The parallax between the two and, your brain also understands, that it falls around off, the edges and, surround. Shape but. If you look at the specular highlight on the top right it's not the same in each eye it's not the same at all this. Is the wrinkle rival one, is, something. That the other one does not and this, is how your brain can understand, okay this is a glassy. Surface, and. You also see if it picks themselves, some head highlights some don't and they, change shape from. Each eye and that also tells you a lot of information about the surface itself through. Occlusion. And you. Can also see that one side of, the lemon you can see some pits that the other eye can't see but, your brain understands, that the lemon continues, around vs. exactly, perfect information and all the other twos are there so your brains like I got it it's a lemon. Instead. If. We use like a depth map we can synthesize the. Second view from a single image I'm still create the parallax, we've got like a a 3d, depth, map of white. And the, black. Bees further away. And. Then we project, that they project the left eye and the radar, all. The original informations, and brick arrival information is gone now it just looks like a round shape with a lemon texture suck on it. And. This is a great idea computer graphics because it makes you easy to do the hard work of bringing one image and then, using a diplomat for to synthesize, the second language but it doesn't really fool you into. Thinking it's real language and, the. Problem is is all of these small details are present many things and when. They're gone it just doesn't look right to our brain it, looks like somebody's. You know made out of clay or made, out of some completely uniform. Surface. As. You can see here the differences, in the highlight on her skin the different reflections, in each line and the subtle parallax, and eyelashes, themselves, you, can see the inclusion, and, transparent, that one i'm i sees a strand that the other one doesn't even see in the teeth to the top you. Can see all of these some, of these hills that when you remove them only mention people stop looking new and they become possible you can't believe put your finger on it and this is the same effect but. By converted. Movies can be off-putting and, why the death is so low and a converted movie because the more deputy introduced the, more obvious that these cues aren't present and the less natural, diverse so. There's. Important problem for us to solve first because everything, about the documentary grows, out of interviews that we captured so, we, meet, bro filmmakers, over, a year ago we ran a modest pig starter campaign and, some people here help us out and we really appreciate it to, raise money, to refine, and develop our own.

Stereoscopic. Volumetric. We do casual technology, and. We're. Calling it the really exciting name of the s 3d, gaming system, stereoscopic, 3d plus step and, it's small, a normal, camera footprint, which, is important, because it keeps our subjects feeling. Comfortable, they don't feel intimidated they're not like they're they're sitting inside the giant green quite a bit and. We can travel with it which makes it possible to go to our subjects instead of making them come to us or, to some specialized, recording sound stage which. Is important, because sometimes. We, just don't have the resources available to focus, on stage. So. Our system. Or current system this is a little bit of an older version of it but it's facing the same now is, a pair, of sony, 4k cameras if you're actually using the recording right now and. These consumer, cameras don't have frame sync unlike, our earlier GoPro quarter. Tech which she's probably saw in some of the video loops at the beginning but. It's an affordability, trade-off, to. Have superior, skin tones which. And low Englishman's which is much more important, in our use case that Frank's, and if we had live resources we've upgraded to a camera that supports frame sync but it isn't really necessary for somebody sitting in front of you and. The Kinect version 2 is still the best and cleanest step camera on the market and things. Like you know it's a real sense and Williams at Kimmel other stuff things that use stereo to depth just. Aren't good enough and. They get confused by, the regular bright way that we're trying to reproduce anyway. And. Then. The depth information, is, recorded onto a tepid laptop and. We use a really great depth capturing. Software called crackle point cloud which. Allows us to record, the depth stream the compressed format which, is very important if you're doing like an uncut four-hour long tape that, you can actually fit in on slate a reasonable, klausman. After. That we use affordable, depositing. Software, Blackmagic. Designs in fusion which i think is built is now into which, resolved which is free if, you're not doing like crazy stuff like this but. I think the paid version it's like 399. It's. Not the $10,000. For you and. It's, actually essentially used by the film industry to convert films into 3d anyway so a lot of the tools are already there and. We ingest. All. Our footage here we synchronize it we color grade it and color match it and. Then we composite, the dip recording which i think is looping now where, we take the depth information and, we project the left eye onto it and the right eye onto it and then we can sit back with a single, camera. Of the both discrete, views and this, reduces, this the stereo parallax, very little and. Then we have an image that has very small amounts of localized, stereo, but. It's only in the small features but the image itself has no. Parallax. On but we also have the dip map that goes along with it so. We output a video texture. That contains eleven image the right image and a depth map with a mattes that's a two-bit mask and then audio all. Inside of a regular HSS, core video file and, we can log and edit it just like a regular video which, is important, you've gotten really long interview, sessions and, it takes a lot of effort, and a lot of reviewing. A lot of logging to, manage that amount of footage. And. Then we then use this video to drive, a vertex shaker which displaces, a segment of like a spherical I mentioned, you can see the, shader here and.

This Way that because it's displacing along a. Spherical, mesh the polygons all are shooting, towards, the viewer head position. And, then what we do is we render the left texture in the left headset. Sweden. The right texture in the right fit set display and that, brings the stereoscopic effect back, into view which of course doesn't really translate on the screen here, and. Then the video files can be local or straight to a client and the examples, that are all here or just 60, megabits a second which is way smaller than like a regular 360, video file and. If you're viewing from just a few feet away and, you don't because, we don't report the back side we have enough information. That we can super sample Olivera displays or not you're not looking between the pixels your every, pixel you're looking at in your headset is representative, the video so it looks very crisp which is important, that, faces don't appear blurry which is another sign effective volume entity so. The you can see well why, not just shoot 3d, 360 video or 3d 180 and. We actually originally. Started prototyping with. 3d video and we quickly discovered that it just isn't good enough and the. Key thing is the inability to, lean. In or out, ruins that feeling, of bi-directional, presence. And. As you naturally change your body language like lean to the side or. Move, over or lean, forward you bottom alignment with the three degrees of freedom video and because, it's 3d, any of the two images you get huge amounts of ice cream so why is that. That's. Because the 3d cameras were fixed in space. There. Goes looking. At yeah, so you can see that the cameras for fixing space and they don't actually represent, where. Your virtual eyes are and. This causes huge amount of waste ratings your fine muscles, are desperately. Trying to align an image that just isn't, aligned. Alone you'll notice this to give or watch like a 3d. Video in the art you took your head to the side and there's something close to you it can be really really uncomfortable. And. The lack of six, degrees of freedom is another problem as you leap forward or back it's, simply not reflected, in what you see and again it embraces the illusion of a person virtually present. And. This, is important, not just from let's, say technical. Weird-looking. Point of view but also pure psychology. So, during this talk many of you did. This to. Listen. A little bit more or like you're thinking you're fine and they just think the information that we are sharing and. This, is what we do in our everyday everyday. Everyday. Lives, and those want, to eat inches, movements. Matter and this, is the fundamental fundamental. Aspect. Of our society societal. Interactions. So leaning in when you want to listen. Closer and catching. Some distance when you need to think about it and digest the stuff in so, especially, with this subject was very not easy subject I want you to be able to move closer when, you hear something that like, you really want to hear. To it or when, you think okay I need, mount time to listen or like to think what she was talking about so. That's why you seeks you and for ask was just. Essential, and. So we make this. So. We make this or topic video online onto 3d space so this way they don't have a lot, of parallax, to misalign as you move around so, they're kind of glued to, the pretty much and, here. You say, you can see how our process, changed over time so, they're like images before. And the, right is after, we. Use this money to update develop, our technology, so for example on the left you can see some issues with enhance they're like very wavy, and.

This Felt very very weird in PR and we were also thinking about some creative ways of addressing it but then, we decided it takes too much attention. Of, the viewer away, from the subject and we decided to spend our time on repairing that and. Then also the changing. Image quality from using how a higher quality, cameras that we started with and. Better. Skin, and orientation, so, these. Are just two things that, we focused, on like face being the most important, and hand, and there, is also some other stuff I. Feel. Some like waving on your body but. Being. Broken, feeling like we, decided, we had to decide what's, important, perhaps we had cake, in a year and they. Said and we. Had to decide what's. First. Of all was doable was. Doable within the reasonable. Time frame because, it's been over a year already and, people keep waiting, for us to be finished, and we would love to finish and. So. We have to decide that and also. We. Have to decide what can we do also being good to people a team. And. People. Want to help us that's awesome, or now that's two of us, so. With the capture technology, so our, next considerations. Are what, to do with the rest of the virtual space unlike. A, lot of other documentary, experiences, in VR we didn't want to place people inside the environment so, whether it would be their living, room or their workplace. We, don't want to collaborate with the, things that distract you and take your attention away a couple, of years ago Stanford, made a study and, the, study showing, that when, we have too much visual information which. Is not part of the story it's. Hard for us to ignore it so they call it the detective. Mode so you investigate. Every single element that's, there, and subconsciously. You worry that it will be important, so for example, if you would put us in VR somebody, would keep looking like all of my phone. Like. Why can I put this is, it important, we play the role in the story we, just wanted to remove all the clutter and that's, why with our capture, technology, we only captured, the images, of the people who share their stories because, they are the most important, for, the experience, and everything, else initially, everything else is just a black void, but.

The, Same Stanford study showed also that, using this 360. Volume makes the viewers remember less. Factual, informations, and so. Into the research the participants, were, watching, with, 90 degrees 180 and 360 and. With 90 degrees they. Remember the most factual. Informations, no matter if they were shown, with visual clues. Or with the, clues. But, they didn't really understand. Emotionally, what the scene is about and who is the person that they are looking at with, their full 360, degree they remembered, less details, but, they understood, much more of the context, and they had the feeling that they know who is the person that died that they are following and. In the choice understanding. Emotions of the people is actually crucial, to. Achieve our goals so having that in mind we, want to both use the whole 360 volume, 360. Space but, without cluttering so, how to do it and. We're very much inspired, by oculus, Studios, year angeleka, in a way that we dynamically. Dynamically. Bring things in and out of the black void so, they also use the audio to create a spatial. Soundscape, that's really emotionally, places, you in the scenes if you didn't watch it we very much recommend, it and, we've. The choice the, up Lloyd are the people so we the, rebook people, so we, record. Them with our technology, but we also plan to use smaller visual, illustrations. Made, with quill, of. Each of the stories so. To stimulate, your imagination so. To also, to enhance enhance, your emotional, connection and, this is just an example, that we did we're, not very good. Artists. In this way so. Another thing that we are looking for is for somebody who can, help us with that I. Mentioned. Initially that, the whole experience is told from the viewers perspective and. So we all know that VR has incredible, ability, of transporting, you to other places are the times but. When, analyzing, the abortion debate we realized that there is about half of the population and, much more than half of this room here that cannot really imagine, how it is to be pregnant to go through this decision even a fear of this decision not just going through that but just thought about it you, can a man you cannot imagine it so, that's why we, decided that we will use VR to, put you there to put you to.

Frame The experience is your, experience. So. The pregnancy the holding decision, my think will be happening to you and also to make it simply, more engaging and interesting. Some. Documentary, filmmakers, Errol Morris being, one of the pioneers, of it try, to get in touch with the viewer to the point where the audience almost sits. In the space and. But no matter how great, they are how many four, four holes they will break there, is still this window, that puts you always, as an observant, as a witness, not, being the part of the story, so with ER we have the opportunity, of putting you directly. Into someone's shoes and we very much want to do that in. Going back also to the Stanford study they. Also identified. Some other aspect, of VR so with changing, the viewers, position, we, change their understanding. Of the scene and of their own role even, if they have no role in the experience, just putting, them in the wrong spot makes them feel uncomfortable or. The taken on content. Concentrating, on the story and, here are examples of the viewer, this. Lady this. Way. So. When, she was putting at the bar she felt that she can observe the classroom, but she was feeling that now she's one of the students, and she has to actually so, my role was to upsell they'll. Done and follow his. History, but when she was put back there she, Jesus villain and she needs to listen to the future when she was put. Here. In. Between the teacher in the classroom she was feeling that oh I'm. In front of the teacher teacher, cannot share. The story the the lesson. Even, if it, did not really, apply, to the story because it was not the interactive, experience with just the media so, just putting the viewer into proper, space from, the perspective. Of the story is extremely important, and this is what we do with the chance so we start with you, and budding this humorous, avatar, we, create the story like it's happening to you and then, on your way you, find those women to share their stories and you sit there and listen, so, we both physically, and mentally with. The story design coaching, naturally, in the situation, of listening. We. Still have a long, way to go, funding. Institutions, and can't well I guess we're all in Canada Canadian. Funding institutions, where we're, all from we're. Now starting only now starting understand, 360, video is a serious, documentary. Being but. They still understand, why and non-commercial. Gammage. And the experiences necessary for a documentary, but. We're continue to work on their own steam and with our personal resources, improvement, for people who are helping us and, hopefully eventually things, like can counsel and come.

To Understand, that VR. Documentaries. 360. Good film necessarily, and. We're always looking for partnerships. Or anybody who's interested, in participating in the project and. You, know people reach out to, us with their personal stories from so many places that we get people from Texas. The last even, as far as DePaul and we're working really hard to. Try and get the documentary, more global voice instead of just our local community, and. That's kind of our current. Focus is, to, film the, interviews before really anything else especially. With the. Changes, in technology, now. That we have our capture figured out now. The focus is filming. The interview so interviews, interviews and, then once, that's all we have to store a book together we have the, greater narrative then it comes back, around work. Back towards to technology. And, thank you very much for coming everybody.

2019-04-10 23:42

Other news