Okay awesome oh. And. That's been testing yes okay. Let. Me welcome. Professor. Alexander. Clicquot. Who, is professor of geographic. Information science. And, affiliate. Professor of information Science. And Technology, at the Pennsylvania. State University. He's. The current cosmos, linear faculty. Scholar in the College of earth and mineral sciences. And. He, has been named the inaugural. Director of, Penn State's Center for immersive, experiences. Something you're building, a new center there I'm collecting inspirations. Including. The. Experiences. At UC Santa Barbara and. University. Of Melbourne, before. He joined Penn State in 2007. He. Found it and you need to say. Kuroh furnaces. Applied. Spatial, intelligence a research, team focusing. On advancing, applied and basic research, on immersive. Technologies. Is augmented. And virtual, reality. And, I think what's really interesting what, we didn't what wasn't really, in the flyer and, I. I wish it was. Is. The nispom, he, needs a lot of experience, in projects. In cognitive, science, and, his, focus of. Research is in spatial, cognition and I think we had some people here who are really interested, in there, so. We. Please. Welcome and, I. Like, to hear your presentation. Thank. You so much it's fantastic, to be here and I really meant it I already, had so, many inspiring conversations. And visiting. Your Center really has left me with a lot of ideas for what, we could do with the center, that is officially. Launched at the end of this, month I. Put. Together a, number of the projects that we. Have been running that Center, on place, and I think that most. Of us can agree that a, place for, anything. Geospatial. Has. An important, or. Is an important meaning and important, concept, this. Is just a flyer from a seminar that I almost gave this spring so I will try again next. Spring again, where. We. Talked about place and place as this combination, of a location, that has meaning, what I find particularly interesting about this approach. Is that it. Is not something that, speaks. To only the social, sciences. But, it also speaks, to physical, sciences, or physical. Scientists. Go to places, value. Being, at a place, immersing. Themselves in, a place to develop a deeper understanding of. For example processes, that help shape a place with. Immersive technologies. Which. We can organize, for example, long this continuum, that has been made. Famous by Milgram. In 1994. The virtuality continuum. Including. Everything from augmented. To. Virtual. Reality and sometimes. We now use the term mixed, reality, because. Of the sophistication, with, which we are able. To scan environments, and to integrate digital, content, into actual environment with, immersive, technologies, we, are able to bring.
Remote. Places, that. Are either distant. In terms of temporal, or, spatial. Aspects. Into. Research. Labs we can change them we can offer new perspectives. That. Have not been available, so, far so. There are two things about immersive, technologies, that really make them different one is actually, this aspect of being, immersed in the data so this. Is something that is. Almost. Completely, new, at least at the scale that we see it at the moment that, we can actually become, hard, off the, representation. Rather than being separated, from so. Instead of seeing data, or, any other thing that we, represent, that we collected about on, a screen, we, step into the data we, are surrounded by the data and we become part, of the representation. Can directly interact with it so, that's essentially, already the second part so, the aspect, of immersion, as well, as the aspect. Of being able. To interact with the data there's, a nice, figure. That, I took from, the internet, and I credited, here and so I hope that that's okay for your YouTube production that. Is symbolizing. This. Idea of presence, presence. Is. The. Sensor. Or the behavior. The emotional, response, to, being. Immersed, and. Chinese. Symbol, for presence. And. I'm not sure that it's entirely true but it's a good story. Comprises. The, different sensors, that, and contribute. To presence, so our auditory. Our visual. Senses, but, also speaks, to a deeper, level of the, emotional, connection, that we feel and then. What I find particularly important, is the undivided. Attention that, we can have while, being immersed at a place so, I think that this is something that again. Immersive. Technologies, are good, at that. They really capture, in, the way that they deliver the con and that other than technologies so far have. Had a challenge, of delivering. In the same way the. One thing that again making, a, making. A point about the possibilities. Is that so. Far we have primarily, seen sort. Of capturing, real-world. Or we have seen sci-fi, movies that allowed us to take a different perspectives, with. Immersive, technologies, we can do, exactly. This now at the. Level where we take, our bodies into that experience so the sci-fi movies, like ant-man or. The. Fantastic, voyages, for those that's, like the older generations, so. Is something that that we can actually now bring. Into, a lab, space bring into experiencing. Data bring into experiencing. Places, in. Sort, of from different angles that have not been had, that we have not done so, far. So. Is that little. Prequel. So. I will. Talk about two things one I just show you a couple of the workflows. That we, have been developing and then, the main aspect, that we are doing is actually empirically, evaluating. Immersive, experiences, to, essentially being able to separate, the wow, effect the, novelty, effect the, or that we all experience, when we put on a headset from, the actual, effectiveness. And. Efficiency, of this. New medium in, communicating. In helping us to make decisions so the, workflow I have two got two, examples one is a immersive. Were claims for geosciences. So, for this project. We, use. Data, from. A, colleague of mine Pete the Femina about. The trinakha gigavolt. Kino on Iceland, it's. Actually. A dormant volcano so, it's not quite as spectacular as it sounds when I say, we. Flew drones into, this volcano so it is actually a volcano. That has an elevator so in. Addition to actually, flying a drone into the volcano, he. Also was, able to haul down terrestrial. Lidar equipment. Into the volcano, and then create, measurements, so. What we did was, we pulled, the. Lidar data as well as data from, photogrammetry. Into. Unity. Game. Engine that allowed us to design. A, system, around the. Data and to. Perform, or help people analyze, the, data in this, kind of immersive, new way so this is an extra DM, from. Iceland, the, sky is something where the students got carried away so that's not bad there's, no I said sky but that's. Okay so it's a creative creative. Freedom there. It's. A little bit slow at the beginning as. Well but. You essentially can then select the, volcano. And be, inside, the volcano this. Is the lighter point cloud with the colors being the different, intensities. Of the data, reflection, we then have a work, work. Solution. And, form of a menu where you can then make use of all, the advantages, that, digital. Environments. Have by, providing additional information by. Reclassifying, for. Example, the lidar data. By. Switching. Back and forth different.
Different. Styles, to select whatever works best for a particular, type of analysis, so here the, switch between a. The. Lidar model itself and then the. Photogrammetry. Model, and the. Quality. Is a. Little bit better on my screen then on the on, the big screen so with. The developments. That we see we can essentially, create more, and more realistic environments, in. In. Here, we. Can perform. All kinds of measurements, so such as measuring, distances. Immersive. Experiences. Are actually the only form, of representation, that, allows us, to experience. What. The actual, size the actual scale of an, object, is they are of course limitations. And. Of course if you are interested in, cutlery, you can of course have, also the, one-to-one, scale on a computer. Screen but. This, structure. Of the volcano is about 600. 600, feet so. In the immersive experience, you can actually blow, it up to the actual scale and then really experience, it as it is one-to-one, which is not possible, essentially, in any other in. Any other. Digital. Environment, that, you put on a put, on the screen, the, second project and I talked, about this quite, a bit with. Several. Of you this morning is, an ACEF project, on visualizing forest, futures it's. A project that is. Located. In Wisconsin, to gather with the Menominee. Who. Are very much in to. Forestry. And forestry is a very central part of their cultural, cultural. Heritage, it's. A project that we have, together with Rob Schiller and Melissa, Lucas. And a couple of other people, what. We try. To accomplish in this project as one part is, to, translate, abstract. Data that. We get from models, such as Landers. Into. Something. That becomes experiential. So many of you now lend us a lot better than then. I do it's an ecology, model, that takes, for example. Input. From climate, change model, and then produces. Output, that. Gives us an idea of how certain, tree species for, example behave. Under. Climate change the. Data that, you normally get from Landers, is, that, for each map cell you. Get biomass. For. A particular tree species which, is rather abstract, so, what we were asking, ourselves then, is what do we need to do in order, to translate, the abstract information. Into. Some parameters. That allow us to create a 3d model and then also, turn this 3d model into something, that we can experience what. We used is the FIA, database, the forest inventory analysis, program. That. Has four specific plots. Species. Age biomass, height weight height. Crown, widths and the, number of trees for. Specific, plots by, regressing, over this, information. We, were then able to, translate. The data that we get from Landis into, the parameters, that we need for, the actual action. Spirit experience, in form of species. Tree, height crown. Waves and then also number, of trees, so. This is a, very, small plot that. From. An overview perspective but if you actually zoom in you. Get actually, the, chance, to see, a forest, or see the data as. It would be for example in, 2050. Under certain climate change scenarios. We, are currently working on turning this into an, immersive. Workbench, as well so where you can for example switch, on and off certain tree species look. At the landscapes, from different. Elevations. And make, the whole thing interactive. As far if a short. Video it's, not the most recent, video, but. It shows that we. Are not Pixar so, but. The quality. That we. Can actually produce as. Scientists. In these projects, or in these kind of workflows, is actually quite substantial.
And With all the discussions, that we had this morning already about. Blender. And other software, so if anything it will become much much more accessible in this, form of communicating. Results. From. Scientific. Analysis, is something that is really in the grasp of of. Many more people these days the. Forest of the future doesn't have, watermarks. That's, just because we are cheap so, we just used the educational, version and not the paid. $3,000. Version, for. This again but yeah you can make it interactive you, can use it for educational, purposes, you, can compare, different different. Scenarios, for, example with them without climate, change you. Can focus, on particular tree species you. Can give different different. Elevated perspectives, so all of this is possible as. A very, new form of interacting. With the place that, we have not had a chance to experience because it, is in the future. So, we do these things and I, with. The examples, I almost stopped here so for all kinds of different different. Places so we have projects. We, create immersive experience, for for. Wetlands laughter. For. Wetlands with, just a drone, image here. And it's we, do primarily, 360, degree. Images we. Have projects, on energy, so this is a fracking site in northern Pennsylvania, we. Do immersive, experience for fire ecology, in California. This is the issue. Wildness, in the lassen forest and, we, also had. A fun project with the Mayan ruins in, Belize. This is two non tonnage in the central. Part central lowlands of of, Belize. I, have. A short video of, a. Slightly different approach, so, often what we do is we have these kind of map examples, that. We use to contextualize, these, these, tours. So. What. You see here is. Just just. An example, of. One. Of the actual. Apps that we create so it works on Android phones oculus. Go or. On, high-end, HTC, vive versions, what. You've just seen is we experiment, with all kinds of aspects.
That We can integrate, here. So, for example offering, elevated. Perspectives, we have a 27, foot tripods that we often take into the field to give an overview perspective if, we're not able to fly drawings and then, what you've seen it here as well is that this new medium actually requires thinking, of how the communication. Storytelling. Works so. If there is for example a particular location. That you, want your audience, to focus on you. Need to find a way to actually guide them to that location because, the beauty, of the 360, immersive, experience, is that you actually give your audience agency, agency. In deciding, to where they want to look the. Downside, is that they. Can, have, take, this agency, and look wherever they want so they don't necessarily focus, on what you want them to focus on so. We started experimenting, and implementing. These visual guidelines, that, help, users. To and also follow for example in array in, a way that you. Normally would follow if you just watch a video where you're kind of confined, to a single perspective, this perspective of, off, the camera. More. Importantly, so. The empirical, evaluation so, and I'm picked out one of the main. Main. Research, thrusts, that we currently have and, that focuses, on immersive, virtual, futures, so. Immersive first a few trips just at the. Beginning so, normally. When we try to educate students. We, are white you have these beautiful spaces here but a lot, of the times we have. Crowded. Lecture halls and. Especially if you talk about something. That happens, in the real world that's relevant for a particular place, you much, more, much. More would like them to actually take them there contextualize, in scaffold, the learning content, with. The relevance. Of the place itself. In-state. Beautiful, State College my. Colleagues, do that but. With limitations so. Normally, you have lab periods of two hours and. That, means that you have a 12 mile radius of the, sides that you potentially can, take students, to or, you have to use the spring break or the summer break for extended, periods so how can we how can we scale, so. We thought, about this for quite a while and came up with, well. Some alternatives, and then also a research, framework, that, allowed, us to kind of structure, not. Just the development but also advancing. Sort of the science behind how we deliver these, kind of immersive, virtual, field trips this, research framework has two components. One. Is what, we call a taxonomy. Or the virtual, field trip taxonomy, and it's, relatively, straightforward so, it's nothing dramatic so, we make a distinction between. Basic. Plus. And advanced, virtual few trips a, basic, one basically. Would replicate the, environment, as it is so this is the most straightforward way of capturing of, a particular, a particular, place. The. Plus one allows. For, capturing, the environment, but from respect. Or in a way that is, not possible in physical reality so, in this case here it's. A, ground, perspective and then on the right side there's an elevated perspective in, case, of this outcrop, you, see that when you elevate the perspective, you out of a sudden see the outcrop in a very very different way than, you would from a ground perspective and the same thing is true if you have a drone image and for example see how, a wetland, is embedded, in the actual in the actual landscape additionally. You could imagine. Bringing, two sides together. That. Are normally is spatially distant, to compare, them compare, them directly still.
All Of those things are possible with, capturing. What the real environment actually, looked, like so potentially, very. Minimal. Development. Cost, and. Then the advanced, version of, Crystal. Button and the, advanced version is then where we have, simulations. For example about a future state of environment, and I showed this before so, if you have a climate, change model, that you used to feed information, into an ecology, model and produce, output, off for, example future state of an environment such as a forest that, would be sort of the Plus, version again. With immersive technologies, in combination, with the 3d, modeling advanced that we have, seen at the moment it is possible, but it requires a fair bit of, development. Cost, the. Second component, of this framework that we, have developed is, what we call sensation. And, so it's a bit of a mouthful so it's the sensing, scalability. Trade-off, continuum, so. Yes. So, what. This essentially, captures, is that, we. Have, so. If we talk about sensing, and Sensibility. Then. We. Have a low-end version, such, as an oculus, gol that has, three degrees of freedom, where. You, meaningfully. Can turn your head but. You not can meaningfully do, any lateral movements, it just doesn't capture that, particular movement on. The high end then or moving towards the higher end we. Have things such as an, HTC, vive and, we have body tracking. I tracking so, we can integrate a lot of sensors, into, the experience, that then also allow for, tuning, and catering. Tailoring, the experience, to, the user and the user experience itself. Of course, with the development, of an oculus, quest for example, we move sort of the lower, end spectrum, to the higher end spectrum as well but, the point is and that's. Why we organized, it as as. A coordinate. System the, point is to be able to say how, much, do, I need to actually invest to get learning. Output, that I desire. So, can I deliver a learning.
Environment A, learning experience or a decision-making, experience, with the low end costs with. A with, a low end was the basic version or do. I need to invest, in high end gear. And do. I need to invest in creating and curating high. End simulated. Advancements. Of sort, of environment, in in, the future so, this, curve is. Sort. Of the. Trade-off curve, between. Sort of making things more. Expensive, and, less accessible. Or. Vice-versa and of, course it could also be in the extremes, so. We could find, out that we really just need an, oculus, goal to, achieve, our goals or we, find out that we really meet both. Very, advanced, and a, very high end high, end system, the, numbers that you see in the, middle here those, are the experiments. That we have run already, so we started, in, 2017. Was, actually an HTC, vive. And. Basic. Version and. Then we have advanced, this to the Plus version and, I come back to that and then, we also reduced, it again to go when the Occulus goal came became, available last summer to. Develop for for. Low-end. Version as well again, was the idea of so what do we need to do to scale, this up to make this a more, regular, experience. In, in. The classroom and, I'm not going to detail there too, much but we also create spin-offs in terms. Of singling. Out particular. Aspects, that, we then. In very, basic research. Experiments. And bezzie research, setups. So. For, this. First. Field trip it's, called the read to a board ego so if you're into, geology, it's a formation. In. Pennsylvania. We. Had an, actual, field trip. And a, virtual field trip so we used the introductory. Geosciences. Class that, allowed us to split the class into two, groups, where then one group would go into the field the other one would go into into. The lab so that we could actually make experiments. A comparison. This. Is the. Festival. Got the. Feed. Side from from. A bath we, used a combination of. 360. Images. So. This is the numbers. That you see here and, then at location, number six we, also used, a 3d. Model derived, from structure for motion so, that the students, were able to do the exact same exercise. That, they do in, the field as, well, which is measuring, the widths of the rock, layers to create stratigraphic. Maps. The. Students, were of this just, the outcrop from the site from, it the distance. They. Were able to use. Google. Street you like navigation. Between the images in form of arrows that only were available in, the order that they, would normally go on the actual field trip, and. There. Were. Parts. Of the outcrop, or in the images where, they were able to get more information again information, that they normally have in their, lab handbook, that they can use for example to. Classify, different grain, sizes, and. Then this is an. Example. Of the outcrop, that we've created the 3d model where. The students, then. Measured. The width of the rock. Layers they. Collect. The data it's. Recorded, and then sent to them by email at the end of their experience. So that they were able to do their homework in, very much the same way that students. In the actual future we're, to do this so what are the results. We. Find that if we ask students, about, their, self, assess learning experience, that, the, students, in the, virtual. Future actually, significantly. More. Agree with the statement such as I learned a lot from this. Few trip, we, also have composite. Scores of. Overall. Enjoyment for. Example there are 4 different questions, that we ask if. We take this together we also find that students, are much, more. Happy. With the with, the virtual future compared to the, actual future and I'm one thing that we were very. Surprised. By because, our minimum requirement, was that this. Should at least not mess up students, grades is that. In this first experiment students, lab grades were also, higher than the. Ones it, when they went to the virtual future compared to the actual future there, can be lots and lots of reasons right so it's not just hey I put on a headset and then all my problems are gone so there, are no mobile phones in, virtual, experience at least we didn't program them in so. There are lots of things that are that, are sort. Of additional, factors, but, we, definitely met. The bar with or exceeded, the bar that we set, ourselves.
It. Was the with this stuff, he. Also, replicated. This study so. In the spring of 2018. We. Had, a. Slightly. More high number of students, but, we also did as we moved the. Basic. Virtual future into the realm of the Plus version future, so in the first one you only had ground images, in the second one we had an. Option for the students to go to the elevated images of 27, feet high. There's, a quickly thing here that happened and it. Was a different instructor. Extremely. Supportive, gave clicker points and everything but at the request. That all students, need to go to the actual future and not, just into. The. Virtual future, so. There's a very very strong belief in the physical sciences, that you cannot deprive students, from going into the field and, quite. Literally getting their hands dirty which has, absolute. Merits so, what we did then is that we had basically the virtual, future as a. Additional. Requirement. Or additional preparation for the actual future which also allowed us to ask slightly, different questions such as how, well did you actually be. Prepared, for, the, actual field trip so, what we find is that there. Was an, overwhelming. Sort. Of enthusiasm, on. The side of the virtual, future participants that they actually felt a lot better prepared, for the actual future everything, else would be very surprising if they didn't because in addition to the lectures they went through this virtual future, but if, anything. It is something that even. Students. Were even instructors. Who believed in going to actual a few trips if, they really want to prepare their students, they could spend, half, an hour in. The lab preparing. Them through for example a virtual future we. Were able to replicate most. Of the other findings as well with. Great success for. Questions. Such as I learned a lot from the field trip as well as the overall enjoyment, where, we did not find differences, this time just a slightly, better. Value. For, was. That grades so, which. Again. That grades are very crude instrument. So. And they did the lab a homework, assignment after they basically went through both. Experiences. As well as lab grades have sort of a certain, tendency. To be sort of inflated. So, we are working on more. Rubric, highest version, of lab grades that allow us to answer and more, in depth in-depth, question, the, one thing that I would like to single out here so that's the question of going. To the what does actually going to the Plus version mean. And. What we, see. Here is a very, sort of simple way of basically, getting onto the 27. Feet perspective, so students were able to look up there, was a rat. Disk, they, clicked on the disk and then they have the overview perspective. From, 27, feet height. We. Measured the response. To that, through. Still. Pressing the robot through. A, very. Slow learner and so so. We measured the response to that through a construct, that's widely, used in the. Media. Of psychology, it's, called the. Spatial, presence, questionnaire. So, it essentially assesses, how present. After, exposure. To a medium, students. Feel, about. A place for example so what we see here might, be compared, is so the FT group is, the first. Actual. Future group we, did not see which, is attina we, saw an improvement. Between. The. Actual. Future group and the basic. Virtual future group if we, then add the elevated, perspective, what, we find is then actually a statistically. Significant, difference if, we compare, the. Plus. Version so, the one with the elevated perspective, to. The, actual. Future version, so by giving. Students, access to, perspectives. That are not possible in, the actual reward we can enhance their. Experience we can give them a better understanding of. The environments, and I will come back to that we actually now at running experiments that help us then to really quantify, not. Just in terms of a qualitative, assessment but, in terms of actually, objective, metrics, what. This then means in terms of spatial learning. We. Continued, with this with, this study and, in the fall of. 2018. And, we ran the study again as. What. We call it. Immersive. Learning in, the wild the. Reason why we call it like this is that again, a quirky, coincidence, and brought, us in the fortunate, situation, that, there, was no actual future possible, anymore there, is construction, going on like everywhere and, they. Widened the road and the, actual, field side is the blast zone at the moment so.
So. There's. No replacement for it so the, course. Actually did not have the chance to send anyone into the field so. And we said hey we have a virtual version of so why don't you take that one in. The fall it worked it does not always work it's kind of amazing that even, there's. A very very intense. Belief, that no, no you cannot just have a virtual version so we're working on that but, in. The fall of 2018, we, had out of a sudden 76, students, that needed to go through the experience, so. In order to, manage. These, higher, numbers of students, we, did two things one, was that. We used all the actually. Publicly, available VR, spaces, on campus so. It's not our lab anymore it, is a place. Called the dreamer II and if, you know Penn State there's a place called creamery so they have this word play with this it's, an emerging, technology environment. Where students. Can go and where they have an HTC, vive we, have an IMAX, and immersive experience nabrit ori where. We, had. Oculus. Go so we developed, a virtual. Future, for oculus goals, we. Have a VR room in two libraries the. Pattee library were just the main library and then also, our, mineral. Sciences library, so, students, instead of coming to our, lab actually, went into publicly. Available spaces, signed, up for a slot and went through the future so that's the ideal situation because, that's basically sort, of the long-term goal that we, don't have to administer how the virtual few trips but that instructors. Basically can sign up and send. Their students into the interface, rooms so. This is just an example so, we. Had to rethink, the. Design of some, of the activities, the. 360, images are very easy but. Then the measuring. That required a little bit more. More. Thought. Through. Because. It, was just a single controller, that, has, very. Different different. Forms. Of. Sensing. Than, for, example being able to use an HTC vive controller. We. Managed, to do that and this, is just an example of how students, on swivel, chairs would. Then have the experience, so in this case we were not confident. Having like a number of students, just standing around and potentially. In bumping. Into each other. There's, a there's a lot of things to say here necessary, it's really in the white it's a study that had. 76, students, of those 76, students, 44.
Actually. Agreed to participate, in the study. Largely. Because they. Had to fill out a questionnaire before. They came to class so. Which, has the consent form and so they, filled out the afterwords form but they didn't fill out the form that we sent to them by email right so. And then we had four locations so, 44, students, we, had actually, 26 in there or. 20-25. In the goal, version so, so. It's a 2 small number to make anything out definite. But what, we were actually. Experiencing. Was something that we did not expect and I just saw one example so. This is, comparing. The. Learning. Experience of the self assess learning experience, of the 3 different semesters, in which we had, have run this and. Again there's a lot more to say about this. And but what we see is that if. We really roll out so this is business. This, is was the first then. We had the second, one which was slightly improved. And. Then these two are the ones that were in the fall with the entire class so. What we see is that, it. That, the. Vibe so. This is the rules where the students, went into all by themselves I actually, had the worst outcome. So, students, at least when they're not having. Had this experience being. Sent into a room in the library it did not work particularly, well, there's, also a kind of a quirky bias in here in that, so, we did the pre-assessment, and post assessment that there was a higher. Number of students, who were skeptical, about immersive. Experience. To start with because it was the entire class so, it was not self, selection, or volunteer, basis, anymore so, we had a fair. Number of what, you might call sort of hardcore physical, scientists, in there who, don't believe in immersive, experiences, and because they were able to sign up by themselves and we have small numbers we had much, larger number, think. Seven Oh compared. To two in. The vice version, compared, to the gold version the. Gold version also was a group experience so, the students, were there an instructor. Not in one, of the our lab assistants, was there and really gave them sort of the tour so. It was an overall very different experience, than just being by themselves in and, Bellaire so. The bottom line is that if we really want to roll. Out immersive. Experiences. As sort of the large scale learning experiences, that. We, have, still a fair bit to go and that it is simply, not quite for everyone, for, students who like it or, we are open to it that's definitely, an excellent. Excellent, opportunity. What, we had hoped to do this semester but what got postpones. For. Organizational. Challenges. That we were facing is to, actually have a class, with, more than one immersive. Experiences, so in the fall we, will have a class where students, go through three. Virtual. Field trips so, and that then will allow us to assess whether. These curves go go. For example up or down so if someone so, they're, all effect people, put the headset on so they've all that's so cool might. Go down but, then sort of being sent to the libraries, by themselves, doing a future actually might go up because when they do it the second times and I know what I'm doing so so. There's a lot of things that will potentially, change, if we, now really use these immersive experiences. As, large-scale in the white kind. Of experiences. So. Just very briefly a couple of, pressing. The Run button, very. Hard so, a couple of current developments, that, then also spin off into, into. What I said that we single out very. Basic, research questions, that surround not. Just immersive, futures but all kinds of immersive. Experiences. So. One thing that we have, started looking into, is locomotion. In VR so. If you have a VR. Experiences. Experience. In most cases your physical, space is smaller than the. Space. That you're representing, so if you have a forest or an outcrop, so even if you have a, a 3d, model, eventually. You run into, the wall so. You need to come up with some form, of locomotion that, allows you to get from A to B, if. B is, outside, sort of your actual, physical space that you have available so, people have come up with all kinds of funky. Solutions, that that, you can do to, get from A to B, what. We did is we, looked into. It, so. What we did is we used. The. Standard versions. So. That. Is available in the wife and. We, actually implemented. This into, the desktop as well so what you most. People know from playing computer games on desktop is that you, have. Continuous. Translation. So you have a you. Have a movement that is smooth that is continuous, for. Example using a mouse and your entire arrow keys.
In. The HTC, vive if you do that you, would definitely get cyber. Sick or motion sick there's, a there's, a trick that Google, implemented, Google Earth VR where. They, reduce, the field of view and they blur the ground if you're moving and that. Actually does the trick but that's only came up after, actually we did the first. Experiment, on this one or the. Alternative is you can reduce, the, motion. To. Something, that's a meter per second, so it's really really really slow, so. In, the vive the most common form is the teleportation. If, you, do teleportation. Though you lose a lot of information right so the, information that we're normally used to in form of optical flow is not present anymore, but. You. Have the headset on and you get a lot of proprioceptive, information about, the environment as well because. Turning. Your head is actually meaningful meaningful, - so, we were actually kind of convinced, that the vive again. You put the headset on you said this is so cool in comparison, to just looking, at the screen would, outperform, any desktop, version, and, of, course we were proven wrong, so. This is actually not the case so. In this case here app is actually bad, so, this is the error there, the students, the participants made afterwards. So they learned an environment, which is called certain, that. We reimplemented. Into. An, environment. Where, you learn. Landmarks. Along a route and then length marks between two routes and. The. Green is the HTC. Vive version. And, then the blue, ones are the desktop, teleportation. As well, as the desktop, continuous. Translation, and the continuous, motion, there. Are no statistically. Significant, differences, which is good but if anything, is were. Sort of a concernings, of increase, in the pointing, error that the participant, made in, the. In device version, so what, this ultimately means, is that as. Awesome. As it is to be immersed if you, combine it with a form of locomotion. That, we are not used to you might actually use or you might actually lose the advantages, we. Are currently trying to explore this in in, greater depth. The. The. Other aspect that I briefly wanted to mention. Is scale so. I mentioned that when, you basically give someone an elevated, perspective you, increase, the scale that is accessible, to, a. User, at a single, vantage point and that, is awesome, because it. Allows for offloading, information. That you normally need to keep in, your memory. In your working memory into, the environment. So. We. Have. Seen that it had positive, effects in the virtual, field trip so we designed an. Experiment that is more controlled, in, form of a maze so. The, maze has again, the same two distinctions. Ground. Perspective and an elevated perspective at. The elevated perspective, the number of landmarks, available. Is. Increased. So that you can actually see more than one landmark at the same same. Time which we thought would have students. To create, a better mental. Representation. Of of, the, maze, we. Are currently running, the. Experiment. As well to increase the number of of, participants. That we have and, we're also currently, running. Experiment. Where we only have the elevated perspective, but not the ground perspective because. What. Because. What we found is that we. A. Lot of things actually were as expected, so, what you see here is that, we separated, this participants. Into a high and low spatial ability, students. And the. Light, blue ones, or. The blue dark blue ones the highest spatial ability, students. Outperform. The. Low spatial ability, students but what, we see and again although there's no statistically, significant, differences is that while, the low, spatial ability, students actually did show an improvement when. We gave them the ground, and elevated, perspective the, high, spatial. Ability student actually did not show an improvement and if anything they. Got they got worse we did this actually in two trials so, we also find that students, over time get better but. Again this also replicates, this this. Finding that especially. The high spatial ability, students did not improve. Drastically. What. We had to do though is again so this is then sort of the nitty-gritty parts, of designing, an experiment where you don't want to. Change. Things that are not part of the experiment is that you actually need to keep their time constant. Right, in order you don't want to give students who also go to the elevated perspective more time so. What we actually find is that so if you go through the, maze, and then, you have to split your time at every other location, to go up and down again that. Actually so we measured this as well as the cognitive load that actually increased, the cognitive, load of the, participants, so, as well intentioned as giving elevated, Prospectus is if. You turn if you put it into this kind of rigorous context, often of an, experiment.
Of A controlled, experiment it actually might not turn out the way that you so we're, currently a running. The experiment, again was just the elevated perspective and then also have another paradigm, that you're using, where students. Have free exploration. And basically, be measured at the time it, takes them to learn this environment so again it's something that I think is a very important, message that as, much, as we love, the immerse, experiences, an obvious and an, advocate for them it. Is not as straightforward to, show that they actually have the advantages. That we would like them like. Them to see. How. Am i doing on time. Okay. I can I can I can stop. At. Any time so one of the one of the things that again I just wanted to briefly show. As an development is that we also develop, joint experiences. So. We are for example connected. 12. To 15. Oculus. Goes into. The same experience, so, we just ran a study, on that with, the energy, side that I showed at the beginning so. How, this looks is that you essentially have elite, client, and, then you have a following, client, so, this is the leaf Kline and this is the following client it's, the same principle, with, the difference, that the lead client, now can basically, was. In control of the, location, again, the leaf line is not in control of where the student, is looking right so this. Again then requires, thinking, through so. How do we how do we get students, to focus on, the. Aspects. That we are actually talking about so what we implemented. Is and that, should show up in a in. A second here is. This. Pointing. Mode that is, it money, yep, so it's a it's the pointing mode so, the lead client basically can set a location. That's, of relevant, in the experience, and then the following, client, would see the will. See the arrows again coming up guiding, the. Participant. User, to. That particular, to, that particular location, and. That's something that, again, for an experience is essentially like Google. Expeditions, but. Flexible, and, we can use our own content. And we can do our own own. Development, so. That has been working, working. Particularly, well for the, for, the energy side. How, much mental capacity, to you have more. One more video okay, so can something completely different at the end but there's also an important, and important, message, so, we did also a study on second, language now right, so language, learning again, is best supported by, immersing. Yourself in, the environment, where, you want to learn the language off right that's why we send students overseas to learn language, rather than studying the book so. The, same could then apply for creating. Immersive environments. That are more flexible that allow, us to immerse, ourselves in, that environment so we did that and we created two environments, actually one. Is a kitchen. That's, fully interactive, and. One, is a. One. Was a zoo that, was not, interactive. So, because, you don't cut it with crocodiles, and. Tigers. And lions so, but. The, kitchen. Is interactive, so you can switch on the water. Again. It's the, interactivity, is not with gloves it's just with the controllers, but. You can point to you, can point to objects. It then reads the, name. In Mandarin so to, you you, can yes. Strangely. Enough most, people like to trash, the kitchen so it's really about it's, one of those earnest versio okay throw these things around so that's. I'm. Not we have done not any Studies on that but I would say that 70%. Of people at some point discover. Here I can throw this stuff so, so. And that's that's basically a, sort of the the, general the general setup and. Again without going into too much detail, so one is the kitchen definitely performed, better than the, embassy. So the adding this interactivity. Was. Was. Helpful, and, then.
The. Most important, thing that I just wanted to briefly point. Out is that. We. Did not find so we, found. Differences, but only after we split the group. Of participants into, again. High and low achieving. Students. In terms of language learning so we assessed this before, they did the with the study. So what we find is that if, students, have sort of a high aptitude for language learning then, it. Doesn't matter what kind of environment, we give them and actually, what we gave them was not, a textbook in comparison, but we gave them the. Same environment, as a desktop, experience so. Putting, on the headset for, high. Achieving. Students, did not make a difference for low achieving students though we do find that students. With the. Virtual. Realities though the blue one actually. Had a significantly. Higher score, than than. The students with just. The virtual world but accesses. Through a desktop so another, important, thing that certain. Learners or certain decision-making, might, be best, supported. Through, an immersive experiences. But. It may not be sort, of the one-size-fits-all for. Everyone. Well. That's essentially, it. So I thank you for your attention and I think all my collaborators. And especially all my students without. Whom all of this would not be possible so thank you. Yes. Yes. Yes. So. This is a sense. There are two routes, that we try to explore, at the moment so, they're, easy, court, what easiest route is the joint experience, that we developed and that's, something that you. Can use in exactly the same way where. You, take your students on a tour you, explain, in some sync to them you're either in the same room or because, everything, goes to a server anyway they, can actually be anywhere, so you can really take anyone you like onto, that onto that trip and you, can they. Can here they can hear you and you can and you can hear them so if it is really about asking, questions and, then, getting responses, so that is that is basically technology. That's there that that, is not a not a problem at all if, you want to integrate this into an, experience. That is standalone, so where the student can go to the. Library, sign, up for an immersive experience because, for example for. Some parts, of the experience you need sort of the high end, interactivity. That only, an HTC. Vive room-scale. Experience, would allow, then. Then. There. Will be possible. Possibilities. For that but, that's probably a little bit further off so if you look into for. Example speech. Recognition, and, simulation. Training, then. There. Are already a lot of a lot of opportunities. That the. More the, more constrained. The. More. Constrained, the, answer, question, answer. Sequence. Is the. Easier it will be to realize, that so, if you have a completely, open conversation, that's I think far off but. If you have a certain. Sort, of idea, of what the responses. Should be even, as a conversational. Sort of agent, that you can implement that's that's already possible so, but it's not like completely, flexible so if they if they, have a question that's completely, out, of the, blue unrelated. To anything, else related to a movie they watched last night then, of course there's no system, that would be able to answer that but sort of whistling limitations. Conversation. And agents actually have come a very very long way and would. Allow you to actually mimic, or at least, mimic some of that that, as well but if you are happy to be in. The room and you can take the students with you then that's already technology. That's there that's, also not even confined.
To A single, location but you can do it at different, locations, it's actually one of the things that at Penn, State has, a lot of Appeal those, Penn State has the main campus, and then, it has 23. Satellite, campuses. Across the Commonwealth, and they. Sometimes have very very different learning. Experiences. So. Having something that creates. Centralized. Of. Material. Really, allows us then in this forum to share it because they, either, need to sort of adjust their scheduling, which might be like a very, way way to vend demanding, tasks but, it could also be a resource, where the faculty, can then send students to a dedicated, VR room and they, still can have at, least sort, of somewhat, of an, a few trip even though it's not the actual future but they can have a shared, experience or. An experience, that is the same at other campuses as well. Yes. Yes. So there there are lots of different up so this yes perfect, spot on so the questions. And responses that we have, and. We, put it on our to-do, list and agenda so. There's there, are options, for note. Keeping, again. It depends on if you have a fully immersive one you really don't want to start, sketching, with I mean there's, there's. Google tilt, brush right so you can sketch like wild in 3d, for example. There's. Ways, of like doing that. Somewhat. On a on a notebook as well what. I think is a better, opportunity, for, these kind of exercise, is actually going to sort, of the mixed reality version, of that so, even. When you have sort of the system, such as as, an, octa. Is. An oculus rift or an. HTC, vive, you. Can have like little cameras, that you put to, the front they basically stream, directly sort, of the room content, to, the. To the, to. The student so. You can have a desk, at the side where if you have a sketching, environment, you can actually we. Use the normal sketching. Betty or the normal hand fighting that you do which is tremendously. More sophisticated, than anything that you have at, the moment at least if you have sort of fine-grained one so, that is a that is a possibility, that. You can that he can explore the. Other possibility, is to have sort of a like, if you have in lots of computer, games as well you have sort of this ability to have a camera with you you, can take pictures and. You can select maybe from a. Prefabricated. Way, of sort of annotating, the pictures, so. If you can just select from a text like copy and paste put it into your own notebook, and, keep the items in there and that's of course a possibility, as well but it's not the, actual activity. Of sketching, something. Yes. So there are solutions and these are the questions that we also get for. The touch one it's the same thing so it's a it's a way of having. Basically. A mixed reality experience, so, if it's important, to fear. For example different grain sizes ideally. You would have a lock somewhere, that. You can integrate into the experience, and again the two versions, of doing so would be to either, have. A mixed, reality experience, where you can basically have at some point just the video stream.
Sort. Of to, you or if you go, super high end you can have in a holo lens for example, and you use that and. Have, the actual lock there or you, can create a digital version off the rock as well attach, a attach. A sensor and basically. Have the rock and you can pick it up we, would see the rock as a, 3d. Model for example using structure from motion but. You have the sensory input through your through your hands as well so, there are solutions again. If you think about the that's, why we have the sensation right so, the, so this is where we try to sort of maneuver. At the moment so we can do all of those things but, they, come at a cost, so. The question then is so for, which learning, experience, do we actually need these and which, learning experiences, are perfectly. Fine was just sort of the more basic. Yeah. No no it's, alright so yeah no no no I like so I I turn, actually the strange thing for me is I was a lab rat and then, ever since I do the immersive experience, I actually go on all these few trips so so and so that's a, yes so yes. Yes. Yes. Yes.
2019-04-20