hello everybody and welcome to another lecture of 6837 today we're going to pretty much conclude our discussion of computer graphics by considering the very last piece of the computer graphics puzzle which is the output device specifically today we're going to focus a little bit on the mechanics of digital display technology in order to show at least a very rough outline of how the work that you've put into producing bits describing an image actually produces something in the visual world right so now we've completed our entire graphics pipeline from a scene description to our rendering and animation to the graphics card and all the way out to the display device now as a quick disclaimer at the beginning of today's lecture i should warn all of you guys that i am not an expert on hardware if you go to my webpage you'll see that the research that my students and i carry out is very much sort of theoretical rather than applied especially judged by the hardware access so we're going to have some informal descriptions of what's going on in output devices but we're not going to dive into many of the technical specs largely because your instructor doesn't know a whole lot about them that said you are currently in a unique department uh relative to other computer science departments throughout the united states and the world in that we are a department of electrical engineering and computer science so even without leaving course six here at mit you can probably find really interesting courses that go through this kind of material for an entire semester instead of one kind of superficial lecture at the end of a graphics course okay so let's get started today our outline is pretty straightforward we'll talk about two-dimensional display technology that's probably what you're looking at right now as you listen to today's lecture we'll talk a little bit about 3d displays and some of the emerging technology that exists in that world and we'll conclude with a bit of discussion of virtual reality and augmented reality displays which are becoming increasingly popular especially as the loop between vision and graphics becomes ever tighter and as hardware becomes efficient enough to make for a responsive virtual reality display system so let's get started we'll start by talking about 2d displays which i think is pretty much where most of our core 6837 has been targeted so a good keyword to know from the graphics universe which is used quite a bit especially when you start reading about gaming engines and so on is the term stack of course this isn't just a graphics term this exists just about anywhere in computer science right the stack is the big pile of software and eventually hardware tools that all get linked together to make for one giant computer system now for the most part our course has focused on the upper part of this stack right i would say 6.837 is like kind of here right and we've talked about developing algorithms and software that mostly interface with opengl to produce an upper level application or layer on the stack like a game of course these days actually there's sometimes a layer in between here so tools like unity are designed to wrap around your low-level graphics libraries like opengl and directx but of course sitting underneath all of these software tools are many interesting and to your theoretical instructor terrifying pieces of the graphics software stack that are incredibly important and this includes the operating system kernel the device driver right which is how your computer mediates its communication with the graphics card and with the display and then finally there is this hardware layer all the way at the bottom which includes the gpu which we've talked about as an abstraction quite a bit in this course and the chipset right your cpu and other things inside of your computer and of course the miraculous thing that happens in graphics and many other disciplines is that we can operate in this upper level and pretty much disregard what's going on underneath because such fabulous engineering has happened underneath uh in these different parts of the graphics stack that really improve what's going on and allow us to think at a high level rather than thinking about writing system code every single time we want to write something in graphics it's actually a relatively new development i mean it used to be when you wrote graphics code you really had to have pretty good knowledge of what was going on in your hardware so today's lecture is going to be something like down here um you know uh once you're you're done processing all the bits and bytes in your graphics code uh the very last thing you have to do is actually send them back out into the real world in the form of photons and things that our eye can see and so of course there are many different options for two-dimensional displays for example uh that i think we see all around us right it used to be that crt was the sort of dominant technology it's cathode cathode ray tube we'll discuss that a little bit um lcd i think is probably what you're looking at now right like your laptop screen there's led displays like the giant things at football fields plasma oled dlp electronic paper and so on there's so many different display technologies out there and one of the really cool things is that in basically a single for example hdmi output on your computer you're able to interface with any of them so we're going to give a bit of a superficial description of what goes on in some of these technologies just so you have some idea of how your display is actually producing photons at the end of the day now the first one that we're gonna start with is crt or cathode ray tube displays uh these are like those giant tvs with the big back that you used to see in the 80s and 90s i grew up with these but now they're less popular um but they're also sort of the simplest ones to understand and they explain some of the history of graphics technology so the basic uh idea in a crt display is described on the slide here so what's going on is that in the back of this big empty space in the interior of your tv or your computer monitor if it's an old crt is an electron gun uh that can produce an electron beam and then those beams get focused and deflected onto a screen uh which is of course the thing that's producing the display so what goes on in the display is that you send these electrons out to the crt in just a fixed pattern this is called the raster which explains rasterization by the way the phrase ski online makes a lot of sense too right because it's just kind of going like this up and down the screen and if you send something from the electron beam to the display when you're at that point in the raster it lights up for a little fraction of a second and that is using a physical phenomenon known as phosphorescence which is basically the process by which energy is absorbed by substance and then released slowly as light now this entire sentence is important to parse correctly so remember there's an electron beam in the back of your display it sends an electron up that runs into the display and now the display lights up at that pixel for just a tiny bit of a second now of course when we talk about relatively slowly this is relative to the speed of light which is extremely fast but the reason why that slow release of light energy is so critical is because the electron beam has to go through its entire raster pattern all the way up and down the screen before it returns to that pixel again to potentially light it up one more time and so essentially the light persists at that location as the electron beam is doing its business elsewhere until it gets an opportunity to return and turn that pixel back on or off now of course because we're using an electron beam you can do some kind of fun things with old tvs do not do this to your laptop it will not work and your laptop will die um and don't do it close to a hard drive but there's some fun experiments that kind of illustrate what's going on in a crt display in fact here's one that was actually filmed at mit hopefully this will load so here there's a tv uh image it's an old crt i don't know why they felt like that and say that and there's also a film of a clock which is being put right into the tv now we have a strong magnet we bring it close to the screen what do you think is going to happen the shape of the clock actually distorts in response to the pose of the magnet again i cannot emphasize enough do not try this experiment at home unless you are confident that you have a crt display and that there's nothing else nearby that can be affected by this giant magnet so let's think for a moment about why this experiment is successful well essentially what's going on is that the uh position of the magnet is affecting the path of those electrons as they leave the electron beam and run into the display so basically what you're doing is rather than having the electrons follow the intended path they get misdirected by the magnet and run into the wrong pixel which then lights up and that explains the distortion in the image that you see so anyway this was the old display technology and it was some of the most popular displays until maybe 10 15 years ago but these days i think what we see a lot more is a liquid crystal display or lcd so the idea of an lcd is described on this slide here essentially it's composed of two polarizing filters which are perpendicular to one another now when you have polarized light right that's effectively just kind of restricting the wave of the light to move in a particular plane if it then runs through a polarizing filter that goes in the opposite direction well nothing comes out what happens in between these two fixed polarized filters i guess one should be rotated 90 degrees to the other is that there's a particular material unsurprisingly called a liquid crystal or a twisted pneumatic where essentially what goes on is the material can be in one of two states one is that it kind of twists from one side to the other in which case it can maybe redirect the the polarization and the other pose is where it's flat like this in which case the polarized light stays polarized it doesn't rotate and it runs into the filter on the other side which is 90 degrees off so no light comes out of the display so essentially what's going on is that the twisted pneumatic cell can either change the polarization of the light so that it fits through the other polarizing filter that corresponds to the pixel being turned on or it can stay the same in which case the pixel is turned off so rather than having me a non-expert try to explain some of the topics in today's lecture we're going to borrow from a few other youtube videos i've credited them with the link at the bottom of the slide here but we'll just let our our friend explain in more detail maybe if you were to zoom in on the monitor you were using to watch this video you would see thousands of small red green and blue dots three of these grouped together make up one picture element or pixel for short when you split up each pixel you will see a backlight three color filters and polarizers as light leaves the backlight it travels along different planes including the horizontal and vertical planes the first polarizer only allows light traveling along the horizontal plane to pass through it and onto the color filters with another polarizer that only allows light to pass along the vertical axis all the horizontal light waves are blocked so no light reaches the color filters this is where the liquid crystals come into play if you were to look closely at one of these liquid crystals you would see a transparent electrode on the front and the back as well as etched glass on the front and the back liquid crystals typically orient themselves in random directions until the horizontally etched glass in the rear and the vertically etched glass in the front force them to twist into a predictable pattern as light passes through the compressed liquid crystal it naturally follows the path of the molecules so that any light traveling along the horizontal plane exits along the vertical plane by reintroducing the electrodes and passing electricity through them we can get the molecules to align themselves in the direction of the electric flow causing light to no longer twist when passing through the liquid crystal when we place these crystals back into the pixel we can see that the light from the backlight will pass through the horizontal filter get twisted into a vertical position by the liquid crystals and then flow through the vertical polarizer to the color filters however if we turn on the electrodes then the light is no longer twisted by the liquid crystals and stops at the vertical polarizer by adjusting the amount of electricity that flows through each set of electrodes we can control how much light reaches each color filter and therefore what color is seen on the display we generally talk about color okay so there's a sort of a nicer explanation all right so uh a few more little details about lcd displays uh they've generally fallen to two categories when it comes to the actual source of light and that has to do with whether the light is coming from outside of the display or inside of the display right so a reflective uh lcd would be something like uh you know the the clock that you often see the digital clocks where essentially the light source is just the outside world and then there comes into a mirror and gets bounced back off um whereas in a transmissive lcd like in your laptop screen essentially there's a fluorescent uh lamp that's sitting underneath this screen which is producing the the light this is called a backlight of course to get color inside of the pixel in your your lcd display uh that's just an additional filter that sits on top so really you know with a lot of these display technologies it's just modulating the brightness of the light you know going from dark to light and then the color is just a big pattern of rgb sitting on top of the screen um so here is like a 300x view of a typical lcd display and of course when you view it from enough of a distance all those colors blend together and what you see is the composited result in fact you might recall from a few lectures ago this allows you to do some interesting things like sub pixel rendering which actually anti-aliases the red green and blue channels separately so that uh you can get even better anti-aliasing effect than you could on the true pixel grid this is basically acknowledging the fact that the red green and blue channels are slightly displaced from one another okay so that's an lcd display another really important type of display is an led display um from my perspective this is a sort of brain dead way you might make a monitor um in fact it works especially at large scale but you need space between your leds so if you go uh you know watch a football game when the pandemic is over then essentially sometimes you see these gigantic screens that are like hanging over the uh the field um these are often led displays which are really just giant arrays of light bulbs led bulbs uh that can be packaged together uh and are controlled like a display i think this is the simplest one to understand but there are also many other types of displays so another cool one is a plasma display panel pdp i think this gets used a lot in large big screen tvs so in a plasma display now you have a big array of little cells that are all filled with some gas like neon or xenon and what happens is when you apply voltage across the cell suddenly the gas gets excited and lights up so just like our lcd uh discussion let's uh see some discussion of plasma [Music] the answer is the plasma each subpixel is filled with a mixture of gas xenon and neon when an electrical impulse of about 300 volts rushes through a subpixel on its way to the electrodes electrons from the gas mixture are violently torn off and suddenly float freely that drastically changes the state of the mixture it's no longer gas it's now plasma it's a highly energized state of matter but it's a state that lasts only as long as the electrical discharge as soon as the discharge ends the freed electrons immediately return to their places and the plasma once again becomes gas what's important is that as they return to their places the electrons release their surplus energy in the form of ultraviolet rays it's these rays that excite the subpixel which gives off light that combines with the light given off by the two other sub pixels and together they light up the pixel every second the plasma screen sends more than 2 billion electrical impulses into the subpixels in order to turn the gas there into plasma the sole purpose being the ultraviolet emissions given off once the plasma returns to its gas state all that so pictures can appear pixel by pixel on the screen okay so there's a more enthusiastic explanation of plasma then your instructor could probably uh a muster here sort of a similar display from an abstract level though i'm sure from an engineering level is quite different it's an organic led which uses instead of uh gauss's uses some kind of organic material to produce light under uh voltage here so again there's there's no backlight which can make for a really nice black color you know potentially thin size and high contrast so there's a lot of nice uh properties although uh the cost can be uh higher and sometimes the lifespan is lower i think uh for cell phones sometimes this is being applied um but i could be wrong now finally i thought i would mention one of my favorite display technologies you know somehow in my mind it like works like a rube goldberg machine you know like one of these crazy ballets of different mechanical parts moving around that shouldn't coordinate but somehow do and that's the dlp projector normally at this point in time in class i would point out the dlp projector in the back of our classroom but of course today you're probably watching our lecture on an lcd screen but these are the projectors that you see in classrooms and movie theaters and so on and i don't know when i read the descriptions of these things they just they sound like miracles to me i i wish i could be inside of a dlp just watching all the parts move around so the basic idea here is that in a dlp projector you have a few different moving parts you have a light which is being produced with maybe a wheel that's spinning around and pulsing at a particular rate and so the wheel spinning around could have different color filters so that it's like kind of cycling between red green blue red green blue red gleam blue um and then that light does not get sent directly out of the projector but instead if we take a look at this diagram on the lower right what happens is the light comes out of the light source and runs into a chip which is on the bottom of the projector and then it gets reflected back out or it doesn't so it ends up in one of two places it either gets reflected out into the real world or it gets reflected to just a light absorber which is also within oops within the projector and so here's what this little mirror looks like and the incredible thing i mean there's one of these per pixel in your display and the idea is that the light comes in from the fixed light source and depending on how this mirror is pointed either the light bounces off it gets sent into the real world or the mirror rotates and then the light just gets wasted it gets run into the light absorber now in principle this is straightforward right i could stand in front of a light with a mirror and just redirect it toward the wall or your face and you would receive light or you wouldn't the really incredible thing about the dlp is that this is happening at like 30 frames a second right so there are these little itty-bitty uh mirrors that are moving around in the interior of your dlp projector at a rate that's so fast it produces an image that feels still on the display so anyway i just find this totally incredible uh but the that's uh the reason why you're able to get such a bright image with your projector is that essentially the light source is decoupled from the display right the light source is just a really really bright light bulb that's being shined onto these mirrors and and because of that you can afford to have a much brighter light than what would have to go through your screen and like an lcd which would be a you know backlight of course there are other displays you might not think about so another one would be your kindle or your e-ink kind of reflective display these are displays that tend not to have backlights and behave almost similarly to the what the reflective lcds that we talked about before e-ink displays are passively updated so the basic idea here is that they're these little capsules which are filled with some solution that's suspended in a die and so you can apply voltage to basically swap up and down which die is on top right so you can either move the white particles to the top or the bottom of each of the micro capsules and that controls whether or not the display is black or white so you apply that voltage the display changes and then it just stays that way you don't need to use additional electricity and so for displays where the update rate is low but the image is going to have to stay there a long time so for example your ebook right you display a page and then it takes some time to read this can be a really nice technology you can get some version of color by like putting a filter on top of these screens but they tend to be quite weak um you really need a backlight to make that work especially because now you have one third of the pixels to work with for each color so anyway that's an extremely quick uh summary of some 2d displays let's talk about some 3d ones uh next so of course 3d displays are really tricky you know and display technology has been around for a long time many of these things rely on all kinds of different properties of the human visual system including you know phenomena like stereopsis where essentially by using two eyes we can get a better version of the world than than with just one um which is essentially you know due to 19th century uh scientists who were the first to explain uh binocular vision so for example uh charles wheatstone and and friends now on a flat display achieving stereo 3d images is is really tricky uh and there are many different technologies that can allow you to do that that have different trade-offs in terms of what 3d effects you're actually able to perceive and which ones we kind of leave out and hope that your brain might fill in these can fall into different categories so for example there are active displays um so an example might be glasses that shudder between the left and the right eye right so that means that the glasses and the display on the screen have to kind of coordinate with each other or maybe there's some passive ones like polarized glasses so maybe my left eye has polarization vertically my right eye has polarization horizontally um or anaglyph rendering where like you put a little lens on the top of your computer screen to direct rays out but not forward toward the two eyes so that you can view a 3d effect so long as you're looking at the screen from a particular location so all of these displays are trying to negotiate different cues that our brain uses to sense the 3d world and there are many of these so for example to sense depth sometimes we use binocular disparity basically the idea that images look different because of the locations of your two eyes we also use a number of ocular depth cues like the rotation of your eyes and and the accommodation so vergence is the idea you have to rotate your eyes closer to see something closer to the camera accommodation has to do with the focus and we also uh see depth cues that are just visible even in one eye right like pictorial depth cues including occlusion like what object is on top uh size right the left circle kind of feels closer to the camera than the right one shadows and so on and so the big challenge of engineering a display that tries to fool your brain into thinking that you see 3d is that you would like all of these different cues to be consistent with one another of course doing that is basically the same as setting up a 3d scene so typically these displays have to give something up so for example sometimes disparity and occlusion conflict like maybe you have remember um we talked about disparity as having different images projected into your two eyes and there are a number of technologies that accomplish that but if they're sitting on top of a computer screen then what's going to happen right along the edge of the screen is that suddenly the 3d objects get cut off and you just see whatever's behind the screen which can really screw up you know disparity versus occlusion style conflicts with one another right in this case the computer screen appears to be behind the lily pad that it's rendering but then of course at the edge of the screen suddenly the screen's on top because we can't display the lily pad outside of the the little rectangle of your screen and so that could break the illusion of depth and create a bit of a contradiction that your brain has to resolve of course there are many different challenges here so for example asking your viewers to focus on things that are too close to the screen can be really painful especially if you're asking them to basically cross their eyes in a virgin sense so there can be some mismatch between the accommodation and stereopsis that can really make a challenge as you move away from your display so and of course this depends on the viewer so for example i personally am pretty sensitive to this stuff and and um sometimes you know there's this sort of conflict between focusing on a screen that's behind you but having the vergence of an object that's right in front of your eye so sometimes it turns out that there's some really clever technologies out there that may actually go back and do some correction and improve the image so that there's some feeling of depth even if it's a little bit inaccurate but reduces the eye strain this is really critical especially for 3d movies so when you go into the 3d movie theater when they're open again um essentially what can happen is um you know for example maybe you put a camera right underneath an object that's dropping towards your face um that can cause some really extreme discomfort with divergence of your eyes you know you're basically crossing your eyes at some point but maybe you can get that effect but still modulate the velocity of the object as it moves toward your face to avoid that level of discomfort there are other types of displays out there that try to get 3d effects not just by wearing glasses but by having other styles of redirecting light so another clever one is an auto stereoscopic display this is basically the idea that you can get binocular parallax without wearing 3d glasses by redirecting light in a very careful fashion so here i show you two different examples of that one is a lenticular display that's what we show you on the left hand side and one is a parallax display the basic idea here is that you have some lcd screen that's sitting underneath but now rather than just looking directly at that screen we're going to put a new layer on top of that screen which is basically just some fixed either lens like what we see here or even just blockage like we see on the right hand side and the basic idea is that we want to redirect slightly different signals to the two eyes and so what these uh lenticular objects allow you to do these lenses is essentially diffract refract the light in directions other than straight out of your screen which is what you would want for this kind of display or similarly for a parallax barrier basically you're just doing the same thing but now using a physical object blocking light moving in a direction that you don't want and so in an auto stereoscopic display the advantage is that you don't have to wear 3d glasses but the disadvantage is that you have to have your eyes in the right place right so notice that these images are designed for two particular eye positions that are demonstrated if we were to move the viewer backward suddenly the image wouldn't really make sense anymore and so these are displays that are optimized for viewing at a very particular distance which obviously wouldn't work for example in a 3d movie theater with viewers sitting at all kinds of locations but for things like the nintendo 3ds or this uh lg optimus 3d if i recall that phone wasn't terribly popular um basically these little mini 3d displays where you're always holding it roughly at arm's length these sort of techniques can can make a lot of sense and you can even switch off between 2d and 3d mode by just turning on and off the uh the different um enabling or disabling the the parallax barrier that is in front of the lcd uh now these particular displays are optimized for just two eye positions so one particular location of a viewer you can take that to the limit and try to redirect things every which way by using lenticular things that essentially send light in very controllable set of directions this can make for unobtrusive displays with wider kind of range of views the only problem is that essentially the resolution of these displays decreases substantially why is that well maybe i have you know 10 pixels in a row but each pixel is now only controlling the light rays that are moving in one individual direction that are only getting viewed from one particular angle so in particular you've basically divided the resolution of your display by 10. and so you really have to pack a lot of pixels in there to get a reasonable multi-view auto stereoscopic display say that 10 times fast um that said there are many different technologies that have attempted this over the years and some of these screens are pretty compelling now a different way that you could pull off the same effect which i think is kind of fun is uh shown here so here's a 360 degree multi-view auto stereoscopic display i think this is so cool essentially what's going on in this video clip is that you have a spinning mirror right so the mirror is determining what direction the light is coming out and somebody's done a really careful calculation uh so that the mirror is synced up with the projector and what's coming out in that particular direction is what you should need to have a 360 degree display every once in a while i think you see these things at like conferences and hotel displays and in fact you can even make them fast enough for some manipulation these didn't really catch on i mean the main challenge with a display like this is that you have a giant spinning mirror at an extremely rapid rate so that means that there's a part that could fail and moreover you you can't really stick your hands in there right so you to be quite careful to make sure that you don't expose this to the outside world and the you know the resolution is a little bit low but maybe that could change over time okay finally uh there are some techniques out there for 3d displays i think these are somehow the most natural or straightforward um that directly try to display volumes so in some sense the spinning display that i just mentioned for you is uh can be understood that way and uh you know these usually mean that you don't have an occlusion but you just have some rotating part that is just sweeping the entire display volume so maybe instead of having a mirror in this case you have a projection screen that's rotating around and this is what's enabling you to make a light source at any location in the 3d volume right so in the previous slide we had a spinning mirror now we have a spinning screen that isn't trying to redirect the light the idea is just that when the screen is in a particular position and then you project light onto it it appears as if the light is coming out of that point okay are also some static volume displays that don't have to spin around um so very simple one of those is basically like taking a computer screen taking five computer screens and stacking them right on top of each other uh and so long as those screens are transparent then maybe you can't get a 3d effect but you can kind of get like a two and a half d effect by having a few layers of 3d information which could be good enough but really the most popular 3d ish display that we see today is in virtual reality and augmented reality displays now in case you're curious there is a difference between these two terms a virtual reality or vr display essentially is going to cover up your eyes and replace what is in your visual field with just a totally new image right so i think that's what we uh kind of envision when we think about virtual reality augmented reality right just a pair of opaque glasses with two computer screens beeping directly into our eye augmented reality on the other hand are typically displays that you can see through but augment your visual field with other interesting information like maybe you know puts a label on top of some information that you're seeing um augmented reality displays can be useful for sort of enhancing our experience in the real world rather than replacing it which is what virtual reality would do now these days that terminology has gotten a little bit fuzzier in particular there's one new brand of augmented reality which is quite popular and that happens on your cell phone so of course many of our cell phones now are equipped with cameras and so one thing you could do is kind of make your cell phone in some sense disappear right and the way that you can do that is you take in uh images from the camera outside of your cell phone and then you just display the same image on the phone itself so in other words the phone is sort of pretending like the phone doesn't exist right it's just displaying what would have been on the other side of the phone um so that creates a new opportunity for an augmented reality display where it's still an opaque display but you can abstractly kind of feel like you're seeing through it because it's just translating the camera signal into output but in that process it also augments the signal with other information so i think that's typically considered augmented reality even though it's not the same set of light rays that are going all the way from the outside universe to your eye now of course there are many popular vr and ar displays this guy seems to be enjoying his oculus vr there's a holo lens from microsoft already these slides are out of date they uh they change every year and the history of virtual reality actually goes back really longer than you might think so in some sense the earliest vr displays were in the early 1800s these are these devices called stereoscopes where essentially you take maybe two photographs or you draw two images which are stereoscopically displaced from one another and then you hold this uh little wood viewer up to your eyes looks kinda like an opera viewer to me and then when you do that essentially what you're doing is filtering out those two images and displaying them to your two eyes and you can experience a whole new static 3d world through this extremely rudimentary display now the second thing you see here which i've always found these images to be totally terrifying even though i know they're a positive development in the vr ar world our experiments at mit by ivan sutherland arguably making one of the very first vr devices um which is all the way back in the 1960s i don't know if people realized that and then just lately there have been an explosion of virtual reality and now a augmented reality displays thanks to a combination of a few factors right there's much lower latency than there used to be um and also the display technology is getting a lot better uh of course there are other interesting landmarks in the history as well uh one of my favorites is the uh the first head mounted uh display uh like what you see here uh on the left hand side so if you you can combine that with the stereoscope before now you don't have to hold on to it anymore you can wear a hat and have your own 3d display i've always been extremely curious not curious enough to actually look it up but if anybody has a good explanation for me why your stereoscopic display really needs a pointy tip and like a muzzle i would love to know but i digress of course there were other uh virtual reality augmented reality technologies that have come out over the year um game boy actually had one all the way back in the 90s it was not very successful i think the latency was really painful the the the the resolution wasn't so great um but these days uh technologies are really exploding every which way oculus sony facebook all these companies have their own virtual reality because oculus and facebook are the same now and so there's actually a proliferation of different technologies with different trade-offs in terms of field of view and so on so there were a number of steps in the evolution of this technology that enabled what we have today you know the the technology in the 1960s from ivan sutherland was one of the first electronic or digital displays that we have evidence of um there was in the 1980s and 90s people thought about haptics and and human computer interaction more you know they kind of realized that the human part of the computer loop is actually pretty important and we should understand how humans interact with computers to make better virtual reality devices and then lately especially driven by cell phone technology in fact there's a really goofy virtual reality device that just slots a cell phone in front of your eyes right the google cardboard but we now have low cost high resolution and low latency displays which really have enabled the uh technologies that we see now and these are coupled by the way with tracking and other and other important tasks but even some of the basic ideas were around in the 1960s um so here we see a head-mounted three-dimensional display including two little crts rendering head tracking interaction model generation and all this happened in 1968 i think this was the sword of damocles as well ivan sutherland called this crazy device but it's just absolutely amazing how much stuff was anticipated um in in making a display that roughly follows the same pieces that we have today right i mean these are the same components that go into the oculus or anything else in fact if you look at the teardown of for example this oculus dk2 is a couple years out of date you'll see basically the same pieces that were engineered back in the 60s when this technology was just beginning to appear now there are many different ways to compare different virtual reality displays um and when you go out in the market and you want to purchase one of your own some things to think about uh include the field of view um i think is really the one of the main ones as well as the latency which is like how quickly it can update uh and so on the field of view is essentially the width of the image that's displayed in front of you like at some point you're going to perceive the edge of the screen and just have black on the side so for example this version of the oculus had 110 degree field of view which i think is is pretty good now these virtual reality displays that i'm showing you their job in life is to occupy your entire field of view if they can right it's just a function of how good the display technology is there are other technologies out there that are attempting to sort of do the opposite and be less intrusive so one of the ones that was promising although it got discontinued but my bad is it'll reappear someday was the the google glass i think they still sell it to like medical people i'm not sure um so the google glass is definitely on the augmented reality side of the spectrum and this was intended basically as a little augmentation to your eyeglasses so that you could just beam your email directly into your brain at all parts of the day while still perceiving the outside world i think unfortunately the value proposition of this device relative to just holding your phone in front of your face was maybe not um awesome but the technology behind the google glass was pretty cool so the basic idea here is that you know you have your your your viewer um yeah so there's their eye they're looking out into the world uh and you have a led and the basic idea here is that there's some beam splitter that's sitting in between and what gets leveraged here is an effect called the pepper's ghost effect the idea that if you have bright enough light then it'll get reflected so this is an idea that goes all the way back to the 1860s so basically this was the technology that first appeared i think in like theater tricks so that you could have characters like ghosts my three-year-old niece calls them gusts but in any event um the basic idea of the pepper's ghost effect is that maybe you have a mirror in your scene that's pointing uh you know toward uh something that is extremely dimly lit and so or maybe uh not a mirror i'm sorry a uh a piece of glass so you have a piece of glass which is pointing toward a dimly lit part of your seam then basically what you're going to see is right through the piece of glass right you'll just see the stuff on the other side so that's like the situation shown on the left you have your spotlight on some scene your viewer is somewhere back here and yeah there's a piece of glass but you're just seeing right through it and now you want a ghost to show up so you turn on a light and that light adds light to the something that in the reflected direction of the piece of glass which was before just dark so now suddenly there's a light source and the light source because it's bright enough gets reflected off of the surface of the glass and back toward your eye and so this is a clever way to make like a ghost suddenly appear in your scene although in order to do this you needed two copies of your scene with very similar structure let's see so here's a fun demo uh using pepper's ghost effect in combination with legos located off to one side where it's out of view the plexiglas is set at a 45 degree angle to both the audience and the ghost at this angle the background remains clearly visible but the plexiglas also partially reflects an image of the ghost to the audience it looks like there's a transparent ghost in the scene in front of them it's pretty simple so now i'm going to show you some ways that you can implement this technique the simplest way to apply it is in photographs and video i think it's pretty obvious start by setting up your technique a large sheet of plexiglas then you need a weight of the backdrop but you do need to use a flashlight to help illuminate the ghost the result is a transparent figure in the window that stares at people as they walk by your house creating a ghost in a doorway uses basically the same procedure but because the opening is much larger you need a much larger sheet of plexiglass well that's how to make peppers ghost try it out and have fun there you go how you can make peppers cat no of course that's one version of augmented reality uh in the google glass a different one like we already mentioned uh was to make a phone that you can basically just see through now the really amazing thing is that our cell phones so here i've got an iphone over time they've been equipped with more and more cameras in the back of your phone and these are cameras that are not just sensing visible light but also things like depth and so what that allows our ar technology to do our augmented reality technology is to have a really good understanding of the 3d structure of the scene that also is coming in in the the light sensor and that way you can augment it in interesting ways so for example uh i think the some furniture companies now when you go to their website you can like hold up your phone to an area inside of your house and it'll put a piece of furniture in the room and so it's such a way that you can actually kind of walk around it and it tracks with the ground the reason it's able to do that is that it's tracking your location in space maybe even occlusion with objects that would be closer to you than the piece of furniture and it's able to do that quickly enough that it can output the result in the display of your phone so your phone is like a little window into a slightly different reality it's a pretty cool technology and i think this is the one that's most likely to change very quickly in the next year or two so of course when you're comparing head mounted displays other 3d technology there's so many different considerations that you can look at everything from height weight fit the eye box i release the uh you know whether or not you have um a big field of view a small field of view whether it's augmented virtual um and and where the field of view is located um all of these different technologies uh have basically evolved to try and cover different use cases for this technology everything from basically a pair of smart glasses right like the google glass whose job in life is to uh just augment the world around you and give you a little extra information to augmented reality which maybe is covering up a more substantial part of the world to virtual reality where you're just replacing everything outside of your eye with something else now of course this roughly correlates with the field of view right the bigger the field of view the more stuff you're covering up uh similarly the angle resolution really depends on on what kind of technology you're using right if you want to have the google class you need to be able to read your email and maybe you need a high resolution small image whereas in some of these vr technologies a bit of a lower resolution big image is more important similarly for pixel accounts so in any event this is just a quick summary of some of the important ideas in output devices uh with like the ones that we've been interfacing with all semester in 6837 but we really haven't considered the details of how they work obviously i'm not going to test you on all the details of these different displays but i think it's important to have some intuition for what's going on and why the bits and bytes and real numbers that your graphics card is producing actually can be produced out there in the real world so with that i think this is actually our last lecture of 6837 it's been a pleasure to teach all of you guys hopefully i'll see some of you in my graduate level courses and good luck with your final projects
2021-01-03