the following is a conversation with Mark Zuckerberg inside the metaverse Mark and I are hundreds of miles apart from each other in physical space but it feels like we're in the same room because we appear to each other as photorealistic Kodak avatars in 3D with spatial audio this technology is incredible and I think it's the future of how human beings connect to each other in a deeply meaningful way on the internet these avatars can capture many of the nuances of facial expressions that we use we humans use to communicate emotion to each other now I just need to work on upgrading my emotions expressing capabilities of the underlying human this is the Lux Friedman podcast and now dear friends here's Mark Zuckerberg [Laughter] this is so great lighting change wow oh yeah we can put the light anywhere and it doesn't feel awkward to be really close to you no it does I actually moved you I moved you back a few feet before you got into the headset you were like right here I don't know if people can see this but this is incredible the realism here is just incredible where am I where are you Mark where where are we you're in Austin right no I mean this place we're shrouded by Darkness with ultra realistic face and it just feels like we're in the same room this is really the most incredible thing I've ever seen and sorry to be in your personal space I mean we have done suggested before yeah no I was I was commenting to the team before that even that I feel like we've choked each other from further distances than it feels like we are right now I mean this is just really incredible I don't know how to describe it with words it really feels like it feels like we're in the same room yeah it feels like the future this is truly truly incredible I just wanted to take it in I'm still getting used to it it's like it's you it's really you but you're not here with me right you're there wearing a headset now I'm wearing a headset it's it's really really incredible so what um can you describe what it takes currently for us to appear so photorealistic to each other yeah so I mean for for background we both did these scans for uh This research project that that we have at meta called Kodak avatars and the idea is that instead of actually instead of our avatars being cartoony um and instead of actually transmitting a video what it does is we've sort of scanned ourselves in a lot of different expressions and we've built a computer model of sort of each of our faces and and bodies and the different Expressions that we make and collapse that into a codec that then when you have the headset on your head it can it sees your face it's easier expression and it can basically send um an encoded version of what you're supposed to look like over the wire so um so in addition to being photorealistic it's also actually much more bandwidth efficient than transmitting a a full video or especially a 3D immersive video of a whole scene like this and it captures everything like the flaws like to me the subtleties of the human face like even the flaws that's like that's all amazing it makes you uh it makes us so much more immersive it makes you realize that like Perfection isn't the thing that leads to immersion it's like the little subtle flaws like freckles and like variations in color and just yeah wrinkles roses yeah asymmetry and just the different like the corners of the eyes like what your eyes do when you smile all that kind of stuff yeah eyes are a huge part of it yeah I mean there's all the studies that most of communication even when people are speaking is not actually the words that they're saying right it's kind of the expression and all that so we try to capture that with the kind of classical um expressive Avatar system that we have that's the kind of more cartoon designed one you can you can kind of put those kind of Expressions on those faces as well but there's obviously a certain realism that comes with delivering kind of this photorealistic experience that um I I don't know I just think it's really magical I mean this gets to kind of the core of what the vision around virtual and augmented reality is of like delivering a sense of presence is if you're there together no matter where you actually are in the world and I mean this this uh experience I think is a good embodiment of that where it's like coming or in two completely different states halfway across the country and it just like you know looks like you're just sitting right in front of me it's uh it's pretty wild yeah yeah I can't it's I'm almost getting emotional it's like it feels like a totally it's fundamentally new experience like for me to have this kind of conversations with loved ones that would just change everything maybe just to elaborate so the I went to Pittsburgh and went through the whole scanning procedure which has so much incredible uh technology so software and Hardware going on um but it is a lengthy process so what's your vision for the future of this uh in terms of making this more accessible to people you know it starts off with a small number of people doing these um very detailed scans right which is this that's the version that you did and that I did and you know before there are a lot of people who who've done this kind of a scan for we probably need to kind of over collect Expressions um when we're doing the scanning because we haven't figured out how much we can reduce that down to a really streamlined process and extrapolate from the the scans that have already been done but you know the goal and we have a project that's working on this already is just to do a very quick scan with your cell phone where you just take your phone kind of wave it in front of your face for a couple of minutes um you know say a few sentences make a bunch of Expressions but overall have the whole process just be two to three minutes and then produce something that's of the quality of what we have right now so I think that that's one of the big challenges that remains and right now we have the ability to do the scans if you you know have hours to sit for one and with today is technology I mean you're using a meta headset that exists it's a product that's kind of for sale now you can drive these with that um but the production of of um these scans in a very efficient way is one of the last pieces that we still need to really nail and then obviously there's all the experiences around it I mean right now we're kind of sitting in a dark room which um you know is you know familiar for for your podcast but I think part of the the vision for this over time is um is you know not just having this be like a video call I mean that's fine it's it's cool or it feels like it's immersive but um you know you can you can do a video call on your phone the thing that you can do in the metaverse that is different from what you can do on a phone is like doing stuff where you're physically there together and and participating in things together and we could play games like this um we could have meetings uh like this in in the future once you mix um once you get mixed reality and augmented reality we could have codec avatars like this and go into a meeting and have some people physically there and have some people show up in this photorealistic form uh superimposed on the on the physical environment I've got stuff like that is going to be super powerful so we got to still build out all those kind of applications and the use cases around it but I don't know I think it's going to be a pretty wild uh next few years around this I mean I just I'm actually almost at a loss of Wars this is just so incredible this is truly incredible I I hope that people like watching this can get a glimpse of like how incredible it is it really feels like we're in the same room like there is that um I guess there's an uncanny valley that seems to have been crossed here like it looks like you yeah I mean I think there's still a bunch of tuning that I think we'll want to do where different people emote to different extents right so I think one of the big questions is you know like when you smile how wide is your smile and how wide do you want your smile to be um and I think getting that to be tuned on a per person basis is um is going to be one of the things that we that we're going to need to figure out um you know it's like to what extent do you want to give people control over that um you know some people might try to you know might might prefer version of themselves that's more emotive in their Avatar than their actual faces you know so for example you know I always get a lot of critique and for um for for having like a relatively stiff uh expression but you know I mean I I might I might feel pretty happy but just make a pretty small smile so I mean maybe you know for me I would it's actually you know it's like I'd wanna have my avatar really be able to better Express um like how I'm feeling then then than how I can do physically so I think there's a question about how you want to tune that but uh but overall yeah I mean you we want to start from the Baseline of capturing how people actually emote and express themselves and I mean I think the the initial version of this is has been pretty impressive and like you said um I do think we're we're kind of beyond the The Uncanny Valley here where it and it does feel like you it doesn't feel uh it doesn't feel weird or anything like that I mean that's going to be the meme that the two most monotone people are in a metaverse together but I think that actually makes it more difficult like the the amazing thing here is that the subtleties of the expression of the eyes you know people say I'm monotone and emotionless but I'm not it's just this it may be my expression of emotion is more subtle usually like with the eyes and that's one of the things I've noticed is just how expressive the subtle movement of the corners of the eyes are in terms of displaying happiness or boredom or all that kind of stuff I'm curious to see just because I've never done one of these before I've never done a podcast as as one of these codec avatars um and I'm curious to see what what how what people think of it because you know one of the issues that we've had in some of the VR and and mixed reality work is it tends to feel a lot more profound when you're in it than the 2D videos capturing the experience so I think that this one because it's photorealistic um may look kind of as amazing in 2D for people watching it as it as it feels I think to be in it but we've certainly had this this um this issue where a lot of the other things just it's like you feel the sense of immersion when you're in it that that doesn't quite translate to a 2d screen but I don't know I'm curious to see to see what people think yeah I'm curious to see if people could see that um like my heart is actually beating fast now this is super interesting like the that such intimacy of conversation could be achieved remotely there's been you know I don't do remote podcasts for this reason and this is like breaks all of that this feels like just an incredible transition to something else to the different kind of communication breaks all barriers like Geographic physical barriers uh would you mentioned do you have a sense of timeline in terms of how many difficult things have to be solved to make this more accessible to like scanning with a smartphone yeah I mean I think we'll probably roll this out progressively over time so it's not gonna be like we roll it out and one day everyone has a codec Avatar we want to get more people scanned and into the system and then we want to start uh integrating it into each one of our apps right making it so that you know I think that for a lot of the work style things productivity I think that this is going to make a ton of sense in a lot of game environments I mean this could be fine but games tend to have their own style right where you almost want to fit more with the aesthetic style of the of the game um but I think for doing meetings and one of the things that we get a lot of feedback on work rooms where people are pretty Blown Away by the experience and this feeling that you can like be remote but feel like you're physically there around a table with people but then you know we get some feedback that people have a hard time with the fact that the avatars are so expressive and and and don't feel you know as as realistic in that environment so I think something like this um could make a very big difference for those remote meetings and especially with Quest 3 coming out which is going to be the first mainstream mixed reality product right where you're really taking digital um you know expressions of either a person or or objects and overlaying them on the physical world um I think the ability to do kind of remote meetings and and things like that where you're like just remote hanging sessions with friends I mean I I think that's going to be very exciting so yeah rolling it out over the next over the next few years it's not ready to be like a a kind of mainstream product yet but um we just want to we'll keep tuning in keep getting more scans in there and keep you know and kind of rolling it out into more of the features but yeah I mean definitely in the next in the next um few years you'll be seeing a bunch more experiences like this yeah I would love to see some celebrities scanned and some non-celebrities uh I just just more people to experience this I would love to see that this is something I mean on my mind is bro I'm literally at a loss of words because it's very difficult to just convey how incredible this is how how like how I feel the emotion how I feel the presence how I feel like the subtleties of the emotion in terms of like work meetings or any kind of in terms of podcasts this is like this is awesome and I don't even need your arms or legs is that well we gotta get bad I mean that's that's its own challenge um and and part of the question is also so you have the scan then it takes a certain amount of compute to go drive that both for the sensors on the headset and um and then rendering it so one of the things that we're working through is what is the level of fidelity that is optimal right you could do the full body in in kind of a codec and that can be quite intensive but um but one of the things that we're we're thinking about is like all right maybe you can kind of Stitch a somewhat lower Fidelity version of your body would still still have the main kind of the major movements um but uh but but your face is really the thing that we have the most resolution on right in terms of being able to read and express emotions I mean like you said if you move your you know eyebrows like a millimeter I mean that really changes the expression and what you're you're emoting whereas you know I mean moving your your arm like a an inch probably doesn't matter quite as much so so yeah so I think that we'll we do want to get all of that into here and and that'll be some of the work over the next period as well so you mentioned Quest three that's coming out I've gotten a chance to try that too that's awesome so the how did you pull off the mixture so it's not just virtual reality it's mixed reality yeah I mean I think it's gonna be it's gonna be the first mainstream um mixed reality device I mean obviously we ship Quest Pro um last year but it was fifteen hundred dollars um and well part of what I'm super proud of is you know we try to innovate not just on pushing the state of the art and delivering new capabilities but making it so it can be available to everyone and you know we have this and it's coming out it's 500 and um in in some ways I think the mixed reality is actually better in quest 3 than it was um than what we're using right now in quest Pro so I'm really proud of the team for being able to deliver that kind of an innovation and get it out but you know some of this is just um software you tune over time and get to be better part of it is you put together a product and you figure out what are the bottlenecks in terms of making it a good experience so we got the resolution for the Mixed reality cameras and sensors to be multiple times better in quest 3 and we just figured that that made a very big difference when we saw the experience that we were able to put together for Quest Pro and part of it is also that you know Qualcomm just came out with their next Generation chipset for for VR and Mr that we worked with them on a kind of custom version of it but that was available this year for Quest 3 and it wasn't available in quest Pro so you know in a way I'm Quest 3 even though it's not you know the pro product actually has a stronger chipset in it than the pro line at a third of the cost so um so I'm really excited to get this in people's hands it um it does all the VR stuff that that Quest 2 and the others have done too it does it better because the display is better and and the chip is is better so you'll get better graphics it's 40 thinner so it's um so just more comfortable as well but but the Mr is really the big capability shift and and part of what's exciting about the whole Space right now is you know this isn't like smartphones where you know companies put out a new smartphone every year and you can almost barely tell the difference between that and the the one the year before it now for this each time we put out a new headset it has like a major new capability and and the big one now um is is mixed reality the ability to basically take digital representations of people or objects and and superimpose them on the world and basically you know I mean there's that one version of this is you're gonna kind of have these augments or or Holograms and experiences that you can kind of bring into your living room or a meeting space or an office um another thing that I just think is going to be a much kind of simpler innovation is that there are a lot of VR experiences today that don't need to be fully immersive and you know if you're playing a shooter game or you're doing a fitness experience now sometimes people get worried about swinging their arms around like am I going to hit a lamp or something you know it's and and am I going to run into something so having that in mixed reality actually is just a lot more comfortable for people right you kind of still get the immersion and the 3D experience um and you can you can have an experience that just wouldn't be possible in the physical world alone but by being anchored to and be able to see the physical world around you it's like it just feels so much safer and more secure and I think a lot of people are really going to enjoy that too so yeah I'm really excited to see how people use it but yeah Quest 3 coming out um later this fall yeah and I got to experience it with other people sitting around and there's a lot of furniture and so you get to see that furniture you get to see those people and you get to see those people like enjoy the Ridiculousness of you like swinging your arms I mean person will be their friends of yours even if they make fun of you they uh there's a lot of love behind that and that you guys I got to experience that so that's a really fundamentally different experience than just pure VR with like with zombies coming out of walls and yeah it's like someone shooting at you and you hide behind your real couch in order to duck the the fire yeah it's incredible how it's all integrated but also like subtle stuff like in a room with no windows you can add Windows to it and you can look outside as the zombies run towards you but like it's still nice view outside yeah yeah it's just it's really and so that's pulled off by having cameras on the outside of the headset that do the pass-through in that technology is incredible to do that on a oh yeah headset yeah it's not just the cameras you basically need to you need multiple cameras to capture the different angles and and sort of the three-dimensional space and then it's a pretty complex compute problem an AI problem to map that to your perspective right because the cameras aren't exactly where your eyes are because no two people's eyes are you're not going to be in exactly the same place you kind of need to to get that to to to line up um and then do that basically in real time and then generate something that looks that kind of feels natural um and then superimpose whatever digital objects you want to put there so it's yeah it's a very interesting technical Challenge and um and we'll continue tuning this for for the years to come as well but uh but I'm pretty excited to to get this out because I think quest 3 is going to be the first device like this with that millions of people are going to get that's mixed reality and it's only when you have millions of people using something that you start getting the whole developer Community really starting to experiment and build stuff because now they're going to be people who actually use it um so I think we'll get you know we got some of that flywheel going with Quest Pro but I think it'll really get accelerated once Quest 3 gets out there so yeah I'm I'm pretty excited about this one plus there's hand tracking without so you don't need to have a control so this count the cameras aren't interested in the pass-through uh of the entire physical reality around you is also tracking the details of your hands in order to use that for like gesture recognition this kind of stuff yeah we've been able to get way further on hand recognition in a shorter period of time than I expected so that's been pretty cool I don't know did you see the the demo experience that we built around um piano like yeah the piano learning to play piano yeah it's incredible you're basically playing piano on a table and it's that's without any controller and like how well it matches physical reality with no latency and it's not it's tracking your hands with no latency and it's tracking all the people around you with no latency integrating physical reality and digital reality obviously that connects exactly to this Kodak Avatar which is in parallel allows us to have ultra realistic copies of ourselves in this mixed reality it's uh so like it's all converging towards like an incredible digital experience in the metaverse to me obviously I love the intimacy of conversation so even this is awesome but do you have other ideas of what this unlocks of like something like Kodak Avatar unlocks in terms of applications in terms of uh things we're able to do well there's what you can do with avatars overall in terms of superimposing digital objects on the physical world um and then there's kind of psychologically what is having photorealistic do um you know so I I think we're moving towards a world where you know we're going to have something that looks like normal glasses where you can just see you see the physical world but you also see Holograms and in that world I think that they're going to be you know not too far off you know maybe you know by the end of this decade we'll be living in a world where there are kind of as many Holograms when you walk into a room as there are physical objects and it really raises this interesting question about what our um about you know a lot of people have this phrase where they they call the physical world the real world and you know I kind of think increasingly in the physical world is super important but I actually think the real world is the combination of the physical world and the digital worlds coming together but until this technology they were sort of separate right it's like you access the digital world through a screen right and you know maybe it's a small screen that you carry around or it's a bigger screen when you sit down at your desk and strap in for a long session but um but they're kind of fundamentally divorced and disconnected and I think part of what this technology is going to do is bring those together into a single coherent experience of what the modern real world is which is it's got to be physical because that we're physical beings so the physical world is is always going to be super important but but increasingly I think a lot of the things that we kind of think of um can be digital Holograms I mean any screen that you have can be a hologram um you know any media in any book art um you know it can basically be just as effective as a hologram as a physical object any game they were playing a board game or um or any kind of physical game cards um you know ping pong things like that they're they're often a lot better as Holograms because you could just kind of snap your fingers and them and have them um show up you know it's like you have a ping pong table show up in your living room but then you can snap your fingers and have it be gone um so that's super powerful um so I think that it's actually an amazing thought experiment of like how many physical things we have today that could actually be better as interactive Holograms and that I think the the most important thing obviously is people so the ability to you know have these mixed Hangouts whether they're social or meetings where you know you show up to a conference room you're wearing glasses um or a headset in the very near term but you know hopefully by you know for the next five years glasses or so and um and you know you're there physically some people are there physically um but other people are just there as Holograms and it feels like it's them um who are right there and and also by the way another thing that I think is going to be fascinating about being able to blend together the digital and physical worlds in this way is we're also going to be able to embody um AIS as well so I think you'll also have meetings in the future where you're basically you know maybe you're sitting there physically and then you have you know a couple of other people who are there's Holograms and then you have like Bob the AI who's an engineer on your team who's helping with things and he can now be embodied as a um you know as as a a realistic Avatar as well and just join the meeting um in that way so I think that that that's going to be pretty compelling um as well so then okay so what can you do with photorealistic avatars compared to um kind of the more expressive ones that we have today well I think a lot of this actually comes down to acceptance of the technology um and because all the stuff that we're doing I mean the the motion of your eyebrows the motion of your eyes the cheeks and all of that there's actually no reason why you couldn't do that on an expressive Avatar too I mean it wouldn't look exactly like you but you could make a cartoon version of yourself and still have it be almost as expressive but I I do think that there's this bridge between the current state of of most our interactions in the physical world and where we're getting in the future with this kind of hybrid physical and digital world where I think it's gonna be a lot easier for people to kind of take some of these experiences seriously with the photorealistic avatars to start and then I'm actually really curious to see where it goes longer term I could see a world where people stick to the photo realistic and maybe they modify them to make them a little bit more interesting but maybe fundamentally we like photorealistic things um but I could also see a world that once people get used to the photo the photorealistic avatars and they get used to these experiences that I I actually think that there could be a world where people actually prefer um being able to express themselves in kind of non you know ways that aren't so tied to their physical reality and so I that's one of the things that I'm really curious about and I don't know in a bunch of our internal experiments on this one of the things that has I thought was psychologically pretty interesting is people have no issues blending photorealistic stuff and not so you know we could have a you know for this specific scene that we're in now we we happen to sort of be in a dark room um I think part of that aesthetic decision I think was based on the way you like to do your podcast but we've we've done experiences like this um where you have like a cartoony background but photorealistic people who you're talking to and we seem to like people just seem to just think that that is completely normal right it doesn't bother you it doesn't feel like it's weird um another thing that that we've experienced with is um is basically you have a photorealistic avatar that you're talking to and then right next to them you have an expressive kind of Cartoon Avatar and that actually is pretty normal too right it's it's like it's not that weird right to to basically being interacting with with different people in different modes like that so I'm not sure I think it'll be an interesting question to what extent these photorealistic avatars are like a key part of just transitioning from being comfortable in the physical world to this kind of new modern real world that that kind of includes both the digital and physical or if this is like the long-term way that it stays um that's that's a I mean I think that they're going to be useless for both the expressive and the photorealistic over time I just don't know what the balance is going to be yeah it's a really good interesting philosophical question but to me in the short term the photorealistic is amazing to what I would prefer like you said to work on but like on a beach with a beer just to see a buddy of mine remotely on a chair next to me drinking a beer I mean that as realistic as possible is an incredible experience so I don't want any fake hats on him I don't want any just chilling with it with a friend drinking beer looking at the ocean while not being in the same place together I mean that yeah that experience is just it's a it's a fundamentally uh it's just a high quality experience of friendship whatever we seek in friendship it seems to be present there in the same kind of realism I'm seeing right now this is this is totally a game changer so to me this is I can see myself sticking with this for a long time yeah and and I mean it's also it's novel and it's also a technological feat right it's like being able to pull this off is like it's a it's like a pretty impressive and I I think to some degree it's just this kind of like awesome experience yeah um but I'm already sorry to interrupt I'm already forgetting that you're not real like this really so I am novel it's this is just a an avatar version of me but but philosophical question yes but I mean but here's some of the so I put this on this morning and I was like all right like it's like okay so this my hair is a little shorter in this than my physical hair is right now I probably need to go get a haircut um and like and I actually I did happen to shave this morning but but if I hadn't you know I could still have this photorealistic Avatar that is that is more cleanly shaven right even if I'm you know a few days in um physically so I do think that they're gonna start to be these subtle questions that seep in where the the Avatar is realistic um in in the sense of this is kind of what you looked like at the time of capture but it's not necessarily temporarily accurate to exactly what you look like at this moment and I knew they're gonna end up being um a bunch of questions that come from that over time that I think are going to be fascinating too you mean just like the nature of identity of who we are are we the yeah people you know how people do like like summer Beachbody will people be for the scan they'll try to lose some weight and look their best and sexiest with the nice hair and everything like that I mean it does um it does raise the question of you know if a lot of people interacting with the digital version of ourselves who are we really are we the The Entity driving the Avatar are we the Avatar well I mean I think our physical bodies also fluctuate and change over time too so I think there's a similar question of like which version of that are we right there's there's like the yeah I mean it's and it's interesting identity question because all right it's like I don't know it's like weight fluctuates or things like that it's like I I think most people don't tend to think of themselves as the uh well I don't know it's an interesting psychological question some maybe some people maybe a lot of people do think about themselves as the kind of worst version um but yeah but I think a lot of people probably think about themselves the best version and and uh and then it's like what you are on a day-to-day basis doesn't necessarily map to to um to either of those so I know that's yeah it is there will definitely be a bunch of a bunch of social scientists and and folks will have to you know in in psychologists are really there's going to be a lot to understand about how our perception of ourselves and others um has shifted from this well this might be a bit of a complicated and a dark question but one of the first feelings I had experiencing this is I would love to talk to loved ones and the next question I have is I would love to talk to people who are no longer here that are loved ones so like if you look into the future is that something you think about who people who pass away but they can still exist in the metaverse you can still have you know talk to your father talk to your grandfather and grandmother and a mother once they pass away the power of that experience is one of the first things my mind jumped to because it's like this is so real yeah I think that there are a lot of norms and things that people have to figure out around that there's probably some balance where you know if someone is has lost a loved one and is grieving there there may be the way is in which you know being able to interact or relive certain memories could be helpful but then there's also probably an extent to which it could become unhealthy and I mean I'm not an expert in that so I think we'd have to study that and understand it in more detail we have you know a fair amount of experience with how to handle death and identity in people's digital content through social media already unfortunately right whether you know there's um you know unfortunately you know people who use our services die every day and their families you know often want to have access to their profiles and we have whole protocols that we go through where there are certain parts of it that um that we try to memorialize so that way the the family can can get access to it that way the account doesn't just go away immediately but then there are other things that are you know important kind of private things that that person has like we're not going to give the family access to someone's messages you know for example so um so yeah I think that there's there's some best practices I think from the current digital world that will carry over but um but yeah I think that this will enable some different things another version of this is is how this intersects with AIS right because and one of the the things that that were really focused on is you know we we want there to we want the world to evolve in a way where there isn't like a single AI super intelligence but where you know a lot of people are empowered by having AI tools to to do their jobs and you know make their lives better and if you're a Creator right and or if you run a you know podcast like you do then you know you have a big community of people who are super interested to talk to you I know you'd love to you know cultivate that community and you interact with them online outside of the the podcast as well but I mean there's way more demand both to interact with you and I'm sure you'd love to interact with the community more but you just are limited by the number of hours in the day so you know at some point I think making it so that you could build um an AI version of yourself that could interact with people you know not after you die but but while you're here to help um you know help help people kind of fulfill this desire to interact with you and your desire to build a community and there's a a lot of interesting questions around that um and you know that's obviously it's not just in in the metaverse I think you know we'd want to make that work you know across all the messaging platforms you know WhatsApp and messenger and Instagram Direct but you know there's certainly you know a version of that where if you could have an avatar version of yourself in the metaverse that people could interact with and you could Define that sort of an AI um version where you know people who know that they're interacting with an AI that it's not you know the the kind of physical version of you but maybe that AI even if they know it's an AI is the next best thing because they're probably not gonna you know necessarily all get to interact with you directly I I think that could be a really compelling experience there's a lot of things that we need to get right about it um that you know it's we're not ready to release the the version that a Creator can can kind of build a version of themselves yet but we're starting to experiment with it in terms of releasing a number of AIS that people can interact with in different ways um and I think that that is is also just going to be a very powerful um you know set of capabilities that people have over time so you've made major strides in developing these early AI personalities um with the idea where you can talk to them Across The Meta apps and have like interesting Unique Kind of conversations what can you describe your vision there in these early strides and what are some technical challenges there yeah so I mean a lot of the vision comes from this idea that you know I I don't I don't think we necessarily want there to be like one big super intelligence we want to empower everyone to both you know have more fun accomplish their business goals you know it's just everything that they're trying to do and you know we don't tend to have you know one person that we work with on everything and I don't think in the future we're going to have you know one AI that we work with I think you're going to want a variety of these um so there are a bunch of different uses um if some will be kind of more assistant oriented there's a sort of the kind of plain and simple one that we that we're building is called just meta AI it's simple it you can chat with it in any of your threads um it doesn't have a face right it's just it's um it's just kind of more vanilla and and neutral and and kind of factual but it can help you with a bunch of stuff then there were a bunch of cases that are more kind of business oriented let's say you want to contact a a small business um you know similarly you know that business probably doesn't want to have to staff someone to man the phones and you probably don't want to wait on the phone to talk to someone but they're having someone who you can just like talk to in a natural way who can you know help you if you're having an issue with a product or if you want to make a reservation or if you want to you want to buy something online having the ability to to do that and have a natural conversation rather navigate some website or have to call someone and wait on hold um it's gonna be really good both for the businesses and for for normal people who want to interact with businesses so I think stuff like that makes sense um then there are going to be a bunch of use cases that I think are just fun right so I think people are gonna I think there will be AIS that I can tell jokes you can put them into chat thread with friends I mean I think a lot of this because we're like a social company right I mean we're you know fundamentally around helping people connect in different ways and part of what I'm what I'm excited about is how do you enable these kind of AIS to facilitate connection between two people or more you know put them in a group chat you know make the group chat more interesting um around whatever your interests are sports fashion um Trivia video games I love the idea of of playing I think you mentioned Baldur's Gate an incredible game just having an AI that you played together with and you I mean that could that seems like a small thing but it could deeply enrich the like gaming experience I do think that AIS will be will make the NPCs a lot better in games too so that's a a separate thing that I'm pretty excited about but um but yeah I mean one of the one of the AIS that we've built that just in our internal testing people have loved the most is like a like an adventure text based um get like a dungeon master yeah um nice and and I I think um you know part of what what has been fun and we we talked about this a bit but we've gotten some like real kind of cultural figures to play a bunch of these folks and be the embodiment in the Avatar of them so um so Snoop Dogg is the dungeon master which I think is just hilarious in terms of the next steps of you know if you mention you mentioned Snoop to create a snoop AI so basically AI personality replica a copy or not a copy maybe um inspired by Snoop what are the some of the technical challenges of that what does that experience look like for Snoop to be able to yeah so that so starting off creating new personas is easier because it doesn't need to stick exactly to what you know that physical person would want how they'd want to be represented right it's like it's just a new character that we created so even though there's a snoop in that case is you know he's um you know he's basically an actor right he's playing the the dungeon master but it's not Snoop Dogg right it's it's um you know over the the dungeon master is um if you want to actually make it so that you have an AI embodying a real creator there's a whole set of things that you need to do to make sure that that AI is not going to say things that the creator doesn't want right and um and that the AI is gonna you know know things and be able to represent things in the way that the Creator would want um the way that the Creator would know um so I think that it's less of a it's less of a question around like having the Avatar Express them I mean that that I think where you know it's like well we have our kind of V1 of that that will release soon um after connect but you know that'll get better over time but a lot of this is really just about continuing to make the models for these AIS it's that they're just more and more I don't know you could say like reliable or predictable in terms of what they'll communicate so that way you know when you want to create the Lex um assistant AI that that your community can talk to um you can you know it's you don't program them like normal computers you're training them they're AI models not not not um kind of normal computer programs but um but you want to get it to be predictable enough so that way you can set some parameters for it and even if it isn't perfect all the time um you want to generally be able to stay within those bounds so that's a lot of um what what I think we need to nail for for the creators and that's why that one's actually a much harder problem I think than starting with with uh with new characters that you're creating from scratch so that one I think will probably um start releasing sometime next year um not this year but experimenting with existing characters and the assistant and games and a bunch of different personalities and experimenting with some small businesses um I think that that stuff we'll be ready to do this year and we're rolling it out you know basically right after connect yeah I'm deeply entertained by the possibility of me sitting down with myself and saying hey man like you need to stop the dad jokes or or whatever the idea of a podcast between you and AI assistant Lex podcast I mean there's uh just even the experience of a Kodak Avatar being able to freeze yourself like to basically first mimic yourself so everything you do you get to see yourself do it that's a surreal experience it feels like uh if I was like an ape looking in a mirror for the first time realizing like oh that's you but then freezing that and being able to look around like I'm looking at you it's a I don't know how to put it into Wars but it just feels like a fundamentally new experience like I'm seeing maybe color for the first time I'm seeing I'm experiencing a new way of seeing the world for the first time because it's physical reality but it's digital like and realizing that that's possible is just it's it's blowing my mind it's just really exciting because I I live most of my life you know before the internet and and experiencing the internet experiencing voice communication video communication you think like well there's a ceiling to this but this is making me feel like uh there might not be there might be that blend of physical reality and digital reality that's actually what the future is yeah I think it's a weird experience it's just it feels like the early days of like a totally new way of living and like there's a lot of people that kind of complain well you know the internet is not that's not reality you need to turn all that off and go you know in nature but this feels like this will make those people happy I feel like because it feels real the flaws and everything yeah well I mean a big part of how we're trying to design this these new Computing products is that they should be physical but I think part that's a big part of the issue with computers and TVs and even phones is like yeah I mean maybe you can interact with them in different places but they're they're fundamentally like you're sitting you're you're still and I mean people are just not meant to be that way I mean I think you and I have this shared passion for sports and martial arts and doing stuff that we were just moving around it's like so much of what makes us people is like you know you move around you're not we're not just like a brain in a tank right it's the where you know The Human Experience is a physical one then um so it's not just about having the immersive expression of the digital world it's about being able to really natively bring that together and and it I do really think that the the real world is this mix of the physical and the digital right the digital is there's too much digital at this point for it to just be siled to a small screen but the physical is too important so you don't want to just sit down all day long um at a desk so I I think that this is uh yeah I do think that this is the future this is I think the kind of philosophical way that I would want the world to work in the future is a much more coherently Blended physical and digital world there might be some difficult philosophical and unethical questions we have to figure out as a society uh maybe you can comment on this so the the metaverse seems to enable sort of unlock a lot of experiences that we don't have in the physical world and the question is like what is and isn't allowed in the metaverse you know in video games we allow uh all kinds of crazy stuff and in physical reality you know a lot of that is illegal so where's that line where's that gray area between video game and physical reality do you have a sense of that well I think I mean there are there are content policies and things like that right in terms of what what people are allowed to create but I mean a lot of the rules around physical I think who try to have a society that is as free as possible meaning that people can do as much of what they want unless you're going to do damage to other people and and infringe on on their rights and the idea of damage is somewhat different in a in a digital environment I mean when I get into you know uh some world with my friends the the first thing we start doing is shooting each other which obviously we would not do in the physical world because you'd hurt each other um but in in a game that's like just a it's almost you know it's like just fun and um and even like the lobby of a game right it's like it just it's not even bearing on the game which is kind of like a funny um sort of humorous thing to do so it's like is that is that problematic I don't think so because it's it's fundamentally it's not you're not causing harm in that world so I think that the um part of the question that I think we need to figure out is what are the ways where things could have been harmful in the physical world that we will now be freed from that and therefore there should be fewer restrictions in the Digital World um and then there might be new ways in which there could be harm in the digital world that there weren't the case before so there's more anonymity right it's um you know when you when you show up to uh you know a restaurant or something it's like all the Norms where you pay the bill at the end it's because you know you you have one identity and you know the I you know if you if you stiff them then like you know life is a repeat game and that's not going to work out well for you but you know in a digital world where you can be anonymous and show up in different ways um I think the incentive to act like a good citizen can be a lot less and that causes a lot of issues and toxic Behavior so if that needs to get sorted out um so I think in terms of what is allowed I think you want to just look at what what what are the the damages but then there's also other things that are not related to kind of harm you know less about what should be allowed and more about what will be possible that are more about the laws of physics so right it's like if you wanted to travel uh to see me in person you'd have to get on a plane and and that would like you know take a few hours to get here whereas you know we could just jump in a conference room and you know put on these headsets and we've basically teleported into a space where we're you know it feels like we're together that's a very novel experience that um that it it breaks down some things that previously would have defied the laws of physics for what it would take to get together and I think that that will create a lot of new opportunities right so um and one of the things that I'm curious about is you know there are all these debates right now about you know remote work or people being together and you know I think this gets us a lot closer to being able to work physically in different places but actually have it feel like we're together um so you know I think that the dream is that is that people will one day be able to just work wherever they want but we'll have all the same opportunities because you'll be able to feel like you're physically together I think we're not there today with with um with just video conferencing and the basic technologies that we have but I think part of the idea is that with something like this over time you could get closer to that and that would open up a lot of opportunities right because then people could live physically where they want while still being able to get the benefits of being physically or kind of feeling like you're together with people at work all the ways that that helps to build more culture and build better relationships and build trust which I think are real issues that if you're not seeing people you know in in person ever so yeah I don't know I think it's going to be it's very hard from first principles to think about all the implications of um of a technology like this and you know all the good and and and the things that you need to mitigate so you try to do your best to kind of Envision what things are going to be like and accentuate the things that they're going to be awesome and hopefully mitigate some of the the downside things but I you know it's the reality is that we're gonna be building this out one year at a time it's going to take a while um so we're going to just get to see how how it evolves and and what developers and different folks do with it uh if you could comment this might be a a bit of a very specific technical question but llama 2 is incredible it's the you've you've released it recently um there's already been a lot of exciting developments around it is there what what's your sense about its release and is there a llama three uh in the future yeah I mean I think on the last podcast that we did together we were talking about the debate that we were having around open sourcing llama too and I'm I'm glad that we did um you know I think at this point yeah there's the the value of open sourcing a foundation model like llama2 is significantly greater than um than the than the risks and in my view I mean we did we spent a lot of time we took a very rigorous assessment of that and red teaming it um but I'm I'm very glad that we release llama too I think the reception has been um it's it's just been really exciting to see how excited people have have been uh about it and it's gotten way more you know downloads and usage than I than I I would have even expected I was pretty optimistic about it um that's that's been great um llama three uh I mean there's always another model that we're training so I mean it's you know for right now you know we built we train llama two and we released it as an open source model and right now the priority is building that into a bunch of the consumer products all the different AIS and um in a bunch of different um products that we're basically building as consumer products because llama2 by itself it's not a consumer product right it's more of a piece of infrastructure that people could could build things with so that's been the big priority is kind of continuing to fine-tune and um and and kind of just get llama two and and it's um and it's little the branches that we've built off of it ready for Consumer products that hopefully you know hundreds of millions of people will will enjoy using those those products in billions one day but yeah I mean we're also working on on the Future Foundation models and um and I don't have anything new or news news on that I don't know and I don't know exactly when it's going to be ready um I think just like we had a debate around llama 2 and open sourcing it um I think we'll we'll need to have a similar debate and process to Red Team this and make sure that this is safe but and my hope is that we'll be able to to open source this next version when it's ready too but um but that's not that we're not we're not you know close to doing that this month I mean this is um that's just it's a thing that we're we're still somewhat early and working on well in general thank you so much for open sourcing llama 2 and for being transparent about all the exciting developments around AI I feel like that's contributing to a really awesome conversation about where we go with AI and obviously it's really interesting to see all the same kind of Technology integrated into these personalized AI systems of with the with the AI personas which I think will when you put in people's hands and they get to have conversations with these AI personas you get to see like interesting failure cases like where the things are dumb or they go into weird directions or and we get to learn as a society together what's what's too far what's interesting what's fun how much personalization is good how much generic is good and we get to learn all of this you probably don't know this yourself like we have to all figure it out by using it right yeah I mean part of what we're trying to do with the initial AIS launch is um having a diversity of different use cases just so that people can try different things because I don't know what's going to work I mean are people going to like playing in the text-based adventure games are they going to you know like having a comedian who who can add jokes um to to threads or they can want to interact with historical figures um you know we made we made one of Jane Austen and one of Marcus Aurelius and I'm curious to see how th
2023-10-09