NVIDIA Special Address at SIGGRAPH 2022
ready to go once upon a time it was all fun and games [Music] nvidia started off by making chips for video game machines graphics became serious business when people started using nvidia for blockbuster movies medical imaging devices and the world's most powerful supercomputers and then one day researchers discovered that our technology was perfect for ai [Music] today nvidia is the engine of ai [Music] engineering the most advanced chips and systems and software that makes them sing [Music] so robots can lend us a hand cars can drive themselves and even the earth can have a digital twin we live to tackle the world's biggest challenges [Music] and don't worry we still love our fun and games [Music] a decade ago nvidia's gpu became the engine of deep learning the foundational technology of modern ai artificial intelligence the automation of intelligent skills uses computers to write software no human can researchers are inventing groundbreaking advances at an incredible pace across computer vision speech natural language processing conversation recommenders robotics and more ai is revolutionizing the largest industries including computer graphics nvidia researchers applying the ai capabilities that our gpus enable are now reinventing 3d graphics the incredible advances of ai and 3d graphics have laid the foundation for the next computing platform the next big evolution of the internet the metaverse welcome to siggraph 2022 we have a lot to share with you today advances by nvidia research new technologies products and collaboration across three major areas of work computer graphics the metaverse and robotics all are interconnected and can trace their giant leaps to ai nearly a quarter of a century ago nvidia introduced the world's first gpu and real-time programmable shader a new type of graphics processor that did not run fixed function pipelines but rather executed programs called shaders programmable shading gpus revolutionized 3d and made possible the beautiful graphics we see in games today four years ago 21 years after our invention of the gpu at siggraph 2018 we launched nvidia rtx a brand new gpu architecture that extends the rasterization and programmable shading based gtx architecture with two new processors accelerated ray tracing on rt cores and deep learning on tensor cores rtx reset computer graphics and open new frontiers for computer scientists the advances in new algorithms many first introduced at siggraph have been nothing short of amazing from nvidia's gogan ai image creator and rtxdi global elimination to asc character animation to audio to face nvidia's ai research is impacting every aspect of computer graphics rtx the fusion of programmable shading ray tracing and ai has started the next era neural graphics neurographics will be integral to how artists create 3d worlds and how the world is animated and rendered one of the most impactful neural graphics inventions is dlss and ai that has learned to enhance the resolution of motion graphics today you will hear sonja share the inventions and breakthroughs of nvidia researchers she will show you the art of the possible and a glimpse into the future with neural graphics neurographics is one of the essential pillars of the emerging metaverse what is the metaverse simply it's the next evolution of the internet commercialized two decades ago the internet was about web pages hyperlinked over a network a decade ago web 2.0 emerged and the internet was about cloud services connected to applications that are oftentimes enjoyed on mobile devices now web 3.0 is here the metaverse is the internet in 3d a network of connected persistent virtual worlds the metaverse will extend 2d web pages into 3d spaces and worlds hyperlinking will evolve into hyper jumping between 3d worlds like games today 3d worlds are experienced through 2d displays and tvs and occasion with vr and ar glasses what are metaverse applications they're already here fashion designers furniture and goods makers and retailers offer virtual 3d products you can try with augmented reality telcos are creating digital twins of their radio networks to optimize and deploy radio towers companies are creating digital twins of warehouses and factories to optimize their layout and logistics and nvidia is building a digital twin of the earth to predict the climate decades into the future the metaverse will grow organically as the internet did continuously and simultaneously across all industries but exponentially because of computing's compounding and network effects and as with the internet the metaverse is a computing platform that requires a new programming model a new computing architecture and new standards html is the standard language of the 2d web usd universal scene description an open and extensible language of 3d worlds invented by pixar is likely the best language for the metaverse nvidia omniverse is a usd platform a toolkit for building metaverse applications and a compute engine to run virtual worlds rev will share a work and collaboration with the industry to advance usd steven will talk about the algorithms and the compute engine of omniverse one of the most exciting applications of omniverse is robotics systems that perceive their environment reason and plan there are many forms of robots in development self-driving cars manufacturing arms warehouse inventory movers agriculture planters and weeders and logistics picking machines robotics is the next wave of ai and omniverse is essential to our work to design train and operate robots one of the most widely used robots will be a digital human or avatar avatars will populate virtual worlds to help us create and build things be the brand ambassador and customer service agent help you find something on a website take your order at a drive through or recommend a retirement or insurance plan creating avatars requires state-of-the-art ai models that can see hear understand language be knowledgeable converse and animate simon will talk about our digital human research and our work to democratize the creation and deployment of avatars we have a lot to show you today sonia make it so thank you jensen it's great to be here with you at cigra 2022 neurographics intertwines ai and graphics paving the way for a future graphics pipeline that is amenable to learning from data this will enhance results help automate design choices and provide new opportunities for artists and creators that have yet to be imagined ultimately neural graphics will redefine how virtual worlds are created simulated and experienced by users there are many important challenges to address in the world of computer graphics let's talk about a few of this and the solutions provided by neural graphics that can revolutionize the field content creation is a time consuming process the exquisite craft of artists remains essential but combining air and graphics can help significantly streamline this process one way to save time is to take pictures of a scene or an object and try to reconstruct it in 3d we can think of reconstruction as the inverse of the rendering process going from images back to a 3d representation by parameterizing the 3d scene with neural networks and optimizing this representation using classical rendering techniques in the loop you can achieve high quality results in a single unifying framework another challenge is character animation which is essential to bring virtual worlds to life physics-based simulation makes the virtual world closer to the real world but building controllers for the simulated character is extremely difficult reinforcement learning automates a controller development process by having virtual characters learn how to move in a physically simulated environment by imitating human motion data and the results are astonishing achieving an immersive and seamless experience of the 3d digital world is critical for virtual reality users especially as we move towards a fully realized metaphors currently most vr users access the 3d digital worlds by wearing bulky head mounted displays with the advent of powerful ar algorithms we can co-design the optics of this hardware the display size and quality the optimized design can deliver full color 3d holographic images in less than half the size of existing thin vr displays let's take a look at the amazing work our researchers have been doing in the domain of neurographics [Music] [Music] [Music] [Music] [Music] aion graphics work in tandem and breakthroughs in each lead to enhance results when combined together it's a graph is presenting 16 papers that are advancing both neurographics and the mathematical foundations on graphics representing collaborations with 56 researchers across 20 universities we are extremely proud that two of these papers have been honored with the best paper award at siggraph today nvidia is releasing new research and tools to apply the power of neural graphics towards the creation and animation of virtual worlds we're excited to introduce cowl and wisp a research-oriented library for neural fields providing a common suite of tools and a framework that aims to accelerate new research re-implementing many of the existing work in neural fields can be done in just few lines of code with this work our goal is to enable and inspire fast-paced progress in neural and foundational graphics we look forward to seeing many more exciting breakthroughs at the next c graph next let's hear from rev stephen to talk about the metaverse and virtual worlds the metaverse is the next era in the evolution of the internet a 3d spatial overlay of the web linking the digital world to our physical world in this new iteration of the internet websites will become interconnected 3d spaces akin to the world we live in and experience every day many of these virtual worlds will be reflections of the real world linked and synchronized in real time many of these virtual worlds will be designed for entertainment socializing and gaming matching the real world's laws of physics in some cases but often choosing to break them to make the experiences more fun xr devices and robots will act as portals between our physical world and virtual worlds humans will portal into a virtual world with vr and ar devices while ais will portal out to our world via physical robots just like in the infancy of the internet no one can predict exactly how and how large the metaverse will grow but today we know we can lay the foundations the foundations of the metaverse require two things first a standard open and extensible way to describe all of the things in the virtual worlds of the metaverse similar to html's purpose in today's 2d web and second a computing platform designed for the creation and simulation of virtual worlds the next era of the internet the 3d internet or metaverse if you will needs a standard way of describing all things within the 3d worlds we believe universal scene description invented and open source by pixar is the standard scene description for the next era of the internet usd is far more than a static file format it's a 3d composition engine with apis for composing editing querying rendering collaborating and simulating virtual worlds usd is unique giving 3d artists designers developers and world builders the ability to work in non-destructive layered workflows the framework is highly extensible providing the ability to build custom schemas for specific workloads or industry applications we have a rich and long-term vision for usd looking at the 2d web we witnessed html's remarkable progress from html 1.0 in 1993 where it could only describe simplistic web pages to html5 enabling rich interactive media and dynamic applications usd is quickly evolving along a similar path from its origins in m e as a static description of large virtual worlds towards a system for dynamic procedural and real-time worlds we see the long view of usd and are fully committed to helping accelerate its development to reach that future sooner our most recent contributions include a custom mdl schema that can represent physically accurate materials and specify material parameters we also upgraded usd from python 2 to python 3 bindings and along with apple and pixar we extended usd to standardized support for rigid body physics we envision usd evolving as a complement to existing 3d standards most notably gltf today gltf is an important 3d file format used across a vast number of 3d applications and delivery on the web we're helping take the initial step towards harmonization of usd and gltf with our development of an open source usd file format plugin with this plugin creators can directly leverage the powerful layering composition and non-destructive editing capabilities of usd with their existing gltf asset libraries our next milestones aim to make usd performant for real-time large-scale virtual worlds and industrial digital twins this includes building support for international character sets geospatial coordinates and real-time streaming of iot data we're also enhancing the usd software stack to enable high-speed incremental updates and real-time proceduralism we will continuously test and open source these builds for our ecosystem of isv partners and customers let me show you a sneak peek of what we've been doing nvidia is enhancing usd to support extremely large and complex digital twins everything from sprawling factories to global scale climate change digital twins need to operate in full design fidelity at real-time speeds and optimize for different devices here are the ways that nvidia is extending usd to scale to the scene complexity of digital twins the source usd is compiled into fabric a gpu accelerated deeply vectorized data representation for real-time updates shown here in drive sim just-in-time optimizations like mesh merging and material distillation reduce scene complexity while preserving visual fidelity and can be applied at load time without modifying the source data this is crucial for making a ground truth representation of a digital twin available to both a supercomputer like ovx as well as a general consumer grid device let's take a look at an example that shows the performance of a digital twin in usd this virtual factory data set from the lotus sd digital factory team contains millions of prims in usd with nvidia's just-in-time scene optimizers the first pixels for this dataset appear in just a few seconds these optimizers can be configured to device requirements on the fly without the optimizers the data set takes minutes to load plays back at 7 fps and can exhaust gpu resources on lower end systems let's take a closer look at the optimization process in traditional usd rendering pipelines hydra copies the usd data into gpu buffers as is this is where optimizers come in we can load the entire scene headlessly into system ram and then merge meshes distill materials and stream geometry to the renderer in this case geometry streaming prioritizes the draw order based on camera heuristics such a solid angle with these just in time optimizations we achieve a 10 time speed up in load time and a 10 time speed up in playback frame rate this is just the beginning for where we can take just-in-time scene optimizations future work could include out-of-core scene optimization to relax system ram requirements and much more usd is the ground truth for digital twins scalable to any level of available computing power we're not only contributing to the development of usd but are continually testing and pushing its limits we built our omniverse platform as a usd engine and an open toolkit for building custom usd pipelines we built drive sim and isaac sim for autonomous vehicles and general robotic simulations on top of omniverse these demanding use cases require real-time performance physical accuracy and extremely large virtual worlds we're also working closely with partners in retail automotive energy telco and more to evolve usd and better serve their domains to further accelerate usd development and adoption we're building an open source usd compatibility testing and certification suite developers can test their builds of usd and certify that the custom usd components produce an expected result we'll provide a testing suite as well as a usd build system accessible by developers anywhere we want everyone to help build and advance usd we've built a wealth of resources available online for free including pre-compiled binaries for linux windows and pi pi we host usd view and usd builds in the omniverse launcher and we provide usd ready scenes on-demand tutorials documentation and instructor-led courses we're excited to introduce simulation-ready usd assets purpose-built for industrial digital twins and ai training workflows the usd ecosystem is vast with contributions from numerous leading technology companies not only in m e but in aec manufacturing and even robotics industries we're partnering with many of these companies to evolve usd and can't wait to see more names on the list i've covered the first requirement of the metaverse a standard way of describing all things in the 3d web usd and now steve parker will introduce our computing platform for virtual worlds just like the internet has a compute and networking engine the metaverse requires a computing platform to support linking the digital and physical world this platform for virtual worlds requires specific enabling technologies we have been building the core technologies of omniverse over the past two decades mdl material definition language to simulate physically accurate materials physics and advanced real-time physics engine and rtx the world's first hybrid rendering engine and ai system that enables us to simulate light and matter with physical accuracy in real time ai to assist in building or autonomously generate worlds make predictions and automate beyond human abilities and of course usd the powerful scene description standard for virtual worlds these core technology pillars are powered by nvidia high performance computing from the edge to the cloud let me show you the foundational technology on which omniverse is built [Music] so [Music] [Music] [Music] [Applause] [Music] see you in omniverse we have several exciting developments in core graphics technologies today we are celebrating 10 years of mdl the material standard for industrial workflows with mdl we unlock material representations from current silos allowing them to traverse software ecosystems we open source the mdl sdk at siggraph 2018. now we are open sourcing the mdl distiller and glsl back-end technologies further broadening the reach of mdl and enabling developers to bring mdl support to their preferred renderers mdl is a flexible language that can be used to define complex physically accurate materials like cloth with complex highlight patterns it defines the ground truth of a material with properties such as energy conservation that enables robust physical simulation many renderers can't render such complex materials and need to simplify or translate into their own material model other renderers may wish to remove some of the material complexity to achieve a performance target the mdl distiller automates such simplification of mdl materials the distiller relies on the mathematical robustness of the core mdl definition and provides mechanisms to manipulate the material layers like a symbolic algebra tool would manipulate algebraic equations so now material artists can author one single truth high quality material without the need to make compromises for simpler renderers the new open source glsl backend brings mdl support to renderer developers building on opengl or vulkan closing the gap to established graphics api standards with the mdl distiller and glsl backend we will see many more developers leveraging the power of mdl openvdb is an academy award-winning industry standard for memory efficient representations of sparse 3d volumetric data the visual effects industry uses openvdb to simulate and render water fire smoke and clouds last year we announced nano vdb bringing gpu acceleration to openvdb today we are announcing neural vdb the next evolution of openvdb let's see it up close whereas openvdb uses a hierarchical tree structure neural vdb introduces neural representations of both values and the underlying tree structure this dramatically reduces the volume's memory footprint allowing users to interact with extremely large and complex volumetric data sets in real time as well as transmit and share them more efficiently neural vdb also carries over the gpu acceleration of nano vdb which nvidia introduced in 2021 while these side-by-side examples look identical the memory footprint of the neural vdb representation is up to 100 times smaller to speed up training by up to 2x neural vdb allows the weights of the previous frame to be used for the subsequent frame neural vdb also enables temporal coherency or smooth encoding by using the network result from the previous frame this reduces the need for post-production effects like motion blur today openvdb has grown beyond its entertainment use cases into healthcare industrial manufacturing and design scientific computing and visualization robotics and machine learning applications by dramatically reducing memory requirements accelerating training and enabling temporal coherency neural vdb opens the door to scientific and industrial use cases including massive complex volume data sets for ai enabled medical imaging large-scale digital twin simulations and more now moving on to omniverse we have several new developments to share with you in omniverse kit and kit-based applications like omniverse create we have major updates to physics in omniverse with scalable sdf soft body simulation particle cloth simulation and soft contact models bringing real-time real real-world physical accuracy to virtual worlds we are also introducing omni live workflows a major development that delivers non-destructive live workflows at increased speed and performance to users connecting and collaborating between different third-party applications omnilive also enables custom versions of usd to live sync seamlessly making omniverse connectors much easier to develop we also have a new customizable viewport improved user interfaces enhanced review tools and major releases to our free 3d asset library omniverse now has several free usd scenes and content packs to get world builders started faster than ever omniverse audio to face is now available with full facial animation and emotion control omniverse machinima now has new content from beyond the wire postscriptum and shadow warrior machinima now has easily accessible ai enabled animation tools like audio to face and audio to gesture nvidia's modulus physics machine learning framework is now available as an omniverse extension delivering near real-time performance modulus trained physics ml models are a thousand to a hundred thousand times faster while providing unprecedented accuracy closer to high fidelity simulations working closely with our research teams we are introducing new ai powered artist and creator tools to omniverse ai toybox animal modeler a diffusion model based ai tool lets artists and creators iterate on an animal's form with point clouds and then generate a 3d mesh nvidia gogan is coming to 3d worlds with gogan 360. generating 8k 360 degree panoramas you can easily load into an omniverse scene lastly omniverse deep search is now available for enterprise customers letting teams use ai to help them intuitively search through massive untagged asset databases deep search lets you search on qualitative or vague inputs bringing up accurate results for red rusty barrel deep search works even if the usd data does not contain any tags or other metadata shared virtual worlds will enable the next wave of ai and will profoundly impact today's industries omniverse is where all of nvidia's technology comes together to realize this opportunity we are building out more developer tools technologies for constructing custom usd pipelines and are enabling full design fidelity visualization of usd scenes we're continuing our work in bringing multi-gpu multi-node hyper scalability to simulate large-scale scenes without compromising physical accuracy we are also continually infusing the latest ai into capturing generating composing simulating and optimizing these virtual worlds and to realize the potential of the metaverse for industrial and scientific use cases we continue to forge links from the physical world to digital worlds through sensors and iot devices omniverse is a network of networks nvidia and our partners are continually developing robust live usd portals from design simulation and cad software ecosystems to omniverse we have already developed several connectors for major design and content creation ecosystems today we are announcing several new connectors including blender autodesk alias and civil 3d siemens jt sim scale and open geospatial consortium connecting users across industries and disciplines you can also try the new ptc creo visual components and side effects houdini connectors they are now available in beta beyond these the omniverse usd ecosystem continues to grow with downloads and users growing nearly three times in one year and to 112 connections spreading across huge software ecosystems our partners continuously release new updates to the omniverse ready connections maxon's redshift hydro renderer is now available as is otoyo octane letting artists and designers use their preferred renders directly in omniverse epilogue sync twin built on omniverse is a new suite of tools and services to enable development of industrial digital twins preview 3d a 3d scanning partner now has usd support enabling workflows in omniverse and with siemens accelerator as part of the omniverse network industrial customers can unlock the power of the metaverse for a new era of digital twins our community of a hundred and fifty thousand omniverse users are everywhere portaling into omniverse with their rtx enabled studio laptops gaming pcs professional workstations and ovx servers with the next wave of omniverse worlds moving to the cloud omniverse is available for free download it at nvidia.com and start creating
today now let me hand it off to my colleague simon to talk about digital humans thank you steven at its core an avatar is a virtual robot that can perceive plan and act avatars will be everywhere they'll become more personal and intuitive eventually it'll be as natural as talking to another person however creating digital humans is complex we need breakthroughs in natural language processing speech and vision while simultaneously processing complex facial and body animations sophisticated materials and rendering all in real time everything must dynamically update and react to us in milliseconds just like human conversations nvidia's work on digital human spans across the company from research and visualization to animation and simulation from ai models all the way to deployment let me show you audio to face our facial animation ai created by close collaboration between nvidia research engineering and creative teams audio to face is an ai model that can create facial animation directly from voices our new version has just been released it has some significant new updates that fastly improve how people create high quality facial animation let me show you what i can do [Music] it's so dark where where is it i can't i can't see it i can't see anything but i know it's there waiting waiting the beige hue on the waters of the lock impressed all including the french queen before she heard that symphony again just as young arthur wanted okay i have a few jokes here what do you call a fish without any eyes we have an exciting road map for omniverse audio to face we just added more features to analyze and automatically transfer your emotions to your avatar we're expanding connections and availability of audio to face to other engines and platforms let me show you where we're headed first we're going to expand our multi-language support we're looking at improving different people's voice adaptability so no matter what type of voice input goes in the network will create the predictive facial animation even more accurately we're also going to provide a training sdk so users can train their own data serving more particular needs people might have to customize their avatar and looking even further ahead here's our vision it starts with simplifying how to create our own 3d avatar with our own likeness using just a single photo here we see a full cloud-based solution where you can drag a photo into this app and the 3d model with textures generated on the fly very easy and quick and these 3d models are not nerf or point cloud data they're actually animation ready meshes with a clean topology ready to be animated immediately with audio to face next we're developing an anatomically based high fidelity muscle simulation approach to generate any type of facial motion on any character the facial animation you see on the right is created through muscle stimulation only there are no other hand tweaks or animations applied we can teach the muscles how to behave and they learn how to move based on a video input from the left not only that this is a generalized solution that works for many different types of digital humans we can train and transfer these muscles to all kinds of digital humans enabling us to generate motion without the need to motion capture or do 4d scans lastly here's one of the most complex digital assets to create and simulate similar to image to 3d we can now generate strand-based hair from a single photo what we're showing here is an end-to-end system from generating hair to simulation for grooming hair we're combining a full physics-based system that allows you to interact and edit your virtual groom just like in the real world we want to democratize building interactive avatars for every platform today we're announcing omniverse avatar cloud engine omniverse ace ace is a collection of cloud-based ai models and services for developers to easily build avatars ita encompasses nvidia's body of avatar technologies from vision ai speech ai natural language processing to audio to face and audio to motion where you can animate your avatar's face with spoken words all running as apis in the cloud with omniverse ace developers can build configure and deploy their avatar application across any engine in any public or private cloud ace will democratize the ability to build and deploy realistic ai driven intelligent avatars let me show you what this looks like in this demo ace is powering a metahuman and unreal engine in real time that includes natural language processing a custom voice model and animation behaviors let me show you how it works hello hello my name is arjun i am a digital avatar what does it mean to be a digital avatar i'd be happy to explain in computing an avatar is a graphical representation of a user or the user's character or persona a digital avatar can also be a virtual assistant like myself can you tell me more avatars are the digital representatives in the virtual world the characters we control in video games can be thought of as avatars and increasingly we will adopt avatars as we shop socialize learn and work in the connected online environments that we are calling the metaverse let me show you another avatar in metaverse this is sam i've heard a lot about this thing called the metaverse can you explain it to me the metaverse is the 3d evolution of the internet a network of connected persistent virtual worlds i'd love to know more the metaverse will extend 3d web pages into 3d worlds and connect the digital world to the physical world the metaverse has multiple applications whether for entertainment socializing and gaming or industrial and scientific use cases to build digital twins thank you goodbye sure i am here if you have any more questions with omniverse ace third-party developers everywhere can create their own avatars that run in the cloud or on a device leveraging nvidia's entire body of digital human work our ai models can connect to all of the applications and engines out there and power the avatars in their environments all through the cloud with omniverse ace we're simplifying the creation of ai accelerated avatars we have much more in store for you in the future this is just the beginning now back to you jensen ai the most powerful technology force of our time will revolutionize every field of computer science including computer graphics nvidia's rtx is the engine of neural graphics rtx and neurographics is like a time machine the breakthroughs you saw today would have otherwise taken another decade the power of neural graphics to create worlds and synthesize images is an essential pillar of the metaverse the metaverse is the next evolution of the internet the 3d internet today we made three major announcements first we announced the significant release of omniverse including new toolkits for creating usd applications and new engine technologies for running virtual worlds nvidia omniverse runs on usd the leading candidate for the standard language of the metaverse second we also announced collaborations with industry leaders and an open source suite of usd interoperability tests virtual worlds are essential to designing training and operating robots the next wave of ai avatars will be among the most popular of the many forms of robots there will be billions of avatars our third announcement nvidia ace avatar cloud engine is a suite of ai models used to create avatars of all kinds ace runs in the cloud or in embedded systems the announcements we made today further advance the metaverse a new computing platform with new programming models new architectures and new standards our industry is set for the next wave the combination of ai and computer graphics will power the metaverse the next evolution of the internet have a great sig graph now approaching your destination [Music] hi welcome to nvidia [Music] [Applause] [Music] [Applause] [Music] [Applause] [Music] this is [Music] you
2022-08-15 17:36
