Bringing Generative AI to Life with NVIDIA Jetson

Bringing Generative AI to Life with NVIDIA Jetson

Show Video

Hello everyone, welcome and  thank you for joining us today.  I'm Dusty from NVIDIA and I'm really  excited to be here and share some amazing   advances with you for bringing the latest  generative AI models and LLMs to the edge.  Since the introduction of transformers and then  the Ampere A100 GPU in 2020, we're all aware of   the hockey stick growth that's occurred with  the size of the models and their intelligence,   which feels approaching that of a human. It's been a huge leap forward in a  

relatively short period of time, and  things have really seemed to hit warp   speed with the open sourcing of cornerstone  foundational models like Lama and Lama 2.  There's a huge community out there of  researchers, developers, artists, hobbyists,   all working furiously on this day and night. It's incredibly exciting,   and the pace of innovation is  hard-slash-impossible to keep up with. 

There's a whole world out there to dive  into and discover, and not just LLMs,   but vision language models and multimodality,  and zero-shot vision transformers alone are   a complete game-changer for computer vision. Thank you to everyone out there who's contributed   to this field in some way or another. Many have been at it for years or decades,   and thanks to them, it feels like the future  of AI just kind of, poof, arrived overnight. 

Well, the time's now for us to see the moment and  bring this out into the world and do some good.  Naturally, due to the extraordinary compute and  memory requirements, running these huge models   on consumer-grade hardware is fraught  with challenges, and understandably,   there's been comparatively little focus on  the work done in deploying LLMs and generative   models locally outside of the cloud, and  even into the field in embedded devices.  Yet in the face of it all, there's a  growing cohort of people doing just that.  Shout out to our local llama  and our stable diffusion.  At the edge, Jetson makes the perfect platform for  this because it has up to 64GB of unified memory   with Jetson AGX-ORIN and 2048 CUDA cores,  or 275 Teraops of performance in a small,   power-efficient form factor. Why bother though? 

Why not just do it all in the cloud? Well, the reasons for that are the same that   they've always been with edge computing. Latency, bandwidth, privacy, security,   and availability. One of the most impactful   areas that underlies the other applications  shown here is human-machine interaction,   or the ability to converse naturally and have  the robot autonomously complete tasks for you.  As we'll see, you really need to be  geared for latency, especially when   real-time audio and vision are involved, and  of course anything that's safety critical.  It would also just seem good  to know how to run this stuff   yourself while keeping all your data local. And fortunately, there's a massive compute  

stack openly available for doing just that. We've been hard at work for a while now on Jetpack   6, and it's easily the biggest upgrade that we've  done to the Jetson platform software architecture,   which is now underpinned by an upstream Linux  kernel and OS distro that you can choose.  And we've decoupled the version of  CUDA from the underlying L4T BSP   so that you can freely install different  versions of CUDA, CUDNN, and TensorRT too.  We provide optimized builds and containers for  a large number of machine learning frameworks   like PyTorch and TensorFlow, and now  all the LLM and VIT libraries too.  There are world-class pre-trained models  available to download from TAO, NGC, and Hugging   Face that can all be run on Jetpack with unmatched  performance, along with edge devices like Jetson.  We're bringing more components and services  from Metropolis to Jetson for video analytics   with DeepStream, and we just released Isaac  ROS 2.0 with highly optimized vision gems,  

including SLAM and zero-copy transport between  ROS nodes called Nitrous for autonomous robots.  Jetpack 6 will be out later this month  and supports Jetson-Orin devices,   and going forward should provide a much  easier ability to upgrade in the future.  With that, let's dig into the actual models  that we're going to show you how to run today.  First up are open vocabulary vision  transformers like CLIP, LVIT, and SAM   that can detect and segment practically anything  that you prompt them for using natural language. 

Then LLMs followed by BLMs or vision language  models, multimodal agents, and vector databases   for giving them a long-term memory and  ability to be grounded with real-time data.  Finally, streaming speech recognition  and synthesis to tie it all together.  All of this we're going to  run on board Jetson Orin. 

So we've optimized several  critical VITs with TensorRT   to provide real-time performance on Jetson. These have higher accuracy, they're zero-shot,   and are open vocabulary, meaning they  can be prompted with natural language   expressing context and aren't limited to  a pre-trained number of object classes.  Clip is a foundational multimodal text  and image embedding that allows you to   easily compare the two once they're encoded  and easily predict the closest matches.  For example, you can supply an image  along with a set of complex labels and   it'll tell you which label is the most similar  contextually without needing further training   on object classes, meaning it's zero shot. CLIP has been broadly adopted as an encoder   or backbone among more complex VITs and vision  language models and generates the embeddings   for similarity search in vector databases. Likewise, ALVIT and SAM or segment anything   are also used CLIP underneath. AlVIT is for detection,  

whereas SAM is for segmentation. And then over here, EfficientVIT   is an optimized VIT backbone that can be applied  to any of these and provide further acceleration.  Again, these are using TensorRT and  it's available today in Jetpack 5 and   of course will be in Jetpack 6 as well. So, let's dig into some of these.  We have a demo video here showing  the capabilities of Alibite VIT.  So, you can see you could just type in what you  wanted to detect and it will start producing   those bounding boxes should it find them. Previously, you would have had to capture   your own data set, annotate it, train  a model like SSD, MobileNet, or YOLO. 

on your training data set and it would have  a limited number of object classes in it.  Well, you know, Alvit was based on clip  which just has a huge amount of images   and different objects in it so you can query it for practically anything here,   and it's a real game changer not to have to  train your own models for each and every last   detection scenario that you want to do. This is a really impressive technology   to be able to deploy in real time on Jetson,  getting up to 95 frames per second on AGX-ORIN.  So not only that, but when you combine  the detections from LVIT with a secondary   clip classifier, you can do further  semantic analysis on each object.  So here you see within each Detection ROI, it's doing further sub-detections.  In brackets means you want it to use  ALVIT, and in parentheses that we'll see   in a second, you want it to use CLIP. So this is not dissimilar to primary,  

secondary detection pipelines that we've done in  the past, but in this case, it's all zero-shot   open vocabulary and much more expressive. So you can see here not only is it detecting   these different objects, but it can  also classify them individually.  So happy face, sad face, other  types of things like that.  And you can perform some very powerful detections  and subclassifications this way with just writing   a simple query on your video stream. No code required even.  So all the code for running LVIT in  real time is available on GitHub here. 

It's called the NanoL project because we've  taken the original LVIT models and optimized   them with TensorRT, which is how we took it  from, I think, 0.9 frames per second to 95.  And there are various different backbones that  you can run different variants of Alvi IT that   have more accuracy, less accuracy,  higher performance, lower performance.  And this also is a very simple Python  API where you just put in your image,   put in your text prompts, and it spits out the  bounding boxes and their classifications for that.  So I highly recommend that you check out  this project on GitHub, if nothing else,   that you take away today, because  object detection is still, by and far,   the most popular computer vision algorithm run,  and this can completely revolutionize that.  So the segmentation analog of this  is called SAM or segment anything   model and it works very similarly. Basically you just provide some control   points like click on different parts  of the image that you want to segment   and it will automatically segment all of those  different blobs for you no matter what they are. 

It used to be that you would have to manually  go and make a segmentation data set, train   the model on that, and those segmentation data  sets were very resource intensive to annotate.  But now, you know, you can click on  practically anything and it'll segment for you.  And when you combine it with tracking, there's  another project called TAM, or track anything, and   that can do segment anything over video for you. We also have containers for this available on  

Jetson that you can take and run today. So everything that I'm going to cover in   today's presentation is available online, what  we call the Jetson AI Lab, which is a set of   tutorials that we've put together that can  easily guide you through running LLMs, VITs,   VLMs, vector databases, the whole thing. And all these tutorials are backed by   pre-built Docker container images  that we have for all these packages.  And you can take those containers,  combine them, mash them together,   build your own custom versions, and build your own  applications with them and deploy them that way.  So it takes a lot of the guesswork and debugging  from porting what normally are GitHub projects   for x86 and discrete GPUs and porting those  to ARM and AR64 to be able to patch all that   and you don't have to worry about getting  Transformers library to run and so and so. 

And we have a lot of different packages  here available at your disposal and   some really awesome tutorials to follow. And you can chat with your own llama models.  You can run lava and all  the VITs that we cover here.  And also stable diffusion too. Stable diffusion is really a lot   of fun to do image generation with. Okay, so now we're going to jump into   the large language models. Here is a chart of the LLM   performance that we currently have on Jetson. This is for AGX Oren, and it shows a bunch of  

different APIs that we've exhaustively built tried and optimized with CUDA and it turns out   the fastest ones are currently AWQ optimization  out of the MIT Song Han lab and also the   MLC TVM project out of Carnegie Mellon. So these two are by and far the fastest.  Llama C++ is also very popular. We use that too in the community   but MLC is about twice as fast and you  can see here we can run everything from  Lama2 7B up to the full 7DB at interactive rates. And we just showed the Lama models here,   but you can pick basically anything  from Hug and Face that you want to   run and run it through these and do quantization. So these are all running with 4-bit quantization,   so they're much more memory efficient,  which is how we get them to fit.  in the memory capacity because the weight  alone for 70B can be up to, you know,   140 gigabytes if you're doing FP16. But if you 4-bit quantize it,  

you're down to like 35 gigabytes or so. So here's a video of me chatting with Llama270B.  Hi Llama, how are you today? I'm doing well.  Thanks for asking. How about you?  Is there anything you'd like to chat about or ask?  I'm here to help with anything. I actually gave you a voice with text   to speech and automatic speech recognition. So we are actually talking to one another.  Wow. That's amazing. 

I'm glad to hear that we can  communicate in a more natural way.  How's the voice synthesis and speech  recognition working for you so far?  Is there anything you'd like to chat about? Oh, it's working really well.  You're running here on an  NVIDIA Jetson embedded board?  That's interesting. NVIDIA Jetson boards  

are powerful and efficient, making them  a great choice for Edge AI applications.  Are you using any specific software or frameworks  to run the AM models on the Jetson board?  And how are you finding the  performance of the board so far?  That's funny that you already know about it. Yes, you are running on the GPU in real time.  That's impressive.  So this was using Riva for the ASR and  TTS, and runs in real time on Jetson AGX.  As we'll see in some later videos, the  smaller models, they generate text way faster.  But it's just like a lightweight  web GUI that I put together with   WebSockets that's geared for low latency. And it takes the mic from your web browser,  

transmits it to your Jetson locally  over WebSockets, and then sends the  TTS audio back to it. So obviously, being at the edge,   most of these Jetson-based embedded systems have  cameras or other vision sensors attached to them.  And as such, everybody in the community is very  interested in vision language models like Lava,   MiniGPT4, and a lot of these multimodal  embeddings that are coming out.  And how they all work is essentially they use an  embedding model like CLIP that will combine text   and images into one common embedding space where  contextually, you know, concepts are very similar. 

So if you have a picture of a dog and the word  dog in this multidimensional embedding space,   those two vectors are found very  similar location to each other, i.e.  they convey the same thought  or sentiment to the LLM.  And then after that embedding is complete, in the  case of Lava, it uses literally the same clip in   VIT encoder that we mentioned previously. There is a small projection layer that maps   from the clip embedding dimensions  into the Llama embedding dimensions. 

And then they also fine tune the Llama model to be  able to understand those combined embeddings more.  And what we found is if you use the larger clip  model that uses 336x336 resolution instead of   224x224, it's able to extract much smaller  details and even read text out of images.  And there are lots of other  image embeddings out there.  It's a very active area of development,  like ImageBind, for example, which   combines way more than just images and text. That can do audio, inertial measurement units,  

point clouds, all types of  modalities that the LLMs can   be trained on and be able to interact with. Essentially, what we're doing is enabling   the LLMs with all of the different senses so they're able to assimilate a holistic   worldview and a perception world model so  they can greater understand things rather   than just have to do it all through text. So you can see here the performance of   the Lava models is very similar  to that of the base Lava models.  It is actually the exact same model architecture. It's just a smidge slower, a few tokens per second   slower, because it turns out that these image  embeddings are a lot of tokens, like 512 tokens   for a 336x336 image embedding, or I think it's 256 tokens for a   224 by 224 embedding. So, and all of those  

tokens come at the beginning of the text chat. So every one of those, you know, your text input   is at least 512 tokens long as opposed to just a  blank normal text chat starting at zero tokens.  So that's why it's just a slightly slower, but  otherwise it gets all the same performance.  So with the latest Lava 1.5 model that came out  that some of you might have seen, it's got some  

really exciting new features in it, including  the ability to output constrained JSON format.  So you can essentially tell it, you know,  detect certain objects or I want you to   pick out this and that and give it to  me in JSON so you're able to parse it   programmatically and actually do things with it. And whereas the VITs you needed to prompt with   specific things like I wanted to detect a  face or hands or whatnot, in this example   you can just tell it to detect everything and  it will spit out bounding boxes of all objects   or whatever it is that you're trying to do. And this will be very powerful for making   closed-loop visualizations that are  deployed in real-time embedded systems.  For example, smart traffic intersection  crosswalk monitors, or blind assistant devices,   or baby crib monitors, or anything that It is an open-ended problem that you want   to extract information from without having  to train all of your own models to do that.  In addition to things like autonomous  navigation, being able to query it on like,   hey, where's this path go? Are there obstacles in my way?  Are those obstacles dynamic? All types of things like that. 

And this is really exciting. Another point to point out   is this whole model, Lava 1.5, is a huge  improvement over the previous versions of it.  but it was still only trained  in one day on eight A100 GPUs.  So if you collect your own data set,  it's actually within the realm of   possibility to fine tune one of these  yourselves for your own applications. 

So here's a video demo of chatting  with Lava 1.5 on AGX Oren.  What kind of fruits do you see? In the image, I see a variety of fruits,   including oranges, bananas, grapes, and apples. How much do the peaches cost?  The peaches cost 2.50. How much do the oranges cost?  The oranges cost 1.50. What breed of dogs are in the picture?  There are two dogs in the picture. One is a husky and the other is a  

golden retriever. Where are they?  The dogs are sitting on a rocky  hillside, likely in a mountainous area.  What does this sign say? The sign says, no outlet private road.  What kind of trees are these? The trees in the image are pine trees. 

Where's the path go? The path is in the woods,   surrounded by trees and vegetation. What kind of bird is this?  The bird in the image is a robin. Who has the ball?  The boy in the black and red shirt has the ball. What game are they playing?  They are playing soccer. Can you describe the image? 

The image features a young girl  riding a brown horse in a corral.  The girl is wearing a red shirt and  is positioned on the horse's back.  The horse is galloping around the corral,  showcasing its agility and strength.  The scene captures the excitement and joy  of the girl's horse riding experience.  Here's a look at some of the example  code that powers these demos. 

And it's basically a lightweight  wrapper that I put around MLC and AWQ,   since those weren't supported in some  of the other LLM libraries out there.  And in addition to all the multimodal embedding  management stuff, which we'll talk about.  But it's got a very simple  API for text generation.  Essentially, you load the model, it will  quantize it for you if it's not already done.  You create this chat history stack and then you  can append either text prompts or images to that. 

It will automatically perform the embeddings  for you, depending on what data type.  those inputs are, and then it  generates a stream of output tokens.  So everything we do here is geared for real-time  streaming, so you can get that data presented   to the user as soon as possible. And then you basically just output   the bot response to the chat history too. Now, let's say you want to do your own chat   management dialogue, you can totally do that. You can just pass in strings of text to  

the model generation function. What you should be aware of is   that the chat histories work best when you  keep a cohesive stack going, because then   you don't have to go back and constantly  regenerate every single token in the chat.  For example, we know Lama 2 models  have a max token length of 4096.  But if you were to generate the full 4096 length  chat every time, it would take really long. 

Instead, you can keep that cached  in what's called the KB cache.  And if you do that between requests, then  the whole state of that chat is kept.  You can run multiple chats simultaneously,  but it's highly recommended  to keep the chat stack flowing as opposed to  going back and forth and just assimilating   it all from scratch every time. And the reason for that is because  

there are two stages to LLM text generation. The first is the decode or what's called prefill,   where it takes your input context  and has to essentially do a forward   pass over every token in there. And this is a lot faster than the   generation stage, but it still adds up. when you're talking full 4096 tokens here.  So you can see if we're running, you know,  Llama70B on a full 4096 token length chat,   it'll take 40 seconds to prefill that whole thing. That's before it even starts responding.  But if you only prefill the latest input, you're  looking at, you know, a fraction of a second.  That's typically like the dot dot dot that  shows up in the chat or like agent is typing. 

What it's actually doing is prefilling your  input into it before it can start generation.  So this is why managing the KV cache between  requests is actually crucially important   to keep a very consistent chat flow going. Likewise, here's a look at the token generation   speed and how that varies with the input length. That does decrease slightly as well once  

you get up into the higher token length. So that's something to account for as well.  So obviously a big concern with all of this is  what are the memory utilization requirements,   and that along with the token generation  speed are what really drive everyone   very heavily towards quantization methods. So a lot of these LLM APIs that we talked about,   like Llama C++ and others, AutoGPTQ, XLlama,  they have lots of different quantization methods.  You can go everywhere from two bits  to eight bits, or most of the time,  Below 4 bits you start to see degradation  in performance, but at you know Q4 A16   quantization I've not really seen any  difference in output from that which   is which is really good because it takes Lama  70B from being 130 gigabytes of memory usage to  Down to only 32 gigs and that is  much more deployable for Jetson   and likewise for the smaller Jetson is too. You can run Lama 2 7B on an O-Rin 8 gigabyte  

board or Lama 2 13B on the O-Rin X 16 gigabyte. And we can see that here that we have a whole   lineup of different Jetson  modules that you can deploy.  And each of these, conveniently,  has a typical model size that is   well-fitted to its memory capacity. So as I mentioned, the 7B models  

are a good fit for the Oren Nano 8 gigabyte. 13b models are a good fit for the orenx 16   gigabyte and so on and so you can basically mix  and match the level of intelligence that your   application requires along with its performance  and other swap C requirements like size, weight,   power and cost of the embedded system that you're  deploying and be able to pick the Jetson module   that's appropriate for you to deploy those. So a few slides ago, I showed some code   that was basically a low-level API for  doing text generation with the LLMs.  Once you start getting more complicated  and adding things like ASR and TTS and   Retrieval Augmented Generation and all these  plugins, the pipelines get very complex. 

And if you're only making one bespoke  application, you can absolutely code all of   that in one Python application yourself. In fact, the first version of that   Llamaspeak demo I did was just like that. But eventually, when you start iterating,   make different versions of that, let's say I  want to have one that's multimodal or I want   to do a closed-loop visual agent, you're going to  start having a lot of boilerplate code in there.  So I've written a slightly higher level API on top.  of that text generation function that you  can implement all these different plugins in.  It's very lightweight. It's very low latency. 

It's meant to get out of your way and make this  all easier instead of harder without sacrificing   even one token per second of generation speed. And with this, you can very easily chain together   all of these different text and image processing  methods and use them with other APIs as well.  So these are just two basic examples of  what the pipeline definitions look like,   and it can be a completely directed, open-ended,  multi-input, multi-output graph, and get some   pretty complicated setups going in there. So another cool aspect that this enables is   what we refer to as inline plugins, or  the ability for the LLM to dynamically   generate code for APIs that you define to it. For example, how is it supposed to know what   the time is, or do internet search queries, or  what the weather is, or perform actions like   turn the robot left or right? All of those core platform   functions you can define in the system prompt and   explain the APIs to the LLM, and when it  needs to, it'll dynamically call that.  This is good to do on top of retrieval augmented  generation, which we'll talk about in a second,   because it doesn't just retrieve  based on the user's previous input. 

It can do so as it's currently generating  the output and insert that into the   output as it's going on. That is a good benefit of  you know, maintaining lower level access to APIs  as opposed to, you know, just going back to the   cloud for everything because you need the ability  to stop token generation, run the plugin and then   insert that into the output and then continue  the bot output in addition to doing things like   token healing or implementing guardrails  and guidance functions, things like that.  All of that is very good  to have very good granular  access to the token generation so you can stop  it right when you need to and then restart it   completely asynchronously so you don't destroy  all of the low latency pipelining that you have.  This is just a basic example here of  when I was toying around with this   to see if it actually could generate it. And this was just with the Llama 2 7B using   its baked in code generation abilities. And this is like basic stuff for it.  I do recommend using JSON format, even though it's more verbose   and results in more tokens, especially if you have  functions that have multiple parameters with them,   because JSON allows it to keep the order of the  parameters straight, so it doesn't confuse them. 

If you don't really have parameters or just  very simple, like the example shown here,   you can just do like a simple Python API. And there are other  Open-source framework plug-in agent frameworks  that you can use like Lang chain and hugging   face agents and Microsoft Jarvis that do  this as well They're not quite as geared   for low latency low overhead, which is like  why I've gone and done this but At the same   time those can do very similar things to In fact, the Hugging Face Agents API has   the LLM generate Python code directly, which  then it runs in a limited sandboxed interpreter.  So it's able to interface directly with Python  APIs, which is really cool and is not hard to do. 

In this case, I prefer just to keep it JSON or  text and manually parse that and call the plugins.  So, you know, it's not  having full access to Python.  So I've mentioned retrieval augmented  generation a few times and that is big   not only in enterprise applications where you  want to index a huge amount of documents or   PDFs and be able to have the LLM query against  those because remember the context length is   max of 4096 with LLAMA2 or there's a lot of you know, rotary embedded encodings that go   up to 16K or 32K or even more. But there's always a limit.  And you might have hundreds of thousands of pages  of documentation that you want to index against.  And when we start talking about multimodal,  you can have huge databases of thousands   or millions of images and video  that you want to index against. 

And not all of that can be included. So what happens is basically you take   the user's input query and search  your vector database for that.  It uses very similar technologies  to LLMs and the VIT encodings.  In fact, clip embeddings are used  in a demo that I'll show you next.  But it essentially uses similarity search  to determine what objects in the database  are closest to your query. And it's a very similar concept to the   multimodal embedding spaces, how that all works. And there's some very fast libraries out there  

for this called FICE and Rapids RAP that are  able to index like billions of records and   retrieve them lightning fast based on your queries.  And those are very good libraries to use. And I've used those on Jetson here to do   a multimodal image search vector database. I basically made this demo to prove out the   abilities of the clip transformer encoder  and just to be able to understand what   I could actually query for retrieval augmented  generation before integrating that into the LLMs.  So you can see here, you can not only query with  text, but you can do image searches as well. 

And it's pretty advanced image  search, and it's completely real-time,   real-time refresh here running on Jetson. This is indexed on the MSCOCO dataset,   275K images from MSCOCO. That took about, I think it was   like five or six hours to index the whole thing. But the actual retrieval search only takes on the  

order of like 10 or 20 milliseconds, which means  it's not going to add lag to your LLM generation   pipeline, which is very important because we  don't really want more than a couple seconds   of lag between the user's query and the response,  especially if it's a verbal communication ongoing.  So here's a chart of the retrieval  augmented generation search.  based on how many items are in your database. And some of these databases can get really huge,   especially with like corporate  documents, things like that. 

At the edge, I think it'll be smaller because you  only have so much space available on the device.  But you can see here, it's just on the order  of milliseconds for most people's applications.  And I also break out different  embedding dimensions here too. 

So some of the higher end  embeddings like ImageBind use  every single image or text as a 1024 element  vector that describes it in this multi-dimensional   embedding space. Clip uses 768.  Clip large uses 768. So then that was the one shown in the demo here.  So this scales very well. I think it would be pretty   rare that you would get up to 10 million entries on an   embedded device like Jetson. But hey, if you're doing lots   of data aggregation, have 30 HD camera  streams coming on, it's entirely possible.  And it's still only on the order of a  fifth of a second or less to do all that,   which is completely reasonable. All right, so tying it out with Riva  

to coordinate how I actually made these demos with  the text-to-speech, Riva is an awesome SDK that's   openly available from NVIDIA that incorporates  state-of-the-art audio transformers that we've   trained along with TensorRT Acceleration. And it's completely streaming.  It does streaming, ASR, and TTS. You can do 18 ASR streams in real time or 57  TTS streams in real time on Jetson AGX Orin and  nobody's really going to do that many streams at   the edge unless you have like multi-microphone  devices or set up something like that.  But what it does mean is when you're  only doing one stream, you're going to   take like less than 10% of the GPU to be  running all that, which is great because   that means that our LLM token generation rate is only going to go down by less than 10% there.  Because the LLMs, they will consume 100% of  the GPU, all that you throw out with them. 

And Riva has lots of   different ASR and TTS models that it ships. It also has neural machine translation and I've   seen some people do some really cool demos with  this where you can do live translation between   different languages and it turns out that a lot  of the LLMs like Llama are trained in English  There are some LLMs out there that are  multilingual, but if you're working with   an LLM that's trained in English, but you want  to be able to converse in other languages,   you can use neural machine translation in your  pipeline to essentially translate between ASR,   the LLM, and then the LLM output back into the  language that you want before it goes to the TTS.  Another really cool thing that Riva has is  these what's called SSML expressions for TTS.  So you can speed up or speed down or  change the pitch or add in like emojis   or laughs or all types of cool things  to make the voice sound more realistic.  And overall it sounds really good for  just being done on an edge device locally. 

None of the demos that I've showed  you so far rely on any cloud compute   or off-board compute whatsoever. You can run these entirely without   any internet connection once you download the  containers or have your application built.  Here is a pipeline block diagram  of essentially what the interactive   verbal chat management looks like. And it turns out there's a lot of nuance   to live chat back and forth, mainly the ability  to interrupt the LLM while it's outputting.  We know these LLMs, they like to  talk and they will just keep going. 

And you can instruct them to be very  concise in their output, but in general,   they will like to ramble on a little bit. And it's important, like in the video, to be able   to speak over them and have the LLM either resume And if it turns out you didn't want to query it   or, you know, stop itself when  you ask it another question.  And the best way that I found to do this is  a multi-threaded asynchronous model where   there's just a bunch of queues, everything is  going into these queues and being processed  and you need the ability to interrupt and  clear those cues based on things that happen.  So for example, the Riva ASR, it outputs what's  called the partial transcript as you're talking.  Those are in the videos where you see the  little bubbles come up and it's always   changing because it's constantly redefining  with beamforming what it thinks you said.  But then when you get to the end of a sentence,  it does what's called the final transcript. 

And that final transcript is what  actually gets submitted to the LLM.  But if the partial transcript starts  rolling in and you speak more than   like a handful of words, it will pause the LLM. And if it doesn't get any more transcript in like   a second or two, the LLM can resume speaking. If it does go final, if the ASR transcript does  

go final, then that previous LLM response is  canceled and a new one is put in its place,   which is important because then you don't want  it to keep spending time generating an old one.  when you are already on to  answering your next question.  So there it turns out there is a lot of nuance  in here and we tried to do it with the least   amount of heuristics as possible because that's  just lends itself to corner cases and in general   it's really nice and fun to be able to chat back and forth with these models.  And I really highly encourage you  to go on to the Jetson AI lab,   download these containers, start  playing around with the different   models and discover their personalities and  go from there, build your own applications.  And I think before long, we'll see these out  there on robots in real world embedded systems.  So yeah, Let's all do it together. 

If you need any help or support at any time, I'm  always available on GitHub, the forums, LinkedIn,   my email, DustinF at, and we're all  out there to help each other and keep it going.  So yeah, it's great to be part of this community. And thanks again so much for joining us today.  At this point, we're going to get set up for Q&A. So if you haven't already, please feel   free to type in your questions. If you are watching a replay of this,   yeah, feel free to ask questions another  way on the forums or GitHub or anywhere. 

All right. Thanks.

2023-11-29 15:37

Show Video

Other news