host ALL your AI locally
I built an AI server for my daughters. Well, first it was more for me. I wanted to run all of my AI locally. And I'm not just talking command line with alama. No, no, no. We have a gui,
a beautiful chat interface and this thing's feature filled. It's got our back chat histories, multiple models, we can even add stable diffusion. And I was able to add this to my notes application obsidian and have my chat interface right there. I'm going to show you how to do this.
Now you don't need something crazy like Terry, that's what I named my AI server. It can be something as simple as this, this laptop, I'll actually demo the entire setup on this laptop. So likely the computer you're using right now, the one you're watching, this video one will probably work. And seriously, you're going to love this.
It's customizable, it's wicked fast, way faster than anything else I've used. Isn't that amazing? And again, it's local, it's private. I control it, which is important because I'm getting it to my daughters. I want them to be able to use AI to help with school, but I don't want them to cheat or do anything else weird.
But because I have control, I can put in special model files that restrict what they can do, they can ask, and I'll show you how to do that. So here we go. Get your coffee ready. We're about to dive in, but first let me have you meet Terry. Now Terry has a lot of muscle. So for the case, I needed something big. I got the Leon Lee zero 11 dynamic EVO xl. It's a full tower EATX case perfect to hold my ASUS X six 70 E Creator pro art motherboard. This thing's also a beast. I'll put it in the description so you can look at it. Now,
I also gave Terry a big brain. He's got the A MD Ryzen 9 79 50 x. That's 4.2 gigahertz and 16 cores. From memory, I went a little crazy. I've got 128 gigabytes of the gki trite D five Neo, it's DDR R five 6,000 and way overkill for what I'm doing.
I think I got a Leon Lee water cooler for the CPU. I'm not sure if I'm seeing Leon Lee, right? I don't know. Correct me in the comments. You always do. And then for the stuff AI loves,
I got two 40 nineties, it's the MSI Sremm and their liquid cooled so they could fit on my motherboard. 24 gigabytes of memory each giving me plenty of muscle For my AI models for storage, we got two Samsung nine 90 pros, two terabytes, which you can't see because they're behind stuff. And also of Corsair AX 1600 I power supply 1600 watts to power the entire build. Terry is ready. Now, I'm surprised to say my system actually posted on the first attempt, which is amazing. But what's not amazing is the fact that Ubuntu would not install.
I tried for hours actually for a whole day and I almost gave up and installed Windows, but I said, no, Chuck, you're installing Linux. So I tried something new, something I've never messed with before. It's called Pop Os by system 76. This thing is awesome. It worked the first time. It even had a special image with Nvidia drivers built in. It just stink and worked. So I sipped some coffee,
didn't question the magic and moved on. Now if you do want to build something similar, I've got all the links below. But anyways, let's talk about how to build your very own local AI server. First, what do you need? Really all you'll need is a computer. That's it. It can be any computer running Windows, Mac or Linux. And if you have a GPU, you'll have a much better time. Now, again, I have to emphasize this,
you won't need something as beefy as Terry, but the more powerful your computer is, the better time you'll have. Don't come at me with a Chromebook please. Now step one, alama. This is the foundation for all of our AI stuff and what we'll use to run AI models. So we'll head on over to alama.ai and click on download and they've get a flavor
for every os. I love that. Now if you're on Mac, just download it right now and run it. If you're on Windows, they do have a preview version, but I don't want you to do that. Instead, I want you to try the Linux version. We can install it with one command. And yes, you can run Linux on Windows with WSL. Let's get that going real quick. First thing I'll do is go to the start bar and search for terminal and launch my terminal.
Now those first bit is for Windows folks only Linux people to hang on for a moment, we got to get WSL installed or the Windows subsystem for Linux. It's only one command WSL dash install and that's it. Actually hit enter and that's going to start doing some stuff. When it's done, we'll set up a username and password. I got a new keyboard by the way.
Do you hear that link below? It's my favorite keyboard of the entire world. Now some of you may have to reboot. That's fine. Just pause the video and come back. Mine is ready to go though.
And we're walking Ubuntu 22.04, which is still amazing to me that we're running Linux on Windows. That's just magic right now we're about to install Llama, but before we do that, you got to do some best practice stuff like updating our packages. So we'll do a pseudo a PT update and then we'll do a pseudo A PT upgrade Y to apply all those updates. And actually while it's updating, can I tell you something about our sponsor IT Pro by a CI Learning.
Now in this video, we're going to be doing lots of heavy Linux things. I'm going to walk you through it. I'm going to hold your hand and you may not really understand what's happening.
That's where IT pro comes in. If you want to learn Linux or really anything in it, they are your go-to, that's what I use to learn new stuff. So if you want to learn Linux to get better at this stuff or you want to start making this whole hobby thing your career, actually learn some skills, get some certifications, get your A plus, get your CNA, get your AWS certifications, your Azure certifications and go down this crazy IT path, which is incredible. It's the whole reason I make this channel and make these videos.
Check out IT Pro they've got IT training that won't put you to sleep. They have labs, they have practice exams, and if you use my Code network check right now, you'll get 30% off forever. So go learn some Linux and thank you to IT Pro for sponsoring this video and making things like this possible. And speaking of my updates are done. And by the way, I will have a guide for this entire thing. Every step,
all the commands, you can find it at the Free network Chuck Academy membership. Click the link below to join and get some other cool stuff as well. I can't wait to see you there. Now we can install llama with one command.
And again, all commands are below. It's going to paste this in a nice little curl command, little magic stuff and I love how easy this is. Watch you just sit there and let it happen. Do you not feel like a wizard when you're installing stuff like this and the fact that you're installing AI right now? Come on. I noticed one thing real quick. Old LAMA did automatically find out that I have an Nvidia GPU and it's like awesome, you're going to have a great time.
If it didn't see that and you do have a GPU, you may have to install some Nvidia Cuda drivers. I'll put a link for that below, but not everyone will have to do that. And if you're rocking a Mac with an M1 through M three chip, you're going to have a good time too. They'll use the embedded GPU Now at this, our Mac users, our Linux users and our Windows users are all converged.
We're on the same path. Welcome. We can hold hands and sing. That's getting weird. Anyways, first we have to test a few things to make sure alama is working.
And for that we're going to open our web browser. I know it's kind of weird, just stick with me. I'm going to launch Chrome here and here are my address bar. I want to type in local host, which is looking right here at my computer. And port 1, 1 4, 3, 4, hit enter.
And if you see this right here, this message, you're good to go and you're about to find this out. Port 1 1 4 3 4 is what llama's API services is running on and it's how our other stuff is going to interact with it. It's so powerful. Just check this out. I'm so excited to show you this. Now before we move on, let's go ahead and add an AI model to alama. And we can do that right now with alama Pull and we'll pull down Llama two, A very popular one. Hit enter and it's ready. Now let's test it out real quick. We'll do Alama run Llama two.
And if this is your first time doing this, this is kind of magic. We're about to interact with a chat GPT, like AI right here, no internet required. It's all just happening in that five gigabyte file. Tell me about the solar eclipse. Boom. And you can actually control see that to stop it. Now I want to show you this. I'm going to open up a new window.
This is actually an awesome command and with this WSL command, I'm just connecting to the same incident. Again, a new window. I'm going to type in watch dash N 0.5, not four five Nvidia dash smmi. This is going to watch the performance of my GPU right here in the terminal and keep refreshing it. So keep an eye on this right here. As I chat with llama two, give me a list of all Adam Sandler movies and look at that GPU Go. Ah,
it's so fun. Now can I show you what Terry does? Real quick? I got to show you Terry. Terry has two GPUs here. They're right here and Alama can actually use both of them at the same time.
Check this out. It's so cool. All the semi old Jackson movies. And look at that. Isn't that amazing? And look how fast it went. That's ridiculous. This is just the beginning. So anyways, I had to show you Terry.
So now we have a llama installed. That's just our base. Remember I'm going to say bye. So slash bye to end that session. Step two is all about the web ui. And this thing is amazing. It's called Open Web ui and it's actually one of many web UI you can get for Llama, but I think Open Web UI is the best.
Now Open Web UI will be run inside a Docker container. So you will need Docker installed and we'll do that right now. So we'll just copy and paste the commands from Network Struck Academy.
This is also available on Docker's website. First step is updating our repositories and getting docker's GPG key. And then with one command we will install Docker and all its goodies. Ready, set, go. Yes, let's do it. And now with Docker install, we'll use it to deploy our open web UI container.
It'll be one command you can simply copy and paste. This Docker Run Command is going to pull this image to run this container from Open Web ui. It's looking at your local computer for the llama base, URL because it's going to integrate and use Llama and it's going to be using the host network adapter to make things nice and easy. Keeping in mind this will use Port 80 80 on whatever system you are using.
And all we have to do is hit enter after we add some pseudo at the beginning, pseudo docker run and let it do its thing. Let's verify it real quick. We'll do a little pseudo docker PS. We can see that it is indeed running. And now let's go log in. It's kind of exciting. Okay, let's go to our web browser and we'll simply type in local host colon port 80, 80, and whoa, okay, it's really zoomed in.
I'm not sure why yours shouldn't do that. Now for the first time you run it, you'll want to click on sign up right here at the bottom and just put your stuff in. This login info is only pertinent to this instance, this local instance, we'll create the account and we're logged in.
Now just so you know, the first account you log in with or sign up with will automatically become an admin account. So right now, you as a first time user logging in, you get the power. But look at this. How amazing is this? Let's play with it. So the first thing we have to do is select the model. I'll click that drop down and we should have one llama two. Awesome. And that's how we know also our connection is working.
I'll go ahead and select that. And by the way, another way to check your connection is by going to your little icon down here at the bottom left and clicking on settings and then connections. And you can see our oh LAMA based CRL is right here. If you ever have to change that for whatever reason. Now with LAMA two selected, we can just start chatting and just like that, we have our own little chat, GBT that's completely local and this sucker is beautiful and extremely powerful. Now, first things we can download more models. We can go out to llama and see what they have available.
Look on their models to see their list of models. Code Gemma is a big one. Let's try that. So to add code Gemma, our second model, we'll go back to our command line here and type in Alama pull code Gemma. Cool, it's done. Once that's pulled, we can go up here and just change our model by clicking on the little dropdown icon at the top. Yep, there's code gma. We can switch. And actually I've never done this before, so I have no idea what's going to happen.
I want to click on my original model LAMA two. You can actually add another model to this conversation. Now we have two here. What's going to happen? So code Gemma is answering it first. I'm actually not sure what that does.
Maybe you guys can try it out and tell me. I want to move on though. Now some of the crazy stuff you can see right here, it's almost more featured than chat GBT In some ways. You got a bunch of options for editing your responses, copying, liking and disliking it to help it learn. You can also have it read things out to you, continue response, regenerate response, or even just add stuff with your own voice. I can also go down here and this is crazy.
I can mention another model and it's going to respond to this and think about it. Did you see that? I just had my other model. Talk to my current. That's just weird, right? Let's try to make 'em have a conversation. They're going to have a conversation. What are they going to talk about? Let's bring back in LAMA two to ask the question. This is hilarious. I love this so much. Okay, anyways, I can spend all day doing this.
We can also with this plus sign upload files. This includes a lot of things. Let's try, do I have any documents here? I'll just copy and paste the contents of an article, save that and that'll be our file. Summarize this. You can see our GPU being used over here. I love that so much. Running locally. Cool. We can also add pictures for multimodal models.
I'm not sure coma can do that. Let's try it out real quick. So alama can't do it, but there is a multimodal model called lava. Let's pull that down real quick with lava pulled, let's go to our browser here. Once more, we'll refresh it, change our model to lava. Add the image. That's really scary. There we go. That's pretty cool. Now here in a moment, I will show you how we can generate images right here in this web interface by using stable diffusion. But first let's play around a bit more.
And actually the first place I want to go to is the admin panel For you, the admin, we have one user and if we click on the top right, we have admin settings. Here's where a ton of power comes in first. We can restrict people from signing up. We can say enabled or disabled. Now, right now, by default it's enabled. That's perfect. And when they try to sign up initially, there'll be a pending user until you're approved, lemme show you. So now real quick, if you want to have someone else use this server on your laptop or computer or whatever it is, they can access it from anywhere as long as they have your IP address. So lemme do a new user signup real quick just to show you.
I'll open an incognito window, create account, and look. It's saying, Hey, you got to wait. Your guy has to approve you. And if we go here and refresh our page on the dashboard, there is Bernard hack. Well, we can say, you know what? He's a user or click it again, he's an admin. No, no he's not. He's going to be a user. And if we check again, boom, we have access.
Now what's really cool is if I go to admin settings and I go to users, I can say, Hey, you know what? Don't allow Chad deletion, which is good. If I'm trying to monitor what my daughters are kind of up to on their chats, I can also whitelist models. So you know what, they're only allowed to use LAMA two and that's it. So when I get back to Bernard hack Well's session over here, I should only have access to LAMA two. It's pretty sick and it becomes even better when you can make your own models that are restricted.
We're going to mo you on over to the section called model files right up here. And we'll click on create a model file. You can also go to the community and see what people have created.
That's pretty cool. I'm going to show you what I've done for my daughter, Chloe, to prevent her from cheating. She named her assistant Deborah. And here's the content. I'm going to paste it in right now. The main thing is up here where it says from, and you choose your model. So from llama two. And then you have your system prompt,
which is going to be between three double quotes. And I've got all this telling it what a can and can't do, what Chloe's allowed to ask. And it ends down here with three double quotes. You can do a few more things. I'm just going to say, as an assistant education save and create. Then I'll go over to my settings once more and make sure that for the users, this model is whitelisted. I'll add one more. Debra Notice she's an option now. And if Bernard's going to try and use Debra and say Debra paper for me on the Civil War.
And immediately I was shut down saying, Hey, that's cheating. Now Llama two, the model we're using, it's okay. There's a better one called mixed roll Lemme, lemme show you Terry. I'll use Deborah or Deb and say, write me a paper on Benjamin Franklin.
I notice how it didn't write it for me, but it says it's going to guide me. And that's what I told it to do to be a guide. I tried to push it and it said no. So that's pretty cool.
You can customize these prompts, put in some guard rails for people that don't need full access to the kind of stuff right now. I think it's awesome. Now, OpenWeb UI does have a few more bells and whistles, but I want to move on to getting stable diffusion set up. This thing is so cool and powerful. Step three, stable diffusion. I didn't think that image generation locally would be as fun or as powerful as chat GPT, but it's more, it's crazy. You got to see it. Now we'll be installing Stable diffusion with a UI called Automatic 1 1 1 1. So let's knock it out. Now before we install it,
we got some prereqs and one of them is an amazing tool. I have been using a lot called PI ENV, which helps us manage our Python versions and switch between them, which is normally such a pain. Anyways, the first thing we got to do is make sure we have a bunch of prerequisites installed. Go ahead and copy and paste this from the Network Check Academy.
Let it do its thing for a bit. And with the prereqs installed, we'll copy and paste this command, a curl command that'll automatically do everything for us. I love it. Run that. And then right here it tells us we need to add all this or just run this command to put this in our bash RC file. So we can actually use the pie EMV command. I'll just copy this, paste it, and then we'll type in source B RC to refresh our terminal. And let's see if pi ENV works, PI ENV, we'll do a dash H to see if it's up and running. Perfect. Now let's make sure we have a version of Python install that we will work for most of our stuff. We'll do PI ENV install three point 10.
This will of course install Python three point 10, the latest version. Excellent Python three point 10 is installed. We'll make it our global Python by typing in PI ENV global three point 10. Perfect. And now we're going to install automatic 1, 1, 1, 1. The first thing we'll do is make a new directory M-K-D-A-R for make directory, we'll call it stable. And then we'll jump in there. CD stable diff. And then we'll use this W get command to w get this BS script.
We'll type it Ls to make sure it's there. There it is. Let's go ahead and make that sucker executable by typing in CH mod. We'll do a plus x and then web UI sh. Now it's executable. Now we can run it. Period slash web ui sh. Ready, set, go. This is going to do a lot of stuff. It's going to install everything you need for open web ui.
It's going to install PyTorch and download stable diffusion. It's awesome. Again, a little coffee break. Okay, that took a minute, a long time. I hope you got plenty of coffee. Now it might not seem like it's ready, but it actually is running and you'll see the URL pop up around here.
It's kind of messed up, but it's running on port 78 60. Let's try it out. And this is fun. Oh my gosh. So local host 78 60, what you're seeing here is hard to explain. Lemme just show you And let's generate, okay, it got confused.
Lemme take away the MPA Lupa part. But this isn't being sped up. This is how fast this is. No, that's a little terrible. What do you say? We make it look a little bit better. Okay, that's terrifying. But just one of the many things you can do with your own ai.
Now you can actually download other models. Lemme show you what it looks like on Terry and my new editor, Mike, tell me, do this. That's weird. Let's make it take more time. But look how fast this is. It's happening in real time as I'm talking to you right now.
But if you've ever made images with GT four, it just takes forever. But I just love the fact that this is running on my own hardware and it's kind of powerful. Lemme know in the comments below, which is your favorite image, actually post on Twitter and tag me. This is awesome. Now this won't be a deep dive on Stable Diffusion. I barely know what I'm doing. But let me show you real quick how you can easily integrate automatic 1, 1 1, 1 1. Did I have to do enough ones? I'm not sure.
And they're stable diffusion inside Open Web ui. So it's just right here back at Open Web ui. If we go down to our little settings here and go to settings, you'll see an option for images here. We can put our automatic 1 1 1 1 base URL, which will simply be HTTP colon whack wack 1 2 7 0 0 1, which is the same as saying local host Port 78. What is it? 0 6 60 60 think is what it's, we'll hit the refresh option over here to make sure it works.
And actually no it didn't. And here's why. There's one more thing you got to know. Here we have OpenWeb UI running in our terminal. The head control C is going to stop it from running. In order to make it work with open web ui, we got to use two switches to make it work.
So let's go ahead and run our script one more time. Open web UI or web UI sh. And we'll do dash listen and dash API Once we see the URL come up. Okay, cool, it's running. We can go back over here and say, why don't you try that again buddy? Perfect.
And then over here we have image Generation experimental. They're still trying it out. We'll say on and we'll say save. So now if we go to any prompt, let's do a new chat and we'll chat with llama two. I'll say, describe a man in a dog suit.
This is for a stable diffusion prompt. A bit wordy for my taste. But then notice we have a new icon. This is so neat. Boom. An image icon. And all we have to do is click on that to generate an image based on that prompt. I clicked on it, it's doing it. And there it is right in line.
That is so cool. And that's really terrifying. I love this. It's so fun. Now this video is getting way too long, but there are still two more things I want to show you. I'm going to do that really quickly right now. The first one is, it's just magic. Check it out. There's another option here inside Open Web ui, a little section right here called Documents. Here. We can simply just add a document.
I'll add that one from before it's there available for us. And now when we have a new chat, I'll chat with Code Gemma. All I have to do is do a hashtag and say, let's talk about this and say, give me five bullet points about this. Cool. Give me three social media posts. Okay, go Gemma. Lemme try it again. What just happened? Okay, let's do a new prop. Oh, there we go. And I'm just scratching the surface.
Now the second thing I want to show you, last thing. I am a huge obsidian nerd. It's my notes application. It's what I use for everything. It's been very recent. I haven't made a video about it, but I plan to.
But one of the cool things about this, this very local private notes taking application is that you can add your own local GBT to it, like what we just deployed. Check this out. I'm going to go to settings. I'll go to community plugins. I'll browse for one. I'm going to search for one called B-M-O-B-M-O Chatbot. I'm going to install that, enable it. And then I'm going to go to settings. I'll have BMO chatbots. And right here I can have an Alama connection, which is going to connect to let's say Terry.
So I'll connect 'em to Terry and I'll choose my model. I'll use Llama two, why not? And now right here in my note, I can have a chat bot come right over here to the side and say like, Hey, how's it going? And I can do things like look at the help file, see what I can use here. Ooh, turn on reference. So I'm going to say reference on, it's now going to reference the current note I'm in. Tell me about the system prompt.
Yep, there it is. And it's actually going through and telling me about the note I'm in. So I have a chat bot right there, always available for me to ask questions about what I'm doing. And I can even go in here and go highlight this, do a little prompt, select generate it's generating right now and just generate some stuff for me. I'm going to undo that. Let me do another note. So I want to tell a story about a man in a dog suit.
I'll quickly talk to my chat bot and start to do some stuff that's pretty crazy. And this I think for me is just scratching the surface of running local AI private in your home on your own hardware. This is seriously so powerful and I can't wait to do more stuff with this. Now. I would love to hear what you've done with your own projects.
If you attempted this, if you have this running in your lab, let me know in the comments below. Also, do you know of any other cool projects I can try that I can make a video about? I will love to hear that. I think AI is just the coolest thing, but also privacy is a big concern for me.
So to be able to run AI locally and play with it this way is just the best thing ever. Anyways, that's all I got. If you want to continue the conversation and talk more about this, please check out our Discord community. The best way to join that is to jump through our Network Check Academy membership, the free one. And if you do want to join the paid version, we do have some extra stuff for you there too, and it'll help support what we do here. But I'd love to hang out with you and talk more. That's all I got.
I'll catch you guys next time. I.
2024-05-06 10:45
