Hello, I'm Rob Hirschfeld, CEO and co founder of rockin and this is the give your Pixie wings. This presentation it's actually one of two parts. This one we're going to talk about the basics of how Pixie works. And the second part we're going to go into alternatives and other ways to get provisioning done. So if you're advanced jump head. Otherwise hang on we are we've got a lot of material to cover. This is based on a talk that I gave for SRV con. But this is going to be rackin focus. So we're going to talk about rack end and how rack end does these things. So as your host, I
want to guide you through what is going on with Pixie provision it is absolutely essential to rack and and racking customers and people who are using digital rebar to handle provisioning. Digital rebar is a distributed infrastructure is code management platform. Which means that we're helping people run data centers better. That's that's what we do. Core an essential part of that is digital rebar provision, which is our provisioning abstraction layer. So it handles Pixie on bare metal, really on any infrastructure that you can throw at it. Using a variety of techniques. In this talk, we're going to be deeply focused on the Pixi bare metal boot process. But everything I'm showing you can be applied more generally to other types of infrastructure also. So let's dig it. In concept provisioning super, super easy, all we're trying to do is put an
operating system onto a computer could be a switch, it could be a server really doesn't matter. And while it's conceptually easy, it's actually really, really hard. And it's really hard because we have to deal with bootstrapping, which is literally starting a system up from nothing. Supposed to be hard. So you have to have a process to do that. And the first parts of this process are going to be fragile. firmware limitations on servers, you know, you can only do what the server is able to do, and then work around it. And these are variations right when we're talking to customers at rack end. They're dealing with different types of servers, different vendors, different protocols. Some of them are using old legacy BIOS, some of them are using UEFI BIOS, some of them are turning it off altogether. Some of them using redfish, some are freezing vendor,
it is crazy heterogeneous. And that's okay. We have ways to cope with all that. But that's what makes it hard. That's why we spend a lot of time doing what we're doing. That's why it's very hard to just take a TFTP server and a DHCP server and build a real provisioning infrastructure, we'll explain that. Networking is always hard networking is networking security, we're going to talk about secure boot and section two, important critical, you need to think about it. These systems especially as we get to edge and their zero trust have to be done right. And then provisioning is not complete. If you don't do post configuration, you can't have 100 copies of the exact same thing. Every machine has to have an identity and logins and credentials and IP addresses that are right and applications installed that are working correctly. So there's a lot to think about. And
that's what this talk is going to cover. It's not even going to go into things that rackin spends a lot of time helping customers do that are also super important to getting your data center and your infrastructure running well so you know as you think about this, we have yes answers to how do I actually know what my system is configured at can I fix it and validate that it's the thing that is there is what I think it is that it didn't lose some dims or machine drivers failed or somebody cabled it wrong, all those things are absolutely essential to good operational process. And we can help you hardware configuration read and BIOS can read and BIOS out of band management. Also, you know as getting your your BIOS flashed even on your Nic cards or your raid controllers absolutely essential not covered in this talk. Things that we're happy to talk about is rockin but deep and
you need to get this other provisioning pieces right before you go there naming and address and credentials and Jackson all absolutely can't run infrastructure if you don't get all of these things right and I'm sorry there's no hall pass of worry about only a little bit of it. You got to get it all right. The good news is we spent a lot of time helping customers get it all right and we have out of the box stuff that handles most all of this so it's nice in this talk since dragon I can tell you yes we do that we try to make it easy. We do it all as infrastructures code. Check this out. Check it out. It's really powerful. But before I get there may teach you a little bit about how all this stuff works. So exploring provisioning basically most people when they think of Pixie they use as a proxy for net boot net. You'll hear Pixie I Pixie oni pro and IE which is for
switching kickstart presets. cede all very common. That's the first part of this talk. Image deploy and esoteric flavors are in the second talk a whole new whole new section will talk about Packer. Writing boot partitions, cloud init kexec, secure boot BMC will add an Raspberry Pi, as a special addendum into this bring you up to speed on all of this stuff. Those are critically important. Because really, the basics are great. Most of our customers are really working to get into these more advanced, faster, more robust and resilient provisioning styles. So part two, but
bear with me. All these things actually lead to kernel init process. So no matter what we're doing, at some point, you're bootstrapping the kernel, kickstart pre seed, weasel, Windows has a net boot partition. I'm going to call it kickstart just to keep things simple. That's the Red Hat Santos way to do it. But it's the same concept. It's a kernel initialization kernel in it process that we have to get to once we've got to that the road gets less bumpy. But that's where we got to be. And there's challenges after this is our internal what we show to customers, when they start asking really hard questions and their network teams get involved. We have a series of these, this is one of I think seven or eight swimlane charts that show all of the process and pieces of how provisioning flows with documentation and things like that. That's my way
too much of an eye chart for this. So don't try to read it at the moment. And then over here, this is the streamlined version of it. Still an eye chart where we actually walk through all the things that digital rebar goes back and forth and hands off during the provisioning process. super important. Also, we have a whole bunch of training materials on that too. We're not there yet. Before we go further, I do want to show you what this process actually looks like. So this is a pixie boot it running in a virtual machine to our in memory OS called sledgehammer which is super optimized to go quickly. I just started the OS running, it's going to DC DHCP it's going to pick up I Pixy Linux. Hi, Peter oven, we see his name all the time loading
stage one and stage two of those and doing the kernel in it. Let's let's remember, this is highly optimized, it runs in memory, it doesn't have to do a lot of the normal kickstart. And so it completes really quickly even so you can see there's a ton of things getting set and updated. Turning on a stage demons and things like that to actually get the system going to get to a command line prompt. That is a full provisioning process. We're going to start much simpler with some some big animal pictures. Because you need to understand this first. Anyway, bootstrapping is a multi stage process. So let's look at each one. The first thing a server has to do in this process is just get on the network. So firmware on the
server called PLC, often embedded in the NIC card controller will get on the network. So it'll use a DHCP Dynamic Host Discovery Protocol or control protocol. To get an IP address. That protocol has a whole bunch of options baked into it. It's UDP, which means it's just sending datagram packets, it's not a TCP service. So you don't have to have a lot of the other infrastructure you'd need. It sends back a next server instruction, which is
telling it where to find the next bootstrapping instructions and some options like gateways and its IP address and all sorts of information. Digital rebar has a lot of control over those instructions, we build in a DHCP server. It's not required, but we highly recommend using it because we control this process and then feed it forward into the rest of the provisioning operations for you. But no matter how you're doing it, you need to have DHCP respond with some next server instruction to that that initial Pixie bootloader. From there, it's going to download URL Pixy Linux or some other bootloader. Usually I'll pick the Linux that is going to get the system started. So that's what we call stage one bootloader really
small amount of code but it's enough to actually have a real network stack. Really start doing regular provisioning operations. The Pixie is just smart enough to download that one program and start it once you've started that program, you can get a better bootloader stage two or stage two bootloader it will bring in an eye Pixie EFI file will actually have some real operational pieces for it can switch to HTTP or better HTTPS to then start bringing down the process. We do that because it's much faster, much more controlled. It's more secure. The faster you get out of that first stage bootloader, the better. And so we we
split this up into two stage two stage boot loaders, we were really early doing this, we've been doing it for a long time. But it's getting to be pretty common practice to do this to stage bootloader. Or if you have a modern system and you're using the latest BIOS, just switch into a I Pixie bootloader. And you can skip that first bootloader Pixie boot load and jump right to this step doesn't save you that much time, but it's more secure. And then finally, once you've gotten to that bootloader, then you can go in and get a real kernel right transition to doing that kernel in it. And that kicks actual kickstart program process. And we'll talk more about what that looks like as we go. But the thing to keep in mind here is that this is actually for operating systems that you've started, every one of these stages is really a new operating system, you might not think of it like an operating system, but that's what it is. And so it's going through this process, it actually generates DHCP requests each time this
happens, new leases, potentially you have to have a system that understands that each one of these boots is not a completely new operating system, because they're going to come in without without having the lease without knowing what's happened before it and walk through this. But the bootstrap process is and it's helpful to think of it this way, is actually staging through multiple operating systems getting to more and more complex operating system until you finally have that Oh, your Linux or Windows or VMware kernel that you're that you're actually going to start being able to build on top of and like I said, you can skip that that one of those kernels but you're only getting so far in this. And the funny thing is, is that at that point, you're really not doing Pixie anymore, you're doing I Pixie, but everybody just calls it Pixie so when you say Pixie people think provision and they're they're pretty much equivalent. Okay, so now that we've gone through that, I do want to show you what that whole kickstart process looks like. I'm about to run a CentOS install just a regular one out of our out of our system and start that boot process. So we're going to take that same machine and let it run through the process. And what you'll see here is it's going through this exact same Kickstarter preseed ours, first
stage second stage bootloaders. But in this case, it's then going to start the Santos bootloader. So you'll see this acento since the sledgehammer, it's doing its init rd, and then running our initialize system, it's very similar. sledgehammer is actually based on centers. So you'll see a lot of same scroll, but then it will switch into a system process that is going to be much slower. This is where it actually starts in a condo, which is the kickstart process to install the Hello OS. And it's gonna take a while to go do all these things. And this is with external libraries turned off. So it's not even trying to drag things over the internet installer. So this is Anaconda. And walking now through the whole install process. And
you can see there's quite a bit of bits and pieces to this. So the magic of time lapse has meant that you didn't have to sit through the five minutes that I just did waiting for this to complete. And not only did it have to do the full install, then it did a reboot on top of that. So now we're watching it, reboot from the disk to install the operating system. So this is not as a paint drying process. But you have to do it. This is how operating systems get installed and what they actually look like when they go so you just got to watch CentOS being installed in just five minutes, which all things considered is pretty darn fast. That's it now you understand everything you need to know about Pixie booting, right? Sadly, no. provisioning is much more than Pixie as I've been
saying. All we're doing when we get to a pixie or an AI Pixie system is we're just getting that kernel loaded. Once that kernel is loaded, we still have a long, long road in front of us. So from there, we actually have to do a kickstart right once again, preseed weasyl, whatever else you're going to do, we call it pre seed, or succeed as a hybrid inside of digital rebar. You'll see that in some of our templates. And from there, what you'll what you'll actually get is configuring templates. So the challenge with kickstart, is that even though it's smarter than Pixie, it actually needs to know what the system is that it's installing to it needs to know the NIC Nic drivers Nic configuration to start to turn on advanced functions, it needs to know the storage configurations that are there. It needs to know what output console to us, right? Is it is it
using serial out on 123? Is it you do care about USB drives all that stuff that's configured in your system, you need to know about right if you're trying to use some add on Nic cards instead of the baseboard ones, you have to program those. That's what the kickstart system is going to do. It goes through all that process, just to get a real operating system with real IO started. Once again, digital rebar has a whole bunch of templates, we call them boot M's, where we influence this whole process and help make it go. Everything I'm showing you is really what's in the budem process. And it's complex. It's the most complex thing about digital rebar. And yet, we've done it for you. So for the most part, you'll be able to take booms that already exist, apply them into the system and never learn how they work behind the scenes. Isn't that nice? Some people,
you will find yourself digging into Buddha and some will help you. But for the most part, our shelf works off the shelf stuff works has been battle hardened, tried and tested for years. And that's a good thing. So after that, then you actually start doing operating system installation in earnest, you're not out of the woods, you actually have to download a whole bunch of packages and install the things and pieces that you need. And still after that, then you have to start doing things like getting your credentials and apps and access, you have to set up passwords and name the systems and get on the right networks that you want to get on. All that's in post config. So a lot of steps have to be have to happen here. And that's really important. So when we look at this process, that's these configuration templates that we have going on. One of the things to note in this is that when we're talking about the system as a whole ISOs are minimal. And they're actually pretty stale. So
when we take an ISO, which is an operating system image setup, people don't put everything that's that they need in there. A lot of the vendors especially canonical really makes that a very minimal set. And they assume that you're going to start downloading packages with an internet connection off of their repos and mirrors. And it's going to start slurping those things down, that's going to be slow, it's going to be painful.
If you have a real firewall or proxy server, things like that, this might not work at all, you might have to have local mirrors, you probably security policy requires you to have local mirrors, we have ways to encounter all those things. But that's the next step here. You're gonna have to be done either way. Even if you just start using sledgehammer you're gonna have you're gonna need additional packages, sledgehammers are, we'll talk about it, it's our it's our way of doing this process in a very minimal way that you can get running pretty much on any system ever.
We've been really careful about that. And then once you're done, then you start doing post configuration operations and making all the rest of the process run through. And what digital rebar is really about, ultimately is connecting all these steps together. So when we go to automate it, what we're building is a workflow that makes all of these pieces fit together very cleanly. And that's really important for provisioning. Just installing an OS or you know, just putting an image out there is not provisioning, in our opinion, it's it's really just installing an OS provisioning means getting a system ready to roll. Right when we think of if you go all the way back to the original meaning of provisioning, which would be packing your wagon
for a trip and getting everything you need to be successful. You have to have everything in the wagon. And that's that's what provisioning is to us. That's that's how we approach solving these problems. And for us, it's actually packing all caravan because you have to have all the wagons working together. Sometimes you got to figure out what goes in each wagon. And that's part of digital reverse magic. And we do that with
infrastructures code. So everything we're building, it's all in our code repositories. It's all locked, you don't twiddle the bits. You might influence things with a profile. that's by design. But for the most part, it's infrastructures code, it's version it's managed. And that's important than the other thing that you have to do to do this well and we do a lot of is out of band management. So this is a great process. NET boot go all the way through. But if you're managing the system on an ongoing basis, you also need to manage it out of band, which means power cycle, remote console BMC there's tons of different protocols redfish is becoming more popular, but it's not universal. It's not actually common
across all vendors. There's advanced features that pop in there, and you have to figure all this stuff out. So part of what digital rebar is able to do is not just do this kickstart pre seed type template net boot. We also then do processes that include out of band management so that you can cycle machines. In Section two, we'll talk about some advanced things that use media attach out of bmcs, to boot machines, a lot of ways to use out of band management. And you need to if you're in a data center or an edge out of band management is your way to assert final control the systems,
especially if you've lost or you're a service provider, and you've given up control of that system out of band management is your backdoor to get back into and control the system, shut it down, tell it to reboot and reflash everything and you're good to go again. So very important components in the overall provisioning workflows and pieces. And so I've been talking about this, let me actually show you what some provisioning templates look like. So before we go further, I want to show you what these provisioning templates look like. This is a view in digital rebars Ux that is new. So if you're an old user, this is great new functionality that we brought in in the four six release timeframe. And what you'll see here is these are all the content packs, I'm just going to go into the primary content pack community content, and we're going to look at some of the Boo dems, which is where this provisioning template pieces work. And if I look at one of our standard ones, let me start with sledgehammer.
Looking at the raw object, what you'll see is all sorts of parameters to get injected into the system at boot time, you'll see templates that we have that actually go through and you know, do different actions of bringing up the system, set the path for TFTP boot. And then actually we'll start this is Raspberry Pi configuration file, that unique thing. inject different addresses. So what we've done here is we've created a standard set of best practices around boot provision. If you look at our one for CentOS eight, you'll see something very similar. Here's a whole bunch of
standard templates that get pulled in, we can I can show you those, we'll find them in just a minute. optional parameters to get injected different loaders that are necessary. If I come down into the templates, I can start looking at here's the Santos a kickstart. And once again, this is a standard kickstart. But the benefit here is that we've actually standardized all these pieces parameterised them have taken a lot of the headaches out. So if you are building and setting these variables, they should work across different operating systems where appropriate where they don't, you're going to be able to do your own. This is why we did something called kick seed, so that we can standardize the Debian, Ubuntu and CentOS REL families. And this is true universally, but this is deep, detailed stuff. Go to the digital rebar content repos, check out some of the
things we're doing. But there's a lot going on here. And that is the point of this talk is for you to sort of see just how much actually goes into every single step of building a provisioning operation. So I hope that was helpful to show you some of the pieces and parts that actually go into building digital rebar
templates. Before we end this section, I do want to give you some idea of questions that we normally get for digital rebar. And then this will serve as a good starting point. For the next section. I hope this has really answered some of your key questions. Three, two. So I hope that seeing some digital rebar templates was helpful for you. A lot of thought goes into how these things are built, just digital rebar. Go Lang templates, you know, take some getting used to we're doing variable substitution, we're really pulling data in from an infrastructures code perspective. But it's a really good investment. Because when you're building systems using the data you have, they're
much more resilient, you can go a lot faster, and you can actually make systems that account for how reality of your data center is that that's really what digital rebar is about. As we conclude section one, this talk, I do want to go through what typical questions are. We do see the same questions over and over again. And they're smart questions, right? Why is this so fragile? I really spend a lot of time trying to explain how provisioning gets To be so fragile, right? bootstrapping is complex, it's hard to rely on firmware that might have been written before you started your it career. And so these are these are systems that have worked this way for a long time. And they have to be Charan fed. And that's just
the way it is in some cases. But, you know, understanding it helps you build resiliency into your system. We do see people who want to do Pixie over Wi Fi all the time, or think about it. Wi Fi requires credentials before you get on the network. It's not an unsecured network, or if it is, secure your network. So it doesn't really work. From a bootstrapping perspective. If you're injecting information in before you've Pixie booted, you've defeated the purpose of Pixie you could probably do it, you could probably throw these images over the wire, not really what we'd recommend from from that perspective, so most Pixie is wired network VLAN comes up all the time. You know, these are these are bioses. They're
pretty simple. They don't understand VLAN tagging, as a starting point, you can do it there are ways especially if you're using ua fi or advanced like I Pixie, you can VLAN tag. If you do want VLANs, what we see happening is either you do it at the switch and you're not exposing it at the the system itself just handling trunking at the switch. Or what you're doing is you're switching in and out. So we've had customers who will leave their switches in a VLAN, zero untagged VLAN for this boot provision. And as soon as that process is completed, they use automation triggered out of digital rebar to then put you into a VLAN. And turn off that trunking is a little bit riskier, because you might be in a situation where you do
not able to Pixie boot because that that traffic can't be forwarded. But it addresses people security. So you create a small window, and then you get off of that window as fast as possible. Which, you know, that's sometimes that's how security is there's a small insecure window and then you move out of it as fast as you possibly can. And we do a whole bunch of work to help people minimize their exposure from those perspectives. People want to know about containerizing Digital rebar or the whole process. Yes, we help customers do that all the time, we don't actually recommend it as a first thing because this is UDP DHCP TFTP require datagrams, not TCP, which isn't as friendly with Docker. And so you end up having to be very careful about how you configure and give security to those containers. So does work. But if you don't need
it, don't just add it for the glory of containerizing things. But if that's your process, definitely we'll help you can make it work. Setting bison right Oh yes, that's multiple hours of presentations about this. Ultimately, it's you know, not that bad. We we run tools to set rate and BIOS and a whole bunch of profiles and configurations. definitely doable. But that is not Pixie, it's really actually not even provisioning.
It is a system lifecycle Configuration Utility, we tend to prefer to do it in band meaning from from our sledgehammer boot OS, we then do run the tools to do rain BIOS configuration. As opposed to setting things out of band we do also do out of band configuration using redfish are the vendor specific track I lo whatever protocols. But those don't scale as well when you have to do hundreds of systems in parallel. And they don't give you as much feedback as we get when we do it in band. So a lot of different ways to skin that cat. love to talk about it, and help you make yourself not have to learn how to do it. Because there's not a lot of profit in learning how to set BIOS. Not at least for our customers, they want to get it done, make it reliable and move to a more important business challenge. And then finally,
how can I make this whole stuff faster? Because right now what I'm showing you is not particularly fast, especially if you're booting a machine and you're waiting for it to post BIOS. You know, can you improve your end to end performance because a lot of people spend a lot of time waiting for servers to get provisioned. And we get it. That is your tie in for session two, so that we can talk about this more in depth. We're going to cover those topics and more. So please check out section two of this talk. I hope this has been helpful. If you have questions please come ask us. Join the racket slack. Or email us send us a postcard or a fairy with a imbedded in a paper airplane. We'd love to hear from You talk about this stuff. Thanks a lot. This has been Rob Hirschfeld with rackin. See you in the next presentation.
2021-01-15