welcome to cloudnative Compass a podcast to help you navigate the vast landscape of the cloud native ecosystem we're your hosts I'm David Flanagan a technology Magpie that can't stop playing with new shiny things I'm Laura Santa Maria a forever learner who is constantly breaking production eppf is turning complete and can be written and rest so obviously I'm sold that's just two of the things we learned on today's episode want to learn more contrary to not so popular belief ebpf is not some secret Berkeley green project that's out there to eat your refrigerator but it is a gateway to becoming a Linux kernel maintainer if you're curious about what EP ebpf is why it matters how to pronounce it and how badly you can break your kernel when trying to learn it this is the episode for you as we talk with Liz rice the author of learning ebpf and chief open source officer at isovalent all right Thank You Les could you please say hello and tell us a little bit more about you hi yeah so my name is Liz rice I am chief open source officer at isovalent which is the company that originally created psyllium and uh a lot of people will have heard of psyllium being based on ebpf and earlier this year I published a book about you people called learning ebpf I feel like I may have answered more than one question in one go there hey no that's fine that's fine a little bit of context just us perfect yeah good contest considering we're about to talk about ebpf and go into a little bit more detail there so yeah awesome all right well let's just start you know not everyone is familiar with vbpf so could you give us the trdr what do people need to know to understand the rest of the conversation to do so they don't need to know what it stands for um it stands for extended Berkeley packet filter but honestly forget that it doesn't really help because it does a lot more than packet filtering now so we tend to say that acronym doesn't really mean anything anymore and what it really is is the ability to run programs within the in within the kernel within the operating system kernel so we can dynamically change the way that the kernel behaves by loading these evpf programs and I think when I say that I have to make sure that people really know what I mean when I say the colonel the kernel is the part of the operating system that interfaces between our applications and the hardware that they can you know processor and its peripherals so if you are writing to a file you're doing anything over the network writing anything to screen even allocating memory the kernel has to get involved your application can't do it directly it has to ask the kernel for help and the kernels also coordinating all the different processes that might be running on the machine and that means the kernel is involved whenever you're doing anything interesting really so it's a really great place to write things like observability tools and security tools and we can do that with ebpf and we also get to customize the way that the kernel behaves for the things that it takes on things like the networking stack we can modify the way that behaves with ebpf so it's really powerful and a really interesting way to instrument all of your different applications that are running on that one kernel nice and here I was hoping you could I was gonna say here I was hoping you could actually explain why I said Berkeley in the middle of all of it but that's a whole thing yeah I could explain anything else the original um pocket filter paper um was written by two people whose names I can't quite remember right now um but they were at Berkeley at the time so it was okay okay that it all originates from says like Lawrence Barkley lab or however he's playing with that's called yeah you know just for fun for somebody like me who's like why does it say Berkeley okay anyway [Music] so I I'm curious then did you join Isa valent because you were really interested in eppf or are you now really interested in eppf because you've joined ISO villain like what came first no I really got interested in ebpf the first time I heard of it I saw Thomas Graf who is the CTO of ice surveillance uh talking about psyllium and evpf at dockercon back in 2017. and at the time I thought well that's pretty interesting technology and at that point it was really cutting edge in the kernel you needed you know it wasn't available to most people in production it wasn't available to hardly anybody in the Linux distributions that they were using back then but I thought this is a really interesting technology and um I'd kind of kept an eye on it a couple of years later I started working on a project that was um sort of using ubpf and I was also as part of learning about it myself I was going out and doing talks I find that you know the best way to make sure you really understand it is to try try and explain something to somebody else so yeah I'd started doing talks about evpf as well and through that um actually got invited back to The ebpf Summit in I guess 2020 which I surveillance put on sort of on behalf of the ebpf community and there was so much really cool stuff going on you know in the world of ebpf and particularly at I surveillance it turns out I hadn't realized before that Summit just how much I said I don't have been involved with ebpf right from the get-go So Daniel borkman who's one of the three maintainers of ebpf in the kernel was you know one of the early engineers at I surveillance and he's still there so you know we're so embedded in the way that ebpf has developed over the years it's you know I really do get to work with the the people who created it and the people who've been using it and had the vision for building things like psyllium you know so yeah I joined ice surveillance because it's just absolutely full of really cool people doing really fun things with EPF awesome makes sense so you started experimenting with bbpf in 2017 we're now in the latter half of 2023 which just seems absurd to me now but like over those years you've seen the adoption grow as we all have especially across the industry and even the cncf with projects like pixie and Falco and of course psyllium as well um why has that adoption grown so quickly for a relatively Niche I don't know is that a niche technology I think it is like I don't know I don't know it seems like it's kind of everywhere so yeah it's one of the I I guess expertise in it is pretty Niche but a lot of people are using it without really even knowing that they're using it I mean there's probably people using psyllium who don't realize that it's based on evpf certainly a lot of people will be using things like TCP dump and never really sort of thought about evpf and that's fine that's you know there's so many really um powerful tools that have been built um I I think you know things that uh Brendan Greg have popularized um you know in the kind of 2017 period you know he was already out there talking about how Netflix were using evpf for um observability purposes for tracing for you know diagnosing and then improving performance issues and uh you know really popularizing the power of ebpf um the reason why I think there was that kind of sudden upturn in adoption is the fact that the the level of ebpf support in the kernel has reached a point I think around the 4.18 kernel version around that kind of time frame is where you really start getting sufficient to ebpf support to do really interesting things and the more modern your kernel is the more additional capabilities in evpf and probably lots of other areas of the kernel as well um and but there is this real turning point when I would say particularly when Rel was probably the last of the distributions to um kind of it's It's always you know relatively cautious about upgrading to to new versions of the kernel and at the point where really all of the distributions were using the modern enough kernel that meant you could just deploy these ebpf based Tools in production regardless of your distro and I think that really made a huge difference to the adoption yeah awesome so you mentioned the Berkeley packet filter and for the people that are not aware it's like a networking thing that allows you to do iptables like stuff in the car I'm not trying to go into it any more details next I'll I'll make an absolutely best of it but it does networking stuff right it blocks packets it rewrits packets it does a lot of stuff but ebpf has kind of grown beyond that now we're seeing it used for a whole variety of different Technologies like Falco and pexy um is there like what's the right way to freeze this question really I should have had it prepared but why has it extended Beyond this why does it have these new capabilities what is it enabling within the kernel for people why is it interesting to you and to others I know that's a very broad question yeah so I think the original idea of packet filtering was to be able to look at each incoming packet and say you know make decisions about what to do with it with that packet um and I think in the very first place it was really just do am I interested in sort of seeing this packet I maybe I want to filter you know packets that are going to a particular Port so that I can count them or something like that um so it was making fairly simple decisions about you know what kind of what to do with these with these packet filters the extended part involved um I think a few different trains of thought one was the idea that if you extended this sort of relatively small instruction set that could be used to to examine packets if you turn that into something a bit more kind of like a virtual machine instruction set um you know when if you look at BPF byte code it's very reminiscent of like machine code um you know it's all about registers and loading values into registers and and comparing them and jumping to other instructions it's very very similar to machine code so there was this idea that having a a virtual machine in the kernel could allow you to do all sorts of interesting things there is the idea that maybe you could attach these programs to other points in the kernel not just to incoming packets but you could make decisions or change the behavior at other points in the kernel um and I think the last major thing that distinguished extended from from its predecessors is what's called ebpf Maps and maps are these data structures that you can access from within an ebpf program and you can share them between ebpf programs and you can also access them using system calls from user space so it's it's a way of exchanging information between user space and ebpf programs or between multiple different ebpf programs and all those things kind of combined has turned out to be really powerful to the extent that one of my colleagues recently did a talk at um one of the kind of Linux kernel developer conferences where he showed that evpf is now cheering complete which is pretty cool question just on that right because you know the talks I've seen you know from yourself and others in this space when you talk about ebpf one of the the things that's always mentioned is the fact that the ebpf sandbox the virtual machine can and shouldn't never crash is that is that still true even with the ability for the ebpf programs to communicate with each other and with user space programs yeah exactly so the reason we're able to make that claim is because of a thing in the kernel called the ebpf verifier so as you load a program into the kernel it goes through this verification process which is really analyzing all the possible paths through the program and uh ensuring that well first of all it will run to completion so this a long time ago that used to mean no Loops at all now that's been kind of improved and optimized and you can have loops um it's checking for things like there's only a limited set of what's called helper functions that you can make from an ebpf program and the set that you can call depend on really the event that triggered it so if you were being triggered because a network packet arrived then you can call helper functions that are related to looking at that Network packet but you can't for example ask for what's the use of space process associated with this packet because there is no user space process Associated at that time whereas if you were in an ebpf program attached to a user Space Program making a system call then you absolutely could ask a helper function to give you the the process ID so the verify is checking that all this sort of contextual um helper functions are being used appropriately and that memory access is safe that if you're going to dereference a pointer you have to explicitly check that it's not nil before you do so because dereferencing a nil pointer if you know anybody has ever written a c program they will have crashed their C program by the referencing ml pointer I guarantee it um yeah so the verify is really just checking that that program is safe to run safe in the sense that it can't crash the machine that memory access is safe um of course it can't tell the difference between uh you know maybe I'm a legitimate networking packet that's uh a legitimate ebpf program that's filtering Network packets uh you know maybe I'm protecting against DDOS attacks or maybe I'm a malicious evpf program and I I'm just throwing away packets for fun yeah the verify I can't tell the difference between those two things but when we talk about being safe to run in this context we really mean it's not going to crash or hang the machine okay cool anything worth it yeah go ahead go ahead Liz no keep going keep going well it's the the thing that really distinguishes ebpf from writing a kernel module so you always could extend the kernel always some time you've been able to extend the kernel by writing kernel modules but people are pretty reluctant to install kernel modules because it's if there is some kind of bug in it if it does crash it's going to bring down the whole machine and there's no um kind of safety net like what the ebpf verifier is bringing makes sense so out of curiosity like obviously we're talking a lot more in-depth kernel kind of things and obviously you referenced if you've ever developed a c program you know what this is like in a machine code and things like that who would you say are probably the most common user that you encounter or like the the people who are really using this the most who do you think this is the most relevant for I guess right now so lots of people will use ebpf through tooling that's built on it so I mean David mentioned you know there's various different projects in the cncf there's all the um BPF trace and and the BCC family of tools that people can use on the command line to do observability there's there's lots of different tools that have been built on ebpf as a platform and I think for the vast majority of people that's how they'll really experience it they'll use things like psyllium or um you know or Falco or pixie and they may be interested in the fact that it's unit using ubpf but they don't actually have to get involved in the details which turns out to be a really good thing because I mean I'm I'm the sort of person who really wants to understand you know like how the sausage is made I I I want to kind of get inside and get a feel for like how is this really working you can learn about ebtf programming uh you know it's relatively easy to get you know things like a hello world or you know some basic networking capabilities running in ebpf programs that you've written yourself but you do quite quickly start hitting the point where you're interacting with kernel data structures and at that point you kind of need to understand what those data structures represent and what the effect of you changing them might have and so it it does quite quickly turn into kernel programming so I kind of say that on the one hand most people just don't need to know anything about the details of it at all but if they're interested it's relatively accessible for people who are comfortable with programming to to kind of dip your toe in and then if you really want to become an expert ebpf programmer which I by no means consider myself at all that really does start to require kind of Kernel expertise but fortunately I work with lots of people I always who have that kind of expertise so would you say this is the the gateway to actually becoming a Linux maintainer honestly yes yes [Laughter] I've not done it but it never crossed my mind that I would ever make a contribution to the Linux card right now I've kind of in that world I sort of start to think you know if I had another 25 hours in the day the Temptation is high to go get involved I understand I've asked my one obligatory question David now it's your turn no I I like that I like that discussion right because it's it's one of those questions like um I do a couple of talks on that touch only BPF right I don't go deep on it because I'm not that smart but I always do the same demos and it's the i o visor uh bpfcc tools demo um specifically I show off exact snip and open Snoop because you know an SRE platform devopsy world is quite interesting and important from a security perspective and an automation perspective to be able to show when certain sensitive files are opened on a disk and ebpf makes that really really simple and another really cool demo is just by using exact Snoop you can actually monitor for sudo and set uidbit one binaries on the machine and so when people Elevate their privileges you can get notifications for that kind of stuff too and the question is always like how much do I need to know about ebpf to then start doing tools similar to that and then you show them the source codes and there's like 20 lanes of python it's not a lot to do these kind of things and I think that's because I feel like people can start to build ebpf programs and traces without going deep into it in the same way that with containers we can all run containers on our machine but you don't really need to understand what a control group is or namespaces anymore and I feel like ebpf may make that same transition probably already has made that same transition so I'm going to flip that around a little bit and throw the question to Liz and people do see these demos they listen to this and they're like okay ebpf is really cool what are the languages and the sdks that they can go and start to work with right away to experiment with the new tech yeah so you kind of have to answer that question in two parts there's the actual ebpf program itself that's going to run in the kernel and then there's the user space part that might interact with it there are some occasions where you don't even need to use a space part so for example if you're doing networking functions sometimes they don't need any kind of user space interaction because you can just load them into the kernel and they can they can do what they need to do but usually we're going to have both these parts for the kernel the program's going to be in ultimately it's going to be in ebpf bike or byte code form talked about there being these bytecode instructions that look like machine code you could just write the machine code by hand apparently there are people who do that um but uh you know but for for me I I would rather write in a in a slightly higher language than that and the languages are restricted by being they have to be able to compile down to bytecode so uh the compilers that support it right now are clang GCC with both of which can compile C and also the rust compiler um I'm not aware of there being other programs that support BPF by code as a Target so uh yeah C or rust really become your choice there um there is a little bit of a caveat in that there's this project called iovizer BCC David mentioned the the tools and and things like open Snoop and exact Snoop that come from that project and BCC gives you some friendly kind of macros such that you can write your code in a sort of hybrid of python and c and it takes care of a few takes care of a few things for you from your from your C programs but then there's the user space side of things and there you have a much wider choice I mean really you're you're not restricted at all except that you probably want some sdks that will make system calls for you and make it easier for you to interact with the ebps program through that syscall interface and there are there's a go SDK there's a rust SDK that in fact there's a couple of go ones um and there's a c one which is probably I would say today the most widely used called lib BPF um psyllium uses uh go we have a go ebpf library but I think a lot of the projects outside outside of that probably directly using lib BPF yeah you said one thing there that I completely disagree with and is you said you have a choice between rust and C that's not a choice I knew this was coming but I knew this was coming I knew it of course talk about rest I'm gonna just break in before he gets going on it um you know it it occurs to me you there was the mention of like you know when containers came around and things like that things kind of change and as there are more and more languages that people are familiar with that you can compile down to the ebpf byte code how often do you find people getting into trouble like they used to do when containers first came around because they didn't quite know what they were doing but they kind of got it enough to get away with it so how often do you find people getting in trouble and like how do you get them out like that's always my questions like how do you troubleshoot this thing especially when this is like kernel level and you can really really mess things up fast yeah I I would say that there's probably two major ways that people get caught out um one of them is around the kind of the tool chain and installing things that are compatible with each other because you know every kernel version has different evpf support and then you need you know maybe your user space libraries that maybe are or aren't compatible and different distributions of Linux might have different um you know versions of different uh either lib BPF or or tools things like the the BCC tools or particularly there's a thing called BPF tool that you can use for managing uh BPF programs and making sure that you've got a compatible set of the packages the source code the kernel the tools that you want can trip you up in numerous different ways um and the other thing that catches people out once they got everything installed and everything seems to be compatible and they've started compiling some code and then they go to load it into the kernel and they hit verification errors and um I I would say over over the last few years there's been a lot of improvement in the kind of information that the verify gives you about why it's subjecting to whatever it is it's objecting to but I have certainly that there's a blog post somewhere that describes the verifier as a Fickle fickle Beast I think it is nice yes all right so you know we've covered a lot about evpf so far um and just kind of to understand the landscape right now like it's heavily used for networking selium has gone all in on ebpf even towards the service measure angler now uh as a villain even now have tetragon going for the security angle and trying to help people with that and you know we're seeing like pixie and Falco to do more security and monitoring automated observability all of this stuff is there as I don't know if like this new BPF maturity curve right but as people start to do more of it and the skills become more aware or people are using it more and well ebpf creep into like our day-to-day application code do you see people using ebpf to write you know their cmses or their you know proprietary applications like I I don't know what those use cases are I don't know if they exist but well ebpf become more than what it is today which is a bit of a forward thinking question but maybe they do and also what would be used for but I think one interesting parallel is the way that um networking capabilities that used to be in user space have migrated into the kernel um TCP stack you know I am old enough to remember when that was more commonly in user space you know that you'd use a TCP Library um and now we we just expect the colonel to take care of that and I think what ebpf will allow us to do is to gradually move more of that kind of functionality into the kernel but in a way that doesn't require everybody to take the leap at the same time because we don't all have to be running the same EPF programs um so I think something like service mesh is a really great example where um psyllium as a as a cni a networking component for for kubernetes is in this really great sort of position in the kernel to be able to you know pick up Network packets and put them where they need to be and observe them and report on them and do you kind of security related operations on them all of which are very much the kind of things that we expect from a service mesh today we can't do everything in the kernel I mean theoretically I think we could but in practice I think all the kind of layer 7 operations we're using a user space proxy to do that we're using Envoy to do that um I think over time you know I expect that in you know some number of years time we all of that code will be in the kernel but maybe that you know kubernetes will be in the kernel too who knows maybe that's the kind of future fancy rewriting kubernetes perhaps we should all do it in Rust in evpf David's looking like he might actually do that yeah no he might he might I mean I am not personally written any kubernetes components than Russ but people are exploring that these days um so you never you never know right I don't trust that statement that you're not doing it yet no I I'm not I don't have time I'm too busy talking to you all right well uh is there anything else let's you would like us to throw at you to ask before we we wrap this up the NFL this is sitting on the tip of your tongue waiting to be said sorry yeah no there is one thing I would like to mention which is the upcoming ebpf Summit because if okay we are interested in you know hearing more about what's going on in in the kind of evpf community learning more about how it's being used seeing some of the kind of interesting directions that people are going with it and learning more about the future of ebpf itself ebpf Summit it's online it's free it's a virtual conference this is going to be I think the fourth time that we've held it um it's on September the 13th and uh yeah come come join in if you'll go to ebpf.io you'll find a link to the summit there and uh yeah it's it's always been a really nice kind of community feel event so I'm really looking forward to it all right if you want to shamelessly plug anything else yourself as a villain anything else feel free to mention that no or forever hold your peace we'll make sure all the links end up in the show notes as well I guess it would be remiss of me not to mention learning ebpf well it's available either for you know if you want the pdf version you can download that from I surveillance.com or you can you know order it from your favorite local Bookseller or get it from Amazon if that's your bag shop local exactly all right well thank you so much for your time pleasure thank you Liz thanks for joining us if you want to keep up with us consider subscribing to the podcast on your favorite podcasting app or even go to cloudnativecompass.fm and if you want us to talk with someone specific or cover a specific topic reach out to us on any social media platform until next time when exploring the cloud native landscape on three on three one two three don't forget your comfort forget your compass
2023-09-02