AWS re:Inforce 2023 - Security in the Open: OSS and AWS (SEC201-L)

the past 12 to 18 months like no point in my previous 20 years involvement in open source has really driven home just how pervasive and ubiquitous open source has become open source is truly everywhere and I know that it's in my watches I know that it's in the cars that I drive it's the washing machine in my house and of course all over the software on the computers that we operate I've known that frankly for 20 years I knew that open source was everywhere that it was important and of course open source is important to AWS we build tools top it we built Services atop it our customers love it and so we've got a vested interest in the broader security of Open Source in in the landscape that we operate in but as I said the past 12 to 18 months have been really a huge change in people recognizing just how ubiquitous it is and so synopsis who provide the black ducks software composition analysis tool they're telling us that 96 of code bases that they see contain open source 96 percent I've always known that open source has been ubiquitous but 96 percent so I want to spend some time talking about open source how AWS thinks about it I want to say that it's not just important but talk a little bit about why it's important and how we engage in open source communities I think the why is important because it probably applies to most companies and I want to explain our commitment and our strategy AWS has this long history of benefiting from open source AWS would not look like it does today without open source if you'll remember ec2 it was built Atop The Zen hypervisor S3 was originally built atop Apache Tomcat they were the foundational layers that allowed us to innovate quickly and deliver value to our customers and we think that open source is much better in terms of being able to provide those foundational layers that allow us to innovate and deliver value no one wants to reinvent the wheel and write another web server we have a number of very good open source web servers that allow us to not have to spend the time to re-implement that of course that means that we need to have a very healthy a very sustainable open source Foundation to build a top and healthy means a couple of different things right we obviously we want secure code we want well-tested code that operates well we also want the communities that are building that code to be sustainable and to be healthy we don't want the one person in Nebraska holding up the entire internet and so when we look at when we're looking at how we pay attention to open source you know we recognize that it's important to our business it's important to our builders and we build obviously plenty of things on top of it but it's not just important to AWS it's important to our customers many of our customers tell us that they have open source first mandates in their business meaning that when they're choosing a new technology platform that they are defaulting to open source if one is available and so when with customers repeatedly choosing open source asking us to help with their open source workloads and whether that's a database server whether it's AIML platforms our container orchestration engines we have to make sure that those workloads run really well on top of AWS and for our customers in general open source also is important to the world it has become a common Foundation that we're all dependent upon which means that we have somewhat of a shared destiny because we're all depending upon open source because open source is ubiquitous we have a lot of conversations Mark and I get to to speak to customers frequently and customers are continuously asking us tell us how you think about open source how are you securing your supply chain we're having lots of software supply chain security conversations today and that really comes down to two questions the first is how are you consuming open source safely what what constraints are you putting in to make sure that open source is being consumed in a responsible Manner and then also how are you taking care of these software development process to ensure that all of these things that you're ingesting are secure and people are lumping both of those things into our software supply chain security conversations and so we want to talk a little bit about that as well and show you some of the things that AWS is doing may not be a perfect fit for other folks but at least it'll be very transparent in how we handle open source and how we think about that process Mark you wanna talk about some of those four patients yeah good morning everyone it's great to be here um this presentation is very broadly about how AWS works with the open source Community to make it stronger healthier and more secure but we thought it would make sense to at least briefly talk about how we consume open source and that conversation although a brief part of this overall talk we could do a whole hour just on this topic we'll at least give you some thoughts about both how we operate but thoughts about how you as well could operate there's not one perfect way to do this but we think we have some learnings we'd like to share so just briefly in terms of secure consumption we very much take a Federated approach to this topic as we do with many other topics we don't highly centralize really anything at AWS but we still have Central teams that support certain core functions to make sure that the teams that have sort of direct immediate responsibility are doing their job well and for US Open Source consumptions begins in the engineering teams that need that decide to use open source so the the consumers of Open Source take that core responsibility for the software artifacts they're using whether they develop them themselves or whether they import something from the outside world it's their responsibility to do that in a safe sane and secure fashion with support of a bunch of other teams but still the core responsibility exists in the in the in that initial consumer and in order to make sure that we have ownership of these artifacts once they're brought in we we place that responsibility on sort of that initial consuming team they can hand it off later perhaps if that makes sense but some team has to really own the health and maintenance of that of that core library or what have you and that's the Builder team they need a lot of support to do that well and that's where our Builder experience team which is a central team comes in this is the team that runs the core code repositories build servers build pipelines as well as all the deployment capabilities and a lot of centralized testing that runs on every single code check-in everything from unit tests and so forth but security tests and code scans static code analysis software composition analysis all these tools are run in a central service fashion by by the Builder experience team so that means that any other organization that wants to use this opens say an open source Library the first thing they're going to do is they're going to look to see hey is anybody else using this just like they would want to reuse code that's internal only they're going to want to reuse open source code as well because that means someone's taken initial responsibility for vetting maintaining and updating that system and so that's this is where that other team would go would go to the central code repository and tools find that that capability or that library or that source source code and they're progress from there again with that sort of Federated ownership and control and then finally we have a security team a Central Security engineering team especially on the proactive what we call Proactive security or appsec which does run a lot of the central tooling to make sure that we're again using and consuming and deploying systems and code in a very safe fashion across the board including the open source components and so the Central Security engineering team not only runs a lot of the security checks on that Central build in that Center Builder environment and maintains both proprietary and and third-party and open source security scanning tools and so forth they run for example a this is a little off the topic but a canary service so when you write your code you you also write security and variant types of checks which then the security team has a service which runs those constantly to make sure that certain security and variants of your service in production are constantly doing the right thing returning the right result the expected result from any any call typically around authorization but lots of other security and variants that you want to maintain you build essentially tests that run constantly against the production environment um holding kind of helping hold this all together is our open source program office they provide a kind of the general interface to sort of the correct and proper and healthy interaction with the outside world along with David's team as well um they're the ones that are doing things like making sure that we have good relationships with the right outside Builders um sponsoring and funding things that make sense we'll talk more about security related funding later in the talk and in general creating that that healthy interface between AWS and the broader open source Community to make sure those lines of communication those relationships are there to support the overall health of this of this environment and and again using open source in a secure fashion and we'll talk in a minute about UPS working Upstream so another aspect of all this is if we do see chain need to make changes for improvements or security fixes what have you a key part of what we're going to be doing is making sure that those get pushed Upstream as well so after a little detour if you will although a very important one on consumption we'll go back to sort of how we think about uh how we interact with the broader world and open source and we'll we'll focus on these three pillars work Upstream that means when you're working with something where you're not the kind of primary creator make sure that you are interacting with that original Source in a way that is sponsible and and contributing to the common good there release tools and effectively this is be the Upstream in many cases when you have things that the rest of the community can benefit from but put them out there so others can benefit and then we'll talk also about financial support as well we'll start with a workup stream so we can if we think if we take this kind of long term and Community ordered approach to working in the open source world we can benefit not only ourselves but the broader community and we've been doing this for a long time and increasingly more and more over the years um I'll dive into a few of these we'll dive into a few of these examples in more detail but just to cover a few that in a more kind of quick and anecdotal way I want to call that Zen project this was you know as David mentioned ec2 one of our first certain most foundational Services originally launched on the Zen hypervisor this is back in the days when Intel processors didn't have support for hardware virtualization and people may remember some funky things that were done in those days as in was really uh pioneering with this notion of para virtualization remember you raise your hand if you remember a pair of virtualization it's kind of kind of dead and it's it needs to be dead because it has some issues but it did an amazing job and we worked very closely in that in that world of Zen to make sure that we were deeply involved with improvements patches and so forth um fairly now we've been gradually deprecating our use of Zen as probably many of you know and moving to KVM based the Nitro hypervisor we're still active in the Zinn Community one of the last kind of major contributions we made is in was in the Specter meltdown era when we realized that a lot of shared State at the processor level could relate to certain kinds of side Channel issues and pair of virtualization in particular was very problematic in that regard it was very problematic that the kind of shared State between the kernels and drivers and the para virtualized instances um that you could have problems of of the type that are manifested inspector meltdown so we actually created an internal technology to create a kind of shim layer so that the operating system that's being hypervised thinks it's being para virtualized but actually there's an hvm hardware virtualization layer sort of under the cover so you're essentially cop creating a sort of fake copy of some Zen internals on the the M side and then actual hvm capability between the the guest and the hypervisor this is a technology we call Vixen and we did push this upstream and so now any consumer of it was modified slightly but it did become kind of the standard now so that Zen users can run old-fashioned never updated operating systems that expect para virtualization but actually are using Hardware Primitives to to do that there's many other great examples here before we get into some of these others the one other one I want to call out very recently in the KVM world those of you who are familiar with operating system and hypervisor security you might be familiar with something that Microsoft calls virtualization based security so for the Windows operating system in the hyper-v hypervisor they've developed some technology that they can sort of take parts of Windows and run it in a separate kind of like a separate VM it's not exactly a separate VM but it's a hypervisor protected memory space and that is designed to protect against certain kinds of cross-process attacks essentially that might cause trouble with the things like the local security Authority and windows LSA which maintains all of your passwords and clear text in memory for example um now the problem is that that's a proprietary Microsoft Technology they've released some information about how they did that but it's definitely not open source we developed a VBS capability for KVM which we just recently started pushing Upstream so anyone who uses the Linux is in the Linux ecosystem using the KVM hypervisor will be able to run these sort of advanced and proprietary Windows features and do that on an open source basis so we're very excited about that let's move on to some specifics here we'll start with openjdk so we've had a long history with Java we're a big Java shop continue to be and back now five years ago 2017 time frame began to see a future in which we really felt the need to take a lot more ownership for for Java and our use of java and then the 2017-2018 time frame realized we're not the only large Enterprise that is a little bit concerned about the future of open jdk there was a change of ownership as you may recall in the industry I won't name any names um above the Java environment and um there's a general concern that the open version would might need some additional love and care and and really become more of a full community effort so we joined in that effort we created our own Java distribution we call coretto and since then we've really invested heavily in the open jdk world and we release our coretto capabilities and runtime for the world for free you're welcome to use it it's fully supported by our support organization and by our engineering processes I won't go through all the different things on the timeline for the sake of time you can download the slides but basically it's just the point is we're making is that we've decided to really help the community own jointly on the destiny and the quality and the features of openjdk going forward and making some very very deep investments in that and I'll hand off to David for some more yeah topics I just want to call out coretto has done amazing things for our customers and for the users specifically we just saw a New Relic tell us that that coretto is the most widely consumed open jdk distribution which is mind-boggling but it tells me that customers are actually finding value yeah and what we're what we're providing so I'm super excited about that but I'm a little more excited about rust and how many folks know what rust is okay so rust is a new programming language and I say new it's been around David does the world need another programming language the answer is and more security tools yes it actually does and he'll explain why so rust is a new programming language and I use the term new uh relatively it's certainly newer than C and Java and it has come about offering the performance of C but with a number of additional assurances so the biggest value from a security perspective that we see with rust is thread safety and memory safety and we're going to talk a little more about memory safety in a little while but when we were looking at operating things at scale Mark said we're a large Java shop and we started paying attention to rust in 2017. in 2018 we shipped firecracker which was written primarily in Rust and that is an open source operating system that uh that's really permitting serverless technology so if you're running AWS Lambda firecracker is there we've released that as an open source project for people to to consume uh and that was the first visible notice that we gave the world that we were paying attention to rust we had been using it for some internal things already prior to that more recently we launched a new Linux distribution called bottle rocket and bottle rocket is heavily rust language focused and is designed to provide a really thin small security surface for running containers and so we've also made that open source folks can download and make use of that already in 2019 we announced that we were sponsoring the rust project in 2020 we started hiring a number of rust maintainers because we recognized that rust was going to be important to how AWS was going to develop in the future so we started building up a team of folks who were already invested in the rust programming language who had already made substantial contributions and had leadership positions there shortly after that we helped found with a number of other stakeholders in the community the rust Foundation and today ec2 looks at rust as its language of choice for security sensitive applications such as nitro enclaves and so we think that that rust is incredibly important to what's going to be the future of a number of different workloads inside AWS and we're working we're heavily working we've got two teams who are focused on Upstream contributions into the rust programming language because we think that there is so much benefit not just for AWS and how we build our tools and our services but for the rest of the world because we get into things like memory safety but Mark and I we just talked about programming languages and I won't talk about a third because maybe the world does finally have enough programming languages I want to talk a little further up the stack with kubernetes and kubernetes is really a workload orchestration that primarily the primary workload is containers and it has become incredibly successful and it's become successful by a number of different measures the number of people contributing there are literally thousands of people who work on every single release of kubernetes the number of workloads that now assume that kubernetes is going to be the underlying layer that they're operating on and I can't tell you the number of of new tools that I look at that presuppose that they're going to be operating in a kubernetes environment we obviously recognize this when our customers were telling us that kubernetes was important we have the eks service and obviously provide managed kubernetes but we recognize the need really early on that we needed to get involved in kubernetes and the rest of that cloud native Computing Foundation Arena and so we've been getting involved in places like container D etcd nerd cuddle all of the surrounding technologies that help make kubernetes so useful and so helpful because kubernetes is not just one thing it's essentially using a driver model allowing people to plug in different things like different network controllers different different scheduling technology and that is one of the reasons that it's been so successful is that ability to plug in things but that means lots of help is needed on the outside so NCD container D Etc are are contributing Technologies to that success and we have heavily invested there taken leadership positions in technical steering committees in uh in the governing board and a number of other places and also release some of our own tooling where we recognize gaps that our customers are pointing out to us so Carpenter was recently uh released recently a year ago was recently shown in and provides a lot of value for our customers in trying to scale things on kubernetes but we also recognize that kubernetes does not exist on code contributions alone so we've done a couple of other things one is despite how popular kubernetes is and how fast it's innovating that's creating some upgrade fatigue with customers and so they don't want to upgrade every six months they've asked us for help and we along with others in the community are working on figuring out a longer support time frame for that because customers want to be able to actually operate something not just be in an infinite upgrade Loop we're also though realizing that there needs to be infrastructure to actually test kubernetes in advance and so we committed millions of dollars in Cloud credits every year for continuous integration for the types of testing that ensure that when you deploy a new version of kubernetes that hopefully it'll be pain-free we've been doing that as well as helping with their release infrastructure now and and getting involved there to make sure that you know again the the project does not sustain itself on code alone but that's not the only thing we should be doing right we talked a little bit about uh adding features Mark talked about uh adding features to KVM improving Security on Zen and and that's important and I think that AWS has an obligation or responsibility to release the security improvements that it has and Mark do you want to start joking about that yeah so in many ways again as a shortcut you could say you know work Upstream is one thing be Upstream is another thing of course there's an overlap and a dynamic quality of that but in this case what we're saying is we have developed technology from scratch that we realize is very useful and for the I say from scratch I'm sure there's open source libraries inside of those products but basically Pro you know significant major projects that we then want to release to allow others to benefit from even when they're not using our Cloud platform or any of our tools and so we want to talk about a few of those uh in this session as well we'll dive into I think three of those but I'll call out a few that don't have their own separate slides in this overview slide so a launch from August of last year open cyber security schema framework ocsf this is actually a a set of supporting open source tools but the core release is actually of a standard which allows security tools to interoperate in a much more seamless way we when we would talk to customers and working with a lot of Partners in the security space what we heard over and over again was my security teams spend too much time on data munching and data cleansing they're they're doing this kind of repetitive stuff they're writing tools or they're doing manual things they're cutting and pasting between multiple open windows on a screen just crazy stuff to try to get security tools to actually work together and to do that basic kind of function you need to do often which is I needed to sort of a join across two data sets and it was often very very hard if not impossible so what we realized was if we could create a standard for how tools communicate with one another it'd be a huge benefit to that so now there we did we launched this last August actually a black hat with about 20 other partners all signing up to support ocsf now I think there's up to 60 or 70 companies who are all committed to either emitting or consuming ocsf records from their security products and that will make much easier going forward for security teams to manage the data that's flowing in and out of these tools we also use the ocsf format as the native format of the security link product that we just launched I think a week ago and was in the keynote yesterday so any tool that can emit ocsf can natively feed their data into our security Lake and that was not an accident by the way that we worked on both the standard and and a service together that supports the standard so that's a really I think a really interesting and impactful one David already mentioned bottle rocket bottle rocket you could think of as a secure Linux distribution focused on container security so for example it has a read-only file system there's no reason why file systems need to be updated at the OS level if the only thing it's doing is running containers that are running virtualized file systems on top so it has many other such attributes of very locked down very secure Linux foundation for running container environments um firecracker we already mentioned firecracker is a implements a technology that are called micro VMS so again we don't trust container boundaries as a security boundary in general there's a very bad track record of containers being fully secure in terms of the ability to escape from a container they really weren't designed for that so there's no blame there so we strongly believe in using hardware-based virtualization whenever you have a multi-tenanted or any workload where you need strong isolation the problem is that VMS typically typical virtual machine takes a while to boot seconds maybe a minute because they're big and complicated so firecracker implements a micro VM technology in which a full KVM virtual machine can be booted and executing user space code in less than 1 8 of a second so 125 milliseconds is the design goal for firecracker we've met that consistently over time and that means that you can now launch VMS as quick as you can launch a container essentially or close enough that you can use that as a very foundational property of your container-based or function as a service workload so we use firecracker underneath Lambda and underneath our fargate service but we also release it to the world and there's a lot of other vendors now who use firecracker in their container runtime environments and we're really excited for the success of that um you I think you're going to call one of these out yeah I've got a favorite I I do have a favorite on this slide and it's trusted language extensions so postgresql the open source database has a wonderfully vibrant extension community and that allows postgres to change from a relational database to a vector database to something focused on geospatial and a host of other things there are literally hundreds of extensions to post-resql the database itself and we recognized from some of our own pain and while it didn't directly impact our customers because of defense and depth we saw that a number of folks were calling out the extension space as an attack surface that would allow them to do nefarious things to postgresql databases true in in hosted environments as well as self-managed and so we created trusted language extensions and release that as open source we did that at re invent last year and I'm excited about this because not only does it solve a problem for us and we could have just solved that problem but we released this because we recognized that everyone should have a more secure and a more safe postgresql so the trusted language extensions are probably my favorite thing on on this slide because it's something that uh we've seen a significant problem identified that problem and then work to to not just shut down the the specific issues but to create a safer environment to run those extensions in I'm super excited I'm I'm really excited by seeing how many people are adopting trusted language extensions as the environment in postgresql that have nothing to do with AWS that's awesome it's a great story so we'll dive into a couple of these um open Quantum safe is called out here oqs and we'll talk about that in the context of s2n and some other crypto stuff and then Cedar and maybe I think we have one other example here in terms of our decision to work in the open with core technologies that we develop so you can imagine you run a cloud platform encryption super important it's one of the fundamental properties of data isolation data security that customers rely on that we rely on so you really want to be constantly on the The Cutting Edge of of crypto of cryptography and of encryption Technologies across the board customers obviously they're running applications on top of our stack they want you know super efficient systems with minimum CPU and memory utilization but they want they want state-of-the-art technology underneath so we've been on a journey now for a number of years to develop technology in this space and to open source the technology we developed because we believe it can benefit the broader community I'll start with s2n s2n was one of our first forays into this General space it originally stood for signal to noise I'm not sure if we even spell that out anymore we call it s2n but you can think of s2n as a very stripped down very minimalistic very feature poor intentionally implementation of the TLs protocol feature poor meaning that TLS has a ton of features that most people never use and but to implement those features takes a lot of code and therefore introduces the possibility more bugs this is their wake-up call for us and I think for a lot of people in the industry was the heart bleed uh zero day remember the heart bleed one and raise your hand if you had a heart a bad day on the heart on heart bleed day we did too although we were able to patch a massive Global Fleet of load balancers in less than 24 hours we still had literally hundreds of thousands of hosts that had to be patched in our elb service but that was a wake-up call because we looked at like okay what's the root cause here well it's a bug in open SSL and okay software has bugs but it was a pretty painful one um but what about openssl oh it's 300 000 000 lines of C code it's it's old it's crafty it's got I mean God bless the community for keeping it up and we now we've been financially supporting them ever since and it's done a ton of good for the industry but it's not what we felt really good about in terms of relying on as a core technology for very fundamental security properties of our of our cloud so on since then we've been kind of on a journey to we both Implement a replacement and you know substitute that in and use it as well as release that to the world s2n was one of the first steps there so it's a again just implements the TLs part of the protocol still uses openssl for the cryptography parts of of the handshake and and the encryption um that was some years ago S2 has an interesting example too because this was before rust was kind of up and running and popular um so it's written in C however it's written in a very strange and idiomatic sea that looks just like rust because um you can't do any direct memory access it requires all these kind of interactions and we have formal tests and proofs that show that if any developer does a check-in that's not using the uh the proper memory handling functions that are inside of s2n um sorry your code will get rejected so still using the C language but using it in a very carefully crafted and memory safe fashion a couple of other really interesting facts about s2n first of all it's a project that had I think at the start maybe a little bit more now about 20 000 lines of code 80 000 lines of tests so it's a very very test Centric test heavy project and it was the first test first release to the wild as far as I know at least of any significant piece of code in which formal verification was part of the build and the tester and release process so all the people we hire and you've probably heard in many talks or maybe I've asked a groupie called automated reasoning team or these form of verification capabilities a branch of computer science that has been a bit obscure has been around for a long time it's not obscure anymore at least as far as we're concerned we're using more and more we're using the tools and techniques of the formal verification to prove the correctness of code and s2n if you go to the GitHub repository and you look at the build instructions and you download and try to build it you can actually download a bunch of formal proofs about the correctness of various parts of that code again it doesn't mean the code is bug free necessarily but it's a much higher standard than we've traditionally done for a lot of these systems so that was a big and important foray into this space but we didn't want to stop there so subsequently we've also open sourced our own lib crypto so now we're giving you an open source re-implementation of the core crypto cryptographic Primitives you need for uh for your network as well as other use cases for uh that are traditionally done in the openssl library um I won't go on and on about you know uh lib crypto but it's out there um one of the key things we're doing with lib crypto is we're taking on the burden of fips validation that's one of the the kind of things that people often expect of their cryptographic libraries they want that third party validation and interestingly even around the world we still hear people are very happy with fips validation even though it's a technically a U.S government standard um and but it's hard for an open source team to do that it's expensive it's time consuming it's frustrating it takes a long time it's almost you almost kind of require commercial interest and Commercial commitment to really kind of go through the trouble which many companies do but we're doing that on behalf of the open source users of s2n and of lib crypto we're taking that that through fips validation and we'll keep that up to date as time goes on which is kind of the hard part often you get it through and then you kind of like relax and move on and then your code gets out of date and so you get into this bad cycle so we're doing that with crypto and the final thing I want to call out that's just briefly referenced in this slide and we could spend a whole hour on this post Quantum cryptography so we've made a lot of investments in pqc again the nutshell version which I'm sure most of you know is nobody has a sufficiently coherent quantum computer today to crack asymmetric cryptography but all the experts say that if we can if we can invent one and we don't usually like to bet against engineering teams once problems go from Theory to engineering the engineers are often very good at succeeding at very hard problems and so it's not unlikely that in the future not too distant future there will be a sufficiently powerful quantum computer that could crack today's asymmetric cryptography now the good news is often not mentioned which is symmetric cryptography does not suffer much from quantum computers so if you're using AES 256 for encrypting something that you write to disk as EBS and S3 and Dynamo and all our storage services do quantum computers would speed up Brute forcing of that cracking that crypto by a factor of a hundred a thousand doesn't matter it's there's so much power is required to crack those that that's essentially irrelevant at least as far as we know according to today's experts but asymmetic cryptography totally different it's much much very susceptible to being cracked by a quantum computer using Brute Force techniques and therefore we have to get ahead of this problem before it becomes a problem so we've made a lot of investments in post Quantum cryptography we have production systems today in production using hybrid key exchange which means that it encrypts the inner keys of a TLS session with both elliptical curve and post Quantum so that if one is cracked or the other you still have it's basically defense in depth so a lot of really interesting technology going on in there but again the key thing is we're doing this in the open so that anyone can benefit from all the Investments and the work that we're doing in this in this space speaking of formal verification uh and that because that's one of the key sub themes of this open source release which we just made a month or so ago we've introduced to the world a new another new language hey we need new languages we really felt we did need a new language in this case we looked heart long and hard at existing authorization languages so this is a kind of you know narrow but very important part of the security world is how do I encode permissions about access to systems if I could step back for those of you who like me have worked in sort of identity access management over the years as an industry we've done a pretty good job on the identity side right we have a pretty good story about saml and oidc and blah blah blah that allows me to come across into a system with some sort of validated cryptographically validated user with a set of claims okay now I show up at a system now what happens somebody has to make an authorization decision like what rights does that person have in the system that they're accessing there the industry has done a terrible job of standardization we tried years ago with something called Zach mole if you know what that is boy we're you know we can go have a cup of coffee and talk about that um failed miserably and if you think about today's I.T systems on premises for especially you have these Apple models everywhere SharePoint apples exchange you know different file systems everything has an access control model that's all completely disjoint completely heterogeneous and it's a not a good story in terms of wanting to understand and answer those basic questions who has access to what like that is the Holy Grail of access management systems who has access to what today super hard to answer that question it's easier in the cloud by the way because you have a common identity management system for all the apis still not super easy to answer and we're getting there but at least there's a possibility because of the unification on the on the access side but long digression but we decided that are then an existing IM system works great but was it the right language to expose for a broader a set of use cases we decided not I can go into lots of great details why not we looked at existing open source or standards Technologies they didn't really meet the requirements so we invented a new one called Cedar and one of those many great things about Cedar which I won't go into but one of the most interesting things is it was built by a joint engineering team that included both experts in access management but also formal verification scientists who built both the language for formal verification as well as the implementation the runtime and the libraries those are also formally verified because you really want correctness when it comes to access decisions so this is a really cool technology there's some great blogs on Cedar if you want to dig into the Gory details it includes another technology called differential testing which is a a very interesting sort of probabilistic fuzzing if you will because even with formal verification you still want to do some some testing to make sure that your tools are all working properly so enough on Cedar but it's a very exciting release and we hope the world adopts this and uses it everywhere because even though you know there'll still be some challenges in terms of unification having a common access language is it would be a huge win and we're working hard to make that true yeah you know Mark I was one of the naysayers who said that there was no need for cedar because we had other access languages out there and when when this project came up internally I was one of the folks who said why are we Reinventing the wheel this doesn't make sense but the formal verification of both the engine the access engine and the policy itself really sold me that this was something new that actually needed to exist I'm excited about Cedar I I'm excited about the attention that it's getting as well because people seem genuinely excited about this and those people are not even AWS customers they're recognizing that this is a gap that we have in the security space so I want to talk a little bit about snap change which was another open source release we made last month and snap change came out of a team inside AWS called find and fix are our F2 team and to explain a little bit about what F2 does the F2 team is has a mandate that they are to go look at what AWS and our customers are using the most and then to audit it to find problems specifically security problems in the most commonly used open source tools that AWS and our customers make use of and they've been on this journey for roughly a year now they've built up a team of security researchers and they're trying to figure out how to scale up their capacity to find problems and snap change is a buzzing tool and fuzzing essentially tosses a bunch of strange inputs into various break points in in a piece of software and and frankly fuzzing as a general tool has been responsible for finding lots of problems and they looked around they they're using plenty of fuzzing tools as it is today but when they looked they said you know there's some things that we can improve specifically snap change is a tool that works in an emulator or a hypervisor environment it presumes that the software is actually going to be running as a virtual machine and it is able to inject the the fuzzing via that layer and so I said you know why are you doing this that there are there are already tools that do this and one of the call outs was well those tools today require a modified KVM and or a modified kernel so you've got to load a specific kernel module that does not ship in tree and that's a that's a barrier to some folks for being able to easily deploy this being able to run it at scale the other thing that they wanted to do is they wanted to scale this across multiple cores and so multiple CPUs at a single time and run that allowing folks to hopefully scale up and and make their fuzzing activities a little faster so snap change allows folks to basically use break points in code and and submit all of this uh this fuzzing data into the program and to replay that and so dramatically more scalable and hopefully faster for folks to use this is really targeted at the security researcher audience this is not something that we would expect an average developer to go make use of we're excited about it because the security Community seems to be finding this useful and I I'm I'm really excited that this team is getting a little bit of visibility because they're doing amazing things in terms of finding bugs because their their mandate is both defined and to fix they're not just looking at bugs that they're finding in code and saying hey here's a security vulnerability they're developing proof of concept they're developing a patch and shipping that to minimize the workload on the open source maintainer because we don't want we don't want to stand up a team that basically uh creates a denial of service a social denial of service attack on open source maintainers so we want to provide them as much help as possible when we're going out and looking for security vulnerabilities I'm going to interrupt you for a second go slightly Off Script because I think this is super important and something I learned from you when I first started getting involved more in open source security so by the way David is the president of the Apache foundation in his spare time and uh very very long and deep involvement in open source community and I remember when we first started talking about these kind of security Audits and so forth you sent me this post-mortem of a security audit that was done on some Apache software and it was so enlightening because what you what it made me realize is you can go in and you can find bugs there's no problem finding bugs in large software code bases now what bugs okay great now what do I do where's the resources that are needed to fix the bugs that's a whole different thing and it's also very hard to inject those resources even if they exist into a an existing development team an existing culture and existing way of building and developing software so I found that super enlightening and it's also kind of uh an eye-opener like it's not a matter of finding bugs it's a matter of creating this virtuous cycle of finding things but also developing the skills the capability of the community that owns the software prioritizing those fixes all those things are much harder problem than just finding things and I think that's a super important story around how we need to as a community get better at doing this stuff I I think that report that you mentioned uh was talking about all of the influx of help that happened after heart bleed and I use help in in air quotes right because security is not a one-time thing that you can say all right we're secure today and then walk away it requires that continuous investment where you are where you're making long-term uh investments in not just technology but in communities and making sure that you can um making sure that you can actually shift the culture shift the practice to something more secure it's not you're right it's not about finding bugs if it was that easy we would we would uh we'd be much better today we would probably not be giving a talk today on open source security so I I said this earlier but uh open source projects do not work on code contributions alone and we think that AWS you can see it in our leadership principles size and scale bring great responsibility and we certainly have a responsibility to open source and so we recognize that and one of the mechanisms that we can take use make use of is to support a number of Open Source projects and open source foundations and we do this across a number of places so we support places like the python software Foundation the Apache software Foundation we're members of the Linux foundation and Cloud native Computing foundation and provide a baseline of support via that mechanism you know that that provides us a way to make sure that that base level human infrastructure is in place and that people are able to get things done but I want to dive into how we're thinking about funding security responsibilities because I think that is a separate layer above just funding the open source projects themselves so the python package index if you're not familiar with Pi Pi it is a you're not a python developer if you're not you are you are you know if this is how the world consumes python right it is the package repository if you're a Java developer there's Maven Central if you're a python developer there's the python package index and it is a huge attack surface for folks trying to impact the software security supply chain and we recognized this early on paid attention to it started working with the python software foundation and said what can we do together to improve this and they said well you know today we're doing mainly reactive response to security issues we don't really have a great security strategy for the the package repository itself we're not proactively going out and looking for threats we would like to change that they put together a proposal and brought that to Mark and I and said hey this is what we'd like to do can you fund this and we said absolutely this makes perfect sense we'd love to jump in here because we consume packages out of the Python package index all of our developers who are doing python do that our customers do as well and so does the rest of the world you're not developing in Python if you're not using this and so we have been working with them to help create proactive strategies build up some internal Staffing so that they can actually respond to issues timely and we're not depending upon volunteers in Nebraska to respond to a security issue in the python package must have some individual in mind when you refer to so we won't mention any name we won't miss we won't mention a comet's name but we're not responsible we're not responsible uh if we're not taking into account these distribution points and making sure that we are helping to fund their security because that impacts all of us we have that shared Destiny we talked about earlier we've also and we've hinted around at this a little bit earlier but there's a number of scholarly articles written by people far smarter than I that are looking at the overall timeline of security vulnerabilities and they're reporting that 65 percent of security vulnerabilities in the past decade are memory related two-thirds of security vulnerabilities are from a single class of security threat and one of the reasons we are so interested in Russ is that it provides tooling to ensure memory safety that's not a Panacea for you know rust does not make you magically secure and I don't want to paint that picture but it is a much more assured position from a memory safety perspective and I'm excited about that for what we're developing internally Mark but we looked at at where some of the the entry points in terms of Open Source was and I remember us sitting in your office in Virginia um a year ago now almost and talking about the places where we were seeing attacks and seeing security vulnerabilities in open source and again security is not a point in time thing that you can go say I'm secure today and I don't have to worry about it anymore and so we looked at where can we where can we focus on things that are on a network boundary things that are getting lots of direct internet traffic are there parsing huge files how do we go address that and so we identified four initial projects and asked the prossimo project out of the Internet Security research group to go rewrite those for us and we provided funding to rewrite those so those initial four are the ntp Daemon we asked for a rust implementation of that pseudo and Su because again increasing your authorization level is something that you really don't want to have memory issues in Russell there's a long and sad history of bugs in that there are a lot of memory bugs specifically in in sudo Russell's the the TLs library for the rust programming language we wanted to improve how fast Russell's was going to be able to be production ready we want to make sure that Russell's has already written so no inherent rewriting but we want to mature that rapidly because we think that a TLS implementation that's memory safe is super important if if your encryption fails you've got lots of other problems and then an av1 decoder and av1 is itself an open source success story and we wanted to make sure that uh that this decoder that's processing lots of huge files both audio and video audio video and images is uh is written well and is secure so we've been doing uh we've been investing in a lot of that and finally the industry as a whole has coalesced around the open source security foundation and uh Mark Mark Champion joining openssf years ago right after uh right after it came out we joined that we started investing and spending time there folks from from Mark's office were certainly heavily invested in in trying to advance the state of security in open source but we recognized that there was opportunity to do more and so last year we committed 10 million dollars towards openssf to make investments in long-term sustainable open source security initiatives and we did a couple of those I'll talk about them real quickly Alpha Omega which has two objectives the first is paying attention to the ten thousand most widely used open source components and uh creating automation around finding security vulnerabilities and patching it so they're really heavily focused on automating security as much as possible the other side of Alpha Omega is focused on the top 100. what can we do that has outsized impact across the most heavily consumed open source packages and how do we make that security investment sustainable they're doing great work we're seeing lots of things happening there in a number of different communities where we continue to be excited about that security scorecards though Mark talked about consumption right and we at AWS we delegate open source consumption decisions to builders nobody comes to Mark asking for permission to use another programming language even though he's in the security organization nobody comes to me in the open source organization saying I want to use this thing is it okay and individual developers doing that and we think security scorecards which will provide an automated look at a number of Health metrics and security metrics provides much greater information to base decisions upon so that those developers who are deciding what open source they're consuming can make better informed decisions we're excited about some of the work coming out of security scorecards and look forward to that being more ubiquitous and of course we're we're doing a lot of General support as well so let's close out our session and just as a reminder of our takeaways and our pillars work upstream and contribute to the open source world and provide financial support and for all these things we encourage you the broader Community to join with us in in these areas think about ways in which your organization can work in one or two or three of these ways or come up with new ways to work to support open source join with us come to us with ideas we're very interested if you have funding ideas if you have engineering ideas that you think can help in this world that's super super important we're going to join together to make the world a safer place because we all depend on secure open source and we thank you very much for coming and have a great rest of your time here at reinforce thank you thanks

2023-06-21 10:14

Other news