Right-sizing security across Amazon businesses | Amazon Web Services
- Hi. Welcome to the Executive Insights podcast brought to you by AWS. I'm Clarke Rodgers, Director of Enterprise Strategy, and I'll be your guide through a series of conversations with security leaders. Today I'm joined by Steve Schmidt, Chief Security Officer at Amazon.
Join us for a conversation about all things security at Amazon and AWS. From the creation of the chief security officer role to how we develop and innovate security products internally and ultimately how we protect customers. Please enjoy. Steve, thanks so much for joining me today.
- Happy to be here, thanks. - So it's been a while since we last talked. In fact, I was doing the math last night.
It's about four years ago. - Wow. - You and I were on a remote interview- - [Steve] Yep. - And you were in a different role, right? So you were the chief information security officer at AWS.
Not too long after that interview, Andy stepped up to the CEO role of Amazon. - Right. - And one of the first things he did was bring you on board as the chief security officer. Can you tell me a little bit about why he did that? - Sure, so one of the things that Andy really values is understanding how we protect our customer's information.
And that grew out of necessity in AWS where it's foundational to the business itself. You can't have a business like AWS unless you do security right. And when he got into his new position, he wanted to take that same mindset and apply it to all of our customers across the incredible diversity of businesses that Amazon currently operates. And he also wants to know that there's somebody whose day job every single day is keeping an eye out for that kind of thing. And so he asked me to step up and take that over. - So as chief security officer, there's one word missing from there.
- That's right. - "Information," right? So you're a CSO. We've had a couple customer interviews where they have also, quote, unquote, dropped the I. - [Steve] Right. - Can you tell me a little bit about why it's important to be the chief security officer, not call out "Information" specifically? - So when you look at the protection of information, which is the real currency of our realm, it is something that certainly has the logical component that people are used to with the CISO job.
But increasingly, our adversaries are using humans to go after information that is harder to get to from a logical perspective. - Right. - So there's been this sort of pendulum swing in the industry, and it is... Long ago, if you think about it, people had spies, physical spies, who would go in and grab copies of documents or things like that.
When we went online with computer systems, "Oh heck, there's a new opportunity. Let's go break into those things remotely," etcetera. As businesses get better about securing that information, whether it's on premises or elsewhere, it turns out that people go back to being spies again. - Right. - And so we have to integrate the protection of both our physical assets and our information assets, our logical assets, in order to get a full picture on what we're doing with our customer's data and how we're protecting it, and also our business data.
Because quite often now, our nation-state adversaries are focused not only in getting access to the data that our customers have, but also the information about where are we taking our business next? How are we building the next set of very interesting products or services, whether it's a satellite or a robotic vehicle. Those have incredible value to countries around the world, but also to the businesses that they support. So as security professionals, we have to look at that complete picture of who are our adversaries and how are we going to protect against them? - For sure.
So as you stepped into the CSO role, and now you have physical security reporting up into you, I would imagine the language that the information security professionals speak and the language that the physical security people speak could be on two different planes. How do you go about integrating those two so they're, quote, unquote, speaking the same language and can work effectively together? - Sure. So it's less about the word choice that they have because they have their own lingoes, you're absolutely right. What's most important is understanding what their end goal is, how you're going to measure your progress towards that goal, and to identify most importantly, the handoff points or the gaps between the physical world and the logical world.
Think about it this way. If I'm in the physical world, I prevent people from getting into buildings, or I prevent our adversaries from being hired by the company. But if an adversary does get into a building, is that interesting to the information security side? If they're standing in a lobby, that's one thing. If they're in a closet that has a bunch of network switches in it, that's something entirely different. - [Clarke] For sure. - So we have to link those two together in order to get a proper picture of what's going on.
- Did you have to build any tooling to sort of cross these bridges? - Yeah, it's both a combination of tooling and processes that we had to put together. So tooling is a constant evolution. We're always getting better, in theory, at finding ways to collect, analyze, and use data.
Where it becomes more interesting and more difficult in fact is figuring out how you hand that off between different parts of the business, and how you make sure that they've got access to the things that are necessary to do their job, but don't overwhelm them. And that's where the process component comes in. It's often about the bubbling up of things that could be sensitive, that are sensitive, etcetera, and finding ways to ensure that they're transitioned from one part of the organization to another.
And that really takes highly calibrated people. It takes humans who know that, "Okay, this may be innocuous-seeming in its own right, but when combined with this other thing that we saw, it's really interesting." And unfortunately, there is no system yet that can do that. Maybe with some AI training down in the future, we'll be able to get there, but it's just not there at the moment.
- You started the security program at AWS, so you know AWS backwards and forwards, how to secure it, the threats that are against it, the risk appetite, etcetera. As you moved into the CSO role, you now had to learn about Amazon.com, or what we call internally, "Stores," Whole Foods, Prime Video, MGM, Twitch, all of these different organizations. First, how did you get up to speed on the security profile for each of those businesses and sort of the risk appetite? And then how did you sort of bring it all together so that you felt comfortable that the risk profile for Whole Foods is appropriate, the one for AWS is also appropriate for AWS, how did you sort of figure all that out? - Well, first of all, one of the things that I love about my job is the diversity of the businesses. People often say, "You've been in your position for like 16 years.
That's really unusual for somebody in the security industry. Why is that?" It's because of the diversity of work that this company has. It's an opportunity to continue learning, which is, I love it.
I'm not that young. People say, "How long are you going to keep working? Are you going to retire?" Whatever, and I'm like, "Well, I'm enjoying myself. No, I don't want to. This is really a lot of fun." And it's because you go from building the world's largest cloud provider to putting satellites in space, and running grocery stores. And the diversity there is just an incredible challenge from the business perspective, but it's also an interesting opportunity to leverage the scale of the company to make things less expensive to operate.
When you look at operating security organizations, they are not cheap. But when you can scale it across a business as large as Amazon's, it means the unit cost can be driven down. - For sure. - So every business benefits from the scale of the other businesses. And so finding ways that a grocery store can take advantage of the same security that a satellite business does, which they can in many areas, for example, vulnerability management, whether you're patched on a computer system, is not really fundamentally different when you're building satellites versus operating a grocery store. And so that allows us to do things at a level that a standalone business really could not afford to, and raise the bar for everybody.
And that's part of what my job here is is making sure that we have a standardized bar across the company, whether it's for vulnerability management, or incident response, or any of the other components that go into a typical security organization. And then figuring out what are the bespoke components that have to be put in place for particular businesses because of their unique situations. That way we don't try and apply the sort of one-size-fits-all to everybody because that would just drive costs through the roof. If you look at a grocery business for example, the loss of a unit there has a relatively de-minimize value. - Sure.
- Whereas the loss of a satellite is the opposite. - For sure. - And so we have to tailor the security situation for individual components.
- You have chief information security officers running the security programs for each of these other businesses. How do you hold them accountable to run their security business? - So one of the things that Amazon focuses on a lot across the company is the idea of a single-threaded owner. So somebody whose job it is just to focus on one component of something. And in security, that's why we have a CISO for each individual business is because of two things, one is I want somebody who every day just focuses on that, whether it's Amy Herzog who's in the devices and Kuiper space, or whether it's Chris Betz who's looking after AWS.
But at the same time, I use common measurements across all of those things. For example, I look at a vulnerability management monthly business review, which encompasses the entire company. - Got it. - And it uses the same numbers, the same methodologies, the same presentation methods, etcetera, so we get a common view that's consistent across every one of those businesses. And it allows us to do two things. One is to make sure that we're meeting the bar that we require, and the second is to ensure that we are applying the visibility that we want in every corner of the business.
Because quite often where people have problems is they think, "Oh, this is not an important thing, this is a small piece," etcetera. And that's where the bad guy gets in and we all get bitten. So by giving that sort of 10,000-foot oversight on everything, we make sure that we're doing the things that we need to in every part of the company.
- And then you have centralized systems under the Amazon Security or AMSEC umbrella that everyone can take advantage of? - Right, so there are a lot of things that are fundamentally the same across all of our businesses. The way that you collect certain kinds of data, the way that you analyze that data or report on it, and rather than making every individual business do the same thing over and over and over, we decide to move them into one spot. That allows us to save on things like developer time. So, if you think about running a large scale, we'll go back to vulnerability management, collection engine, for that you've got to have on-call engineers. If you've ever run an on-call organization, which you have, you figured to have one person on call, you've got to have approximately seven people.
- Yes. - In order to do that effectively to accommodate leave, and vacations, and everything else. - We want people to take vacations.
- That's right. And so by spreading that across a bunch of different businesses from one place, it means that we do so more effectively because we get better tooling centrally, and at a lower cost. - What practices have you either developed or followed to report the status of security across all these disparate businesses to the Amazon board? - The Amazon board first of all is really interesting. There are very few boards that have the technical acumen across the population that Amazon's board does, but also Amazon chose several years ago to create a security subcommittee. So unlike a lot of places where security may report to the audit committee, for example, there is a dedicated group of people whose job it is just to look at security on the Amazon board.
That's great, and it also means there's a lot of scrutiny on us in the process. So we've had to build a reporting mechanism which evolves over time. - Sure.
- Because of two reasons. One is because we get better at doing our job on reporting. And two is the board becomes even better-informed over time.
They ask more pointy questions. - [Clarke] Right. - And want to know more specific details about niches in the business.
We generally find that it's important to report up to them on specific components of the business every single time we talk. Are we meeting our security bar in certain places? Additionally, they have things that they're really interested in, "Tell us about this particular part of the business," or "This thing we're getting into we think has a lot of risks, so give us a rundown on blah, blah, blah." And that allows us to have a consistent component, a variable component that's based on their interest. And then we decide to put in something called "current events" at the end, where we take all of the stuff that you've seen in the news, we distill it down to things that we think have particular lessons or points for us, and present that to the board as an informative component at the end. "Here is something that happened out in the industry, here is why we were not affected.
Here is the investment that led to us not being affected." And I think that has tremendous value for the board. Number one, they understand that we're in a good place, but number two, it helps inform investment decisions down the road. So when we can come forward and say "We invested in multi-factor authentication eight years ago, 10 years ago, and that prevented this particular threat actor who got into this other large company, from affecting us." They say, "Okay, great.
What are the other investments we should be planning now that will help us in 10 years to continue to avoid problems?" - What recommendations would you give to your CISO peers or CSO peers who are reporting to their boards to really help sort of drive their points home, speak the language of the board, etcetera? - So the number one thing that I've heard from board members about why they like the way that we do things is we're very careful to avoid jargon. A lot of people in the CISO roles are technical, and they want to report on things in ways that- - The bits and the bytes. - Exactly, are reflective of the way they think about it.
But we've got to remember the board is the customer here. So when we are presenting to them, we have to speak their language, and we have to find ways to explain things in context that makes sense to that particular board. So number one is find a consistent reporting mechanism rather than varying it each time, because that makes it harder for someone to grok it, basically. - Right.
- Number two is figure out what are the really like two or three super important metrics that you want to get across in there every single time. Don't drown people in metrics. For example, we always, always report on vulnerability management. It is the single most important fundamental security control that we operate, and I think anybody operates. And then figure out what are those things that are interesting add-ons at the end, which help you develop the should we invest in, so make that intentional separation between components of the reporting process.
- As a security leader, you have lots of choices about what tools you purchase versus what tools you need to build. Oftentimes, that comes down to scale, right? So, the commercial off-the-shelf software may or may not scale to Amazon's scale, and you need to build something yourself. Over the last year we've talked publicly about tools like MadPot and Mithra and Sonaris.
As the CSO, how do you make the pitch to get the dollars to actually put the engineering resources behind tools like that, and I imagine many other tools that we haven't talked about, how do you make the pitch to say, "This is a worthy of your investment and here's the value that we're going to get out of these?" - We will purchase tools when they're a commodity item. So for example, endpoint detection response. - [Clarke] Right. - We buy that
as opposed to build it ourselves. Why? Because the Mac laptops, the Windows laptops, the Linux laptops that we use are the same that lots of other people use as well. - [Clarke] Sure. - We may have a little bit of different software on them, but it doesn't really differentiate our business.
Whereas we are the only ones who can build a very large scale system like MadPot as an example. Where we can build something that nobody else can is where we tend to invest. The way we do that investment process is just like we do a lot of other things, you prototype it, you try it, you see what works, you guarantee that you're not going to get it right the first time, so you've got to go and re-engineer something, change it a little bit, etcetera.
Now, MadPot has been incredibly successful, but it didn't just pop up overnight. It is an investment of many, many years that started off with a single engineer who said, "Huh, I really like this idea. Let's go see if it can get anything interesting." And then it's turned into this engine that allows us to acquire really, really timely threat intelligence data that we can take out of that, we can process, and we can feed into the security tooling that all of our customers have access to.
And I think that's the part that's most important. So for example, a lot of our customers say, "Ooh, I want raw threat intelligence feed." I'm like, "Well actually, no, you don't." Now, what you really want is the stuff that's relevant to you in the context in which you're operating right now.
The rest of it's just noise. And the volume of it now is so huge that it's pointless to try and digest the whole thing unless you have a business that's like ours. And so, a lot of our customers have found that they really, really like taking things like threat intelligence, and consuming it as part of a managed service. - Got it.
- And that allows them to not have to spend their time winnowing down very large piles of data, or missing context because it's not been applied across multiple customers. And that context thing is important. That's where the centralized visibility that only we have, to go back to the original part about commoditization versus custom-builds, really brings advantage. - And then I guess the internal pitch, for lack of a better word, it's, "This helps us secure AWS/Amazon, and adds benefit to our customers." So, I mean it's a win.
- And in typical Amazon parlance, we'll start off with the customers first and say, "This helps all of our customers better secure themselves." - [Clarke] Right. - And at the same time, it helps us with our business because most of our businesses are Amazon, AWS customers anyway. So, it is beneficial across the board. - Let's switch gears just a little bit.
This past year has been the year of generative AI, right? What are you seeing, or maybe what are we doing inside of Amazon actually using generative AI as a security tool or as part of your security tooling chain? - So the most effective use of generative AI that we've seen so far has been in the application security process itself. - [Clarke] Okay. - So as you're familiar, in AWS, every single feature, app, etcetera, that goes out the door, has a security review before it's launched. Other businesses haven't had that luxury in the past because it's incredibly expensive to do that, especially when you consider that it means literally thousands of security engineers who are focused on that. Most companies out there do not have the number of security engineers in total that we have just doing AppSec work- - Sure. - At Amazon. And so generative AI gives us tremendous leverage in that space.
It's still in the science stage, I'll say, but it's got an enormous amount of promise. We think that over time, we'll probably see a reduction in human workloads in the application security space in the 60 to 70%, which means that we can do things more quickly for lower costs with better consistency in places like AWS, which has always had an investment in evaluating everything. And in areas where there's an opportunity to look at more that's never been evaluated before, it means we can get greater coverage.
So it's sort of a double win. - So you're going to take those human hours that you save with generative AI, and put them on other cybersecurity challenges. - Oh, it's actually going to be in looking at in more depth at services that we've got, and examining more applications across the board. The other piece is generative AI is not going to be the entire solution for a lot of this. What it'll do is it'll winnow out a lot of the sort of low-hanging fruit, if you will. - [Clarke] Right.
- And allow our engineers to focus on the really, really interesting problems that only a human can go into. People think, "Oh, generative AI can solve all problems." Not yet, and I think anybody who tells you it can in the security space is probably not really understanding what's going on. There always needs to be human oversight in that space, at least right now. - [Clarke] Right. - And more importantly, there has to be human judgment on whether the generative AI has made the correct decision or not.
- Speaking, or staying with humans, what makes a great security hire at Amazon or AWS? - I would say the number one thing that we look for in our security hires and most people are thinking, "Is that this particular kind of technical acumen?" No, curiosity. - Tell me more. - What it means is someone who doesn't look at a problem and say, "Oh, okay, this is it." They say, "Well, why did that happen? Well, why did it get in that place in the first place? Why didn't we detect it sooner? Why was it something that a builder could even do wrong to begin with?" That is the kind of person who keeps digging that we're really, really interested in. And you're really familiar with the correction of error- - Yes. - Process that we use, COE, and it has at the bottom what? Five whys.
And it's that same thing is really get down to the root of the problem as opposed to just addressing a symptom of the problem up at the top. - So, curiosity is king. - Curiosity is king, yeah. - If I give you a crystal ball- - Uh-oh. - If you look into that crystal ball, what are CSOs and CISOs going to be focused on in the next year or so? - Well, I think that most people are going to have to focus on generative AI because their businesses are consuming those services at an incredible clip, and they're going to have to do two things.
Number one is simply understand where they're using generative AI. And I think we're in a lucky place in Amazon, and we've got centralized visibility that allows us to see where all of our builders are using generative AI. Most companies aren't that way.
And they've got to go figure that out. The second part is you're going to have to figure out for generative AI, how you're doing RAG, and whether you're enforcing authorization and authentication appropriately throughout that process. That is not trivial.
It's something where the software world doesn't even have it completely nailed down yet. And it's something that is incredibly important to being able to use generative AI in a way that presents only back the data as a result of the prompt that that person on the other end is authorized to have access to at that point in time from the place that they're currently sitting in on the piece of equipment that they're currently using. Most people have not even thought about that problem yet.
- Right. - But in reality, if you look at our internal systems at Amazon right now, when you're on your laptop, we measure a lot of things about you when you're using our internal systems. - [Clarke] Sure. - Literally, is your laptop in the state we expect it in? Have you patched, etcetera? We've all had the little notices- - [Clarke] Yes, I have. - That pop up-
- Yes, I have. - That say "You're going to be quarantined if you don't patch." And where are you in the world? What's your job? And things that most people don't even realize we're looking at like what day of the week is it? And is this an hour that you normally access, or are you coming from an unusual place? And those sorts of things.
- "Is this normal for Clarke?" - That's right. "Is this normal for Clarke, and is this normal for a person in Clarke's job?" - [Clarke] Got it. - And those controls simply don't exist in most places. That's why implementation of guardrails is so important.
I don't care what system you use to do it, you just have to have guardrails in place. And those guardrails have to constantly evolve. As the threat landscape evolves around you, as the science of generative AI changes, and as your business determines more and more what is acceptable use of the data that they're feeding into a generative AI system. - That's fantastic and great advice. We'll see if your predictions come true.
Steve, you are the chief security officer for all of Amazon. It's a high-stress job to say the least. What do you do to keep not only yourself sane, right? And sort of blow off steam, just stay current with the world, sort of tune in, tune out, and then how do you also make sure that your leaders are doing the same thing and taking care of themselves? - Yep. So, the first thing that I focus on with all of our leaders, whether they're new or tenured at the company is how are you doing something to separate yourself from the job? Our jobs are incredibly high stress. They're something which are effectively never off. - [Clarke] Right. - So having a way
to hand off responsibility between people formally and saying, "All right, Clarke, for the next six days you're on vacation. I don't want to hear from you, I don't want you online." And our people are incredibly dedicated. There have been places where literally I've had to threaten to turn off people's computers because like, "You're supposed to be on vacation, stop emailing me. I appreciate your dedication, but the problem is you're going to burn out." And these are marathons.
These are not sprints. And so building that mechanism is super important. The second thing is having something that allows you to do something that's valued to you individually. Some people are musicians, some people climb mountains, some people go fishing. I personally am a volunteer firefighter and paramedic. - [Clarke] Oh, wow.
- It is something that for me is sort of the exact opposite of my day job. So, we're human beings, we like interacting with people. I'm talking to you here face-to-face.
Yay, not on video. That's awesome! But my day job is two things that are very different from that. One is most of what I do in my day job only has effect three, five, 10 years down the road. - [Clarke] Right. - So it's not an immediate reward.
The second is it's all computers with some people thrown in, and those people I can't even see or touch. So with the fun part of my life, the firefighter, paramedic side of things, if I do my individual job well, I can help a person who I can reach out and touch, have a better day. And that gives me that immediate feedback that I crave as a human being. And two, it's very physical, as opposed to something that's purely logical. - Different kind of stress, but probably a healthier one, maybe? - It is. And the irony is that there've been a bunch of studies there that people who have been in the fire service for a long time actually have their pulse rate go down when the sirens are on because it's their happy place, and they're enjoying it.
And let's face it, who doesn't like going down a road with sirens and lights on a truck? It's a lot of fun. - That's awesome. And I would imagine and I'd love to hear more, as CSO, you are faced with decisions and very critical decisions every day. Are you a higher-ranked, or a lower-ranked paramedic, or firefighter, meaning somebody else is making the tough decisions and you're just focusing on your day? - Ironically, I'm actually an assistant chief in the organization itself. But that's more about leadership skills- - Okay. - Than it is
about technical acumen in the other spaces. But I've been doing it for 38 years. - [Clarke] Oh, that's fantastic. - So I've built up a little experience. - Great.
Well, Steve, thanks so much for joining me today. - Thank you, it's been great to be here.
2025-02-24 04:44
