How Machine Learning Supercharges Threat Detection in the Palo Alto SOC | This is How We Do It Ep. 2
Welcome to this is How We Do It, episode number two. And I have Devin with me again. And in this episode we are going to talk about our security operations centers team and how we structure them. And what is the philosophy behind Devin? You have been around for a long time to remember our SOC when it was only a few people. And I remember you saying that you also worked in smaller secure operations or like with smaller security teams. Tell me what,
who are the essential players of a security person center? So if you would have to build up a brand new soc, you would be the only person and you could hire as many as good as you want to be <laugh>. Sure. How would you start doing it? Sure. So my personal experience has been two SOC teams. One was for a company of 80,000 and now here at Palo Alto Networks we have 15,000 employees. When I started at Palo Alto Networks and we started on this journey of building our SOC about six years ago now, we were just two managers and three analysts. That was all we needed. Palo Alto Networks was half the size and prior to that, the security function of what we do was part of it.
So at every organization, the needs for the security team are going to be different. You may have, or you may be a part of the network team and you're responsible for configuring firewall and you take on the role of ensuring that they're set up securely. As companies grow and mature, it does usually happen that IT and security will separate. That happened for us around that 2017 timeframe and now we've grown to the 22 full-time employees that we have here in the SOC today.
And of those 2210 are in the traditional analyst role where they're actually looking at alerts coming off the technology and doing thread hunting. And then the rest of us on the team support those analysts by enabling tooling, logging, giving them the alert data and the insights they need in order to be successful. Is. It true that there's a group of people working internally whose full-time job is to try to take us down? Yes, absolutely. That's our red team. Some organizations choose to hire external penetration testers.
You can also have them internally. And our relationship with our red team or attackers is that they are full-time employees whose job it is to test our defenses. And the way they go about those exercises is on a quarterly basis, they'll pick a target. It's always new so they're not repeating, something that's going to be easy and something that the sock is going to find fast.
They first get approval for it because they want to make sure that leadership is gonna be okay with them potentially attacking an internal system. And they go pretty far, I've seen them make fake employees and in the sock it's literally us looking at the name and going, this sounds like a fake name. They're that good. Sometimes that we were even reaching that point and questioning is this them or not? And they'll go about their attack and secret as long as they can for three months. And during those, three months, if we find something in the sock on the blue team side of the house we call it that we think might be them, we'll ask them and they have to tell us so that we don't spend too much time chasing down our own internal team and then either stop the exercise or finish. And at the end of the exercise we do a debrief where they report on everything that they did, which gives us in the sake a literal checklist where we can go back and say, we saw this, we didn't see this. We need to build a new alert here.
We need to build new automation here. And we get that feedback. And over the past couple years of doing these red team blue team exercises and always picking a new target, we've had a very large repository of good detections come out of those exercises because we wouldn't have known that we had a gap until the attack actually happened. And then we also do what we call a purple team exercise or a tabletop exercise. Hmm. Incredible. And so those red team people, they actually have power user accounts, they can do insider threats, pretty much everything that is on the table. Yeah, they've gotten an advantage over the external attacker because they've already got an account on the network and if they've been here for some time, they already know where the valuable stuff is.
So they can just skip the whole reconnaissance phase. In some cases they still do their own pre-planning, but they have a lot more information than somebody outside the company would <laugh>, let's put it that way. It's not easy to be in a sock. Yeah. And I have a big respect for you.
They, they're a good partner that it is a symbiotic relationship and we all benefit each other towards the same goal. Yeah. Yeah. So looking at various attackers over the years of your time in the Palo Sock, what do you see as the main motive of the bad guys attacking us? Are they ta trying to take us down? Are they trying to exfiltrate information? Like what is their main motive and who, who are they? Some of the biggest attacks we've seen lately are supply chain attacks. This is a really relatively new topic in the past two or three years. SolarWinds again, being one of the big examples where instead of going after the target company, the attackers are going to some organization that services them. So they may not see us as a weak target.
They may know Palo Altos is well protected, so we're gonna go to someone who serves them, one of their vendors, like SolarWinds. And for us we could be the launching pad for those attackers to over 85,000 other customers around the world. So we are in the game not only to protect ourselves, but ensure that we don't become the gateway to our customers suffering and attack. Mm-hmm. <Affirmative>, Devin, if you would, to summarize like what technology you are using, and I know that maybe we can have a separate, episode on this one. Sure. But what technology are you using? Are we eating our own dog food in our SOC? I like to say drinking our own champagne. That's.
I've never tasted dog food, but. Smells better. <laugh>. Yeah. I, I'm more on the drinking, drinking my own
champagne side of the house. But yes, absolutely. We are what we call the first customer of all of our own products. We're involved in betas before they get released. Our product teams work very closely with us to take our feedback and ensure that that is considered in roadmap decisions. And then we are using every single piece of the Palo Alto Networks platform and we are even using some third-party partner technologies where we don't have our own. And in previous episode I mentioned we're,
Proofpoint customer for email security and Proofpoint actually integrates with Palo Alto Network's own Wildfire, which is our cloud-hosted sandbox. So we can take emails coming through Proofpoint and actually detonate them in Palo Alto Network's own Wildfire technology and then allow Proofpoint and other vendors to make decisions based on that data. We'll put a slide up to show the full technology stack and how it all integrates. But generally, the single pane of glass in the SOC is Cortex Exor. That's where the SOC team spends most of their time today. Their tickets come in, and open a ticket in the incident management module, the SOAR playbooks run and assist in the analysis and response tasks. And then the threat intel management module allows us to continually ingest indicators that help feed the autonomous decision-making in those SOAR playbooks. And then all of the data generally coming from Cortex XSIAM,
all of our sensors and enforcement points along the bottom row all feed their logs and alerts into Cortex XSIAM, which is using machine learning to stitch those alerts together into a complete incident picture so the SOC isn't flooded with too many alerts. And then those make their way into Cortex XSOAR for the SOAR playbooks and the SOC analysts to do the response. Some of those sensor and enforcement points we've got along the bottom for the network.
We've got our next generation firewall ingesting all of the network activity on our users and our user endpoints and our server endpoints is Cortex, XDR, it's compatible with Windows, Linux, and Mac giving us all of that very rich endpoint data and allowing us to take action. And then for all of our cloud services under the Prisma brand, we've got a couple of different products that help us monitor containers, workloads, cloud configuration and settings. So together, when all of those logs make their way into XSIAM, and the machine learning deduces them into an important attack story for us. That is what is eventually making its way into the SOC. And then we've also got Cortex Xpanse, which is giving us the outside in view.
So we've got a lot of sensors inside showing us what we already know about, but because we've grown by acquisition, there's always the chance that we've got old test or demo environments still lingering out on somebody else's cloud account. Shadow IT is a big concern. The stuff that we don't know about, we can't protect.
So Cortex Xpanse is going out into the cloud and finding all of those exposures that we may not have known were out there and allowing us to get control of them before they become a problem on. The public internet. On the public internet. Is it true that even if it's not under our domain, Xpanse picks it up because it looks like it's ours? Yes. So the, ability for Xpanse to find traffic uses a number of different data sources. Um,
and what we're able to do is look at other internet traffic through various taps that Xpanse has out on the public internet to identify traffic that isn't just coming to us. And we're actually able to use that data to show customers that they may have become a victim of a certain type of attack. We actually did that when SolarWinds happened and, certain environments started beaconing to the C2. We knew where the C2 was, so we were able to use Xpanse to identify customers of ours who had been compromised by the SolarWinds attack and didn't even know it yet. And we were able to proactively go to them and give them the resources that they needed to respond and get control of their environment. Again. Incredible. How would you do all this without the attack surface management? Like before we did, um, before we acquired expense? Yeah.
Did you have any as m solutions? We didn't. That was a gap for us. We had tried using other tools to do basic seed and discovery just based on keyword, something more like brand abuse. there's a lot of tools out there today that can help you track if your organization's trademarks name logo are being used and then that has the benefit of finding phishing attacks usually by accident people using your logo on a name that isn't yours, that kind of thing. So that's not really discovering unknowns. You're giving it the seed information and say, look for this exact information.
But where Xpanse, the attack surface management is unique is it takes that seed information but then it branches out from there and it looks at a lot more data, kind of like your search engine crawler, you know, you can go and get results from a search engine like that cuz it's already indexed at all and Xpanse is doing the same thing. Even with historical information. I I heard that things are that Xpanse that it's attributable to you. Hmm. Even picks up those things.
Absolutely. It's um, very powerful. I can imagine that knowing, knowing what you know now, <laugh> Yes. You probably wouldn't do this without ASM. Certainly.
And I mentioned that it is a good use case for us at Palo Alto Networks because of our rapid growth because we've acquired so many companies. The products that you see on our architecture slide today, some of them are the happy marriage of four or five different acquisitions of other tools that became what our products are today. And every single one of those tools used to be a completely independent company that used a different cloud provider that had different tooling, some of it facing the internet, some of it not. And when you are actively undergoing the merger and acquisition of two to three companies at any given time, which is usually what Palo Alto Networks has going on, there's always the chance that they've forgotten something out there that they forgot that they had this environment or that there was an old test environment owned by someone who left their company before we acquired them. Yeah. Those unknowns are what we really are, focused on finding with Xpanse. Yeah.
So the rate of automation and the, the volume of alerts, is it, is it increasing? Is it this big or this massive because on the dark side, attackers are getting more efficient. There's open source, there's automation, like do you come across machine-learning or AI on the other side attacking us, scanning things, trying to get using machine, you know, machine capacity, machine powers Yeah. To, to hack us instead of one individual poking through ports and figuring out tactics. Absolutely. We're already there and, and have been for many years. Most of the reconnaissance that is happening now today is automated. Automated scans, if you've ever looked at the logs for an inter internet facing device, even if it's your home router and you just see nonstop scanning activity.
So we, the attackers have automated the reconnaissance in some cases they've even, automated the malware. They're using machine learning to create new malware. So we need to fight fire with fire and we're at that stage where we know some or many of the attacks that we are seeing have automation behind them. We need to meet that at the same level. And it has increased drastically both because of the number of employees that we've now got at the company has doubled in five years. So our attack surface has grown larger.
We have more logs and alerts by the millions. So we need to start continuing to scale that automation so we can handle all of that because it's not going to slow down anytime soon. <laugh>. How do they use machine learning to build a malware? Is it refining a malware to be non recognizable or is it like, tell me a little more about that. Sure.
Generally there's families of malware. So you've got different groups of malware that has the same core function or they get onto your network in the same way they're delivered in the same way and that's why they're classified in the different families that they are. And so when we have the machine learning trying to create a new version that's not going to get detected. Some older detection technology is still just looking at things like the file hash, is the file exactly the same as it was previously, yes or no? And that's how they decide to prevent. It's really archaic. And so these malware families will now auto-generate new versions of themselves so they look different and bypass more basic security controls and they continue to do that rapidly.
So if you are playing the whack-a-mole game in security and you've got one file that looks like this and then you search your environment and you don't see any other copies of it and then you get another alert that did the same thing, but the file looks completely different, maybe different name, maybe there's something changed under the hood, but that's the automation they're using to try to evade detection faster than the detection can identify that it's changed. Incredible. Do you have any evidence, seeing that there's an AI on the other side trying to attack us or like, is, is this a theory or have you guys ever come across any evidence to show that there we are actually fighting with a very smart and fast ai? Well, an example that's really hot right now is ChatGPT. Have you popped any questions in ChatGPT yet? Yeah, so one of the big attacks we see today is social engineering and that may be someone emailing you, texting you, calling you, trying to trick you into giving away your credentials. Now we've got publicly accessible language-based machine learning and AI that can have a full conversation with you and you may not even be able to tell that it's AI behind that.
And a full conversation with a million people at the same time. Correct. And, probably something that humans would never be able to do. Correct. So now they will, if they go for 0.1% at that rate. Interesting. Yeah, well Google's got the service where you can get it to call a restaurant and make a reservation for you. You just hit go and it calls the restaurant and they don't even know that they're talking to a bot. In some cases.
We're already seeing that happen in day-to-day life. So we can only assume and we definitely have knowledge that attackers are using the same technology, they're very forward-thinking, they're always looking for the next best thing and the next best thing right now is more machine learning and more ai. How, how are you gearing up for that age? Or are you already geared up? The biggest thing that's changed for us is the amount of tools that we have to use.
We have a history going back five years of building this SOC approaching six years now. And the amount of icons we had on this slide that showed the technology and how it integrates has actually been decreasing. We've been doing a lot of acquisition activity, but we've been merging features and products. So now is the SOC. We have very few tools that we need to manage making our jobs a lot simpler and the simplicity that we're introducing is allowing us to focus our energy back into the response. working those alerts and hunts rather than trying to manage a suite of tools that doesn't talk, or that is difficult to integrate. We're able to out of the box, get really good data and take action on it with minimal input from us.
And that is going to be the key component that allows us to scale. We're very confident. We're at the point today where we could get tens of thousands or hundreds of thousands more alerts. We could double in size again as a company, but we already have the technology that's going to be able to handle that increase in volume without having to change anything today.
So speaking of acquisitions, um, does our SOC activity inform our acquisition targets? <laugh> what question to ask in a public interview but are, you know, in your experience, are we, you know, have we been acquiring companies that came up as a need in your work in the security operations that this is something that we gotta have IT and every other company have to have it? Like ASM being one of them? Our product and engineering teams from day one have been very interested in what our SOC team sees as valuable. We literally are their first customer. And so when they're, noodling over a roadmap item or new feature, rather than just guessing, they will come to us and they'll ask us where our gaps are. They'll watch us use our existing tools. So I know for a fact the feedback that we've given them has influenced where Palo Alto Networks decides to expand capability. Absolutely. And it's really special to be in that spot where you can help influence that because you could probably pick out a few examples of companies where, they maybe aren't listening to what the, people who are using the products are saying. And when you have the repeated feedback over and over and you don't adapt to that feedback, it looks like you're not listening, but I've never felt in our SOC team that we're not being listened to. In fact,
they will often ask for more information than I can possibly give cuz they're so eager for that feedback. And I think that's one of the main reasons we've been so successful in this platform approach in getting a sensor into every area of the network that the SOC needs to be successful. Because when our SOC team has said, we really need this, we could use it yesterday, those teams move and fast.
Yeah. What a great place to work at. Yes. Devin, thank you so much for the interview. Of course, my pleasure. Thanks for having me. Two amazing episodes and in the next episode we'll be talking about machine-learning and AI with Billy Hewlett, our head of AI research. Until then, hasta la vista.