Warning: State-sponsored campaigns target global network infrastructure
If a Telco network in the UK goes down bad things start to happen and they will happen very, very quickly, whether that's food distribution or energy management or a whole bunch of other things, actually this becomes a non-negotiable for resilience of a nation, let alone just an individual organisation. It's not just informing customers of the problem, it's like you guys have to do something about this. Goodness, how long have you been in security for any length of time? You still know that patching is the first thing you do and pretty much all of them are not zero-eight. They're all known vulnerabilities that somebody's not patched. Now again, people are like, wow, patch is stuff and you're fine.
We all know patching is very, very, very hard. So I think it's the same, it's exactly the same kind of conversation really. Hey everyone, it's David Bombal coming to you from the Cisco Officers in London.
Mark, welcome. Mark, tell us a bit about yourself because we haven't met before. No.
I believe you can't tell me everything because you'd have to kill me, right? That's basically it, yeah. Yeah, I am James Bond. I like it. No, no, no, I'm really, really not. Where do I start? 26 years in Cisco.
I joined Cisco as a graduate back in the late 90s and I've kind of been here ever since. It doesn't feel like 26 years. I've done lots of different roles inside Cisco, mostly in and around security and mostly in and around government.
So yeah, that's what I was sort of hinting at, right? Yeah. Do you do a lot of advisory stuff for the UK government and stuff, right? Yeah, so I guess over the last couple of years I moved into a role, our team that we know as security and trust organization inside Cisco. We call them STO because we like our acronyms, as you well know. And what STO do is a couple of different things, but primarily the focus I have is that security and trust look at the security of our products, not the features and functions, but how we build them, which is really important to a lot of our customers.
And then the kind of regulatory landscape, so that's where I then spend a lot of time with the UK government because it's about how the market is going to be regulated from a cybersecurity point of view. And therefore what we then have to do as an organization to sort of go and meet that regulation for all of our different customers across the UK. So what we're talking about today is, and I found this on Twitter or X as you call it. I saw you repost this, right? Yeah, okay, yep. That's good.
So I think that sums it up really nicely because I remember people posting, my route has been up for five years. My route has been up for 10 years. Can you beat this record? And that sums it up nicely, right? Because that's a big problem. Yeah, it is.
And on the one hand, you feel a huge degree of pride, a piece of kit that your organization has built and sold has actually got such reliability that it can be up for five and 10 and 15 and 20 years. That's amazing. And that is a kind of good news story. But of course the flip side is, I mean, nobody's patched it in five or 10 or 15 years. And of course, that's less of a good news story. And I think that's part of the challenge.
And whilst I know we'll spend a lot of time talking about this. And of course, the problem is is the folks don't always think about patching of network kit. They think very much about endpoints. And even that is often not done particularly well in a lot of areas.
And it's even less well done in the networking space. Because I mean, there's this document on Cisco's website that was reading state-sponsored campaigns, target global network infrastructure. So I mean, rather than us reading it, I'll link this below for anyone who wants to read it. Can you tell us what this is about? Because this is a big emerging thing it sounds like.
Yeah, well, it's interesting you use the word emerging because my first experience with what you'd call router malware or attacks against routers is probably 15 years ago. So this is not a new problem. But it's one of those that sort of bobbles to the surface when there's a bit of an instant and then drops back. But in essence, it's this idea or the recognition that we're waiting for. A modern day router is just the Linux box. I mean, iOS Xe is Linux running BinOS, which is Cisco iOS as a demon.
And so all of a sudden, you've now got a different paradigm than maybe you would have had with classic iOS, which was a bit more complicated, but still a computer ultimately. And therefore, it suffers all the same things that you windows and you Mac and you're Android and your iOS devices all do, which is that vulnerabilities are discovered and need to be patched and maintained. I think we have a unique problem in the networking space, though, in that not only do people not patch, but also people hold onto their equipment much longer. And I think that's a sort of unique to that domain problem where people will run their networking equipment for five, 10, 15 years well beyond the present of life and its end of support date because it just works. So there is always that tension of, well, if it's kind of working and I don't need to go any faster. 100%.
It's not broken down fixed. That's it. So it's already an emerging problem, but I think we're on one of those, those not ebbs, those flows at the moment where it's coming up. And I think that the fundamental challenges we're starting to see a shift in the same way that we see the threat landscape shifting all the time. We're seeing a shift towards people or at least we're seeing a shift towards discovering that there has been a pivot towards targeting of network devices in particular end of support, end of life, on patched and so on and so on. The easy sort of low hanging fruit arguably that exists out there on the internet.
And of course from there, there's lots of things you can kind of do. You can use it as a jumping off point. You could use it as a point of capture and intersection for data because that's what readers do, right? So yeah, so we're definitely on one of those flow moments with this at the moment. So I mean, is it because it's nation state, is that the new change or has that always been an issue? No.
Why is it suddenly like making the news kind of like now? I think it's probably a combination of the things. We're definitely seeing the attribution. So the UK National Cybersecurity Centre, the NCSC, directly attributed Russia, Russian APT Actors against something called Jaguar Tooth, which was discovered earlier in this year and that malware, as you say, was attributed to Russia. So we can nation state in a way. It's a little bit of sensationalism, I guess, because to a certain extent that helps drive the message.
If we say that our nation state actors, China, Russia, the two that are often called out in these type of actors, then that tends to bring attention to it, which is what we need. And I think it's also the attack of motivation. You've got to think about what the motivation might be to attack a networking device as well. Is your average cybercrime gang going to go and target a rooster or a switch? Is it going to generate revenue for them? As you well know, that's the big motivation in that space. So I think it's probably a combination of a few things.
I think it's making sure there's the awareness there because it is an, it's not an emergent problem. It's something that's been around a while. But the nation state thing, I think, just helps drive the awareness.
I mean, it was interesting reading us because they're talking about the old, old, famous SNMP, right? Yeah. 100%. And that's to the point that I mentioned earlier. Now, this is a vulnerability from 2017.
Yep. An irrellatively easily exploitable one of that. And so it's one of those difficult tensions. If you've got devices out there on the internet, that we all know things get connected in, they get configured, they're given point in time, and then sometimes they just get left.
And they just continue to work, doing their job. I think in that case, I even had to Google what the device was. And just so despite my 26 years, I couldn't quite, it's an AS, which means it's probably an access server. Oh, yeah.
No, it's a dial-up box, basically, one of the old dial-up boxes. So yeah, it's, that's what they attacked, right? That's what they attacked. They targeted one of the old access servers. So for those who've got kind of pedigree in Cisco, if you remember the old dial-up mode and banks that we all remember and, you know, and love for those who are of a certain age, probably not many of your viewers is I'm guessing, but probably you and I will remember it.
That's what it was. And there's so much of this kit out there. I mean, you just have to look at Showdown and realize how much Cisco kits expose on the internet. If it's on the internet and you can poke it, that's probably a bad sign anyway, because generally you don't put interfaces that are easily accessible, you know, telnet SSH on the internet. And that's the problem. Their SNMP was open to the internet from a ability bank.
I mean, I'm glad you said that because on YouTube, you know, we have, I have comments on YouTube. Let's say, comment, most common comment is, this is dumb, this will never happen. Yeah, and, you know, I'm of the opinion that I think users aren't dumb. Yeah. You know, I think there's, you know, if we were all back 10, 15 years ago, people would say, well, it was dumb users. And I've had the privilege of working across lots of different sectors of the public sector and very few of them are dumb.
But what they are is resource and cash strapped and sometimes skills poor. And those things aren't dumb, you know, I don't know about you, but I can't perform open heart surgery. No, no, no, no, not normally. No skill you have, but yeah, if a cardiothoracic surgeon rocks up and says, I don't really understand how to secure this thing, you're a dumb user. They're really not dumb.
But the problem is, is like anything, we expect people to inhabit the world that we do, and therefore we build things with a mindset that assumes that, well, of course, why wouldn't you know how to click a certificate and make sure you understand what that means and do this? And why do we make it so hard? And to a certain extent, this is a similar kind of problem. You know, you might imagine that your average network operator ought to know better. But they're just like you and I, you know, people make mistakes. They're rushed. They're, you know, doing a thousand different things at the same time.
And so it's easy to write down what's the next task, you know. So that's the challenge. I think that's really good that you said that. Can you get more technical though? Because here they talk about SNMP, right? But the different versions of SNMP. And this was, was it SNMP version one? This was version one.
Yeah. Yeah. Version one, which goodness is quite old.
It's not secure. Clear text. Clear text.
All you need is the community string and a way you go. And that's private normally. Normally public private, especially if you've probably, if you've put SNMP on the internet, you've probably left it as public private.
And that's the problem. And again, these things, you know, we've got this massive legacy problem with a lot of the Networks where people are just leaving things in the ground too long and not saying, wait a minute, I am running SNMPV1 or VTC, which still, you know, people think is a bit more secure and it is a little bit more secure, but it's not really. V3. You know, but I mean, goodness, I couldn't even tell you whether the version of iOS they were running.
I think it was a version 12, possibly I think, which again, if you've been around for as long as we do, we know how old that is. Was V3 even around then? Or was it bleeding edge? And that's the problem. So, you know, move to V3, you solve a whole chunk of those problems. But we all know that we should probably use SSH, you know, SSHV2 for all of our interactions with the router, but how many people just stick with Telnet because it's easier. Yeah.
It's interesting that you say that again, because again on YouTube, a lot of the comments I get is like, it's dumb to show Telnet, no one's going to use Telnet. But reality is different, right? Oh, 100%. And I think it's unfair to say that people are dumb. There's a whole bunch of reasons.
Sometimes it is just naivety and sloppiness and laziness. But sometimes it's, you know, again, they just don't know any better. Or there's a pressure or actually 10 years ago when it was designed, Telnet was okay, because we didn't see the same kind of threats as we did. Or maybe they're tooling and their management environment, which is legacy, still drives them towards using Telnet.
There's a whole raft of reasons why people continue to use them. Sometimes it is just, you know what, they've just been a bit dumb or they've just been a bit naive. But more often than not, there's some reason for it. And threats change, right? Threats change, management environments change.
And again, you know, these networks can be hugely complex. And if you've got a network admin or a couple of network admins that are doing this, as well as doing all the desktop patching and maybe dealing with endpoints and maybe dealing with IP telephony and wife, how much time in the average day have they got to think, right, okay, I've got to go patch that network box, which is still doing what it's doing and it's still working fine. It's easy to get missed because like you said, you plug it in, you configure it and then people forget about it because it just works. And that's it. And whilst again, you and I and probably many of you would sit there and say, what about our home routers? You know, well, I don't know, but my mum took our home router out of a box and plugged it in. And she shouldn't have to do any more than that.
But the problem is we say, well, you've got to change the wifi password, you've got to change the admin. Why should my mum, who's 70 on Tuesday next week, this week actually, and why should she have to worry about all these things? She should be able to just plug it in and it should just work and be secure. And so there's a whole different conversation, which I know isn't necessary for today. This whole secure by default, secure by design, mantra that's been sort of, again, there's been of an upswell in that as a mindset, another big part of my role is, how do we make this stuff easy to consume, secure out the box and secure by design and secure by default? That's a big, big focus because you eliminate so many vulnerabilities and so many problems. You know, I know I'm going slightly off track, but buffer overflows, right? Buffer overflows, you write too much into too small a buffer and bad things start to happen. Now that bad thing might be the device crashing or a box crashing, but if I can exploit it in a particular way, you know I can then get, come on, I could get execution.
Buffer overflows can be mitigated through lots and lots of different software capabilities. You know, we can implement a whole bunch of non-exgutable stacks and SLR and all these things are built because fundamentally memory safe code isn't, you know, or memory safe architectures just aren't there because we didn't have to worry about it 50, 60, 70 years ago. Maybe going a bit too far back, but you know what I mean, aging myself now. But there's a whole project at the moment that's come out, I think of Cambridge University and an arm where they're looking at memory safe CPUs.
I think they call it the Morello Cherry. If you could take every single CPU out of every single piece of hardware we've got and put one of those in, you would eliminate an entire class of vulnerability, an entire class, which probably still accounts for 50, 60% of the vulnerabilities we see every single year. So yeah, there's a whole whole different conversation. I don't know, it's nothing to do with network devices and, you know, nation states.
But it's good to know where it's going though. I mean, it's good to know that stuff's coming because that's, I mean, why haven't we solved those problems? It's been around like you said for a long time. Because it's a big problem to solve, right? It's a legacy problem again of, well, wait a minute. So we've now got to rip out or at least plan in the use of an entirely new hardware architecture, which now needs new compilers and new dev chain, new children, new code.
All of this stuff really isn't easy to shift, right? I mean, we're struggling to get people to patch things. And in most cases, that's pretty straightforward. Well, we're talking about an entire ecosystem of change and that, that, that is a multi, many multi-year kind of shift. So in this document, they talk about, and the document from Cisco, they talk about SNMP and then they also mention other management protocols. So are there better ones to use today? Yeah. So we talk about netconf and restconf for being two alternatives.
They're just more modern paradigms again. You look at it. Yeah, they're all encrypted. They're more efficient.
They're more modern alternatives are using SNMP. But again, it's that legacy of, well, what about your tool chain? Does that support restconf for netconf as well? This is a problem because I mean, would the recommendation like just take out listening people, that's not going to work for everyone? And that, again, is, is part of the challenge. You can't just switch this stuff off in the same way you often can't just switch telnet off. You might have a script that's running around, going around sucking up all your configs as part of your config management. So that means now somebody else has got to touch that and they've got to touch every device to turn off tell that.
Okay, and what's the actual net gain? Well, it's more secure. Well, yeah, but if I think that I'm never going to get compromised, then why would I really go burn two, three, four days of dev time to build a new tool changement? Well, I would somebody do that if they don't think it's going to be a problem when I've just got to patch all my endpoints and I'm probably then better off or I've got to implement MFA and all these other things that are probably higher priority in my stack. But yeah, so I think that's it. What a lot of these boils down to is it's exploiting the legacy. You know, it's that low hanging fruit that's been put out there.
SNMP, tell me. You know, all protocols, you know, if you've got HTTP left open on your, on your router or your firewall or whatever it might be and somebody's left it because somebody might have done some management on it at one point and maybe not switched it off. You know, RDP. You know, let's probably not even open that particular kind of worms, but again, another classic problem, yeah, SMB.
Yeah. You know, let's stick SMB on the internet. Yeah, really? But somebody will have done it maybe accidentally, but often time it lived in a deliberate decision because they needed to do something else.
And they didn't think about security ramification. So I mean, in the document they mentioned stuff to look for, one, there was interesting GRE tunnels. What are they like? What are the, like, things to look for? Like, I mean, we see this as a problem. Yeah, Cisco, is trying to raise awareness to this.
And I believe there's a whole, what is it, coalition of multiple companies, right? So the center for cybersecurity policy and law, who are a group in the US essentially has instantiated this consortium really to address this problem. So we're a founding member along with the likes of Juniper and Fortinet and BT and Verizon. And there's one or two others in there.
And the idea is sort of recognizing that there's a problem here. You know, we've all got this problem. It's not just a unique problem to Cisco. But recognizing there's a problem and this coalition is really forming around trying to, I guess, get a sense of the scale of the problem. And then actually come up with a set of meaningful, tangible things that organizations can do to try and do something in this space. Because again, it's another one of those.
We've been talking about patching these stuff for ages. Yeah, it still doesn't happen. So we've got to try and think of some new and novel ideas as to how we actually move the needle on improving the resilience here. Because fundamentally, networks now underpin our economic well-being.
You know, if a Telco network in the UK goes down, bad things start to happen. And they will happen very, very quickly. Whether that's food distribution or energy management or a whole bunch of other things. So, you know, actually, this becomes a non-negotiable for resilience of a nation.
Let alone just an individual organization. I'm glad you mentioned that because in the document, I'll put a list up on screen of like vendors. It's like everyone who's well-known in the space. Yeah. That there's this Russian group that we're trying to attack these devices, right? Yeah, it wasn't just Cisco. It's like everyone.
Yeah, 100%. And I think that's it. It's an important point to say is that, you know, I suppose we have a somewhat unique position in the A we've been around probably longer than most of the others. You know, one would imagine there's probably more of our boxes out there.
Others might disagree, but I think we've probably still got a big legacy footprint out there. And that's why, you know, in the same way that a lot of people might target Microsoft, maybe versus Mac OS in the endpoint game, it's a game of numbers, right? But it's certainly not unique to us. And in fact, you know, outside of us in the enterprise space, there's definitely been a lot more focus on especially VPN endpoints. And you'll have seen some vulnerabilities announced in some of, you know, some of the other. So tell us sorry, just for people who haven't seen it.
Yeah, so, so I think, I mean, good and goodness. I remember 14, it had one where they had vulnerabilities in their VPN stack and their edge devices. There was another one. I think Juniper had some as well. I mean, we've had some, but the point is is that these things are on the internet.
They're very visible. And so people are using them to again, pivot into enterprises as a hop on point. So I mean, there was talking about not having to affair on VPNs. And that's what they've been targeting, right? Yeah, so that one's that one's very much about, I suppose exploiting creds. Because obviously if you don't have your multifactor, then it's easy.
You've got your creds and where you go. The other side of his actual vulnerabilities in the VPN implementation, which then is being exploited to either get access to that device and then hop in. Or otherwise, you know, effectively pivot yourself into the organization.
And that's that's the sort of two sides of the VPN as a, as a, as an exploitable endpoint. And we've seen a lot of that noise. The, the other side has been targeting sort of small office, home office devices as well. So we do see people like D-link and TP-link and links and, you know, all the other brands that you tend to see on the end of broadband lines.
And the motivation there is, because they're not enterprise focused, the motivation there is often the attackers want to use them as effectively part of their, their, their attack chains. So they'll proxy through them. No, proxy through them. Okay. So I think to the corporate network, so rather than me going direct to you, what I'll do is I'll compromise a thousand home routers and I'll bounce between them all.
So therefore tracking me down and I might pull my C2 back, my control traffic back through that. You know, again, you might use them for DDoS, but if you're a nation state, your motivation is going to be slightly different. And so therefore you might use that to hide your C2.
And again, that's documented in some of the, I think sister talk about it. And I think we talk about it in some of our articles as well as, you know, there's the sort of enterprise endpoint, the Cisco router, but also then those other endpoints become interesting as part of that broader landscape of compromised network devices. So I mean, the coalition is to try and is to bring visibility to this problem because I mean, we always hear the noise of IoT or we don't end point, you know, your PC, you need to get patched, whatever your phone.
But we don't always hear about this. So this is to try and shine a light on this. Is that right? It's definitely shining a light. And I think it's shining a light with a view to the action, you know, something that's actionable at the end of it. Yeah, because I want to get to that. So what, what do we do? So Cisco are part of this coalition, a government so involved, but like what is the government street do? Like someone like me in an enterprise, what should I be looking for? What should I be doing? Because some of the stuff mentioned, yeah, but it's complex.
I challenge complex. I mean, it's, it's, it's, it's, yeah. And I think that's part of the part of the problem. Again, you've got a stack of a thousand things to do when you walk in the door every morning.
And, you know, there's going to be arguably, there's going to be a bunch of things that are going to be more impactful from a security point of view, whether that's patching your endpoints or turning on MFA or looking at some logs and so on and so forth. Your network stuff is probably going to be a little bit down the track. So part of that coalition is absolutely about saying, you know, actually realize these things can be compromised. People are actively targeting. And so you do need to kind of raise the priority.
But in terms of what can be done about it, I mean, first and foremost, patching, you know, it's the most dull boring conversation in the world. And it's the one that we always troughed out because it's an easy answer. But that's, it's, it's an easy answer.
And it's the right answer, you know, the, the, the Jaguar tooth malware was exploiting a vulnerability from 2017. Drop on a new version of iOS and that's gone, you know, and okay, dropping on a new version of iOS is in a lab. That's really easy, right? You know, it's a line of conflict. Yeah. And you're done. And then you've got to reboot.
Real world. Yeah. And the reboot thing thing. Yeah.
And some might say that Cisco code occasionally has one or two or undocumented features in it. And so therefore moving from version X to version Y might introduce some challenges in your network stability. I'm not going to comment on those things.
I'm sure you're a few as well. But that's the reality. That's software, right? Software is complex.
And so just simply patching or or in case of iOS or Xe and these things, you know, rolling the entire image, these things are, are easy to say and sometimes hard to do. And it's also what happens if that router is, I don't know, on a on a low bandwidth link in the middle of a field. You know, so what do I roll an engineer to site or I try and drown load it? What happens if it goes wrong? There's a whole plethora of reasons as to why people don't do that patching.
But it still remains our biggest defense against a lot of these kinds of attacks is just remain to be, you know, being up to date and maintained. You know, second things look off the creds. You know, one of the a lot of the compromises we've seen, we talk, I mean, Jaguar 2000 isn't, isn't, isn't one of them. But there are other examples that we, we don't name, but there are other examples where we've seen compromises against devices and it's not been because they've exploited some, you know, old vulnerability.
It's because the creds have been compromised. Now, how have they compromised them? Who knows? It's very, very hard to know. But we all know that people will, I don't know, sync creds with Google accounts and then Google accounts get popped or maybe they've got the monotachic server and the tachic server has been popped because somebody's managed to get in through another way and and and there's a whole bunch of reasons how creds can get popped.
So looking after those creds, again, easy to say sometimes really hard to do. But again, putting MFA in front of some of that can be, can be very helpful. If you're using jump boxes, make sure jump boxes are all hardened where you're doing, you know, using jump boxes for management of these network devices.
So that's kind of the credentials and you're patching sorted out. And then it's, then it's the rest, um, using secure protocols. We talked about SNMPV1 versus V2C and the V3 netconf, restcomf, um, you know, using modern equivalents or modern slash secure equivalents of these older, older telemetry protocols has got to be fundamental. And again, there are tools out there.
There's probably a stack of open source tools. I'm sure that we'll just go off and just slip, configs up and tell you, wait a minute. You've got SNMP open here, here and here. Just go change it again. Easy to say, not always easy to do. But they are that this is the reality of being, you know, being secure.
It doesn't need some God box being purchased and God bit. We love people to buy God boxes from Cisco, but the reality is this is not a technology problem. This is, this is a process and policy problem that people have got to get their heads around and then just in the same way you wouldn't consider running, I don't know, Windows 98 still on your laptop and stick on the internet. On people do, but it's, we know it's not a good idea.
It's the, it's the same argument and it's getting people into parity with that mindset. So I think those are the things, you know, patching, cred management, MFA, secure protocols, don't put those protocols on the internet. If you are using in-cycle protocols, don't put, don't point it at the internet and allow it to be successful. I'm glad you mentioned so down. Yeah, because, because how easy is that, right? It's really, really trivial to just go on showdown and find Cisco iOS boxes that are on the internet and there are many, many out there. Not so many in the UK, I've got to say, I've done a few searches there, you know, but there are many overseas and those overseas ones are probably connected to organizations that might have a UK footprint, you know, because that's the other thing is that that might just be the box somebody forgot that's connected to a defense contractor or a big pharmaceutical organization.
And that's it. We're, you know, we're away at that point. It's interesting that you mentioned the home as well, because that's a, I kind of alluded to like that's sometimes how people get into the corporate network. Same thing as like a remote office or another country, right? Well, that's it. I mean, if I've got your home router, then, you know, again, especially with hybrid working and more home working these days, you know, once again, it's, it's, you're now extending that enterprise boundary into the home. And so the security needs to go with it. And we know that that's going to be a less secure environment for a lot of people.
And we know that people tend to be, you know, much more relaxed or at least less diligent with their home credentials. And again, because you're using your home, you know, your corporate laptop at home, you connect it to your corporate Wi-Fi, which means connected to your router, which means blah, blah, blah. You know, again, if your company's patching isn't up to date, that's, that's now a path into that organization. So now you always have to temper these things, though, because the flip side of this is actually doing some of that is still non-truthy or, you know, we've always got to remember the, the biggest proportion of kind of criminality. Let's just call it criminality out there on the internet still remains to be cybercrime, ransomware, financially motivated, you know, the nation state stuff generally isn't financially motivated, but of course, by definition, it then represents a much smaller portion.
So the likelihood of somebody going after you from a nation state point of you to pivot through your home router to get to you, I think I'm going to do that. Or are they just going to go bribe the cleaner? I know what I do. I suspect you would probably do the same. Why would I go through all that technical heartache when all I can do is just bribe somebody? We always love the Mr. Robot slash technical stuff, but yeah, the technical stuff's great. Don't get me wrong.
I love the technical stuff. Hey, anything that's on TV, usually portraying cyber in technical stuff, except Mr. Robot. He didn't do too bad a job. But the reality is that social engineering component. And I know that can be quite a diverse, you know, kind of topic. The reality is is the bad guys tend to want to use the minimal amount of effort possible to achieve their aims.
And I think a lot of times when I've spent time consulting with clients, you can get very kind of caught up in the wrong set of problems. And so you kind of need to get it back into, well, I'm really going to kind do this through your Wi-Fi network. I'm not going to do something else instead like, I don't know, drop a fishing email on your on your desktop and click through. So yeah, it's it's always an interesting one. And that's the thing on the on the route of things.
The this isn't an epidemic, you know, this isn't every single route just compromised, but it is something that we need to start to think about more clearly and more deliberately in a way that we maybe haven't done in the past because there hasn't been that reliance. And a lot of this still stem back to that. These things are now part of our critical national infrastructure. So we can't we can't we can't ignore it. Because you know, this is everything up to an including life. You know, I spent 10 years in my career working with the NHS.
You know, when their networks go down bad things really, really do happen. You know, we saw WannaCry back from what 2016, you know, and OK, that wasn't people can keep compromising route. The impact was very, very clear. So that's the kind of impact and that's why we've got to start talking about this and get people aware and taking action more importantly.
Talking about it's fine, but the action is the important thing. So the actions are what you mentioned before. The actions are what I mentioned before.
Credentials, patching, insecure protocols. And then there's also I suppose the bits that I didn't get on to because I got sidetracked like I do. But the other bits are things like monitoring.
You know, GRE tunnels you mentioned. Yeah, it's good to say because in this document they mention GRE tunnels and a few things that are like interesting. Yeah. So monitoring for weird stuff like that, right? Yeah, I mean, you know, if you don't... If you should never GRE tunnel if you're in the UK necessarily going to somewhere in the forties perhaps. Yeah.
Or if you do, you know, maybe check if it's normal. Yeah. And that's kind of the thing is, you know, we've got plenty of tools out there.
You know, NetFlow obviously been the classic one to be able to, you know, throw into an anomaly detection tool that will start to tell you what isn't normal. Yeah. Is it perfect? No. But it's a downside better than just, you know, walking around with blinkers on.
You know, so if you've got GRE coming out of your router and it's not configured for GRE as far as you know, that might want to ring a few alarm bells. But again, similarly, if you've got traffic flowing to a set of IPs that you're not familiar with or a set of IPs that are geolicated in a particular country, again, they're easy to spoof. So they're not, they're not the smoking gun. But they all build that picture up. But the only way you're going to do that is by catching the information and reviewing it regularly. So again, people are doing that for endpoints.
There's lots of tooling out there for endpoints. This is all about saying actually we've got to pivot this and think actually there's a there's a bigger holistic picture here we've got to do. We've got to we've got to start to look at. I just want to, I mean, you can get into the, into the weeds here. You've mentioned, mentioned monitoring.
Is anything else you want to specifically mention? Tacax was the big one. Yeah. That you mentioned jump posts.
Yeah, jump posts, Tacax. Yeah, I mean, all of those things are all about monitoring and hardening. You know, if you, if you're not maintaining the integrity of your jump posts, that's a problem. If you tack X servers running a version of Cisco, SQACS from, you know, from the early 2000s, probably not a good thing and so on and so on.
They all, it's all about applying those basic principles of cyber hygiene to all of these different elements because if I'm an attacker and I want your network, I'm going to go for, you know, if I want it for not just ransomware, I'm going to go, well, even if I want it for ransomware, I'm still going to have to go go for your crangles, which is your AD. You know, if I'm in your network environment, guess what? I'm going to go for your network management system. That's what I'm going to pop. You know, it's all the wins, anybody, you know, that's actually that that's it.
Keys to the kingdom at that point. And that's the same kind of idea is that they sort of management plain, you know, we talk about network management, you know, management planes and control planes and data planes. We talk about management plane. That management plane is there's often a lot of focus on security, but there's, there's always, there's always bits that are missing.
And they're, they're often missing because people have had to make a compromise. Oh, I like, I mean, they've got to build an out of management network. That means a whole different set switches and roosters and I've got to plug it in and put that's a lot of work and a lot of cost. I guess what people don't like spending money on security. It's a fact. It's interesting.
You mentioned SolarWinds because SolarWinds were supply chain, right? And there's this, the famous story of the NSA embedding stuff into Cisco riders. There are stories in that area. Yes. Yeah.
So I mean, but we were talking offline reality versus like the movies, the chances of like NSA putting stuff into riders or nation states doing that is low compared to them just like fishing. I couldn't tell you honestly, you'll kill me. Yeah. Let me just check. But no, but I mean, reality in general terms, right? They shouldn't be looking like opening every rider and looking at the inside.
That's pretty hard, right? Again, am I going to spend 5,000 pounds or 10,000 pounds on on a low paid cleaner and ask them to drop an implant in a desk area versus taking a device off the shelf, doing stuff with it, making sure that stuff actually works and then make sure it gets to your device. Oh, and then I've got to get data off it. That's like a lot of work to me.
Does it happen? Couldn't tell you. I can't. I couldn't tell you. I just, I don't know.
I don't. I'm not in that. I'm not in that community.
But the reality is is that if I'm going to do stuff, that's I'm going to do all this stuff first. So you mentioned there's a lot of stuff on the list of important things to do. And this is further down. Yeah. So this guy's not falling suddenly today, right? This has been going for a long time.
It's just something to be aware of and we need to relook at. Yeah. I think it's just, it's all about that diligence level. You know, it's, it's an area of the environment that people just haven't historically looked at before. Either because it's too far down the priority stack or actually they just aren't aware that these are general purpose computing device or especially computing device, but they still run an operating system.
They've still got memory of this or what compute. And therefore, but they don't join the dots that actually you could in theory, compromise it and get it to do things that you want it to do. And I think that's often the realization is you can talk to clients and it's like, did you know that this can happen? Really? There is that.
Oh, but I thought it was a router and therefore somehow magically protected. Exactly. There's no shroud of invisibility around this device. It's still just a device.
And I think that's it. It's not, you know, we're not, we're not talking about fire and brimstone raining from the heavens or anything here. This is just be aware.
This is just another vector that the attackers pivot. And we all know that the attackers just pivot onto different things, whether it's supply chain, Alasola wins and a whole bunch of other things that have come up and we now need to think about that. And this is just another one of those next pivots.
Is everybody going to pivot to it? No, because it doesn't, it doesn't always buy you what most people are interested in, which is the kind of ransomware classic cybercrime part of the thing. But it could be an interesting route in if I've compromised a network device. Guess what? I could then use that trusted relationship. Because again, there are often a trusted relationship to pivot further. So it could be a stepping stone. And that's what we have seen.
I think it was the sister report on black tech that came out a couple of weeks ago. I think they sort of talked about compromise of router being a, you know, being a jumping off point for the, for the, for the, for the attack to get to where they ultimately want to get to, which was deeper in the network. So it was almost a means to an end in a way.
Mark, I'm always slow with these things. Explain to me why is this suddenly being focused on? Is there like a catalyst that started this or was it something that you guys have been doing for a long time? I mean, you've kind of alluded to that. But why now? Why are we raising a spotlight on this? I think, I think it's, I think it's an aggregation of two or three different things. You're right. We have been doing this in my introduction. I talked about the security and trust organization.
So that organization, the biggest part of that is our secure development life cycle. So that team is responsible for in a way, hardening our devices. So they're the ones that put in place. All of the coding guidelines to make sure we're using safe sea and, and we're putting in a SLR and non-exec. They do all of that.
Now, they do all of that because we know that these devices could be targeted. And we've been doing that for probably 15 plus years. Yeah.
And a lot of that investment here, like the one secure booth, the trust anchor module, sign firmware images. There's also some really cool stuff to do with chip guard where we can check the identities of all the basics in the box to make sure that they're not counterfeit and so on. There's a whole raft of effort that's gone into that. And part of the frustration is nine times out of 10, our customers don't even know there's a problem that we're trying to solve there. So this is not new from our point of view.
And we've known this stuff has been going around for a long time. I think what is new is that we have seen two things. So we saw the NCS report on jaguar to specifically nation, you know, talking about nation states, sponsored attacks. You know, that's, that's a catalyst in of itself. You know, we look at the current geopolitical landslate, you know, we've got conflicts in in the Middle East. We've got conflicts in Europe.
Those in themselves drive a certain set of concerns and new threats. So I think there's that that's an element. And I think we also found in our own data and our own experiences with customers, we started to discover more of this ourselves. OK. So we saw cases getting raised into Cisco, tech, investigate, investigate, this actually looks like compromised devices.
And you'll see that in the Talos blog, which I guess you'll link to. So you have to do your job for you. Thanks. It's all right. But the Talos blog that was launched around the same sort of time as jaguar to talks more broadly.
It talks jaguar to talks more broadly about the cases that we found ourselves and talks about the kind of, you know, the trade craft and the behaviors that we've seen. And so I think if you kind of put those two or three things together, you get a point in time. Oh, and I think the other thing is the criticality. You know, whilst you could argue that networks have been critical for a long time, I think again, so it's covered. It's just escalating 100%. So you kind of take all those things, throw them into gear.
And it's like, actually, we really do need to start to do something meaningful because the market hasn't shifted. We've still got customers out there that are running equipment that hasn't been patched. It's got 10 year-op time. So nothing's going to change until we start to try and put some focus on it and actually come up with something that's sensible that people can do. That isn't just go patch because go patches there.
But well, how do we help people get to that point? How do we make it easier for people to get to that point? So that requires effort on our behalf and industries behalf as well. In the same way that you saw with, you know, goodness, patching Windows, no fire, Windows 98, Windows 2000 again, was a painful process. Now, I'm not a Windows guy, but I know my Mac, it patching's a relatively trivial task.
And certainly my iPhone just happens. I don't even think about it. Magic happens and it gets updated and I don't need to worry about it. And so I think the industry part of that coalition is how do we take networks further towards that direction? So it's there. So I think that's those are the three things that underpin why we've needed to do it now.
Why now is a good time to be doing it? And you mentioned industry. So it's industry plus government, right? Industry plus government. Like UK government and US government. Yeah. So I think what we'd call the five eyes. So, you know, UK, US, New Zealand, Australia and Canada, I think BSI are in there from Germany.
And I think France might be in there as well, but I'd need to double check the paperwork on it. But again, it's that that's where you need that bit of a push pull. Again, we are a big part of my role right now is working on telecom security. So the UK government came out on the telecom security act a couple of years ago to try and compel the telecom sector to do better things in cyber. Why did they have to do that? Because the market wasn't self-regulating and they didn't do it themselves. So guess what? We've got to come out with a big stick to try and drive the right behaviors and the right outcomes.
And I think what I'm not saying that government needs to intervene in that way right now. I think this is just that actually government may need to intervene. And maybe there are policy points that the government needs to think about putting out to try and drive the right behaviors. Maybe it's a set of good practice guidelines. Maybe it's just something to nudge the market a little bit to make it take action.
You know, in the EU there's a whole raft of new regulation coming out that is going to be very much a big stick. But that's where you need to kind of, well, it's all well and good. We've had trust anchor modules and sign firmware and all these are the great things for a long time. SNPV3 has been around for a long time. What is using it? Why is that? Well, because it's hard and sometimes you just need a bit of a push.
And I think this is exactly the same. It's that same tensions, which is why you need government industry when the kind of regulation space to kind of drive that action. I like that. So I mean, it's not just informing customers of the problem. It's like you guys have to do something about this. Yeah.
And goodness, how long have you been in security for any length of time? You still know that patching is the first thing you do. And it's also the one, you know, I don't know the figures. I think last time I looked at the CV database or, you know, sorry, I looked at general kind of exploits and issues. And pretty much all of them are not zero-daired. You know, they're all known vulnerabilities that somebody's not patched.
Now again, people are like, well, you know, patch is tough and you're fine. We all know patching is very, very, very hard. So I think it's the same, it's exactly the same kind of conversation really. Do you think we'll ever, and I mean, it's just your opinion looking into the future because you can't predict obviously. But do you think we'll ever get to the point where network devices, order update, like a phone or something? It's too critical, right? Well, I smile because there's a conversation I constantly have with some of the folks I know on the government side in particular. I think there is a, there is a Nirvana point there.
If you look at Miraki, you've kind of got that paradigm already there, built into the overall operating model. So, you know, my, my MX at home and my switch and my, my access points just update. I get an email at whatever time at 2am, this is all going to update.
That's great. And I'm a home user, so it's not that critical. But you've got an element of that paradigm already there. You know, are people going to be upgrading, you know, it's just great thousands, ASR 9Ks on an automatic update? Probably not. However, there are things that we should do.
I mean, you know, really simple things I've spoken to folk about is if I'm going to, God forbid, I'm on an XR console and an ASR 9K, how do I even know that my version of code is not current? Wouldn't it be great if there was a bit of Naga where they said, hi, I'm out of date or I haven't been updated in this many months? Now, you could argue that might just get lost in the noise, but it's something. How do we make it go red on a console? Not because it's got a network link failure, but maybe because it's just not been checked. And there's a whole bunch of things that we've done on cross-work trust insights, which could have changed its name since, but cross-work trust insights is one of our management tools in the SP space, which is trying to surface some of these things. You know, what is the integrity of the device? Are the known good values that are coming from the trust anchor checks? Are they what they ought to be? And raising that up through a kind of management port pool is, again, another step that we've taken to say, actually, this stuff is really important to kind of surface and allow people to see, because then you can make an informed choice.
Now, you can still make it poorly, poor informed choice, but I guess it's all about trying to get people to that point. But will we ever get to a point where these things are all too updating? I think we might in the enterprise space. You see automation, SD1, D-NAC, Meraki, all of those things are ranging people more toward abstraction away from the network device. You know, you're my skills on CCIE's of Comptia and, you know, they're becoming less and less relevant.
And I think as much as we might hold onto them dearly, they are the kind of things that are over my dead body. I absolutely would take the CLI away from it. Yeah, but if I've got network like Infrastructure as Code, Network as Code, DevSecOps and DevOps, etc.
And my network infrastructure is completely automated. Do we really need, you know, all of that stuff that we learned when we did our CCIE's? But I mean, I think to sum it up nicely is what this was a great reposited because, you know, the young people could say we dinosaurs, and we grew up in a different time, or got, you know, learning the CLI in a different time. And that paradigm of, oh, my ride has been up for 10 years, I'm a genius. It's changed now to, okay, it needs to be patched much more often. Yeah, it needs to be patched. And that march of automation and orchestration tooling that we've seen over the last five, six, seven, eight years, that is the right direction of travel.
You know, we, one of the, all the hardening guides and everything else that we talk about, you know, you put those things in place, you abstract it away and then not only do you get the visibility because you've got all of that abstracted into some management paradigm of some sort at a top level. But you get the benefit of them being honest, I know I've still got a box over here that's running SNMPV1, and I can then de-risk it because if I don't know it's there, I can't, I can't do anything about it. So yeah, that paradigm, I think, has to shift because otherwise, which is going to be playing whack-a-mole. So Mark, I, you do a lot of stuff and like you mentioned the telecom space, big, big networks.
Yeah. A lot of people watching, perhaps don't touch that network. So the kind of stuff you're talking about, that's required in such a big network. And it sounds like it's going to be filtering down to smaller networks, perhaps.
I think the principles of it should, yeah. Of the nation. Yeah, or touch the CLI. Absolutely.
And in fact, it's probably easier to do in those enterprise environments than it is in the big telecom networks because they're sitting on so much complexity and legacy. You know, I think in a, arguably a more enterprise-oriented network, I think those things are maybe, because of the churn rate and the swap-out cycle of those things. If you're not looking at multi-million dollar investments like you are in say a 9K with four-digit line cards, I think it's possibly an easier ask in the enterprise space. And that's where I think a lot of those, you know, can we get to the point where we update through a single click or we can update with confidence or it's self-updating? I think those paradigms are closer in that space. Mark, thanks so much. I really appreciate your sharing and, unfortunately, I survived this interview. You haven't had to tell me yet.
I appreciate it. We don't know what happens off camera.
2023-12-06 13:27
