How to Be a DevSecOps Engineer (w/ Aras "Russ" Memisyazici)
Russ: I love how marketing people paint what we do as magic. We're not magicians, we're not wizards, okay? So, for instance, the concept of "self-healing" that just really means you're restarting a service, ladies and gentlemen, that's it. Let's turn it off and on again. That's literally what we're doing, except we're doing that at a code level.
Ricky: Today, we have on our show Aras, or “Russ” Memisyazici, a cloud architect and a veteran in the world of DevSecOps. DevSecOps is a newer term that describes the continuous integration and automation between software development, cyber security, and IT operations, which requires deep mastery of all three fields. Russ has a bachelor’s degree in computer science and a master’s in information technology from Virginia Tech, focusing on cyber security and machine learning, along with a score of certifications on top of his 30 plus years experience. He’s a SANS instructor candidate, AWS solutions architect professional, and overall security evangelist looking to help others in the community to adopt best-practices in their organizations. Russ currently works as lead DevSecOps Engineer for a startup called Onna and is also an assistant professor of cyber security at California State University Long Beach.
In this video, we’re going to talk about DevSecOps, the security ramifications for cloud technology, and some ways and means of educating yourself towards becoming a DevSecOps pro. So Russ, thanks so much for coming on the show. Russ: Thank you for having me, Ricky.
This is an honor and a privilege. Ricky: So let's just get right into it and talk about your background, specifically, cloud architecture, DevSecOps. Can you walk us through what is DevSecOps exactly? Russ: It's a fairly new concept, right? And the hilarity is, it's now subsiding and giving way to now what's known as GitOps. But in a nutshell, the idea of DevSecOps is the idea that we're building security into the DevOps process.
For those of you that may not know, we used to have developers, and then we used to have systems engineers, a.k.a the operations people. And these were two disparate systems that were all trying to achieve the same goal. So circa 2016, 2015, somewhere in there, we decided it would be a really good idea in the industry if we could actually combine these two and actually do something well put together. So following the agile methodology, DevOps picked up a huge capability and huge backing by giants like Google, like Facebook, like Netflix, especially those that were geared towards the cloud, started doing DevOps and agile-based development practices. It was all well and great until they started realizing, we have this problem called "hacking" and we're really not doing a good job with security.
So maybe we should include security at the beginning rather than at the end. Which is what most companies tended to do. And that thought gave birth to DevSecOps.
So the idea of SAST, DAST, RASP, IAST, which basically are interactive application security testing, static application security testing, dynamic application security testing, and runtime application security concepts, were born out of DevSecOps. But really, at its core, what does it come down to? It's somebody that knows both the development side of the shop, as well as the operations slash systems engineering side of the shop that has a security focus and can lead both teams can be that bridge between the two to make sure that both sides are doing things in a secure manner. So a DevSecOps lead, in my case, I actually demonstrate what needs to be done.
And if necessary, I train people in this principle, and we help them get things done. So DevSecOps, therefore, becomes a key element in ensuring you have a secure and proper delivery of mission objectives from start to end. Ricky: Awesome. In cybersecurity, most people, first thing they think of is hacking and maybe incident response.
And infrastructure, DevOps, not seen as sexy, which in my opinion is really one of the more underrated aspects of cybersecurity. And so why do you think learning it is so important? Russ: So I'm sure you recall SQL injection, and cross-site scripting, and just your basic reflection attacks and so on and so forth.. Web application penetration testing, web application security testing, just messing around with the websites. If you think about it, the developers have now become both the developer of the application, but they also have to develop with a blue team mindset.
So they're on the defense. They're needing to make sure that their code is secure against these standard types of attacks. So, therefore, it's important for us to educate them in these matters so that they're aware, "Oh yeah, maybe I should put a boundary check on that for-loop," or, "Oh yeah, maybe I should make sure I'm actually sanitizing my inputs, and maybe I should validate those inputs too." So it's concepts like these that we have to help our developers ingrain into their everyday code. So that overall, when the time comes and that code is put into production, it's been battle-tested as much as possible internally.
So that at this point, you're then open only to zero-days that the developer may not have been aware of. But the reality of the matter is, unfortunately, organizations can't really afford to necessarily just scale up their security implementation as much as they would like. They may have an application that perhaps they invested even up to millions, and that application may not be compatible with the latest version of the operating system. And unfortunately, let's assume there's a zero-day about a kernel-level vulnerability where the only way to fix it is to upgrade the operating system. In this case, the organization is stuck.
Now the good news is, there are what we call compensating controls that you can put in place of the patching process. You know what we call virtual patching, right? in the industry. But again, the sad reality of the matter is, most organizations either don't invest heavily enough into these concepts, or they just simply don't have the intelligence and capability to be able to pull it off.
The sad part is, as much as we're trying as cybersecurity experts to help organizations and companies move forward with their needs, we're unfortunately not doing a good enough job to get them to upkeep their existing investment in their existing infrastructure. And as such, that's leading to holes, which then hackers are leaking in through, and you're seeing the results in the news, "Leaked: 5 Million+ Records!" like Spotify being the most recent one that got hit. Spotify did everything right. They just, unfortunately, still got hacked because of the most basic of principles out there, which was maintenance. Ricky: Awesome.
So DevSecOps seems like it spans both a breadth of making sure that the software engineers, web app developers, that their code is sound, but also making sure that your infrastructure and the design is also sound, and nowadays a lot of these IaaS, infrastructure as code, in your playbooks, and everything. People might be using just default templates with keys just scattered everywhere. Russ: Yes, yes, unfortunately.
The cool part is, nowadays, there are tons of tools out there available that allow you to actually scan for default configurations, default tooling, and left behind SSH keys, Git keys, and signing keys. Basically, just secrets in general. And the frustration is, it is so easy to clean up and to make sure that these things are not left in production.
So very recently, we were dealing with this situation, and there were a bunch of development libraries included in a particular function at the time. And I said, "Why on dollar deity's green earth are we actually including these things in production?" And so I ended up having to go from chain to chain to chain until I finally got to the developer. And I said, "Hey, why did you actually leave the libraries?" and he said, "Wait, the libraries are still there? It reminded me of that joke. This commander comes on base, and he notices two soldiers guarding a bench. He asked, he's like, "Why are those guys guarding that bench?" And they're like, "We don't know, sir, but it's been tradition." He goes, "Okay."
So he asked when he gets to his office. "Let me call up the previous commander and ask him." And the commander says, "The commander before me was doing it, so that's why I was keeping up the tradition." So that's interesting.
So he calls the previous commander. At this point, the retired general is in his 80's, and he calls and says, "Sir, I'm very sorry to bother you. But I was just wondering; there's a bench out there with two officers guarding it.
Do you happen to know why this was instituted?" General says, "The paint still hasn't dried?" Here's the thing, at the end of the day, a developer left the development libraries in the code for whatever purpose it was, and he forgot to clean up. But the operations people and the systems people have no idea what their libraries are there for. They're assuming what they were given is final, so they take it, and they put it into production. And that's, right there, the issue. The DevSecOps guy is the one that goes, "Wait, let's do a sanity check on here.
Let's figure out why this is." So then we get to trace back and actually find out, "Oh yeah, that shouldn't have been there in the first place." Ricky: So what types of technologies and skills do you think is important to learn for DevSecOps, infrastructure-related activities? Russ: So the trend these days is for somebody to understand CI/CD as it stands for Continuous Integration, Continuous Delivery.
What's going to make you valuable from the CI/CD side, Jenkins, Travis CI, Circle CI, and even GitLab's own built-in CI that it has, or are you aware of any of the Azure CI side of things. You're going to be catching the headhunter's attention. These are the what I call "top five" of the CI/CD solutions out there. From a language-specific focus, obviously, Bash is still king in the operating system world. So really, that comes down to then understanding your CMDB offerings, which are your configuration management tooling. You have Ansible, you have Puppet, you have Salt, you have Chef, you have Packer, and then you have Python.
Python is your kind of universal glue that you glue these five services with to generate something. So what I usually tell my students and to those that are first time starting out, "The systems operations, the DevOps' ops side, is a lot like building Legos." So I'm going to say, for instance, "Build the latest, say, Star Wars Lego set that I just bought."
The dev side is the one that actually makes those pieces. The ops side is the one that takes those pieces and actually builds the Lego process itself. Now, you can use superglue if you're really not going to break it down again. Or you can, as Lego was intended to be, break it all down and build it back up and do that in an automated fashion as quickly as possible each time. And as you keep doing it each time, you end up improving the process where you may end up even having extra pieces, and you'll still have the same structure. And the leaner the model, the stronger the durability of the tool.
And therefore, the simpler the tool, and therefore the less amount of headache from the tool. "Oh, I'm in SecOps, Russ. Does that mean I'm DevSecOps?" And the answer is, no. I've had people that call themselves, "Oh, I'm a DevOps guy, he's a SecOps guy, he's an ops guy, I'm a dev guy." No, no, no, a DevSecOps person has the experience and capability to deal with these types of situations from all facets.
This person needs to have the experience from the security world, from the development world, and from the operations world. So, in essence, this person is truly a joker of all trades. You can actually put this person in the security side, and they'll be able to do your cybersecurity.
You can take this person and put them in the developer, and they can actually write some code. You can take this person and put them in the ops world, and they can actually start deploying what's been developed and make sure it can make it all the way to operations, and then to production, through the cycles of alpha and beta, and so on and so forth. That's what it takes to be a DevSecOps. Ricky: So what kind of salary ranges do people in the DevSecOps world expect to command? Russ: Depending on the cost of living.
In other words, if you're in the California region versus you're in the East coast side of the world, you can see anywhere from a beginner from $80,000 to $100,000, easy. And right now, the top-tier are making $250K and above, and your median is around, I would say, $150K for those who have been in the DevSecOps world with at least a couple of years of experience, if not a little more than that. So, it's going to depend on their qualifications. So a very good friend of mine back ten years ago said, "Russ, look, for somebody who has four years of experience versus somebody who just finished their bachelor's, you're going to be here, and they're going to be at the bottom." But he said, "If you as that four-year level experience with no bachelor's degree person, don't go after your degree, this person, as they start gaining, they get to continue to go up. Whereas you are stuck here because you don't have that degree."
And I said to myself, "Wow, really, does a bachelor's mean that much in the technology world? Now, this concept that I'm discussing has been argued back and forth so many times. And some people would say, "Look, person X, I know got his high school degree from a GED program, and he's making millions. Well, look, Bill Gates and Mark Zuckerberg, they dropped out of college. They didn't even finish." Yeah, but for starters, those guys dropped out of Harvard.
They didn't drop out of Joe Schmoe's community college. So that kind of tells you what level they're at. And then secondly, do you need a college degree to get to somewhere? No, but it sure does make things a lot easier. I can tell you that.
Because when you're interacting with an interviewer, or when you're interacting with a job application hosting organization, they have their checkmarks that they have to go through. And one of the very first checks they do is education. Close to 90+ percent of DevSecOps positions out there require that you have a bachelor's. Again, if you have a significant level of experience, you perhaps could bypass that requirement for getting a college degree. But that being said, assuming you have at least a bachelor's, you have at least five years of experience in the DevOps world.
And preferably, you've at least built, led, or been a part of a senior team of some sort; that's when you get to call yourself a DevSecOps person. Ricky: What type of work roles, teams, or organizations would people with a strong foundation in DevSecOps have an opportunity to work in? Russ: So startups are clobbering at DevSecOps people. So the good news is, as someone who is currently employed in such a position, I can tell you on a daily basis, I get at least three to four job offers from different people. So I get asked this a lot, "What's the difference between architect, and engineer, professor?" And so I tell them, "An architect is a person that can take the business requirement and translate that into an action plan for an engineer to be able to implement." So that's the biggest difference there. And the only way that talent comes is by experience.
So we all start somewhere, and that somewhere is your junior developer, your junior data analyst, your junior operations person, that's where you're going to start. What I usually, again, tell people is, look at junior data analysts, operations, and developer positions. Look for junior DevOps developer if you're going to be a developer. Look for QA engineers. If you already have experience, let's say you're a veteran developer and you're like, "Ah, just because I typed in some Angular or some Node code, or some TypeScript, now all of a sudden, I get to add $40,000 to my salary base, and I'm not doing that? I'm crazy, I want to take advantage of that too." You can.
There are tons of freebie courses and certificates out there for you to take advantage of. And when you do, then you can apply for essentially instead of a just developer role, look for DevOps role. And it's the same exact thing. Ricky: So then what kind of books, courses, resources would you find useful for learning DevSecOps? And I know the answer is all of them, but are there any ones that are more tailored towards DevSecOps rather than just simply having a developer experience, security experience, and ops IT experience? Russ: Some of the books I would recommend, I can't praise this enough. Gene Kim is a great writer, and the Phoenix Project; it's a classic, and it's absolutely brilliant.
It actually helps you understand DevOps truly. Beyond that, there is the DevsOpsSec book and the Agile Application Security book. So those are my top three that I usually recommend to people. The DevOpsSec book is by Jim Bird. Agile Application Security, Laura Bell is the one that wrote that one.
Every DevSecOps person has a passion, so you can have a DevSecOps person that's really strong in developer experiences, you can have a DevSecOps person that's really passionate about operations, and you can have a DevSecOps person that's really passionate about cybersecurity. So I, myself, I have a cybersecurity background, and that's what I'm most passionate about. So what I usually tell people is grab these three because they're a good enough representation of the triad, and then whatever your shtick, what makes you happy? There's the Hands-on Security in DevOps, that's another good book. Focus on books towards that one with a DevSecOps focused.
And then, if you're about operations, I usually tell people the SRE and the Site Reliability Workbook by the Google's operation team. That one's awesome. And then if you're about cybersecurity, welcome to hell and grab some books from the shelf because we're going to have some fun reading, lots and lots of books.
Learning never ends, guys. Usually, I tell this to all of my students. And I have the need to reiterate that here. Learning never ends.
It's important for you to always stay on top of the latest trends. Especially in the DevSecOps world, things are flying by so fast, so quickly, that you're like, "Wait, I just learned about DevSecOps, and now it's called GitOps? Wait, what about ChatOps? What happened to Slack?" You don't have time to actually stop to focus. Focus on the basics. How do you eat an elephant? And the answer is one bite at a time. Now, how do you actually learn to take a bite out of an elephant? That's where you need to learn the basics.
And so the best way to learn those basics is by reading books and hands on the keyboard. I've had so many students that have learned theoretical knowledge. Oh, my God. They memorized the heck out of what I told them. But then when I asked them, "Okay, so here's a real-life scenario.
Walk me through conceptually, how you would solve this and then break this down into components, and then tell me how you would configure or work on each component and how you would tie these components together." They freeze. And the reason people freeze when they're faced with such a question is because they just don't have that experience to know how that tool does what it does. Like everybody knows. Jenkins is a great CI/CD tool.
But did you know you can actually execute Jenkins in ways it wasn't designed to solve different business challenges? So, for instance, I actually had a guy; so he was maintaining a hospital and all the machines that take your blood pressure, they actually have a specific name, but these things are actually networked devices, and they're literally deployed physically all throughout the hospital, but they all ride on the same wireless network that they had. So what this one person I know actually abused their Jenkins deployments, and anytime a new build of the firmware for these devices came out. He actually used it essentially as a firmware patching tool. And I'm like, "Dude, it's Jenkins!" And he's like, "Yeah, I get to do testing, I get to do pushing, I get to do pulling." And I'm like, "But why?" It was just, that's the way he thought of doing it. Now, how many people do you know have done this? I literally know one person.
But that's the guy I would hire if I had a truly unsolvable challenge or unthought of problem before that needed to be addressed. Because I know that guy has dealt with these tools in-depth that he can actually manipulate it to address a concern in a way that hadn't been addressed before. And that's what makes it so cool. I was doing brown bag lunches to help these people learn about Python so that they could learn at least how to code so that they could then start automating some of the everyday tasks that they had.
In this particular case, this guy had never, in his life, touched code. He had always considered code to be magic. And I'm looking at this guy, I'm like, "You've been in the field for how long? And he was like, "32 years." And I'm like, "You've been an IT pro for 32 years, and you never picked up a single coding book in your life?” He's like, "I didn't have to.
That's what software developers are for." I'm like, "Why?" And see, that's a risk because if you have people operating your production environment, that your whole operations is based on some kind of automation code, some application. So, in other words, you have a guy that has no clue what he's actually providing to the public but accepts, "Okay, I know I need to click this. This has to be green, and that has to say this." What's the intelligence you're expecting out of that guy? And so, that's the challenge. You need to help people stay fresh and stay ahead.
Ricky: Cloud is one of those big buzz words that's been around for a long time now. And I still have a hard time wrapping my head around how to explain what is the cloud and how would you demystify it? Russ: So what I usually tell people about the cloud is, so there is no cloud really, you're just recording and utilizing somebody else's computer. That's all it really is.
You're just renting resources from a computer storage, and memory base, as well as a network. With these four, the beauty of the cloud is you can only rent what you need for as much as you need, for as long as you need. That's the beauty of the cloud. So the idea is, it shifts the thinking from CAPEX to OPEX.
So, in other words, with the old school way, where they say, "I'm going to set up my own infrastructure on-premise, and I need to invest into a data center, or I need to invest into a rack with some CPU and storage." That takes CAPEX. You plan ahead of time, you budget things, you invest, you plan out every three years, five years, ten years, there's a rotation plan, and all of these kinds of things that go into that kind of thought process. With the cloud, it's just a monthly fee. You're only paying for as much as you're using.
Now, however, with that, the biggest challenge with the cloud has always been adapting to the cloud, actually moving your business to the cloud, which is a whole different journey. There are those that are lucky enough like I was to be able to start 100% in the cloud. But the idea that are those that are migrating from the on-prem to the cloud, or those that have to maintain both an on-prem premise foothold, as well as treat the cloud, essentially as an edge data center. For those types of situations, the considerations are pretty critical, and it takes truly an expert to guide these organizations through those journeys.
I've had the honor and privilege of working with a very big insurance company. This company was globally positioned. They had footprints in Europe and a footprint in Asia; they had a massive footprint in the US. So I was brought on as the cybersecurity guy for the cloud, but they had previously hired on people, some very fresh, cloud-talented people who were talking to some very monolithic, archaic-practicing organizational types. So to give you an idea, this organization was still operating with mainframes. And again, there's nothing wrong with mainframes, and if the solution is to maintain a mainframe, then the solutions to maintain a mainframe.
But you cannot expect to bring your baggage into the cloud and expect the cloud to just magically make it go away. That's not how the cloud works. And so from a risk stance, what I kept telling this organization was, you're not going to be able to replicate your existing footprint in the cloud, and then expect the cloud to be your edge data center and work it with it that way. That's not how this works. If you want to actually work with the cloud, you have to think in cloud. And then, they looked at me very blindly, and I said, "Okay, I'll tell you what, the journey into the cloud is what we call lift-and-shift."
That's usually one of the methods. And so the lift and shift argument is you actually replicate your existing footprint in the cloud. Great idea, but really, you're just faking.
And you're really bringing on a lot of unnecessary baggage into the cloud, which is why it ends up being very expensive. But that's meant to be a tool; it's meant to be an approach. Then the idea is, once you're in the cloud, you can start going, "Well, do I really need that piece?" "Let's throw that away.
Do I really need this piece?" "No, I don't anymore. That also goes away." "Do I need that?" "Nope, let's take that and throw it away too." And before you know it, you actually start getting rid of what essentially formed your baggage in the first place.
And you end up with a nice lean cloud-operated model. The idea of transitioning perhaps from old school SOAP requests to a API-driven agile microservices-oriented architecture that actually addresses a client's needs, in the push of a button instantaneously backed by the availability and the rigidity that the cloud provider allows you to take advantage of. That's the way to address the problem.
Ricky: When people think, "My stuff's in the cloud," it's giving away of responsibility. And you assume that somebody else is going to take care of your data, take care of your infrastructure, and all of that. But for an organization, we know that now, their liability is now also your liability. So your role as a DevSecOps engineer is to straddle that line and make sure everything is in the go. You basically introduce additional complexity into what once was a purely on-prem, on-premise environment. Russ: Every cloud provider suits a particular scenario and a particular need.
And as an architect here, one of my goals and one of my challenges is to interpret the business goals and then to design a system that is going to answer that challenge. Now, when you utilize a cloud environment, you have to keep security in mind. Every cloud provider has some form of a watchdog like service slash process.
With Amazon, you have GuardDuty, or you have Inspector, or you have Detective, and so on and so forth. With Google, you have your SecOps tooling. With Azure, again, you have your security services that are all combined and bundled together, and they actually do a great job on logging. You can be sure that any cloud provider you use will have logging capabilities down to tracking your issue to the millisecond.
How else are they going to turn around and charge you for your usage? If they're not measuring it themselves, how can they be sure what you're using to send you a bill for? At the very least, you have to be able to see what's happening, so that you can then protect what needs to be protected, and expose what needs to be exposed to the outside world. Among all the cloud providers out there, Amazon actually does a very good job on highlighting what they call the Shared Responsibility Model. Azure, and GCP, and Alibaba cloud, and Oracle— they all also have adopted a version of that since. But the idea comes down to this; the cloud provider is really responsible for the hardware that they're renting out to.
And they're also responsible for the network security that's behind the layers that you don't have access or control to. But anything they're exposing to you as the consumer, you are responsible for the security of. Now for what it's worth, they have some built-in protections, that even if you were to allow yourself to be shot in the foot, would not necessarily affect the provider themselves, but that does not mean they're doing your security for you. The biggest risk is not understanding that shared responsibility model.
Now, what most organizations keep doing, they think, "If I grabbed this storage bucket thing of theirs, and I just put my data in it, this cloud provider is backing, the security of it, so I don't have to worry about it. And it's also backed up too, so I don't have to worry about it. No, there's actually all different types of storage mechanisms this cloud provider's giving you. Furthermore, the cloud providers telling you, don't do that; instead, make sure you're doing your IAM policies, which stand for Identity Access Management. Your IAM policies tie those and secure those, and make those more specific.
And the cloud provider's also telling you, make sure you're actually utilizing standard cybersecurity best practices as your regulatory compliance requirements are asking you to do. For instance, you need to have a properly regulated and monitored environment. Grab a marketplace image of a Cisco router, or a Palo Alto device, or a Splunk set up. There are tons of solutions out there now that make it very easy to implement for you to have a truly properly, easily set-up IAC SDN environment. And if you can do that, the rest is easy enough to do. Now, unfortunately, most organizations, sadly, don't employ or encourage their existing workforce to go after the more applicable, fresh knowledge.
So, as a result, people end up staling out. They don't know what these concepts are. Ricky: What cloud technologies are you excited about? Russ: So, machine learning and AI is one of my fortes that I love. So specifically, these days, I am all about SecOps with AI.
I love how marketing people paint what we do as magic. Again, for the audience listening to this talk, we're not magicians, we're not wizards, okay? So, for instance, the concept of "self-healing." For those of you that are listening and don't know beyond having heard self-healing, that just really means you're restarting a service, ladies and gentlemen.
That's it. So your website's down. Okay, let's turn it off and on again. That's literally what we're doing, except we're doing that at a code level. Now what we've done with technology as time passes is we're improving on how we do that.
So in the case of SOAR, your Security Orchestration and Automated Response, we're watching for certain conditions. So, in this case, the website's throwing a 500, or a 400, or the DNS is not responding. For some reason, there's an outage.
Literally, step one, stop service web server. Start service web server, that's what's executing. And this is part of then made as a playbook.
So we call it the plunger service, for instance. When your sink is backed up, clogged up, you take a plunger, and you go... That's exactly what you're doing to the webserver. You're actually just plunging that; you're turning it on and off to see what's going to happen.
Two things happen when you do that. One, you start looking at logs after and before the time you did the plunging; the before helps you understand what led to that point. The after helps you watch the beginning of the log. So perhaps, somebody pushed some code that required some connectivity, that pod that you were pushing the code to does not have access to. Hence why the application failed, or perhaps some component it's expecting is not in place, or perhaps during the time of build, some library got missed. There can be tons and tons of reasons why a service is down, or it's not working.
But coming back, that resiliency aspect all comes down to automation, and the solution is more or less generic enough if you can design the architecture accordingly. Now, couple this with AI. So I no longer now need to even say, "Okay, execute this playbook." But instead, I have an AI watching that situation and responding to it. The challenge there becomes, what if the evil guy is doing AI too? So the really cool part is, I was actually recently a part of another chat where we had blue team bots Fighting against red team bots. And it was absolutely brilliantly hilarious to watch these two bots trying to attack and defend against each other at lightning speed.
Ricky, I'll be honest with you. I was actually afraid for a second because I'm going, "Okay, I'm a cyber security guy. My job is I can run a SOC, or I can be on the red team side, and I can do all these different approaches to try to solve the challenge of security. But when you start introducing the element of self-capability, of self-reliance into hacking.
AI at the speed of lightning can apply a way wider and vaster array of approaches to a problem, basically brute force his way into a problem to get a result in comparison to a human. Now, where AI continues to fail is the experience aspect. That comes down to training your model to know how it should approach a particular problem.
We still, thank God, haven't been able to make it as efficient as a human, yet. So that's what excites me in today's world. I'm looking forward to the day where SecOps is truly automated from an AI aspect where I can bash the heck out of the code, patch the heck out of the infrastructure. And make sure that the code that's utilized to spin up the infrastructure and the code that's utilized to solve the business challenge have been battle-tested to death to make sure that there's nothing left for it to be exploited upon. Ricky: So given that, what is your take on certifications then? We've talked about, "Hey, you should get a library card," but there's these things called certifications too.
Another holy war to unleash. Russ: So the first answer really you should be focusing on is what's the field you're going into? So if specifically, your claim to fame is the DevSecOps path, I would say, at the very least, have certs that prove your development background, your security background, and your ops background. So good ones towards this are your Network+, your Security+, and your Linux+, so your CompTia. I call this kind of the "Holy Trinity." These three, they tend to be your basic starter pack, easy to knock out certs to get there. Then from your cloud aspect, Amazon continues to be one of the biggest claims to fame from the cloud provider's aspect.
So getting an Amazon cert is certainly valuable. So if you're in the developer world specifically, I'd recommend the DevOps certs. If you're in the solutions architect, a.k.a operations world, getting your solutions architect. And finally, if you're in your security, getting your security certs.
SANS, I love SANS. I've been interacting with the SANS organization since the year 2000. I've had the privilege to be able to attend SANS courses and interact with them as TAs and instructors, and I've led birds of feather sessions, so on and so forth.
Throughout that, I've gotten to take a lot of their classes and a lot of their certification programs, the ones I would recommend at a very least; the lowest level would be your GSEC, or your GPEN, or your GNFA, or any of really the GIACs, just get at least one, if you're going to be a cybersecurity-focused person. If you're going to be an operations person, it also doesn't hurt to get certified in things like Kubernetes. So Google actually has a Kubernetes cert you can go after.
And also, Azure has Azure Ops as a certification you can get. The point is this. You want to have the idea to prove to whoever's talking to you; you have the experience of What you're being questioned on. So certs help prove experience. Certs do not prove knowledge.
Your knowledge is going to come from either your degree or your experience. All certs prove is that you have played with that environment and you have a clue about what it's doing. But just because you have a clue does not mean you're an expert. Even if you're getting something that's calling you an expert, I'm a solutions architect pro, but would I call myself equivalent to the guy that wrote the service at Amazon, to begin with? No. Would that guy call himself an expert? I've actually interviewed for Amazon, and I can tell you, they don't either. And they look to people outside their teams to see if anybody calls themselves an expert.
It frustrates me to no end. I have 20 plus years of experience in the cybersecurity, IT development, and operations fields. I don't call myself an expert. I've been educated by literally the best in the industry. They don't call themselves an expert, but these Gen Z millennials go out there and take a two-week course and then get a cert.
And then they call themselves "cybersecurity experts.” And I'm like, "Ugh." And what's worse is, organizations hire them because of it. And these people are put in real-life situations like, "Okay, now guard the bank." And I'm like, "Oh boy, does this person have the actual experience and the capability to understand the charge they've been given?" And the answer is, unfortunately, more than enough, no.
A big double-O zero. And now are all certificates that way? No. So, for instance, I did not believe in the CISSP before. I believe in it now, and I'm going to explain why. So when the CISSP first came out, there were a ton of brain dumps out there that I know it was cheating. But a lot of people didn't care.
And they bought these brain dumps, and then they just aced the CISSP, and then they got to call themselves cybersecurity experts and leaders. It frustrated me to no end. So because of that, I never put any value in the CISSP. But over the years, CISSP program has also grown. So now they require mentorship.
Now they actually have a whole board that evaluates their questions, and they keep them up-to-date, and they actually apply some pretty good security practices behind their questions, where you're not going to be asked the same question that was asked previous years. The CISSP does have some requirements. You have to have some annual experience requirements in place. There are some educational requirements in place.
So you really have to be at least this tall to play for them to even consider you for it. So with these in place, they've actually begun now churning out what I would call "more successful" CISSP holders. So to me, for that reason, the CISSP is true and dear to my heart. Start at the basic; getting your certs helps you establish that basic that we're talking about.
Which basic certs? Those that prove your passion. So what I usually tell people is, get at least the most entry-level, basic search you can get from each cloud provider. So with Amazon, get the solutions architect associates with GCP, get the Google cloud practitioner. With Azure, get the, I think they call it the cloud practitioner if I recall correctly. The Azure, again, take a look at my profile; you'll see them certified in all of them. But the point I'm trying to make is, get that basic cert.
For starters, it's very easy. It's very cheap. But the added advantages, it shows to the world how to play in that particular cloud environment. It doesn't necessarily say you're an expert in it, but it does at least show to the world that you're capable of interacting in that world when and if the need arises. And so you combine your cloud knowledge along with these toolings that I've told you, then it's just a matter of time.
Unfortunately, you cannot beat a good old-fashioned keyboarding out there. You have to put in some time, so come up with some fun challenges. I loved Google's Summer of Code challenges that they had.
The idea was, "Hey, you're free during the summer. Why waste that time just partying? Instead, focus on helping humanity to make it better. And in return, depending on your performance, we'll offer you a job." The thing is, it allows the participants to build and establish both an experience with working in a Google-like environments. But also, it helps them gain some really valuable experience.
And that's what I'm really trying to tell today's world is, guys, there is no such thing as free knowledge. Unfortunately, until we invent the matrix plug where I can go, "And now, I know Kung Fu." Until that day comes, all we can really do is continuing to type in, and putting it in, and logging our hours to able to generate, build systems, and services for people to enjoy.
Ricky: You've worked with a lot of different programs like US CyberPatriot, US Cyber Challenge; which one of these programs do you see as having really the most potential? Russ: My personal passion is helping raise the next generation of cybersecurity warriors. That's one of the things I absolutely love. And this is like my hobby, right? My Batman personality is, I'm actually, by day, a lead DevSecOps ninja. And by night, I'm actually playing the part of an educator. Those two worlds, however, collide a lot.
So that's why what I usually tell people is you need to stay fresh. You need to stay ahead one step at a time of what's happening. As the Bruce Wayne side of my world, I'm an educator, and I'm always in the know. Thanks to academia and the resources that I have access to. And then I get to take that knowledge and apply it to my day job as a lead DevSecOps person. So I love the US CyberPatriot program.
So I was actually coaching a high school, and I was their instructor there. Man, I wish there was a CyberPatriot program when I was growing up in high school. Life would have been a lot different. Randy Martini, Doug Logan; these are the people on the East coast side that have helped me interact as an instructor and help me host some of the challenges the USCC required from the cyber range that I built. So when I got to play in these worlds, it gave me the capability to observe what the raw talent that's sitting there is lacking at and then being able to craft and design programs to help them get to this tall to play along. So from that aspect, I absolutely loved the time that I've been able to interact with these programs; NSA's GenCyber is another program I've gotten to interact with where the idea was, we're teaching the teachers.
So high school teachers come to us, and we talk to them. And the hilarity is one of these stories I love talking about. So I give a talk on mobile device security.
One of the challenges I do, I have a Samsung Note 9, and so I hack it ten ways to Sunday. I actually have rooted firmware on there, so there's this piece I do where I'm using Metasploit, and so I'm like, "Okay, so guys, look. I have the camera off, and I have the camera on. And so I turned off the camera as it's recording and behind me, you can see on the projector, I have it synced where it's showing on the screen, what the camera's capturing." So I'm like, "And as you guys can see, I've now turned the phone off.
So I hit the power button, and they actually realize, "Wait a minute, the camera's still off." And they're like, "Yeah, but you didn't actually turn it off; you just put it on standby mode." I'm like, yeah, you're totally correct.
Let me actually turn it off. So I actually hold down the power button. And they see, the Samsung logo comes in and closes, and then I've put it up on my table.
Now the thing is on top of me, right above on the screen; I usually put a piece of paper. That says, "Smile, you're still on camera!" And so, even though it's off, I actually have a dual boot, so I have a separate partition that's booting. So it's still powered on.
So, therefore, it's still showing. So they get to observe that the phone's been turned off, but then the camera's still working. And there's no indicator at all that it's working. The whole point was, this is part of the showmanship; I'm trying to show my audience that there is no such thing as security when it comes to a mobile device. So now here's what I didn't anticipate.
A teacher literally got up in the middle of the session, and they just broke their phone in half. I was like, "That's not what I wanted you to do there." And the guy says, "So this is really the only way to be secure?" That's, yeah. Sure, yeah. You know, 15, 16 years ago, one of my big brothers that I have the privilege of calling, Valdis Kletnieks, he said, "Russ, what's the most secure system out there?" And I was still in my fresh years, and I was like, "I can put a firewall and then I can put an IDS, and I can put a Splunk set up, and I can have all these things core collaborated, and I can be watching on the money.
He's like, "Yeah, but what's the most secure system?" And I was like I just told you, “I can set up TripWire, I can have a baseline and so on." And he was like, "Nope, the most secure computer is one you can dig a six-foot hole, you put the computer in there, you cover that up with cement." I'm like, "But I can't use it!" He's like, "Exactly; security does not mean it's also usable."
And I was like, "Ah..." Now all these years later, I totally get why he asked me that question— depending on the business need, depending on the program that I'm teaching at, the teacher breaking their phone? Yes, they are secure. That's a way of security. Breaking that phone is to make sure you're not hacked. But you don't have a phone anymore. So you've totally destroyed the usability aspect of the solution you were trying to secure.
And unfortunately, in today's world, a lot of people lack that piece. I don't want to throw shade on like, CEH. or OSCP, or anything like that, but these programs had a very offensive red team approach to security. And yes, it is very easy to break the glass, and it is very hard to defend the glass, but just saying, "Oh, I know how to break the glass, therefore, I'm a cyber security expert."
I don't agree with. And that's why usually the programs and certifications that I focused on, and then I've had the privilege of interacting with have been well-balanced programs, which again, things like NSA, GenCyber, US CyberPatriot, USCC, EDUCAUSE, B-Sides, RSA, all these places I've had the privilege of talking and interacting with; these are usually the aspects that I've been intrigued about which led me to interact with these organizations in turn. Ricky: With that being said, what are some of the shortcomings in the education space that you see for cybersecurity? We have all these certifications, these books, there's all these mentors and resources. What's broken? What's wrong? Russ: So, just yesterday, I was actually involved in an interview for a potential instructor into our program. What I see as challenges tend to be the enthusiasm. So let's say I'm interacting with you.
If I was a strong professor with a very good cyber security background, but I kept on going in this tone forever... After a while, the students are just going to fall asleep. So it's important to bring showmanship to what you do. So I have a Mediterranean background, right? So the joke is if you tied an Italians hands, he wouldn't be able to speak that also holds true for Turkish people. The body language and the way that you interact with your students is important.
You need to give them the understanding that they need to be as excited as you are. And so it's important to… Because at the end of the day, we're talking about bits. We're talking about ones and zeros, and we're talking about extremely, sometimes boring concepts. Like for instance, when I'm talking about in networking, about VLANs.
Okay, great. We know there are access-based VLANs, there are trunk-based VLANs, then there are native VLANs. Great, so we know there are three types of VLANs, and we can talk about how one of them actually strips the tags. One of them allows all the strata tags to go through, and the other one's a catch them.
Okay, great. Now, if I touch you, this may stick in your head for, I don't know, a few days or hours, or even seconds, depending on how interested you're actually in the topic. But if I actually flavor this knowledge with experience, and I started talking about how I ended up getting hacked so bad because I had set up all the VLANs as trunk VLANs because I didn't understand any better and I just wanted the tooling to work, then that makes for a story that is actually intriguing to the students. The issue with academia is trying to find nut jobs like me. The crazy in me is because I'm so passionate about these types of things.
And I'm actually interested in helping the new people, the newcomers, understand these topics and to elevate their passion as passionate as I am about the topic. There is no such thing as boring when it comes to cybersecurity, as far as I'm concerned. So I had a friend who had just bought Tesla Model 3, and he invited me, and this guy is actually a Ph.D.
in computer science, by the way. And so he's showing me the ridiculous mode, this and that. And that was so cool, "Oh, it's Spaceballs." And so after that, we step out, and we're talking, and I just had my hand on the hood of the car.
And as we're talking, he says, "Yeah, Russ, do you mind moving your hand?" I'm like, "Dude, sorry, I wasn't gonna scratch the paint or anything." He says, "No, I'm not worried about that. I'm worried that you're going to hack the car." And I looked at this guy for a second, and I'm like, "Nah." I'm like, "Nick, are you? You're joking."
He's like, "Dude with you, I never know." I looked at him, and I said, "Bro, If I could hack a car by touching it, I'm starting a religion, and I'm the prophet of it!" Are you kidding me? But see, that's the problem, even though somebody maybe a Ph.D. in computer science, because they may not have focused on computer security, they have no understanding. So to them, cybersecurity is magic, even though they have a very strong understanding about how computers work. They may see cybersecurity as magic.
And what I keep telling people is the saying that Arthur C. Clark said all those years ago; so succinctly, any sufficiently advanced technology is indistinguishable from magic. And that's the challenge with cybersecurity. When we're talking about cybersecurity, people see it as magic, but it's actually science. What we're doing is we're abusing vulnerabilities in a particular process, and we're pushing them and twisting them and ways to make them work the way that we want. See, one of the reasons why I love the Matrix movie just as much as Ed from the SANS world also does as well.
You think that's air you're breathing?" There are some rules that can be broken, and some rules that just don't exist. So in the case of, for instance, containers. What's a container? A container is a jailed process that is controlled through cgroups. So, in other words, we're tricking a process into thinking it's only limited to the resources we tell it's limited to. But what if we can actually break out of that sandbox? In that right there.
That is the difference between a very boring professor talking about, "how containers are constructed..." And the crazy one talking about real-life examples and giving applicable movie trivia to help drive the point. So the biggest problem with academia, in my opinion, is passion. So I have been extremely lucky to have been educated by some really good educators.
And I am totally following a Buddhist philosophy in this case; what I've been taking, I'm also giving back. I strongly encourage cybersecurity Educational programs for people to seek that have passionate instructors. That, well, even if they're not as interactive as me, perhaps. they're not some very boring drone, just talking about something over and over in a very monotonous tone.
Ricky: So you're really a veteran in the field of, you know, cyber security with all the years you've spent in the trenches, cold server rooms, the bullets flying. What was that journey like for you going from little Russ to big Russ? Russ: When recruiters are calling me, and they're like, "Sir, how many years of experience do you have in cybersecurity?" I'm like, "If you want me to talk about from age eight, 32 years." They're like, "Age eight?" Here's the thing. I had a Commodore 64.
My dad had just gotten one for us. It was very expensive, and I didn't want my little brothers messing with it. So I learned how to code in BASIC. Specifically, to be able to put a password program on it. The idea was, I didn't want my brothers touching my Commodore while I was in school. That was the whole point, so that's where I got my start.
Now, how did I actually gained that knowledge? Back in those days, in the old years of '88, the late 80's, we had this thing called the CHIP magazine, which we were buying monthly, and it had a bunch of source code on the back. And also, we have these computer shops that basically the little nerds get to congregate to play with. So that's where I got my start. And then, this newfangled thing called the Internet started showing up, and I was lucky enough to be at a high school that was picked to be the main hub of Internet for the entire city. And my computer science teacher didn't speak English. I was born in the US.
I'm a Texan by birth. He actually gave me all these books, so in this case, they have a Unisys UNIVAC one. He gave me the operations manual and the administrator's guide, all in English. And he expected me to translate those into Turkish for him. So as I was reading in English and learning, I was also translating into Turkish for him.
So I started, therefore, learning about basic core concepts of Unix and COBOL at the age of 13, 14, somewhere in there. And then, I came to Virginia Tech in 1999 as a cadet, and I accidentally hacked the website of the Corps of Cadets. It really was an accident, but that kind of jump-started my career into IT in the academic world.
And then, from there, I got introduced to SANS in the year 2000. And so I started taking a bunch of SANS courses. So I started from the junior IT roles.
Very first job, I was essentially a cable monkey. I was basically just removing old CAT 3, cabling out of the buildings, and rewiring them with CAT 5 at the time. And so I had to learn like what's the AT&T B standard. And how do you actually wire something up? What does it mean to a patch panel? How do you actually do proper wiring? So there's a subreddit called cabling porn. I think that some of my instructors over the years have to have been followers of that subreddit because you look at some of the crazy cabling designs these guys have done. I'm like, "Okay, I don't want to have this mess spaghetti of cabling, but is cabling really my passion?" So that question I asked myself led me to jumping into databases.
I ended up getting my MCSD and MSDBA in the Microsoft world, and so from there, I started understanding how to actually operate, essentially, as a junior systems administrator. And from a junior sysadmin, I became a sysadmin. Then from a sysadmin, I became a senior sysadmin. Then from a senior sysadmin, I became a systems engineer.
And then, from a systems engineer, this is where I started actually playing with VMware and the virtualization. Usually, people, they're like, "Yeah, what's your visualization experience like Russ? And I'm like, "I was running production services on an Ubuntu 6.06 box running VMware Server 1.09."
And I'm like, does it get any better than that? And I was running production services on these things with a PFsense box fronting it. Because one of the things about academia, they have a budget of zero, and they expect you to do the same thing Google does, or Apple does, or Facebook does. And you're going, "Okay."
Now the cool part is, you have access to all the knowledge of the universe. And you have, technically speaking, less than ideal hardware access to. So you're like, "Okay if I have enough time, I could probably pull something off like that." And so that's what I did.
And so that capability afforded me to be able to go from, essentially, a capable monkey to part of a team of five at the university that I was at before I started my journey in the Cyber Range. So it started with me and a laptop in a conference room where we literally took out a desk and some panels out of the trash. And it was Dr. Raymond, myself, and our admin assistant Ellen, and so it was the three of us. We started the Cyber Range where I became a CSM, which stands for Certified Scrum Master because I wanted to learn about agile. I ended up getting my leadership studies certificate from the business school of Virginia Tech; I then also went after my cloud certs.
And so, at one point of my journey, this was the most craziest of my time; essentially, I was the CTO of the range. On my desk, I had like over 40 resumes in a pile that I had to review. Then on this side here, I had a pile of homework from my students that I had to grade. Then here, I had a pile of paperwork from our department; there was some paperwork, budgetary needs, and so on and so forth. Then my keyboard and my monitor.
Then on this side, I had my notes and my books on what I utilized to build challenges in the Cyber Range. Then on this side, I had a bunch of submissions from various board members who had all of these challenges that they were submitting that I had to review from the technical view to basically provide a feasibility report on whether or not this could be imported, and what would the action plan for that look like to help me start put those into the range. And then, as of all, this was not enough, here I had a bunch of drafts for the talks I was planning on giving them. So literally, I just kept doing this, and this, and back and forth. And by the time I left, two years later, we were teaching 12,000 plus students instead of just four-year colleges; we had included two-year colleges and high school students. And on top of that, I had built a team of 22 in total I was a part of, I hired two direct hires under them to help build a national resource, in this case for the Commonwealth of Virginia, to provide cyber security education to students.
And when I got to that pinnacle of where I was, I looked, and I said to myself, "Wow, this is pretty cool." And you know, the question you asked me, "So how did you get there?" I was like, "It's crazy, the path that I've taken to get there." Would I recommend to everybody to get it? At the age of 40, you see these white hairs over here? That's what you earn by going through the way that I did it. I don't know if there's an easier, a better way to get to it, but I can tell you commit to challenges that can help you rise above.
I loved my time in the Marines, one of the biggest lessons I learned was, if you screw up, don't apologize. Don't dawdle. Instead, look at what you screwed up.
Learn from it and move on. And don't repeat that mistake again, with the concept of always trying, basically, half a size bigger. So let's say you're a junior, fine. The 80/20 that Google talks about where it looks like, Okay, 80% you do for your job you're hired for, and 20% dedicate to the position you actually are gearing towards moving forward with. Basically, take that concept into your daily practices. And before you know it, you actually are there.
They talk about the imposter syndrome all the time. I don't like that they call it "fake it till you make it" because if you put under-qualified people in key critical positions, that's setting your organization up for failure. But if you put somebody who perhaps has medium to high-level experience in a position that requires high-level experience, then the organization gets two things: one salary savings because they don't have to pay as much, but two, they also help grow that person into what they want that person to become. I've been lucky enough to undergo that level of training with different organizations throughout my life.
So then people ask, "Should we go after startups?" I wouldn't recommend playing the startup game until you actually have some experience under your belt because a startup as a very brutal environment because they're held to very rigorous standards and rigorous grading. So I usually recommend to people to start at a well-established organization, go work for Google, go work for Facebook, go work for Yahoo, go work for... take the Instagrams. Find a high-target, well-known organization that, when they look at your resume, they don't need to go, "So what is this company?" You get a few of those under your belt, and before you know it, you'll find yourself in the spot that you actually want, making the target salary that you were shooting for. And again, I'm happy to interact with anybody that wishes to talk to me after this interview.
I'd be happy to coach them and guide them along their process. Because again, everybody's journey is unique. And including myself, I'm always open to opportunities; I'm always interested in that next-level of challenge because that's what keeps me sharp and fresh.
I focus on and welcome those types of offers and challenges from anybody, and I strongly encourage those that are listening to do the same. Ricky: So where can people connect with and find you, Russ? Russ: Well, LinkedIn, obviously, is my public doorbell these days. Beyond that, you can also reac
2021-01-18 08:36
