Along Those Lines: Understanding Essence, a Co-op-Created Cyber-Defense System
[Music] i'm grant brooks from crawford electric cooperative in bourbon missouri welcome to along those lines [Music] this episode's sponsor is ipkey's power partners who wants you to know that ipkeys is a cyber security expert trusted by government agencies who can help you increase utility network resiliency and propelled durms technology learn more at ipkeyspowerpartners.com hi everyone this is a podcast about electric cooperatives the work they do and the challenges they face i'm your host scott hoffman cyber security has been an ongoing concern for businesses including electric cooperatives for decades but in recent years as electric utilities have deployed new connected technologies to monitor and maintain the grid the need for strong cyber security on the operations technology or ot side of the house has grown exponentially to help electric cooperatives with a solution that's customized to their specific needs nrca worked with co-ops and other partners to develop essence a tool that began in 2014 as an innovative way to quickly identify anomalies on downline systems and has evolved to become a robust tool for detection visualization and reporting of potential cyber attacks here to talk about this unique co-op created cyber defense system is emma stewart and auricia's chief scientist emma welcome to the podcast thanks for being on hi how you doing so let's start with a description of essence what is it and how does it work so the essence platform is really a technology which monitors and analyzes the behavior of the operational side of the network the grid network so in an energy delivery environment you have i.t which is where all your business systems live and then you have ot which is where your operational controls those kinds of things that essence is designed to monitor the ot side of the house so basically where the rubber meets the road and then give you visualization through a passive monitoring system of what's happening in your network is it real is something else happening allows you to see anomalies so if say something in your grid physics is changing like somebody opened a switch some kind of automation operated there should be a communications angle and there should be a good physics angle it gives you a view of both of those and can tell you if one is doing something and the other one isn't there's a really good chance that there's been some form of cyber attack on your system so it gives an operator of you almost like a big red dot to say if something is wrong please go take an action of this and super important though it's all passive monitoring there's no active action we take um so basically it's not probing the network it's just giving you this view of your network perfect can you just sort of bring us up to speed on how essence started how did we get involved we were involved on the ground floor of this how did energy get involved what are some of our partners and how did it come to be essence 2.0 from its first iteration so much like any products it goes through different lives so 1.0 and 2.0 or are just the
evolutions of where the essence platform's been going it started its development long before i joined nrca um about eight years ago with my predecessor craig miller and some others in the team essentially developed a passive monitoring sensor called c4 through a doe program department of energy and then they progressed forward through different programs federally funded and other internal funding into this essence platform which also includes this grid state monitoring platform that goes with it so it was accelerated in different ways with federal funds we started with the oe funding most recently in the last few years before essence 2.0 we had the darfur radix program which is where grid state was tested in a contested cyber range environment and then after that we got into essence 2.0 which is where we brought in the federated escalation which is where the data can be collaboratively shared between locations and allow for this collective defense action that's one of the things that actually makes it fairly novel and so it's it's gone through various evolutions we have our stable essence platform and then we have our technology development platform so like most products we've got multiple angles going at the same time that's great can you just sort of walk us through a scenario where a co-op might need essence the capabilities of essence to protect its system start from the intrusion and walk through how the network administrator would get notified and what they might do with that information the way intrusions happen we actually start well before even someone sees it in their system the way people get into the system at the moment you've heard a lot about fishing through all sorts of different angles i think we've done a really good job of teaching people about fishing recently but when somebody has clicked on that email you've given access to your network in some way that you may not realize affects you later on there's an actor that's possibly sitting there that's pivoted potentially into your operational side and they generally sit there for a bit you don't see anything they're they're living there quietly when they decide it's time for them to do something or there's some form of trigger the operator or the system itself would measure some form of deviation so one of the attacks that we might consider is spoofing so your operators are sitting there they expect a switch has taken an action so the automation in the system has told them to open that switch or change voltage on the device or do something to change the state of the system that operator is sitting there the actor decides i'm going to tell you that's happened it hasn't really so the operator thinks the action has taken place otherwise it really hasn't what essence and ot recording does essentially is there should have been a communications action if that didn't happen or that communication didn't even make it out from when the operator requested it it's interesting it'll also give you an action that didn't happen so nothing changed on the communications network something did change somewhere else and now the operator gets a series of alerts on their dashboard that hey we think this is spoofing b your switch is actually still in this state there was no physical reaction on your system see here's what you probably should do so here's your playbook and then the operator can run through that playbook and take a protective action of some kind be that sitting there and giving this series of switching events that somebody needs to perform to isolate that part of the system waiting to get help so calling for essentially the federal side to come support either the fbi or even reporting it to eisec the operator takes that action and takes the appropriate steps to get help and response okay so i mean essentially it's got a constant idea of what a normal state on on your grid is and if there's anything that kind of falls out of line with that it sends a notification and lets the system administrator know that there's an action that needs to be taken is that a fair summation yeah one of the additional things it does do is it gives you what's called a network ontology sometimes people may have seen these before it looks like a big spider diagram it's basically what's connected to what and who's talking to who the example i like to use is that the tweeting fridge if for some accidental reason someone connected their tweeting fridge to your skating network it would pull this up as a warning of hey we have an unknown actor we're sitting there do you know who this fridge is and what they're doing then you can say fred you should leave go back to your day job so it's a fairly simple kind of elegant solution what exactly makes this unique from other methods of cyber security is there a way to kind of quantify how this is different i mean so ot monitoring is definitely a burgeoning and kind of new leading edge type field it's been around forever but in reality there hasn't really been a need for people to have as much monitoring as we're looking at and so all of these technologies that are coming up i mean i can name probably 18 different technologies and vendors that are working to do these kinds of things and all of them have their different capabilities all of them have their unique solution again the reason this is suddenly bursting into the surf limelight is well things have been happening on the ot side that really started in 2017 so everyone's been sort of coming to to get ahead of the curve on this one so essence and the ot technologies all have a lot in common and the thing that makes essence a little bit different is it's actually leveraging it's been developed by the co-ops they've been supporting it and also it uses that co-op network to benefit itself so the more data you have the better the solution can be the quicker you detect something we use the co-op network in this way that the data from essence can escalate to your neighbor so for example i'm sitting in a north carolina right now multiple co-ops here one of them can have essence another one at the other side of the state if one is experiencing a cyber attack on the ot network they can escalate that event data to the other one and say hey are you experiencing the signature that will help for that response it gives them a heads up even before they're attacked or before an event that's happened that they can respond so federated escalation is one of the interesting pieces of it that since joining this program have found to be most interesting um so it's like collective together defense are all we're all in this together as they said things and how many co-ops at this point are actually using the system complicated question um we have uh up to about 15 at the moment but a number of those are mexican distribution and gnt generation transmission entities from the g t side obviously there's a number of just smaller distribution co-ops that are served by them and because of the architecture of the networks the gmt ends up being the one that has the essence node so we're working through how to do this again it's using the co-op network which is awesome but again there's i think we're around 15.
okay can you kind of give us a quick sense of what it takes to deploy this i know there's software is there hardware involved as well any other things that are involved with setting it up so there's a couple pieces of hardware they get installed in the the dmz the demilitarized zone there's our deployment manager that comes with all of it it's uh our carson joe micucci who is awesome he's a marine he comes in and he will run that like a military show to get that thing installed um but essentially it takes about 40 hours of human time from the co-op side and it takes about eight weeks about us or a team of us doing different things everything from the data agreement to the actual deployment plan to the security plan for the device itself because uh you need a security plan for your security appliance and all the way down to the training your operator needs to know how to use it all the way down to training implementation and then upkeep is about okay and you mentioned the demilitarized zoner that's sort of like the gap between the computer side and the actual downline device side okay and once you've got this deployed on the system does it take the system a long time to sort of learn what the normal is or is it fairly instantaneous and then it just kind of learns as it goes so there's a set of rules built in um grid physics don't generally have varied rules there's rules of physics that don't change so it can be deployed immediately with a set of rules because we know that if a voltage is showing at 1.15 per unit or 15 over nominal then we're probably in the wrong system so there's a set of rules that are deployed initially but over time it does learn so we deploy i think 21 rules as it's set up and then as we we deploy more locations and get more data this is the normal baseline of the system let's this is not abnormal but the same thing happens every day so yeah it learns that over time when it's deployed it's functional and then it has the rules to work by and as things evolve it gets touches right it sounds like this system can actually be used for more than just cyber it sounds like it can be used for sort of system monitoring as well is that true it could be i think there's there's a lot of platforms for just clean system monitoring that have been out there so one of the things we've talked about is how do we become interoperable with them because we don't necessarily want to reinvent the wheel so it'd be better for us to evolve and work with the platforms that your system wants that's something we've been working on as part of the maybe the 3.0 version they start having system interactions come in and would it be useful for a non-utility user are there applications for like banks or insurance companies things like that uh it can be useful for different aspects of energy delivery it's really designed for energy delivery systems banks have some pretty cool cyber tools and some of them are actually pretty useful in the you know when money's involved some of those could be used for the grid as well and we are leveraging expertise from different industries i think banking military all of those things have done really excellent work in cyber machine learning and data analytics and the more that we talk to other industries the more we can bring in so we wouldn't ever not say never we wouldn't necessarily deploy with a bank but we would probably deploy to protect the infrastructure serving the bank okay let's take a pause here for a word from our podcast sponsor and when we come back i want to ask you about the colonial pipeline ransomware attack along those lines is made possible by ipk's power partners who wants you to know that ipkeys can improve your cyber posture while increasing durham's resiliency in your network as the number of durham's iot devices increases so does the number of vulnerability points learn more about how ipk solutions with continuous network monitoring can help you improve your network's resiliency and cyber posture to mitigate the risk of a cyber attack and secure your critical infrastructure visit them at ipkeyspowerpartners.com i'm jessica johnson from pooter valley rea in fort collins colorado you are listening to along those lines welcome back one thing i sort of wanted to get a sense of is we all heard about the ransomware attack on the colonial pipeline earlier this year can you walk us through would essence have been effective against that particular attack are there things that as the lead on essence you looked at and saw okay if they had been using essence things might have turned out differently so the the colonial event was definitely attacked on the i.t side of the house so it was
where i mentioned this earlier where the business systems are the billing system was a problem where something like essence is super useful is giving you the data from an ot monitoring device the essence are anything else that operator of that oil or gas or network and basically sit there and say there's no sign of an intrusion even the people like eisac or the fbi can look at it as well and say we think your system is safe continue to operate or we think your system isn't safe you should probably press the red button it's more being able to get that back up and um support than it is being able to prevent the i.t site if that makes sense okay and you just mentioned eis act i i did sort of want to get into the reporting capabilities and the communications capabilities of this and you mentioned that earlier as well how they can talk to other co-ops and let them know that there's a threat you're also talking to federal authorities and state authorities as well what is it about essence that makes that possible and makes it secure and useful in a way that had not been there before so one of the the newer parts of our setup is that we're bringing that event data into what we're calling our threat analysis center so when it comes back into our analysis center this is for looking for the novel threats so the things that we didn't necessarily have the signature for so when things are coming back we might want to look back historically from two years ago and say was that signature actually there we just didn't know what it was like anomalous behavior that we just we couldn't attach to anything that's where people like e isac and analysts from there can come in um part of the 100 days work that i think we'll talk about a bit later was to work out how to partner with those entities and allow them to help us we need analysts it's a thing we need people to help us analyze the system so we can work with the anonymized data being shared again we don't share anyone's pii we don't share even what the name of your company is the data comes up to this anonymized platform their analysts can go perform their magic they can help us with those indicators that then get passed back hey by the way take a look for this in the rest of the data i think it's a collaborative public private setup you don't have to share data with our platform for that purpose but there's benefits yeah and if you could just quickly let our listeners know what is eisek and what is its function and how do co-ops kind of work with them yeah i stack this the electric information sharing and analysis center there's a number of different isac entities and they basically are the analysis team they perform in-depth both eyesight classified analysis and open analysis they get all the reports of what's going on in the grand cyber world and essentially do analysis to say here's a public report what's going on anyone that works in cyber and the electric sector has probably signed up for eisek and yes lots of emails from them and you also mentioned the biden administration's hundred day challenge this was something that the national security councils put out as sort of a voluntary effort to get utilities specifically to assess and improve their cyber posture what exactly did they ask utilities to do when how was essence used to meet that requirement so the initiative started out essentially saying that to be able to address these novel threats on the ot side of the house we really really need to get more data we need to get data from utilities from co-ops from public power to be able to analyze the state of the entire network not just the individual entities that had an agreement for one year they wanted basically i think it was 250 entities covering i think it was over 100 million people they wanted that kind of level of data again wouldn't necessarily say we'll just give you all our data and hope for the best we had to input from the industry side on defining like what data do you need where the states are going to go and how is it going to be collected and that's how the industry partners being nreca and a number of others came together to talk actually openly about how to share data with the government from the ot side of the house so the 100 day initiative was launched on april 16th which happened to be the day after the colonial event which you know if you want to launch cyber initiative that being launched beside another cyber attack is kind of a challenge and it was a hundred days of essentially asking co-ops public power to sign up for deploying one of a number of platforms from this list essence being one of them so we were asked to do this and also help with setting up data sharing agreements working on you know how does the government actually get access to this data and what are they going to do with it there's also aspects of how to train and how to improve cyber posture from the non-monitoring side so the training the assessments the technology plans taking entities that were maybe an earlier stage of cyber maturity or earlier stage of communications maturity and getting them moving along as well here's your next five steps you need to take to improve your cycle cluster the third part of it was looking at the insurance world so in the insurance side there has been ransomware insurance but they were trying to to bring together industry working groups as well through this to basically work at home sure for cyber physical events that part is still ongoing okay as far as essence goes you mentioned earlier that you're always testing it you're always trying to try to improve it give us a sense of what essence 3.0 might look like our focus right now is getting the technology out into the co-op sense we want to get as many people deployed as possible i think we have a list right now of over 60 entities that are considered priority by the federal side because they're over a certain number of customers and we're working through that in a very organized fashion to make sure we get technology out there interesting trying to deploy in the middle with pandemic because it does require people to go to certain places so we're working through that we've worked out new remote deployment capabilities where we essentially talk someone through installing it themselves which is an interesting outcome so that's worked out well but it's our priority now and probably for the next few months is making sure it gets out into the right hands because like i said more data is better for the development following that the stand-up of our threat analysis center is the the big piece getting analysts that can work in there and also making the right partnerships with the federal side so that we would get that whole center and the whole workflow together so that everyone's got the right data we're just moving forward the 3.0 version of it well that completely
depends on what the data looks like when it's coming back to us um as the external entities and the adversaries start evolving we need to evolve as fast as them so we're agile we're trying to move as fast as other people do it to get the right analytics in place as a technology development group we do like to look at the data and say okay here's our brand new thing that we thought we do want to start engaging with some of the larger cyber analysis groups the entities you hear about on the reports that give good cyber threat analysis we want to start engaging with them and bring together some other probably more research site topics but things like uh data standards for information sharing be great to have one yeah so yeah but we want to bring that together just so that we can collaborate more with other entities there's six other platforms that we know co-ops have installed we'd love to collaborate with them as well it's great and i hope you'll come back and give us an update on how it's going because it really is a great example of how co-ops sort of come at a problem differently they use their collaboration they bring forward these really unique solutions so thanks so much for coming on and sharing your expertise on this emma thanks i appreciate it and thanks to you our listeners and to our sponsor ipkeys for more on this and other podcasts visit us at electric.coop until next time for along those lines i'm scott hoffman i'm paige eaton from wood county electric cooperative in quitman texas thanks for listening to along those lines subscribe and rate us on your favorite podcast app [Music]
2021-09-23 12:34
