Hacking generative AI: Limiting security risk in the age of AI
being in security you're constantly having to learn new technology so that you can break it so you can understand how it works under the Ood and AI is is no exception the the biggest piece for us is it it's not just you know a new web application framework or you know a new web server technology it's a brand new attack surface with a lot of really uh interesting new new attacks that can be conducted against it and so it's not something that you test months and you're done no nobody's interested in security until something breaks and then it's all we can think about so on today's episode I want to get inside the AI security mindset with two people who know it best one is a real life hacker who just happens to lead a security team at IBM called xforce yeah you heard what I said xforce the other a senior security architect at AWS joining me now Chris Thompson what's up Chris hey thanks for having me Chris welcome to the show I got to ask you how did you get here I've had the great privilege and honor to build the uh adversary services or the red team at xforce Red over the last eight years um so that team is really focused on hacking into big Banks or defense contractors or nuclear power plants and and you know showing how we can achieve objectives and not get detect it can you tell me a little bit about about the people that you work with there at X Force raid we have over I think 180 hackers here everyone contributes whether that's in vulnerability management or application security testing we're on the red team last year for example we spoke at blackout security conference six times um we put out some really Innovative post exploitation tooling um a lot of open-source security testing toolkits for the community to use Chris we've we've actually seen some things get broken recently you know a lot of big security breaches how has Genai changed hackers are doing and how you work being in security you're constantly having to learn new technology so that you can break it so you can understand how it works under the hood and AI is is no exception the the biggest piece for us is it it's not just you know a new web application framework or you know a new web server technology it's a brand new attack surface with a lot of really uh interesting new new attacks that can be conducted against it and so it's not something that you test and you're done especially the risk is compounded because of how many companies are rushing to slap AI into every product Under the Sun and they're trying to you know have that first mve mover advantage and they're not pausing to really perform proper risk assessments so we're finding when we test these these gen applications and these traditional ml you know use cases is we're we're seeing a lot of these environments have been set up without proper authentication um they're allowing codec execution in the backend um production environment they're allowing us to get access to Enterprise data lakes and you know it's really just a symptom of people trying to move way too fast um and worry about security problems later what does a day look like at X Force typically you know we we would hear from a customer that it's about to roll out you know a new financial application or new foreign exchange platform or maybe a new banking platform and they'd say hey we're we're integrating this chatbot so a customer can prer a wire transfer or look up account history for example and so we're looking at from a traditional web application perspective what are these traditional attack paths that we need to be concerned of and what are the new uh types of attacks that we need to worry about when these multimodal chat Bots and these multi-turn chat Bots that that are interfacing with different apis in the back end so we we need to figure out hey how how is this functionality being called couldn't I talk really ask this chatbot to load um insecure code from the internet could they build you know some sort of code execution flow with within the B chatbot itself um and so there's a lot of of new attacks that we really need to be aware of and threat model as we're testing these applications to ensure that you know we're not seeing customers end up in the news when these companies are rushing to integrate AI you mentioned just how dangerous it can be if you move too quickly with it what percentage would you think are actually secure that's really tough of I I think there's there's so many components to leveraging AI into a solution you have the models that go into these applications that are being trained and tuned and you know downloaded from hugging face and evaluate it you have the model training and tuning environment itself you have the whole m SE Ops pipeline which is new what's your first step when you're starting to hack a new gen project it's taking a kind of a risk-based approach to threat modeling so is this just a a you know a sales chap bot that's going to help somebody you know pick a widget off of a of a a storefront or is this you know a healthc care application that's going to allow you to query previous um diagnosis from your medical history right and so we start to think about how how sensitive is the data that's being pulled into this application from Enterprise production environments and then what um you know different systems is this interfacing with is this making calls to like an ethic Healthcare database is this making calls to like a customer relational management system how are those apis secured you know what's being exposed publicly what's privately what can we hit from within the Gen solution and then we come up with a testing plan we say okay you know this this type of testing is appropriate for this whereas you know this this we can spend a little less time on we also try to push a lot of teams to not just think about the front-end solution because we're seeing a lot of of code execution and platform compromise and and you know a lot of it's private and not hit the news and we just happen to hear about it from different customers or from Partners we think it's really important that that customers start to think about you know again that endtoend flow of how did we evaluate these models how do we know about the AI supply chain and kind of an sbom or software build materials that is part of the solution where are those python packages being downloaded from you know how is who rolled out my AI as a service platform and who configured it and who enabled logging and really helping customers to challenge assumptions around all of that kind of backend development production environment and flow so that way in the future when you know another uh AI supply chain attack happens and some python package got back doored for example customers can quickly say okay I know exactly what I used in my solution and I don't have to worry about it obviously AI is still a very new field so I I need you to be honest with me on this one from your POV do we actually have the people or the skills that we currently need in order to properly secure AI like is there a skills Gap somewhere there's a massive skills Gap if you went on LinkedIn three years ago um the amount of people that said they were data scientists or AI Security Experts versus three months ago is night and day right everyone's claiming to to be an expert in this field or they're rushing to to train up and and the reality is there's just a massive skill shortage there's already huge skill shortage and security already and now this is a new you know brand new space that we need to be constantly you know upscaling our our both our internal security staff but also you know making developers and data scientists and management aware of those those different security risks and and making sure that you know we're we're partnering with the right people that come help test or we building out that skill set in house I'm I'm trying to figure out whose responsibility It ultimately is though like is it the developers responsibility to be thinking about security as they go about building their projects or is it somebody else's yeah great question I think it comes down to a shared responsibility model and having those discussions with your vendors and your different developers and your data scientists you know a lot of people that are training and tuning these models they say that the security of the model is not their responsibility and you know I liken that a lot to you know a web application developer saying I don't need to code an application securely just put a web application firewall in front of it right it's awaf is not a good replacement for secure code just as as a you know AI firewall or guard rails are not you know a good solution for securing models so it's a shared responsibility for whoever is training and tuning those models or supplying those models it's shared responsibility with the develop Vel opers and the production teams to build out you know decent guard rails so you know they might not prevent every attack but you can at least you know have a canary in the coal mine and know if your applications attempting to be abused or whatnot or your Solutions being abused and then it's you know having those conversations with your your AI as a service vendors like you know Azure ml or Watson X or AWS sagemaker you know what testing has gone into this platform how do I show Providence of the models and and the the AI supply chain and that you know that esbon how do I know what testing you've done on this platform or this this application that you've built for me and what expectations do you have of myself for testing my my application as it goes live since you are a hacker and I'm assuming that you enjoy your job do you get excited when you find a gap absolutely um it's what we live for as much as we love you know breaking stuff and finding apps and and circumventing all those controls it's also you know as you are in the industry for a while you kind of get tired of only burning things down and so starting to have those relationships with the blue team and with the developers and saying hey we're tired of finding this bug in every single application you create you know let's take a look earlier on in the development life cycle let's let's solve this from the start so it doesn't make its way into every app these are the controls you should put in place or these the detections you should have and if if those primary controls fail here's how you should be you know approaching this well then what's your favorite thing that you've ever hacked Chris definitely some breaking into some uh military bases back in the day was was was a lot of fun we've haded major major sporting events um most of the big Banks and and you know exchanges in the world uh nuclear power plants defense contractors chemical manufacturers it's it's uh Telos uh law enforcement intercept systems everything you name it we've we've hacked it and I've had the pleasure to lead you know the majority of of uh big engagements well Chris thank you so much for joining us today it's funny I feel like I felt my anxiety Spike some when you were talking but then I also felt relieved at the very end when I like oh but that's okay he's one of us he's one of the good guys so um we really appreciate you thank you for that Insight I feel like we could really just talk about hacking all day so now knowing what we're dealing with in terms of risks let's find out how you deal with it from aws's Resident security expert Mita Saha welcome to the show thank you Albert so let's get straight to it how has gen aai changed your job it has definitely revolutionized the whole security landscape it has introduced both challenges and opportunities specifically in my job I know that now I first need to understand how gen works so that I can understand how can I secure it we also need to make sure given as an architect when I'm trying to design new Solutions or trying to do a migration I need to be more conscious about the tools I I'm using am I able to leverate geni for an efficiency in my current work that is one landscape where we are trying to learn gen learn the benefits of gen and leverage it so it has definitely impacted I would say in a more positive and exciting way because now I'm trying to learn new things and implement it as the same time can you give me an example or two about the kind of challenges that gen can pose you know through the hands of Bad actors for you but then also how you've been able to mobilize the power of gen gen has revolutionized to hold hacking landscape like on one hand it introduced new attack vectors and Amplified the existing threats generating more convincing fishing emails generating more def fake audios or even crafty evasive Mals on the other hand AI power tools can automate and streamline vulnerability detection vulnerabilities has always been there in our security landscape rather in the industry be it any industry so we can now leverage the AI power tools and it can help us automate and streamline the vulnerability detection process it can help us enable faster and more comprehensive security assessments we cyber Security Professionals must adapt rapidly employing AI for defense while simultaneously we have to mitigate the risks possessed by the AI driven threats what's the most common mistake that people make when they're securing their data or their AI as I have been fortunate to also write a white paper have been part of a white paper we have collaborated uh featured a few months ago on securing gen and we have stated in our white paper that only 24% of the current gen projects have a security component in it even though 81% of Executives say secure and trustworthy AI is essential as you can tell this suggests that many organizations are not prioritizing security for their AI initiatives from the start that is a potential oversight like nearly um 70% of Executives say Innovation takes precedence over security when it comes to gen now deprioritizing Security in favor of innovation could lead to some vulnerabilities the number one mistake that we can surely fall into is neglecting the security fundamentals or having an immature security culture in our organization which can leave our organizations ill prepared to address the conventional threats like malwares social engineering that take new forms with chaii you talked about prioritizing Innovation over security so with that prioritization let's say that some of the companies that are listening to us right now um either did do that right they've rushed to get AI ready and they bypassed the security they thought you know what we'll just do this later what can they do now what's the first thing that you would advise that they do in order to in order to Rebound in order to rectify that situation one immediate step that we can take is to conduct a comprehensive security audit risk assessment best in my opinion will be to pause the deployment right now and then conduct the security assessment instead of doing them in parallel but if in some organization in some business if that is not a possibility then at least just run the security assessment in parallel to whatever you're doing right now this audit should be performed by a team of cyber Security Experts can be from your own organization or you can hire from any of the other cyber security forms uh that you trust in and who can identify the vulnerabilities the potential attack vectors the security gaps and the a area of non-compliance with your industry standards and the AI initiative that your organization have taken second will be conducting the right threat modeling to uncover the security gaps in your AI environment and to determine how the policies and controls need to be updated to address those New Gen threats then conducting right kind of penetration testing to simulate those attacks and to identify the potten itial vulnerabilities that can be exploited and then finally I would say evaluating the systems to understand how the data is handled and the data handling practices like Access Control encryption mechanism and other potential points like networking and the most important part of this assessment should be a detailed report as a security um engineer consultant myself I believe that if I just come to my customer or my partner with the problems that won't help them right so I think a a key output of this assessment should be a detailed report I should be able to guide or recommend I'm not saying you to do this and that but I can give you a recommendation if I'm doing an assessment in your experience what's the number one potential AI related attack that's most likely right now and how can that be prevented if I have to call out one top AI related attack that concerns our SE Suite which which I have read on is uh adversarial attacks on the AI systems these attacks involve manipulating the input data to an AI model to deceive the model in a way that it causes the model to make incorrect predictions or classifications and generate harmful outputs these attacks exploit the trust and Reliance we have played based on the AI systems and it can have severe implications on the security and integrity of the systems that organizations depend on so this is going to be a rapid fire round where I'm going to list some different scenarios objects I guess and I'm going to ask you just for a simple yes or no for each of these as I call them on out using geni hackers can compromise the following your phone phone yes your TV oh yes your it system yes oh your car yes depends how how advanced I have uh built my car but yes oh well me this is a lot okay your home internet kind of um I am trying to say no but again if I don't set up the right measures then yes okay you're refrigerator not right now but again yes if I is using a iot the iot version of my refrigerator and finally your dog oh you know what yes probably yes and I can give you a use kiss for all of them but yeah how can J AI hack your dog so de fake audios say I'm working and I am not always home with my dog and I have a device through which I can talk to my dog when I'm not at home now if that device is compromised and compromised with de fake audios the J bot can actually fake my exact voice just not my dog even my friend my mother who knows me very well right and if my mother gets a call not from my phone number because my phone is with myself so the J I bought will not have my phone number but say if I'm calling from a different device and that is the Deep fake audio they do and if my coming back to the dog's reference if I am saying something to my dog and my dog is very well behaved listens to me and I tell him or her to do something in the house which can become very scary I can just say just go to the backyard and go to uh a neighbor's house and get out of the house so deep fake audios is definitely something to look out for so that's why we should be always evaluating whoever is calling us or how it is happening so okay well we're relieving this feeling a lot more confident I guess that was I think that you've given me so much to to chew upon right here so I thank you so much and in fact Mamita I appreciate all the time that you shared with us today this has been beyond insightful I'm going to speak on behalf of the listeners and viewers if I may and say like whoa you know you you've given us a lot to look out for but then also you've given us some great confidence and faith in what can be done utilizing geni appreciate you for that now again I also want to give a shout out to Chris thank you Chris for joining on today's podcast and friends that's it for today so thank you all for listening thank you for watching and of course if you have thoughts please post them in the comments below and I promise we'll see you again soon [Music]
2024-12-02 21:39
