From Navy to Legal Tech: AI, Cybersecurity, and Leadership
>> Daniel Gold: Welcome to BDO's Legal Tech Talk podcast. We're joined by judges and legal professionals to discuss emerging trends, regulatory updates, and the latest headlines. We'll provide tips to help your law firms and legal departments make the most out of legal technology. >> Jared Crafton: Hi, everyone. I'm Jared Crafton, BDO's Forensic Technology Practice leader.
>> Daniel Gold: And I'm Daniel Gold, Forensics Principal and leader of the eDiscovery Managed Services Practice. >> Jared Crafton: Let's get started with this episode's exciting topic. >> Daniel Gold: And welcome back to another edition of BDO's Legal Tech Talk. Today,
we have a special guest joining us again. That's right, again. We had the pleasure of speaking with her briefly at the New York Legal Week conference earlier this year in 2024, and we enjoyed the conference so much that Jared and I said we had to bring her back for a more in depth discussion. Andrea is a partner at Norton Rose Fulbright in the New York office. She serves as the US Head of Technology and the US Head of eDiscovery Information Governance. I'm not quite sure she has enough to do. We'll have to talk to her about that. She's got over a decade's worth of experience in the legal tech space. Andrea has been at the forefront of integrating advanced technologies like AI
into legal practice. Andrea, welcome back to the show. We're so glad you're here. >> Andrea D'Ambra: Thanks for having me. Happy to be here. >> Daniel Gold: Well, when you were with us in person in New York, we talked about a wide range of topics. And for all of our Legal Tech Talk listeners, they may recall some of the things we talked about was Gen AI, because frankly, who is not talking about Gen AI right now? But before we get into any of that, I really would love to know much more about your background. So can you fill us in? Obviously, I've got the great bio brief on who you are,
but if you, in your own words, Andrea, could just give us a snapshot of your background, who you are, and what you've been doing? >> Andrea D'Ambra: Sure. I started out in the Navy. I went to the Naval Academy and then served seven and a half years as a US Naval officer. My last job in the Navy was working at the Fleet Information Warfare center, which was the Navy's sort of computer network defense command at the time. So I got a little bit of exposure to cybersecurity right around the year 2000 when Y2K was coming online. That was a lot of fun. And then I went to law school for three years at William and Mary, graduated, did a one year clerkship on the Eastern district of Pennsylvania, and then moved to my first firm, which was Drinker, Biddle, and Wreath. I always knew I wanted
to do something related to technology because I am generally pretty technologically facile, but I wasn't quite sure what I can remember selling. Somebody I was interviewing with that, I wanted to do cyber law. And they said, that's not a thing. And I said, oh, okay, well, that's awkward. And now it is a thing. So that's kind of nice. But I started out doing IP litigation and general commercial litigation and ended up because I understood technology, getting these really massive cases where there were tons and tons of documents and we had, we were using sort of the first iterations of electronic review. So that was pretty fun. Did that for 10 years. And then David Kessler called me up one day and said, you should come to Norton Rose Fulbright. And I said,
well. And he goes, no, you have to come. And I said, okay, fine. And really was the best decision I could have ever made. Very happy at Norton Rose Fulbright. I'm the head of technology, which is not the head of it. Just so we're all clear, I sort of help steer the firm's outreach to our technology clients because I have a fair number of technology clients. And then of course doing the eDiscovery information governance thing and also privacy and cybersecurity. Although for
me on the privacy side, most of what I'm doing is advising clients on cross border data transfers and working on sort of international litigation, as Jared may remember. And then also on the cyber side, I am doing the data analysis of exfiltrated data. So when hackers steal data from the company, we have a process to very efficiently find the PII personally identifiable information within the exfiltrated data. And we can determine whether we need to report on that or notify. >> Jared Crafton: So there you have huge job, Andrea, huge job. And as you mentioned, we have worked together, which I've always enjoyed. But I'd like to ask you about the beginning of your career. You served your country,
so thank you for that, first of all. Second of all, you know, you were working, I believe you said, at the Institute of Warfare or Warfare Institute. >> Andrea D'Ambra: Fleet Information Warfare Center. Yeah. >> Jared Crafton: So I imagine tons of data, right. Rigid hierarchy,
right. How did that prepare you for working at a big corporate law firm? >> Andrea D'Ambra: Interesting question. So obviously I'm very familiar with the chain of command and I do think that the skills I learned as a naval officer relating to leadership and working with both my peers and my superiors and managing up and then also managing down for the folks who worked for me, those were all really useful skills and I think have served me well working with my friends and colleagues over the years and just having that exposure to what back in 2000, the Navy was doing to protect its systems against, you know, all sorts of outside attackers. You, know, I, knew the lingo, I understood that. You know, obviously it's been a while, but, you know, now that we're working on cyber events, some of it's coming back. >> Daniel Gold: You mentioned something really interesting just a moment ago, and you said one of the skills being in the Navy was both managing up as well as managing down. That's a leadership skill that is worth practicing. It's a craft almost. And
I think that sometimes folks that are within organizations don't quite master that skill. And I think it's a really important skill to be able to effectively, most importantly, effectively manage up. Can you talk a little bit about some of your experiences of how you honed in on that and why you realized that was important, how you became better at it? >> Andrea D'Ambra: Oh, gosh, yeah. It's an incredibly important skill and, you know, it depends on the person you're managing up to, right? I had one partner very early in my career who was just very disorganized. Right. Wonderful partner, did amazing work, brilliant even,
particularly in front of a jury, but on a day to day basis couldn't keep deadlines or any, you know, any sort of things like that. So realizing that that was a challenge for this partner, I had to really sort of focus in on making sure that, you know, I kept the deadlines and I kept, you know, keeping this person on pass so that we didn't run into any problems in other situations. I work with another partner who, has particular issues with conveying information in a way that makes sense to the people that they're asking to do the work. And so I'll take the associate aside and say, okay, when you're told, you know, I want you to write this memo, you write an outline and you take it back to the partner and you show it to the partner and make sure that they sign off on it so that you're all on the same page and you don't do a ton of work and come back and then have them say, oh, that's not what I was looking for. Right. So things like that, where you just know, you know, we all have strengths and weaknesses and trust me that I, I have colleagues who manage me in great ways. If you can sort of figure out what
those strengths and Weaknesses are and then help supplement that. Everybody's happy. >> Jared Crafton: I love your theory of management very similar to mine. And I think, you know, as we manage ourselves even right, we got to manage those strengths and weaknesses. We have to put ourselves in positions to succeed. And whether that's, you know, putting a brilliant minded but disorganized attorney with a detail oriented attorney or, you know, finding other ways, that's essentially what we're trying to do with technology.
Right? We're trying to maximize strengths and minimize our weaknesses and get more efficient. Can you talk a little bit about how you're personally using technology, you know, to accomplish that with your teams? >> Andrea D'Ambra: Obviously, communication wise, we're all using collaborative tools like teams and obviously email. My kids think that my whole job is writing email, which is only partially true. But if we want to start talking about Gen AI, I use it. Not to be clear, I never put in confidential client information into any Gen AI tool. But I am using it to. If I want to say something, I will put into ChatGPT. I want to write an email that makes these points in
a way that is not aggressive. The parameters are. And then I take whatever it gives me and I might, I might tell it to, oh, well, make it more concise or make it less flowery or give it this tone or that tone. And it's really amazing to me how it does it in a way that when I see it I then go, oh, okay, I can take this. And I do a couple of different edits and it's great. I recently did it on a, doing a motion and I wanted to explain a concept that is somewhat sophisticated, right, To a judge that may not, you know, understand. It's statistical sampling. You know how to do
statistical sampling. And so, you know, I was working with Chat GPT and I'm like, you know, let's come up with an analogy here that is going to really explain it in a way that is easy to understand. And we went through a couple iterations. He tried something with cake. I was like, no, we don't want cake. But we finally came up with jelly beans. You know, you reach into the jelly bean jar and you pull a random sample and then, you know, you can generally tell, you know, what jelly beans are in the jar. It was really something that I'm not sure I would have come up with on my own, but worked really well and I was able to form it into something that I think is going to be effective. We'll see. >> Daniel Gold: I'm fascinated by this.
Andrea, what do you think we are learning about ourselves as humans when we go through this prompt iteration. So, and what I mean by that for everyone listening, is in order to be able to utilize Gen AI effectively, we have to be able to ask Gen AI the right questions, we have to put in the right prompts. And so when we narrowly tailor it to become more refined and more specific, and we're talking about tone, we're talking about pitch, we're talking about stylistic changes. What do you think, Andrea? That we are learning about ourselves as human beings through this iteration process, this exchange that we're having with Gen AI? >> Andrea D'Ambra: I think that we're learning that we have a lot of shortcuts in our head that we don't realize are there. When I write the first prompt, I'm like, it's so clear who could not, you know, understand exactly what I'm saying? And then, you know, ChatGPT will come back with something that's not quite there. And I think to myself, well,
I wonder how many other people out there don't know what my shortcuts are right and might not be, understanding what I'm saying. And maybe I really ought to think about that too. >> Daniel Gold: So do you think that through our iteration process with Gen AI, that we are somehow learning to interact with human beings better because we're asking humans better questions? >> Andrea D'Ambra: I think so. I really do. I found that when I'm writing requests and emails, sometimes I really sort of think more about the language that I'm using and not trying to just, like, get to the final ask and, you know, maybe explaining a little bit more about what I'm looking for than I did before, but, you know, I think so. >> Jared Crafton: Well, if I may play armchair prompt engineer therapist for a second, you know, there's been a lot of talk of tone in the last few minutes. Right. And
if I may ask you a personal question, you know, is tone something that you got a lot of feedback on earlier in your career, and now you're more cognizant of it when you're writing these prompts because you think it might be a blind spot? >> Andrea D'Ambra:Andrea D'Ambra A hundred and zillion percent. I can remember being a very young associate, and that was one of the big things that was sort of the feedback that I was getting. It was, you know, the tone's not right. You know, you need to use gentler language or more aggressive. So, yeah, and until you just said that, I didn't even think about it. >> Jared Crafton: Well, I'm somebody who's also had that feedback in my career. And I also approach my prompt engineer in the same way. And I think, you know, this is just a
great example of how powerful, powerful this technology is. And going back to the theme, it's how we maximize our strengths and minimize our weaknesses. Right. If we know tone is an issue for us, we now have a way to check that without going to another human being, which is it's increasing my efficiency a ton. And it sounds like it's really working for you as well. >> Andrea D'Ambra: Yeah, there are just times when I don't know the words to. You know, there are people, brilliant people out there who are at, PR firms and things like that who know how to write the crisis communication, and they know how to use the right language, and I know how to use language, and I'm very, you know, I think I'm fairly persuasive when I'm writing, but there are times when, if I'm asking for a difficult thing or it's an interpersonal thing, that I can go to ChatGPT and I'll say, write this email to this client asking them to pay the bill, because they haven't paid it yet. You know, something like that. And it'll come back in an. It's in a way that I wouldn't have been able to craft myself. or if I did, it would have taken me, like, many,
many iterations. And like you said, sort of going to somebody and being like, does this sound good? But, yeah, it's shortcutting a lot of that, I think, to bring it back. >> Daniel Gold: To some of, what Jared said, by the way. Jared, I may use that expression armchair prompt engineer, therapist. I like that a lot. That was really good, Andrea, earlier in your career as well. One of the things that I noticed with a lot of leaders
that Jared and I interview is that, there's a lot of lessons learned. And you talked about a couple of them managing up. You talked about tone. Are there other things that you've learned along the way that have made you the effective leader that you are today so much? >> Andrea D'Ambra: You know, certainly in the Navy, I learned a lot about how it's really important for everybody to feel like they're part of the team and that we're all sort of in this together and it's not, you know, go away and do this thing and don't bother me. Certainly I've learned a lot about taking ownership of things and making sure that, you know, I, think that was one of my real strengths when I was an associate in particular, is that, you know, if there was a brief and it had to be out by midnight. It was going to get out by midnight, you know, no matter what. I think that served me very well. There are probably a million things, but those are the ones that come to mind.
>> Jared Crafton: You mentioned ownership and, you know, given your current role and the work that I know that you do, which is under heavy, heavy deadlines, right? You have reporting deadlines, regulatory deadlines, all kinds of deadlines. Could you expand upon the ownership of tasks, point a little bit more? Because it's just so critical, right? When you have these deadlines and there's so much pressure to get stuff done, you know, how does owning that, you know, really make a difference for the team? >> Andrea D'Ambra: In a couple of different ways. I'll talk about sort of our cyber events. We actually will divide up the tasks and we have this document where we basically outline, okay, this is what everybody's doing on the team. And Chris, who's one of our partners, he's the global head of our group, he will handle all the investigation remediation because he's brilliant at that and just really knows all that part of it. But he'll leave the data analysis to me and my team. That
allows us to work both cooperatively but also, you know, on different silos so that we're not putting off things that could be done now while, you know, they're dealing with the investigation, remediation, parts of it. And the ownership really comes down to if you have people on your team who are taking ownership of the tasks, they're going to get done and they're going to get done on time. And if for some reason they don't believe that they're going to be able to get done on time, they're finding the resources to fix that problem, right? Because our deadlines, as you mentioned, are not ones that we can push, right? They're ones that we have to meet. You know, people don't thankfully come to me and say to me, well, you know, not going to be able to get this done. High five, good luck. Our team is really responsive and really comes through in the clutch.
>> Jared Crafton: Could you talk a little bit about how that trickles down to your vendors? Because I know that, you know, you are reliant on vendors in some of this process. You know, so how does that trickle down? You know, you're owning it, but you're still reliant on some people that you don't really have control over. >> Andrea D'Ambra: It's a challenge. But I will say the vendors that we work with, that,
you know, consistently that we go back to are the ones who also are taking ownership of their parts of the project. And that's something that I really appreciate. And it goes, you know, not only to making deadlines, but being proactive about, you know, if there's a problem that might come up on the horizon. Right. And making sure that when there is a problem, that it's about fixing the problem and not avoiding blame or placing blame, you know, that sort of thing. But, you know, let's fix the problem and learn from this and move on.
>> Daniel Gold: Legal teams are struggling to manage resources, maintain compliance and adopt new technologies. BDO's legal operations team can help we enhance efficiency, reduce costs and support compliance. So you can run your legal department like a business. Schedule a meeting with us today at bdo.com. How do you end up with a role, Andrea, that has, like, probably this, the most important and pressing issues tied to the legal landscape? I mean, that's to have so many various different aspects, Andrea, under your title there, that has to be a lot, correct? Andrea D'Ambra: Yes. Although it's funny, my practice on a, you know,
month to month, year to year basis will swing somewhat dramatically sometimes. Jared, when we were working together back in 2022, I think I was like 75% cyber at that time. Right now I'm like maybe 15, 20% cyber. And most of what I'm doing is eDiscovery, information governance and privacy. There are a few of us who sort of span a couple of different, you know, sort of the four categories of stuff. And then other people in our group are,
specialists and just do, you know, cyber or privacy? the discovery people tend to be a little bit more fluid just because we can help out on different parts of the cyber and privacy stuff. >> Jared Crafton: Have you found ways to incorporate Gen AI in your work? Because I know last time we spoke and I think, you know, your answers thus far have all really been on the personal side. Right. You're kind of playing around for emails you're writing and this and that. But has there been any advancement in terms of actually, you know, contributing on projects with it? >> Andrea D'Ambra: Well, the motion, you know, drafting that sort of section of the motion was really helpful and that was work related. And then sometimes I'll put in cases and ask it to summarize the cases. Part of what you have to do when you're using any of these AI tools
is make sure that the summarization is actually accurate. Right. So sometimes you'll look at it and you'll say, I m don't know if that's right. Or sometimes it will miss a nuance. I always say it's great for a first draft, you know, and then, you know, you sort of move on from there. >> Jared Crafton: Well, actually, one of the things that we've been playing around a little bit is finding ways to do some bigger projects in Gen AI. That's going to take multiple people working on and collaboration in the tool, which has been proven to be a little difficult for us in figuring out how to, you know, kind of chunk out the pieces of a big project and then bring them back together and check all that. So I'm curious if you, you tried any of that yet? You know, do you have any advice from that perspective? >> Andrea D'Ambra: I really have not done any big projects with it at all yet. You know,
we're still testing out various AI tools within our firm. You know, on the ediscovery side, there, is a lot of talk about, you know, various AI tools. Obviously, we've used rudimentary tar, you know, for 14 years, something like that, at this point, you know, these new sort of generative AI tools that can summarize transcripts of depositions, things like that. That's interesting, too. I haven't used it, but a colleague of mine was using it and he said that the summarization, it's good. It does a good job of sort of, you know, giving you an overview of what was said during. But it doesn't call out like those important, you know,
smoking gun moments in the. It doesn't recognize that that's a really important point that they made. So it might sort of brush over it. I've used it personally, I haven't used it in work, but the AI summarization of meetings, you can get assistant to, summarize what your meeting was. And
what we found was that it does an okay job. It doesn't really always get the context. When we were discussing it among our group, we were like, we need to get better at prompt engineering. When we're in the middle of our meeting and saying stuff like, oh, an action item for next time would be, or, you know, that sort of thing. We are just at the very beginning of what this can do. >> Daniel Gold: What do you think, Andrea, the attorney's responsibility is? And when you're talking about summarizing case law, and it missed some of the salient points of the case law. Many lawyers may not think what, how you think or what you think about Gen AI and may just summarize it. To summarize it and be done with it and drop it into
A brief. But do you think that they've got a responsibility here to take a look at that? >> Andrea D'Ambra: 100%. 100%, yeah. In fact, courts have come out and said that. So you can't rely on it to do your work for you without supervision. Just like you can't rely on,
you know, your subordinates to do work for you without supervision. In the same way that I have to check, you know, the case law in the brief or the motion I'm writing when, you know, my associate gives me a whole bunch of stuff, I have to do that with Gen AI and I better do it too. >> Daniel Gold: So that's interesting that you say that. It's odd that many lawyers have not picked up on the fact that there is this, under the rules of professional responsibility 5.3B,
to oversee non-lawyers. And we're going to call Gen AI non lawyer for a moment. But you've got cases like Mata versus Avianca, Park versus Kim, People versus Graybill, Cruz versus Carlin, Smith versus Farwell. I mean, all within a year have, all talked about attorneys not checking these hallucinated cases before putting it into a brief. And now it's there, codified in public records forever. How do you feel about that? >> Andrea D'Ambra: I think it's unfortunate. I think that some attorneys, I think,
give Gen AI too much credit and maybe think of it as a panacea to their detriment. Obviously, there has been a lot of discussion among the various courts out there about whether they should pass some rule that you have to disclose if you're using some sort of AI tool in your briefing and stuff like that as a way of dealing with this. And I really don't think that's necessary. I think that you, know, I was a clerk a million years ago. We didn't have Gen AI. But I will tell you that there were lots of hallucinated cases and even then, and you know, cases that were cited for things that they didn't actually stand for, things of that nature. So. And there were ways of dealing with it then. Right. You know, so I think the court has the power to deal with it even now, but I think that some of the judges aren't as familiar with these technologies. And it's sort of, a reaction, an immediate reaction,
to sort of try and address this. Hopefully we'll all get more comfortable with it. >> Jared Crafton: To me, the promise of Gen AI is really bridging the gap with laypeople and sophisticated technology. And obviously that's a spectrum. But for me directly, I build A lot of complicated dashboards, right? And these are dashboards that we can train people to use and some people are more comfortable than others and, you know, are happy to get in there and click around, but you can get lost in these things and being able to ask a dashboard a plain English question and get back the answer you're seeking to me is just completely game changing. It's really bridging that gap with technology that, you know, I'm most excited about. And you've made a couple points on this podcast that have really made me think that, you know, it's not just the technology bridging the gap, that we have to bridge the gap as well. When you mentioned a few minutes ago that, you know,
in your teams meetings that you would like to start, you know, more explicitly saying this is an action item or this is a follow up, I mean, that's a great example of how the technology is starting to nudge the human behavior a little bit to help bridge that gap. I love that point. >> Andrea D'Ambra: Just to clarify, we don't use it in our work stuff. That was in personal stuff. We have a lot of clients right now who are considering whether to allow AI summarization. You know, when you see the outputs that are coming out right now, I'm not a fan, because it's creating a record that is not necessarily accurate. And you may be under, you know, if you happen to
be talking about stuff that is subject of a legal hold now you've suddenly got this obligation to keep something that isn't accurate kind of a big concern. But I'm sorry, you had another question. >> Jared Crafton: Well, it's really about bridging that gap, right? And you've given us a few examples already. I'm just curious if you've given that any thought, you know, to how, you know, you and your team are going to work with the technology and how you're changing your behavior because of the technology. >> Andrea D'Ambra: I think that we are thinking about it a lot, you know, ways that we can leverage it. I think that there's a lot of hope that down the road we're going to be able to say to the generative AI tool, go out and find me, you know, all the most important documents related to this claim or that claim and give me a summary of them and, you know, cite to them, that sort of thing, and that it will have a good understanding of what's important and what's not. Right? That sort of distinction, which I think is something that, quite frankly, you know, Gen AI is a baby right now and it's going to grow and it's going to learn and it's going to get, I think, a lot better at that, but it's definitely not there. Now, my one colleague was using one of these generative
AI tools, and she told it, go out and find the document that supports my opponent's position. And it came back with a document. She was like, oh, my God, this is terrible. And, you know, she was really freaking out. And then she opened the document, she's like, oh, it's their complaint. Like, you know, it's got a ways to go. And then how do you validate the results you're getting? Right. So much of what we do, we have to validate,
and so we have to figure out ways to validate that isn't doing all the work all over again. Right. Because now that's not really saving you any time or money. >> Daniel Gold: If you take what Jared asked you just a moment ago and you really expand upon that, just even more, there is this question that's out there right now about how much should an attorney actually know and understand about what's in this black box of gen AI? So, I mean, think about some of the concept we're talking about. For those that are listening that may not be familiar. Some of the things we're talking about is technology that's compounded. It's built upon and layered upon one another, really. From the 90s, from the 2000s, we're talking about natural language
processing, natural language interfaces, we're talking about information retrieval algorithms, query understanding, document indexing, all this amazing technology that was invented 80s, 90s, and 2000s, all led to. And obviously having the technology where it is today, the speed and the machines and virtual servers we could spin up, et cetera. We're now at a place where all that research and all those algorithms and tech can all come to fruition. Now in generative AI, I wonder what your thoughts are on, an attorney's need to understand how that black box works, what's inside of it. Because when. And you mentioned it before, 14 years ago, we were talking about Technology Assisted Review. I'm not quite sure we are farther along, 14 years later, Andrea. And an adoption of Technology Assisted Review. But yet here we are. We've
talked about this with several guests on the podcast before, yet here we are talking about generative AI and everybody's doing it. But should we understand it as lawyers, what's behind it? >> Andrea D'Ambra: I don't think it's necessary for us to understand the algorithm. right. The nuts and bolts that are in there and how they work. But we do have to understand
how to validate the results. Right? And it's the same with tar. It's not necessary to understand exactly how TAR does what it does. I finally think I understand how it does what it does. I don't think it's necessary. But you just can't throw a bunch of documents in there and, you know, take the output and you're done. Right. You have to be able to validate the end
result is accurate and is, you know, what you were looking for. And that's what all those cases came out. Right? All those cases related to not validating the result. I think that even in the tar, I can remember very early when TAR was coming out, some vendor was like, okay, you just throw all your documents in there and then, you know, code a couple, and it's going to spit out all the responsive ones and you're done. And I was like, hey, that seems like a big lift, having to do statistical sampling to make sure that everything is as it's supposed to be in the generative AI set. You're really going to have to go back and look at what the information was that generated the result and whether or not it's actually accurate. >> Jared Crafton: Andrea, it made a ton of sense, and this has been an absolutely fantastic conversation. Thank you for coming on again. It's a pleasure to have you.
>> Andrea D'Ambra: Oh, thank you. >> Daniel Gold: Thanks for joining us on BDO's Legal Tech Talk podcast. >> Jared Crafton: If you're enjoying these podcasts, don't forget to rate, review, and subscribe for more episodes. >> Daniel Gold: Head over to BDO.com for a list of all of our episodes, transcripts, resources, and how to get in touch with us to continue the conversation.
>> Jared Crafton: Until next time, this has been another episode of BDO's Legal Tech Talk.
2024-12-15 18:08
