Well good afternoon talking, to a colorful. Empty screen but I know that, more. To the point we're talking to a group of folks who have taken. Time out of their very busy. Afternoons. It's afternoon wherever you are I suspect, to, join, us for. This webinar, on. Legal. Design and we're very very delighted, to have you here, don't. Draw the lesson, from the fact that I had, the most words in my bio, for. The list to, believe. That I'm the most important, participant, I'm the least important participant I'm your moderator, Dan Rodriguez and I have the the good pleasure and fortune to chair the Council for the ABA center, for, innovation, and I, have a bunch of other day jobs that I won't bore you with but, I will mention, give. An informercial a short one for the ABA center for innovation particularly for those of you who, haven't had the pleasure to. To, learn about what we're up to we are a. Startup. I guess if I could say in in. The biz we're, a relatively, a new, center. Established, by the ABA a couple years ago as an outgrowth of the. Great work from. The Commission on the future of legal services and. The ABA center for innovation was established, by. The ABA by. The pooh-bahs at the ABA to, be a catalyst, for innovation to. Promote the. Great work that's being done in the ABA and outside, the ABA in the profession of law and law and, variety. Of organizations, in the business sector and the technology, sector in order to promote innovation, in the provision of legal services we, are obviously and, we'll hear more about that this afternoon squarely. Concerned, with the access to justice crisis, and doing, our good, work as, those, of us who were involved in the work of the Center on the council and elsewhere in promoting. And facilitating and. Catalyzing, innovation wherever. Its found also. Producing. Materials. Information apps. And. And. Work to really help help along this this, process, this. Is the second webinar. In what, we know will, be a series, of interesting. Conversations I, just want to mention that we have a webinar that is recorded. And on our, network. That we recorded, a few months back a very, interesting conversation between, my. Colleague Dean, Andy Perlman of Suffolk law school and judge, a Scottish, legal a district. Judge in Jefferson. Parish in, Louisiana. Who. Was working on is working on a very interesting smart. Supervision, program the 24th district to reduce recidivism so, I urge you to take a look at that at. That. Webinar. And also to be on our mailing list for future webinars. Well without further ado I'm absolutely delighted to be able to welcome two. Friends, of mine and Friends of the center and, great, assets, to the work that we're doing on the center Margaret. Hagan and David kolleru so Margaret. Hagen - the director of the legal design lab and is, a lecturer, at. Design. The, so called DC school, she, was a fellow at the d.school from 2013. To 14 where, she ran. For legal tech and design. Experimenting. And how design could make legal services, usable, useful. And engaging teachers. At Stanford, a series of project-based, classes, with, interdisciplinary student. Groups tackling, legal challenges, through, user focused, research and design of new, legal products, and services, she, also workshops. To Train legal professionals, in the design process to produce client, focused, innovation, Margo graduated, from Stanford Law School in June 2013, and she holds a bachelor's degree from the University of Chicago a master's, degree from Central, European universe in Budapest, and just, to show you you need four not three a degree, it's HD, from Queen's University Belfast, and, international, politics, David. Call her soul obviously, has a big fan and keeps trying to call him on, the telephone. In. An old technology, as it were Dave, is the director of Suffolk, universities, University. Law School's legal innovation and technology, lab an, attorney and educator by trainee worked, as a public defender data. Scientist, software engineer and high school physics teacher he's. The author of a programming language for lawyers heal and a markup, and award-winning legal, hacker an ABA. Legal, rebel and Vasquez, 50 honoree, Congress. Party and his JD from Boston. University's Law School and received his BA, Cornell and his meed from the Harvard Graduate School of Education. He began his legal practice as, a staff attorney for the mass public, defender, agency, I could, go on but won't because. I want to I want, us to hear from these two great visionaries, and innovators they are going to talk, about some, wonderful projects, they're involved in but I am going to point.
To Waylay them before they get. Into that to ask a big-picture, question. Given the extraordinary extraordinary. Important, and impactful, positions. They both have at, their respective institutions and, talk a little bit about, how, their. Respective, programs, work. On access, to justice issues, just to frame, and shape a little bit more about what, they're up to so let me start with you Margaret. Sure. So we had, a five-year. Engagement now, with the Judicial Council of California where. We've run a series, of almost, ten classes with. Different units inside the Judicial Council today. This, morning I was doing the latest one which is around language access technology. And language access services, most, often we partner with the self-help centers directly, in the county courts or at the administrative. Statewide level and we, do a lot of on-the-ground, work, in the courts so a lot of qualitative, work talking to people about needs. Unfilled. Technology. Possibilities. Whether they would be welcomed or not and, other types of new designs, interventions. Changes, to space changes to process, changes. To professional, rules that, both court staff and. Especially. Litigants. In the court, would. Improve their experience, and would pre improve their justice outcomes and then from all that on-the-ground work we, in the classes, and then in our ongoing projects. We, develop. Working, prototypes, of these new technologies, or new designs, for the most promising ones we actually run pilots, and do proper, rigorous, studies about whether they improve. People's. Ability. To navigate process. Show up at court get, to good outcomes so. Some of the projects, that have come out of all those classes, and exploratory work are text. Message reminders to, guide you through the divorce process in Orange County or, visual. Explainers. For traffic court in Alameda, County in San Francisco County. And then lots of digital, guides to walk people through step-by-step so we try to almost be a sandbox, where court, officials, and self-help Center leaders can. Try out new ideas and, we can also be playing, a more provocative, role, to help them really think in a more challenging way about what innovation might look like in their courts, so. We're going to talk about substance but I I'm, dying to ask you a question about process and I'm gonna bundle, this metaphor, but if. It's a sandbox the creator of that sandbox, in the first instance of the courts themselves right there there you don't work for the courts in, any formal sense so what's been the receptiveness, how what it take, us through how they came to you as it were and your and your group to help. Facilitate this. Well I think before, this so our group started. Running design workshops, for literally you know had to be had a few bottles of wine and some hors d'oeuvres and said hey court leaders in the Bay Area which. Of you either at local County roles as. Judges, as a self-help center directors, as court investigators. Or in the more administrative, offices, in the Judicial Council who. Is interested in new types, of solutions, and also in hearing more from communities. About, what the new direction should be so, we kind of had these open door events, starting in around 2013, and 14 when, I was running my D school fellowship, here and from those open door events we found the judges and especially, the people in Judicial Council who.
Had. Been thinking, around, what innovation, could look like but didn't necessarily have, a methodology, or the. Human. Capacity to like form teams and actually go out into. The field and do a lot of this testing and information, gathering so. We were able to find enough leaders, especially. In the self-help center world I think they're hugely interested, in innovation but. Just don't have the capacity to take, time off from day to day work and. From, chief information officers, who, I think are really forward-thinking, here, in California, about all the data that their groups are sitting on and how it could possibly be, used to to, change policies, to change services. So. I take it you would agree that a lot of the folks you mentioned the use term, leaders are really the unsung heroes, in all of this the folks that understand there, are some there's there's room for innovation, there's, need for innovation know, where to look for, it are willing to show up to these to, these folks not just because of the wine ng. But, because, of the ideas and the opportunities, to really partner, that's terrific, so, there's a lot of pent-up ideas, yeah you had the. Legal in innovation. And technology lab the so-called lit lab also, I take it not a reference to wine. We. You know we do look we do play upon the vernacular, usage, as well yeah definitely gonna have to have fun with her also, have fun with our logo, there. There. You go it's always better if you leave the, flame of knowledge right. Good, but. Yeah at the lit lab what we basically operate a so we're we're interesting so in your. Original question about access to justice we're, unique and that we're actually part, of our clinical program so we're a partnership between our clinical, program and our legal innovation and Technology Institute and. We serve basically as a, consultancy. Inside, the law school staffed, by myself and students. Providing. Data. Science, and legal tech solutions. And consulting, to, both clients, external and internal so. Obviously. Some of our internal clients are our clinics themselves, but, then we also partner, with courts occasionally. Small firms and. Other, entities, to help provide them with projects, and we've only been around not. Quite for a year now so. Our, portfolio. Is it's not quite as broad, as largest. As Margaret's but we have worked how do you advertise your work how do you how do you project yourself out of the commedia and say we're we're basically open for business, webinars. Like these. Also. We try to have a presence. On social media right now we're working on working. With existing partners, that we have through the university and through. Our, networks to define, people that are interested doing, interested in doing interesting things and actually, we haven't had a want. For for, people who are interested in working on stuff so you know he says oh you, want to help, build, a machine. Learning algorithm that codifies some of your daily. Processes to, help predict, whether or not a case is going to be a good match for you everyone. Says okay yeah sure so. If you want to do that please. But. But. Yeah so. So I may be more interested, than the answer this question than most of our guests on this webinar so we need not spend a lot of time on it but I'm a, professional, legal educator, I'm dying. To ask what, what how do you involve students, in this how do you how, do you engage lost. And, also students from as I know is the case in Margaret's lab from. Elsewhere the university to, to, get engaged and how our students utilized in this in this in, this in, the space and your particular labs yes, or work so in my lab the thing that's really nice is being, part of the clinical program experiential, education is, the name of the game already so, what we're basically is taking is that experiencial, education, experience, normally, working, with a client accompanied, with legal matters changing.
That To working with a client on. These legal tech matters so, it's the same model of learn, through doing, and. Helping. Elicit. That ability, for our students to be able to learn how to adapt, to, all. The things that might be thrown in so. It's really by taking. That model of experiential education and saying here is a client they have real, needs you, know this is something that we're gonna be there to help support you and figure it out but, you, have to help them get what they want and what they need and, helping, our students build, the skills they, need to be able to deliver those solutions, to their clients and so and your students, before. We turn to Margaret your student or law students right yeah these are all law students, so, we're one of the things that's, maybe. Interesting, about our program, but not so interesting but sometimes I think confusing, people is, we're not trying to produce, a bunch of attorneys. Who are going to go out and create. Technique. Technology, solutions and technology startups, by, being their own programmers, we're, not trying to turn any of our students into production, coders. What, we're trying to do is give them the skills they need to be able to recognize things out in the world understand, the realm of the possible know. If something's out there you, know can there be esta Tector. Know whether or not that's really real can, they work with technologists, to build solutions that are meaningful can. They bring, attorneys. I to something. And recognize. The bias and the problems. With, a technological, solution can. They understand, the technological, issues. Facing their clients, in, a way that's going to allow them to serve their clients because we all operate in a world now where you. Know a lot of stuff is happening in, this virtual space and you don't understand how that works it's, hard for you to advocate. For your client effectively, so we're really about building people who are able. To be part of interdisciplinary. Teams and be able to sort of serve as a translator, with a foot. In both worlds of long tech and be able to facilitate, those. Projects. That. Only really happen when you get that cross polish if, I could add a third world I don't want to put words in your mouth but also business, as a world right so it's law tech and also business and because, yeah. I mean this. Is where my background is showing I come from public, defender I've put, most of my time. But. Yeah and with the clinic's our focus is very often government, and, nonprofit, but we did have a very interesting project with. A small. Personal. Injury firm where they wanted to take their calls, data. And, figure out what. Was it about a call that made it worth. Worthy. For them to take that personal. On as a client so they could give that client. A benefit. Hand also benefited from the asset as, an office so yeah business as well we'll, take it great. How. About how. Do you involve students, in your in your design, level so. Sure I think over the years we've refined kind of more of a funnel approach where we have a big tent for all of our classes so we advertise all of our classes through, as much cross listing, and the. Wonderful thing about the design school here at Stanford is if you get your class listed through here you can get a very wide. Mix of grad students and undergrads so the most of our classes like, our language access class this quarter we. Have about 15, students only five of those are law students, 10 are coming from other fields especially. Education from. Sociology, and public policy, and really engineering. So, they're usually people who are interested, in making government services, better civic, technology. Or, kind of more holistic, like. Poverty, related work so we find some of those great crossovers, lots. Of expertise, from around campus coming, together so. Those classes, the students get to go on the ground they start. To develop some leadership skills and bringing stakeholders together and, how. To harness technology how, does to go about projects, and if they really love that work and especially if they have projects, that have tested well in class then. They can graduate up to being Ras or project, fellows where they take on even more leadership, roles in going. Out and taking a project, or a class project, level, with just so pretty raw and then getting it ready for a pilot run building, more partnerships, securing, funding, getting. Engineering, development. Done, and. Then we have some.
Like. Wonderful, students, who then kind of own their whole 3l year is spent on taking, a project, fully out to be piloted. Tested, and possibly scaled and. Sometimes, they even teach, classes, and. Build. Their own student, teams that's kind of our shining examples, of students. Really. Taking. Off, what's. It like about bringing these different populations, together these learning, the strange folk ways of, Engineers. In with with lawyer. Future. Lawyers I mean we we, think so much in the law school environment, about training our students in this specialized, language thinking, like a lawyer that, you're you're sort. Of out in the box method, tests, are interrogate, that assumption that you can't bring these multidisciplinary. Folks, together but obviously you can yeah. I think it definitely as a teacher you need to play a facilitator. To enforce, that people don't get bucketed, where it's like the, lawyer, in, the team or the law student in the team is only. Given, room to answer questions on policy, in law and then the engineer on the team is supposed to have all the magic answers, about what the perfect solution is going to be so, the goal there is to help them you, know get more realistic that engineers should not be making huge assumptions, about, how. Easy it is to change the law or to change regulations and, the lawyers should not be making huge, misconceptions. About what machine learning could actually do, like to get out of magical thinking so. We try to keep the teams really small so that they can't segregate out, and that they have to almost teach each other their, own problem-solving. Techniques, that they've been trained in their separate schools with. So, I'm gonna ask you both of you are our. Prophet. Isis, if I could say that the, idea that legal, tech design is human, centered design for, human centered design is at the heart and soul of that as I, understand it could you say a bit both, of you about what, you mean by human, centered design in, this in this particular context, Margaret.
Maybe We'll start with you yeah I'll take a stab so a little bit of it is making. Sure that as we there's. A lot of techno optimism, out there. Maybe it's tempered a little bit with everything going on with elections, and misinformation but generally I think especially. In law are really excited. Okay. We're in a situation, no, I think you, know there's so much energy over the past decade. Especially around how. Government. And civic and legal. Worlds. Will get so much better will be more efficient and more usable, but. That can lead to rushing. Into technology, projects, without making sure that the money is spent well that the technology, is appropriate, and ethical, and. Also, that it that it will actually be used so. That's where we see a lot of value in taking a human centered approach, along with a strong focus on technology's, promise that we're talking to people on the ground we're, having them lay. Out what values, and ethical, kind of guidelines. And. Where the priority, should be honestly, because I think we can see that it's. Really easy in a hackathon moment, to get excited about a project or an idea hope, that it will work and start to invest resources in but we hope to kind of ground that back into the. Community's needs and priorities. Before. You jump in harder. Question which, is on that for a moment and ask you is there a conflict between this focus on human centered design when. Much of the legal tech work and the movement it's going is developing. Toward chatbots, and not, human, devices. And not. Necessarily players replaced by robots but we're moving toward you, solutions an app selection can you square that with yes I, don't I think that's a false dichotomy because the human that's at the center of the design is the person who's getting the service so. Really. We're talking about our clients. So it's, it's it's. Not, dehumanizing. To offer a solution to someone that might be some type of technical solution the, human at the center is what. Is this going to be beneficial, to the. End-user that. End-user might be a professional, using the tool to help them assist clients or it, might be something. Like you were talking about Kay something, like a pro, se tool that, helps someone without, an. Attorney present but, in, either of those cases the, human is the person, who you're serving and, so all. You're doing at a few minutes in your design is asking is this, solution. Actually, serving that person, and the. Best way to do that is to talk to that person and I think that's really the key, thing about the design approach is that it's the scientific method you have a hypothesis, about what's going to solve, a certain problem for someone and then, you tested that by actually getting something in front of them saying does this actually solve it and, I think the reason why Laura, and I are so attached this idea of my, Center designers design, thinking in general is it's, this area of process of actually making sure the thing you think is helping is helping so, it's really building, in a success, in that trick and immunity to see is, this actually working as opposed to just we're making something cool so, let me press on that how do you do your assessments, how, do you and your labs and out of your labs do your assessments, to really to, really check in, on how how, effective, they are and, how truly, client, responsive, they are these devices, are so, it's different for every kind right, so if you have a client, who they're, trying to like the example, I use before create. An algorithm to figure out if we're matching well, with. Clients, so. Well. So why do they want that so one of the reasons you might want that there are a couple reasons you might want to codify your existing experience so. That you can interrogate, it right what are that gut feeling that we have that says we take this climate over that client what, really is driving that so like codifying, that you can interrogate and, say are these things really should. Be are, we may be making biased decisions are we always, taking people who do X over, Y and is that reasonable, I'm, you can also use it to scale right, because maybe you have turnover, in that position and you want people to be able to help make that decision consistently. Across, different, users or you just want to scale to have more people use it so, it's really what their needs aren't so you have to ask the client why do you want this and.
You Don't ever actually don't start with the solution you start with the problem so, if they say well my problem is, we. Want to scale this then that's one set of measures, if, it's we're, worried, about how we don't actually understand why we make these decisions that's, another set of measures and then, if we're talking about sort of the sort. Of a chatbot example you're talking about well, then you know you go to your very traditional, start design processes, of sitting down with clients, and talking to them I mean, that might be either in the development, process well, it's definitely in the development process but then longitudinally. Checking, in with users and saying you, know were you able to find what you were looking for just meet your needs and x y&z, manners so the, way you do it I guess is. You, establish, what is the problem you're trying to solve and then you figure out if you really solved that by having the person with the problem answer, that question and, a, note that was that's. Very much please. Margaret I just yeah that in my lab again this is another thing we've refined over the years to kind of make different methods work for, the, specifics, of especially self-help, and access, to justice, so usually like what we're gonna do at court next week is the students are going to go with their top ten ideas of how language access could, be improved through technology, and they're, going to show very high-level versions. Of their top ten ideas to. Court staff and to people waiting in line and in this self-help Center will pay them little bits of money and they'll rank the ideas against each other so even before we decide where to spend our money we, do these kind of focus. Groups and quick rankings, using something called the Delphi method so, we get like a community. Ranking, of where should we spend our money like participatory, budgeting. And then, from there we do a lot of what David is saying so we do a lot of focus groups walk. Along and ride alongs I would encourage any lawyer to do these kind of things even if you don't have technology-based, solutions but. Basically you invite. Your client, or your target user in and try have them tried to use your work product whether it's an application or, a memo you've written or a. Presentation. You just kind of watch, them and then ask them like, how would you rank this what do you think these, technical, rankings, for, our tech, prod like usability, scales and procedural, justice scales, like ten questions surveys and, then after that then we graduate, to more official like observational. Trials and randomized control trials to see how do people really behave with it in the field and those are kind of more academically, rigorous, evaluations. You. Know this. As. You say also in the works we also do data science work as opposed to just a technological, solution, working there you're you're establishing. Success, metrics, and, your than a measuring in sort of your various, or a standard. Statistical. Sense that, you're looking for things you know like false positives, false negatives. Accuracy. Is not the thing that you're usually looking for by, the example, I always give is you know if you're trying to predict whether or not school will be cancelled because of a snow day if, you sing if you have a model that says no you're. Gonna be a you know accurate ninety-eight, percent of the time because most days are not snow days so, I'm really, sort of digging in and sort of figuring, out what, are the, considering. What are the things that you're worried about are you worried about false positives, are they really costly or false false negatives really costly so it's sort of that screaming, versus diagnostic, test that you know you see in the medical community and so it really is getting to know your client what their needs are and then helping. Refine, a success metric. You. Know as I listen to this maybe this is a naive point but as I listen to this as the shall, we say the older person in this in this chat room were in is that, what you're describing in terms of the metrics and the client centered focus and some of these assessment, tools can, be broadened, well beyond technology-based. Solutions right, you think that you mentioned David for the clinic before seem those who are involved in clinical education. And live client clinics, whatever, they're serving, and however they're serving clients would, would would learn, much from, how you all, are doing assessment. Of success. And measures of success in the context of your more text tech oriented programs.
The. Key thing is that the, really. Important, thing underline is in whatever context, are doing this those. Measures of success that you choose are, going to drive. Whatever. It is you're doing so you have to be really thoughtful about those you. Know if you get into some questions about you know sort of like algorithmic. Fairness right, so if you're building an algorithm, to make some, I'll make some decision if your success metric. Doesn't. Take into account fairness. Whatever, however you want to account for that then you you're not going to optimize for fairness so, um I mean that gets a couple. Senate's they're sort of a good sort of thing that we could talk for hours about but the point is you have to think carefully about what it is you. Want to care about and if you want to care about fairness you have to have a way to codify. That and to, look at it, because. If you're not then you're not taking it seriously, interesting. So, you know one of the cent. Observation, and then a question, what. What attempt, ten, based, solutions, or the use of technology, in in. The space of access to justice as, I read the literature is, not simply, the lack of availability. Of technology, although, of course that's a key part of it but for years and years and years the lack of available, data and big data in particular that. I've cooked is now available. In a two, to an, extraordinary degree and, exponentially. Growing in all, sorts of ways but, I know both of in your work and all you you've, worked in that area. I'm. Gonna I'm gonna I'm gonna shut up. What. Do you think about. Go. For it David so yeah well, so I mean this is this is essentially, a probably, the reason why motors are together. On this, this. Webinar okay so we were at the LSC conference. Last year and. The. This. Year I have to show that big data bit. But, we, were you. Know on that Star Wars know, that. But. Yeah so we. Were having a conversation, Margaret's. Labs been working on a taxonomy, of legal issues and. We're. Having a conversation about where, that she could use. Those to help legal, service centers and like get. People to. The. Services, they need and someone. From Microsoft came by and there was discussion about how, hard it is for natural language processing to. Be able to match people up with things. And I'm like yeah well the main thing is for it from machine learning standpoint, you need is you need data, to, train things and. We had there just been a talk by some guy who, was a moderator, on our, legal advice which is a forum. On the internet, website. Called reddit if you know it a few people read our legal advice for, fun which, is something that murder and I are both hoping, will help us out with our next project, and. I've said. Well you know we just need data why, don't we ask them if we can use theirs and we'll. Have someone come in and, label, it and so what they're did it is is, a bunch of lay legal questions they say someone goes online, say. I'm having a problem with my landlord, they said X Y & Z you. Know what should I do. Yes. All right and we, can have another discussion about you know the answers they get and all that stuff what, we were interested is in the questions because these are real people with real problems asking. Questions the way they ask questions without, all the legal leads and so, what, we've done is we have a project. We're really excited about and we're. Going to tell you all to follow us on Twitter because next week there's no. But. But, we, we set together a project where we're crowdsourcing. The labeling, of this reddit, data and other, data so. That we can use it to create, really high, quality. Labeled. Data in regards, to people's, lay legal questions, meant. To the taxonomy, out. Of the legal, design lab so, that we can then train machine. Learning classifiers on it to actually match people up with, their needs using. AI. Which. Is we're just talking about machine learning which he knows it's. Not a robot lawyer just, be very clear. But. Getting people closer, to where they need, removing. Some of that some. Of that burden, maybe. Market talk more about well, I want, to hear Margaret described but before I. Do that I'm still trying to get over the. As. Someone. Who studied you, know law in the grand method, and read, all these wonderful opinions, by the great jurist, probably, no greater jurist, than the great learned at hand. Perhaps. He's spinning in his grave. This program here. I'll. Set Margaret up for it right so you know how, are we going to label all these questions, right. Oh it's so, difficult. Use. Questions there was someone to energy. To do that oh you know what many, hands make light work if, only it's, too light make light work so yeah we had this vision that we could make it actually fun to read all of these legal advice or, legal help. Posts, and then, categorize, them so that's been one of the big projects David and I had worked on over the summer which were ready to debut next week which, is a game online, called learnin hands in.
Which We present to lawyers, and law students these. Posts, from reddit where people talk for one paragraph or three paragraphs or sometimes even ten or twenty paragraphs, about, the situation, they're in and it's equivalent to somebody walking into a court or a legal aid clinic, I'm, saying here's my problem what. Should I be doing about, this I think it might be legal I don't know so. What the lawsuit or the lawyer player of the game player, of learn and hands does is just. Answers. Questions, about whether they think a certain legal issue is present and right. Now we're asking really high-level questions, so do, you think a family law issue is present, in this long post about, someone. And their mom and their estate plan and all the complications, with their will or do you think a housing law issue is present or do you think. Money. Debt consumer issue is present and with each of those yes or no answers, that our players give, to, the questions that labels. Our data sets and that goes into our the. Building of this machine learning model so basically as we get all of these labels from different players those, are like votes and then we have a consensus. Algorithm that David. Has, deployed. To. Make. Sure that if there's a consensus, if all five or ten people who see a question all agree that this is likely. To have a family law issue present, then, that has been hard, labeled into our dataset as likely having a family law issue if there's more disputes. Than we have more people look at it will. You categorize, from. All the folks playing the game what, their backgrounds, are so you'll know be able tell whether. Lawyers. Who are evaluating, these folks. Or technologists. Or students. Law students or others are. Broken. Down by cohort. David. Do you want to answer that yeah, so we we. Intended to ask some questions about so here's a wonderful thin. Diagram where we made a while back of a project. But. We do intend. To ask. People if they have a background but, we're not we're actually trying to implement a solution that doesn't require people. To have a background so the idea behind this. Sort of wisdom of the crowds is, that if there's enough you. Know as long as people are wrong in different ways then. We're going to go and, we're. Also not set in stone in, our sort of consensus algorithm so, we'll be collecting data and if we find that there's a problem with, young people trying to gain the system or something we, don't have to throw out our old data we, can recalculate how we do things and it's not like a process, where we have to know by three o'clock this afternoon, is there, a issue. Of this type in this text, our. Sort. Of fault tolerance is high we can go back and unwind things. So. Our, hope. Our hypothesis, is that a diverse. Group of people will be able to do the labeling pretty well that. Being said we're focusing the game at, attorneys. Law students, people that have an interest in the law but we're not precluding, anyone's participation. So. I'm not sure if I actually answered your question no, that's great without, getting without hurting, out too much on this could you tell us a little bit about the machine learning part of this what what, technology, exists, at the level of machine learning to help help this game run yeah, so so the game itself is not really using I mean. So we've got this algorithm, in the background to help figure, out whether. Or not something should be classified and. We. Use that to help. Tee. Up the next question so that the question someone's getting is always, the ones gonna give us the most information all.
Right So the idea being if a question comes in and everyone says is a family law we can say oh that's family law and we don't have to bug other people to vote on it but if there's some disagreement then. We bug more people about it so there's sort of that algorithmic. Set up in the game, but it doesn't have the things people usually think about hats machine learning going on machine. Learning comes, from the output of the game which, is this labelled set. Of text which, basically to be a text. And it'll say this. I'd open the tank so how many people found to be present identifying to be question and there we're using very, common. Natural language processing. Methods. So, and. There's, sort of an empirical question of what's going to work best with, this particular, set. Of data. We've. Done some preliminary testing, on some private data sets that we had that were already labeled, and. Very. Simple sort of I again. Not to get sort of too much in the weeds so just, a very simple sort of bag-of-words. Interpretation. That's like a natural. Language processing term, of art or, basically you can sort of think about. Are. There it's, almost like a like. A word cloud yes like oh they're like a word cloud kind of you know Isis, words they're. More frequently than others you, might notice that family. Law matters they, have a lot of names of relationships. Like brother sister. Mother father, and. So you can actually count those, those. Instances, of individual. Words, and then you know. Multiple. Collections, of words so you know, you. Know reasonable. Doubt or you know whatever terms of art might come in insert. Multiple. Engrams, and so that's sort of like one very simple thing and we're just doing text classification so. I say basically can we find. That out to sort of really sort of draw it back because all of that probably means nothing anyone. What. We do is we take the text we, turn it into numbers, somehow, so, that numbers might be counting. Up how, many. Times someone, uses this word or that word or, it might be using a more sophisticated method. Like something called word Tyvek which, people can google and it's like magic anyhow, the, point is we turn the text into, numbers.
And Then, what we do is we basically then. Plot those numbers in some space you, can sort of think about it like some like multi-dimensional. Space like. Much more than three dimensions and. Then we we, draw some surface. Hey, I saw the matrix I know. Let's. Say you had just a to act to to. Two, axes right and so this was how many times someone said, landlord. And this was how many times someone said tenant, right, and so if someone says landlord and tenant a whole bunch of times they're. Gonna be. Up here they don't say they're gonna be down here and then, what we do is we figure out what is that line or. That, curve, and, technically. Some space. Surface. That, divides these, type of questions from. Those, and. Then, we say if you're on this side of the line we're going to call you a family law and, that side of the line we're, gonna call you you know whatever that collection is and. So really we're just taking texts turning our numbers finding. Services. Between, homes. Of things those. Clumps of things however, part. Of the things that we're already labeled hopefully, and it relies upon things actually being caught right so there's an empirical question here as, to whether or not we'll actually be able to find all the issues so, for things like family law. We, probably will be able to find it because people talk about them in the same way. But. There are you know a consumer, protection it might be more difficult in. Which case what that means is you need more training data to. Better be able to figure out what the boundaries of that space. Are and, not to make it sound like it's all just you know counting. Words but that's a very simple way of doing it it's. Very helpful you know, something. That Margaret said and I, about. Folks, finding their way to issues. Where they don't know that we have legal problems and figuring out whether, a problem is a legal problem or no problem, and you, know I can't, help but notice the connection to his, path-breaking research, that. Professor Rebecca sandifer who's, been involved in the access to justice movement of course and it's, part of the American Bar Foundation and, I should say a couple weeks ago received, perfect. Genius grant. Incredible. For her work but, but one of her key insights, is the. The sheer, number. Of individuals, who have problems, they know they have problems they don't know it's a legal problem so, before they even get to the question of access to a lawyer, there's, the powerful, threshold, issue of, whether the issue is or is not a legal problem so, this also seems, to me has some some real insight and salience in connection with that research, yeah. Please. Yeah I think the. Direction, we're going in which I don't want to presume be happening, in the next few months or even in the next year we'll see how quickly, we can build a strong, model and classifier, but, the whole driving thing is we, know that there gonna be more people talking, about life problems, on the internet or other digital spaces so.
How Can we start building this infrastructure, and these classifiers, that. Could live, in. Search engines, on social, media on reddit on anywhere, where people are kind of presenting, if we think of it in the health analogy. And that, we could at least then spot and refer people and say actually it seems that you might have a just disabled event legal. Solution. Is waiting for you so, I think this is kind of an infrastructure, project that David and I are doing now and hopefully can spread out into a larger ecosystem of. Preventative, outreach. Messaging, to people to connect. Them to their legal rights of options. Playing. Up enough the, work of the taxonomy because that's really the key here so. The. The algorithms. We built will be able to classify things into the taxonomy maybe other, people will be able to build things based upon our label data but, it's all just the service of having a common language a common set of standards that, allow, us, to be able to match if, we can figure out what someone's issue raised then, we can match them to the service they need so, for that happen, there has to be this sort of foundational, work of identifying. What the issues are and. So that's really like it means we're building roads. Well. It's and, from a big-picture perspective it's, part, of the essential goal of, technology-enabled, design right which is taking what is after all very, complex, phenomenon, big. Date and the. Tools that can be complicated, but to get to reasonable, solutions, to make life easier. Rather than more difficult, in that in that regard I should, also say both, of you tell me whether that I'm wrong but, this whole movement toward what's sometimes, called gamification, which. Seems so incongruous, right. To lawyers from a different generation what, why are you teaching your students how to play games right. In some look but but I take it this is an example of the gamification movement. Really giving us an important, purchase and, impact. On on. On, the provision of legal services I. Think. I'm taking a lot of inspiration from these other citizen, science projects, that have come out of, physics and cosmology and. Even the captcha movement where other, computer, scientists, and scientists have harnessed the crowd to. Really, build. Good models, find, important, information and annotate a lot of data so, I think the in law can learn how to harness our wonderful, community of lawyers, and law students to do the same kind of citizenry, work to. Make this infrastructure, I was. Having a conversation with some the other day they're talking about how you people. Are trying to do the gamification thing, for law for a long time you know how do you know this is going to work of, course the answer is we don't but they were pointing, to other services, where you know like oh well game of Phi you know, marking. Up cases will be enough like this the, one nice thing we've got going for us is the data set so one of the things within it makes clear about this data set sort, of assumed so. We. Have private data sets that where maybe that, we're classifying, with, secure. Numbers, of people doing labeling, the, reason we can do this with the reddit data is because reddit bills itself as the front page of the internet so. People, are making, these posts, public. With the understanding, that. Everyone. In the world could, see them and then, there are internal. Guidelines. In that subreddit. That actually, specify. That you remove your personal identifying, information and, the, moderators, help to make that happen and actually, we. We. Got our. Data set largely from actually a couple of the moderators Pat. Well. So. Why do you want to make that clear yeah. Yeah yeah so. I'm gonna, chase. A hurdle who I should have started out at the very beginning to thank and I'll thank him now whose, are able deputy director for for, the ABA center for innovation he, is also playing the role as Empress are in.
Terms Of navigating. Questions that that guests on the webinar have so so, feel. Free to ask, in connection with this, wonderful, project learned, in hands again I'm gonna have to get used to the title. Other. Activities, and work in that in that space and, and, while we're while, we're stalling or waiting for some of those questions that came in you, can so tell, us a little bit about what's next what's the next thing. In, terms of some of the collaborations within, your your, group or or with other groups as well is the big thing is that the, game will will do a hard launch next week so, if you want to get your hands on doing this you. Know reading these things, and labeling watch. Margaret and I online a lot you'll, see a link sometime early next week and. Can you send that look I mean I assume you'll broadcast that as widely as you can through your social, network but can they and, we also maybe. I'll talk offline about putting it on the center site so they can you know and I think, actually we talked with chase by the time this video goes out will be launched so it'll, probably be included in the description, of. The video as well interesting. Interesting. So. A, couple. Questions coming in so, this one is on weird hands in particular how do you get people to participate in the study are there any incentives. So. We gave a fight it so it should be fun and rewarding in, itself, hopefully it's I've played it so much, I have so many points you have to come and try to beat me I think about 800,000, points on this platform it's, fun as, a lawyer, to, have. All these kind of on-the-ground stories, and and filter through them so there is a fun element we. Also in our testing rounds and to get large, participation we've, been paying some research assistants, who have been wonderful, and Finn our test, users, of it but. Our goal is that we can also possibly. Get pro bono credits we're talking with some law firms about, whether this might qualify as a pro bono activity, which we think strongly that it should that it really goes to a public good infrastructure. But. Yeah try it out and hopefully it'll be self rewarding enough, yeah and you should, be clear the gamification that includes things like seeing. How good you are compared, to your peers. We're. Not gonna be able to beat Margaret Hagan existed. So. Here's an idea that just came over the transom. May-maybe. Partnering, with major bar prep groups Barbara Kaplan, there's thousands of law schools in law students ready for the bar maybe a good way to train data at a high level someone, suggested and, I think we've talked with some legal aid groups about this too as they're training community volunteers. Or other people who are going to be fielding, all of these questions from the public this is a great way to know. You're doing. That triage right yeah, yeah. Interesting, interesting. So. We're still. Waiting for other questions you've, avoided a little bit elided the question of what's next if I know that you know there's probably top. Secret if you told me you'd have to kill me but. Maybe maybe I can ask it a little more opaque way which, is sort, of what are some of the access to justice crisis. Points where you think you know on your wish list what, are the you know what are the the. Moonshots, as it were in this you know in this area. So. One, of the things that's I, can't, officially. Talk, about but I'm really excited is some partnership. So. Some partnerships, with some prominent, search engine providers, about really, changing, how search results are shown. For. Anyone, who has a query, that indicates, this one of these access to justice like I just got an eviction notice what do I do. How, do I get a restraining order how, do I get my FEMA benefits. So. Hopefully. Working at one of these key source, points, which is people. Searching, online or, on social media how we really change how information is served up to direct them to jurisdiction. Correct. Public. And actionable. Presentations.
2018-12-19