Data Science: Ethical AI and Responsible AI (with FICO) | CXOTalk #724

Data Science: Ethical AI and Responsible AI (with FICO) | CXOTalk #724

Show Video

We persist our model development governance  process and we define it around the blockchain   technology, so it's immutable. Scott Zoldi is the chief   analytics officer of FICO. We're a company that focuses on   predictive models and machine learning models that  help enable intelligent decisioning, decisions   such as fraud detection, decisioning such as  risk. At the heart of our company are analytic   models that are driving these decisioning systems. My role as a chief analytics officer is to drive   the decisions we make with respect to machine  learning and analytic technologies that enable   these decisioning software. That includes a lot of  research in areas such as machine learning as we   address the increasing digital needs out there  with respect to decisioning systems and software. 

You're using a variety of models and  collecting and analyzing lots of data.   You're in financial services, and so the whole  field of responsible AI is very important to you.   When we talk about responsible AI or  ethical AI, what do we actually mean?  Responsible AI is this concept of ensuring that  we have a level of accountability with respect   to the development of these models. I like  to talk about it in four different pieces. 

The first is robust AI, which is essentially  ensuring that we take the time to build models   properly. That means understanding  the data that we are receiving,   ensuring that it's well balanced and not  overtly bias. It includes choosing the right   sort of algorithms and doing the robustness  testing to ensure we build models properly. 

The second part of this is explainability or  interpretability of models. What is driving   that decision in the model? What has  the model learned and is that the right   thing that should be driving decisions? The third for responsible AI would be the   ethical AI component. That is essentially making  sure that it is not having a different impact for   different types of people or groups of people. The final piece, which is largely ignored today,  

is this auditable AI, so being able to audit  how the model was developed and ensure that is   going to operate as it's intended as it starts  to be used to make decisions about customers.  If you do all that, you instill a set  of processes and accountability around   how these models are developed and ensuring  that you're monitoring those to ensure that   they're staying in check as they're making  really important decisions about customers.  We have a very early question from  Twitter. This is from Eric Emin Wood.  

He's wondering to what extent is this concept  of responsible AI part of your work at FICO.  It's really central to my work.  In fact, I'm really pleased;   I've authored now 20 patents in this area, so it's  a huge focus for myself and for the entire firm.  With the types of models that we make (for  example in credit risk), we have to be very,   very certain around how we develop those models.  We have to have a set (corporately defined)   standard for how we develop models at FICO and  enforce that. One of the things that we like to  

focus on there is emphasizing that these types  of models that are impacting customers should not   be left to data scientists artistry but rather,  follow a prescribed and responsible AI framework.  The way that works is that we have a corporate  defined modeling model development governance   process, which is adhered to with respect to the  scientists where we have the checks and balances   where we monitor that and we record progress along  those lines so we can demonstrate not only that we   want to be responsible but we are because we  followed this process. We can enhance that   process over time, so it's a huge focus  for our firm. I think it's going to be a   huge focus for every firm that is going to be  applying AI and machine learning technologies   in the digital world in the coming years. You made a very interesting comment. You said that   ethical AI, responsible AI, and managing these  tradeoffs in the models should not be left to data   science artistry. What did you mean by that? Today, what we generally will find is  

different scientists will have different ways they  want to solve the problem, or maybe they went to   university and they got trained with a particular  type of algorithm that they just really know well,   they enjoy, and they love. Some algorithms  are better for responsible AI than others.  For example, a lot of deep learning models might  be great for image recognition but they may not be   the right decision when you're making a financial  decision where you have to be very prescriptive   around what was it that caused Michael not  to get the best score and maybe impacted the   quality of the loan that he was hoping to get.  Constraining the data scientists to say these are   the technologies that we find responsible,  these are the technologies (from a machine   learning perspective) that are interpretable,  and this is how we're going to build models.  Although there might be 101 different  ways we can build this model,   we're going down these paths. They're corporately  sanctioned for these reasons. We can revisit that   from time to time, but we can't have Sally, Bill,  and Frank all building the model differently   based on the data scientist that's assigned. It has to be really something that is constrained   and something that is continually reviewed in  terms of best practices for a particular firm.  

That's a big part of responsible AI. Otherwise,  you're going to have 101 different sort of reviews   of whether or not you think the model is built  responsibly and ethically versus saying this   is the process that we use and we'll enhance  it over time but we're going to enforce it   and ensure that we have consistency  in the development of the analytics.  To what extent is this data science  artistry, as you were describing, the norm   today relative to the organizational standards  approach that you were just talking about?  The artistry, I think, is the majority of  what's happening in the industry today.   One of the things that I like to talk about,  Michael, is just getting companies to think about   how are their organizations set up from an  analytics perspective. Is it a city-state model   where you have lots of different analytic teams  and they define their own sort of methodologies,   or do you have a centralized, corporate standard? That's pretty rare today, but I think,   increasingly, it'll be more and more common  because these concepts in responsible AI is   actually a board-level and a C-level conversation.  It's not sitting with the data scientists. 

The individual data scientist will not have the  entire purview of how to ensure that their models   are responsible. It really takes that corporately  defined standard, but many organizations don't   have that today. They have piece parts. One of the things that I hope that every   organization will do (if you're thinking  about your own operations today) is to ask   that question. Where is it written down,  and is everyone using the same processes?  Obviously, once we define that, now we can start  having a conversation on how to make it better,   how to innovate around it, or how to  create technology to get more efficient   at getting these technologies built responsibly  such that we can meet our business objectives but   also ensure that we're not harming people because  we have the proper controls in place around that.  We have a question directly on this topic.  This is from Wayne Anderson on Twitter.  

He says, "Why have more companies not formalized  digital ethics and responsible AI programs   at scale? Is it an issue of it's not a priority  or are there larger blockers?" What are the   obstacles towards having this approach  being more prevalent in the business world?  Some of the main blockers, it's just a lack of  visibility around how important this is at the   board level and the C-level. I'll give an analogy. Many organizations today have their chief security   officers coming to the board meetings and talking  about how the organization is protected from cyber   security threats. They'll have regular readouts  because the organizations understand that cyber   security is a tremendous risk to the organization,  their operation, and their reputation.  That same thing can be said of responsible AI,  but we don't have typically chief analytics   officers or heads of analytics teams  talking to boards about the relative   risks of the AI models and machine learning  that they're using in their organizations.   That is probably the biggest impediment is just  awareness of the risks that are being taken on.  Frankly, we've seen a lot of bad examples out  there from very well-known companies. But I think,  

increasingly, we're seeing regulation  increase in terms of what's considered   high-risk and responsible ways of  developing models. I think it'll become   more front of mind for these organizations. They may have to then say, "Well, okay. now we   start to understand that all this AI hype around  the fact, "Just let them all figure it out," is   dangerous. We need to put the controls  in place. Then they can start to ask   the questions around how they are properly  organized such that analytics doesn't have,   let's say, 100 voices in an organization. It has  a single voice and it's a voice that is really  

informing that risk profile that the company  is taking on with respect to their use of AI.  To what extent must data scientists (in your  view) have a nuanced understanding of these   ethical issues and tradeoffs, as distinct from the  functional success of their models and algorithms?  One of the things that I like to try to promote  is this concept of explainability first and   prediction second, which basically means that  we have to stop rewarding data scientists for   high-performing models as if that is  the only business objective we have.   The proper business objective should be that  we understand the model, we can interpret it,   we know that we're following a responsible  AI framework where the models are ethical,   and we're monitoring those models over time. We need to broaden the data scientists'   perspective so they see the full lifecycle of  these models, including how decisions get made,   so that they understand the gravity of the  decisions that get made with these models   and their role in that. Otherwise, what happens  is the business owners become callused because   they say, "Well, the model told me so," and the  data scientists are removed from the business, so   the customers that are impacted, because they're  working in a protected, enclosed data environment.  Showing them the entire implications of  responsible AI and the fact that they have a   really important role to play in ensuring  that they're taking these precautions   to ensure that the models are built  ethically, responsibly, and robustly   such that they can be audited I  think really helps to close this gap. 

Very often, that's when the data scientists start  to say, "Listen. This is way too big for me to   understand on my own. I need some framework to  help me." That starts this conversation around   this model development governance process. Now they say, "Now I have the guide  

rails that enable me to ensure that we're  meeting a standard," that, say, a FICO would   define or their individual company would define  as a proper way of building these models.  We have another question from Twitter relating  to the governance point that you just raised.   I have to say, I love the questions that  come in from social media because we're   having this real-time conversation with  folks who are listening. The audience is   so sophisticated and smart, so insightful. Arsalan Khan, who is a regular listener –   thank you, Arsalan, for that – asks, "As more  and more companies use AI that can affect   millions of people, potentially, should we only  be relying on the good faith of these individuals   and their companies to be responsible?  Should the government be involved?  I think the government will be involved. The way  that I see regulation stepping up is that we have,  

for some time, had a number of companies that  promised to do no evil and then, ultimately,   find these big, big mistakes occurring.  I think the government will take a role.  I think the other major aspect of this is  that it's not going to be good enough to   say that I intend to do well, I had a pledge  I would do well, or I have an ethics board   that makes me feel as if I'm doing well. One of the things that we do at FICO,   for example, is we persist our model development  governance process, and we define it around   blockchain technology. So, it's immutable, meaning  that what backs our intent to be responsible is a   model development governance process. Then what backs the fact that we've actually  

done it is this blockchain technology where  we can persist those steps to the blockchain   so that it's immutable. We've  demonstrated we followed that process.  I think, in the future, we could see a day where  government agencies or regulators will say,   "Can you show me what your responsible AI  development framework is? Can you show me some   level of evidence that you have a  proof of work, that you follow it?"   I think we may see more and more of that. It's been my strong hope that organizations   will develop this themselves. I see some great  headway, particularly in the financial services   space where different organizations are coming  together to try to define that. Very much like   the ISAC model in cybersecurity where different  subgroups of industry have their own cybersecurity   standards that they want to follow. Ultimately, as we see some of the GDPR  

and European legislation around  AI developing and our internal AI   policies being developed around responsible  AI, I think we'll see more and more government   involvement because we're not seeing enough proof  of work that we go beyond words and we actually   have established processes that are adhered to in  many organizations to ensure that proof of work.  Scott, you just mentioned the use of  blockchain to make these responsible AI   decisions immutable (as you described it).  Can you elaborate on that for us a little bit?  What we have in these governments, the  model development governance frameworks are,   what is the model we're trying to build? What was  the data that was used? Who are the scientists   that are involved? What are the steps and the  requirements that we need to see let's say within   different sprints of a model development process? Very often, this is an agile development process   for models. We will persist the  progress through the blockchain. 

If a data scientist has gotten the data  together, we'll say, "What does the data   look like? Have we done some tests around  whether there is any sort of bias in there   or data quality issues?" That will get exposed. It'll be reviewed by an approver. That approver   will either give their approval or deny  approval and ask the data scientists   to go forward and do more work. All those steps get persisted, along with names   [laughter] to the blockchain. So,  it's an accountability measure.  What we're trying to emphasize is that  we are documenting (in that immutable   blockchain) all those steps. So, at the end of  the modeling process, we have ensured that we  

have done all the proper checks that are  part of our process to inspect the data.  We have used the prescribed analytic technologies  in terms of types of machine learning models.   We have used standard codebases that define our  variables. If we've added new variables, it's   gone through a formal review with stakeholders  so we don't learn about it two years later when   we didn't know something was in there. We do the  stability, robustness testing, and ethics testing.   All that gets put to the blockchain. It actually comes down even further that that same   blockchain becomes the source of record that we  monitor models. This is a really important point.  

You're not ethical and you're not  responsible until you start using the model.  I could tell you that I built an ethical  model, and you'd say, "Great, Scott,"   but it doesn't really matter until it starts  to impact customers. And so, an important part   of this (in addition to showing proof of work) is  to monitor that model for things like bias drift.  What would happen is you'd identify what those  key (what we call) latent features are that drive   the behavior of the model, and you'd monitor  that. As soon as those latent features start   to have a distribution that starts to get  misaligned, you have an alerting mechanism.  It's really important that that same  blockchain is the thing that's going to   inform us of how to monitor that model and  maybe how to adjust to changing times—like   the pandemic that we are in and hopefully  getting through in the near future—because   data changes and the way the models respond  to change, so having that visible is really,   really important. It serves lots of purposes,  but it gets down to a pretty atomic level. 

You mentioned using blockchain to therefore help  detect whether bias is creeping in, but your   analytic methods for detecting bias can also be  biased. And so, how do you sidestep that problem?  What we're trying to focus on is, as we  develop the latent features for these models,   that's where we would test. Let's say we have  different groups of individuals. We want to show   that the features that drive the model behave  similarly between different protected classes.  We can do that on the data set that  we see we developed the model on.  

We set criteria under an acceptable amount of  variation across, let's say, protected classes.   Then going forward, though, what we do is those  same measures that allowed us to pass an ethics   review with respect to how the model was built on  historical data are the same exact metrics that   we're monitoring in the production environment. That's how we go through the review of these   latent features and what's accepted or not. Then  we ensure that they continue to stay in line   within the production environment. Michael, we do throw away a lot of   latent features. And so, we may find that a  latent feature seems to make a lot of sense.   You and I might look at it and say, "Yeah, that's  reasonable. There's a reasonable relationship."  

But then the ethics testing would  say, "Oh, but Michael and Scott,   you didn't really realize that these two different  groups have very different types of distributions   and behaviors," even though it might make  a lot of functional sense or notional sense   why it works. And so, that's the process. Yeah, it is to root those out and get out   of things where you and I might – because of  our perspectives, because of the time that   we spend – we might say, "Yeah, it looks  like a great feature," but ultimately,   we need all that tooling in place to say, "Yeah,  maybe it was. Except, no, it's not ethical. You   have to remove it." Then you continue to rebuild  that model and find other latent features.  On this topic, we have another question  from Twitter, from Lisbeth Shaw who asks,   "What if the responsible AI framework  developed by the analytics and AI team   is at odds with the corporate culture  and the business goals of the company?"  This is not an analytics governance standard. It's  really a company governance standard. I think the   first thing that has to occur is you have to find  somebody at the C-level or the executive level   that is going to be the sponsor. We don't want  to have an analytics team that has no voice here. 

That sponsor is not someone like a chief analytics  officer. Then is it the CEO? Is it the chief   risk officer? Is it the CTO that's going to take  that responsibility for essentially the software   within decisioning that impacts customers?  I think that the question is the right one.  I'd say one needs to find that executive sponsor  and it has to be at that C-level that will say   yes. If it's not an analytics person sitting  in that seat then it's going to be someone that  

is going to take on the risks associated with  the usage of those models in terms of business   processes, whether it be a CRO or someone else.  Then we don't have this sort of disconnect between   what the company wants to achieve  and the risks associated with   the AI model. But it's a real problem,  obviously. We've seen many occurrences of this.  If an organization has troubles with that,  then we have to go back to – I like to ask the   question sometimes – "When is it okay not to be  ethical?" Generally, people have a very difficult   time with that question. We want to put ethics  and responsibly first and have companies make   that one of the tenants because, frankly, one can  solve most of the problems in a responsible way   if they take the time to do it without really  impacting the performance of their models.  There are literally an infinite number of models  that I could build to solve a problem. Some are  

good. Some are bad. And some are ugly. We want to  avoid the bad and the ugly and go for the good.   And so, these things are typically  not at odds with one another,   but it's an education thing around we can meet the  business objective while still being responsible.  From that standpoint, it's not much different  than sustainability or other corporate objectives   or corporate investments? Let's put it this way. Absolutely. I think that's the conversation that   has to occur. Frankly, that's the challenge,  to that earlier part of our conversation.  For the last decade, we've been hammered with  "AI will solve every problem that you have,   Michael. Then with cloud computing,  we'll do it cheaply. With open-source,  

I don't really need to write the code." But we didn't stop in all that hype cycle and   all that energy to actually understand that right  now, in a period of time where AI has to grow up,   we have to take the same sort of fiscal  responsibility for the technologies here. These   ones are as dangerous as any technology that we  would use within our businesses, primarily because   the biases and instabilities are very often  hidden unless you really spend time to prescribe   how you're going to build it to expose it to the  light of day and to have a conversation about it.  We have a couple of more questions  from Twitter. Wayne Anderson  

asks, "In the larger data ecosystem, we see  very divergent values in Europe, China, Russia,   and the U.S. for maintaining data. How does this  affect the calculus for governing AI at scale?"  For global companies, it's a key challenge  to try to mediate all these differences.   I guess I'd respond in this way. What  we generally focus on are looking at   the most stringent of data requirements. If you look at GDPR and some of the European   regulations, the ability to get consent, to  ensure that we have an ability to demonstrate   where that data came from and maintain  data lineage is a best practice. Building   to that best practice is probably a good idea  because the United States and other countries   are looking to Europe as a leader in some of  this legislation and best practices. I think,  

in other areas where the standards might be a  little bit lower, the corporation needs to make a   decision around whether they are going to take the  high road, I think, with respect to how they're   going to conduct their business within a country. As an example, very often I'll work with clients,   and the quality of the tags, essentially  whether someone is good or bad, is dirty.   Maybe it combines a credit risk behavior with a  fraud behavior. A client might say, "Just throw  

it into one model because I want to figure out  loss," but those are two very different behaviors.  Very often, we have to say, "No,  we're not going to do that. We have   to be very precise about what that tag is." And so, in addition to trying to adopt maybe   the best practice, one might (from a corporate  perspective) say, "You know what? We're not going   to do the right things in Europe and then take  a lower sort of position or a lower ethics   position in a different region only because  maybe the standards are not as strict there."  That's part of the tradeoff and the decisions  that need to be made. But again, typically,   it's an education thing and it's also going to  be a corporate reputation piece of, will we try   to meet the best practice with respect to things  like the right to consent around data, the right   to explain, and the right to remove data from  models. If you start to think along those as a  

future possible set of constraints  across the majority of the business,   then you're prepared to meet a global demand  with maybe a best practice that you work towards.  Again, Wayne Anderson wants to know, "Can  you achieve bias management in responsible AI   if you aren't prioritizing diversity in the  workforce of machine learning?" – A very   interesting question – "And how do you shake up  the perspective to look for a different view?"  Arsalan Khan comes back and says he believes that  diversity is important when developing AI that   will address biases and perceptions. So, they're  really saying the same thing. How do you weed out   bias if you don't have diversity in the workforce  (among the people who are writing these models)?  Again, there are a couple of pieces  here. Yeah, one wants to have diversity   in the workforce. That's number one. We have to have that focus with respect to the   workforce perspective. That includes differences  in terms of the types of people hired, the parts  

of the world we hire them from, and potentially  different exposures to different parts of the   business. I think that's critically important. With respect to a governance process, these are   the sort of things where this governance  process is revisited on a regular basis.   It could be quarterly, or it could be even more  rapidly if someone has a difference of opinion. 

One of the things that we will routinely do (for  example in our own processes) is reassess. If we   say, "These are the technologies that are not  allowed," as new things develop, we're more   than open to go and to change or to revise our  decision around a particular algorithm. And so,   we have diversity in terms of ensuring there's  flexibility in the corporate model development   governance standards that we use, but also  ensure that every scientist has a voice.  I think the best organizations out there that I've  seen really make a concerted effort to make sure   that each and every person within an organization  has a voice to express a difference of opinion and   to challenge a standard. Usually, if that's done  well, then the scientists will have that diversity   of thought, these will be exposed and discussed,  and the standards get better because of it. 

Another great question coming in from Twitter  (again on this same topic) is, how do you   address the veto power of an executive who  doesn't agree with what the data is saying?  A couple of things. I think there has to be a  tiebreaker environment. One might have the veto   power, but that itself should be something  that would be from an ethics perspective. If   you have the key executives, let's  say that each of them have a veto,   that would be something that we should actually  be reviewed by the independent ethics board   to understand why is that being vetoed. Now, if it's being vetoed because that   business owner doesn't like a set of ramifications  with respect to the business outcomes, that can be   reviewed in the context of what is the risk-based  model. I think risk-based models are probably   the very best way to address this. They're used  extensively in areas like anti-money laundering. 

I think we should try not to get to a flat veto  power. We should look at it from a risk-based   perspective. What is the risk in taking this  decision and why does this individual feel this   way? Then expose it in terms of a set of risk  parameters that the company overall will try to   take versus denying of a particular model or not. There will situations, potentially, where   maybe it's so egregious that the data analytical  organizations might feel that they have to   escalate or they need to really broaden that  conversation. But having that risk framework   in place, I think, will be a lot better than just  having, let's say, an executive that is either   misinformed or potentially bias in their own  view to a particular type of model or an outcome,   to expose it for the entire set of executives to  ponder that risk-based decision they need to make.  In summary, then you're depersonalizing  these decisions, but doesn't that also   require the executive to be willing to  submit to the decisions that are made   by that process and by that risk committee? Correct. I think most executives  

will if there's a process in place. It is very often (at these executive   levels) decisions where not everyone gets  what they want. Some people have to concede.   They'll look at the pros and the cons,  and their concerns will be acknowledged.   Then the group will make that decision. I think that's one of the things, from a corporate  

accountability perspective and, frankly and  effectively, working executive team that they're   already used to. It's just that now we have to  give them the framework so that that conversation   can be had where AI, ethics, and things like  this may not be their core competencies.  We need to help them with what are the  risks, what's at stake. Most of them won't   decide to take down their firewalls because it's  expensive because they understand what the risk   is. The same thing has to occur on the AI side. Education, education, education is another really   big part of this so that they understand.  Most of them, if they're doing their jobs,   will make very good, rational decisions  when they have that framework in place. 

That's a really, really good point that,  in many respects, it's not too different   from recognizing the importance of  the firewall, and we don't just simply   take it down because we feel like it today because  there's something that has to be dealt with. We   have a considered decision process to get there. Yeah, and that's part of that growing up and   that maturity. I do a lot of work in  cybersecurity, and these frameworks are   there for a reason. [Laughter] They're  there to protect us from ourselves and   well-intentioned employees that make mistakes. That's the key thing I hope to see developed  

over the next five years is more of these  frameworks in place. I don't think people   will push against it. I think a lot of people  that are perceived as not supporting ethical   responsible AI just simply don't understand  it or don't understand that they need to pay   as close attention to that as they would, let's  say, the security posture of the organization.  One difference between the firewall example and  when we talk about models and decisions based on   models is the firewall is really like insurance.  It's protecting against a negative event that  

may happen. But when you talk about these models,  you're talking about predictability that can have   a profound benefit on the business outcomes.  That's number one. Number two, the technology,   understanding the model requires a deeper  technical understanding, and the implications   of the model require a deeper technical  understanding than simply the abstract concept,   "Well, we're going to turn off the firewall,"  which is pretty simple, pretty straightforward. 

Agreed. Right, and so I think one of the things  that will help enable this because the executives   also don't understand how firewalls work  or more sophisticated technology in cyber.  I think one of the things that is part of this  is this example of what we call humble AI.  

I could imagine a conversation going like this: "Hey, this is the risk that we have." The company,   as a group – at this executive level and maybe  with the help of the executive committee – says,   "Okay. We are going to take the risk because we  think we're okay with that risk, but this is what   we're going to do. We're going to monitor it in  production. If it crosses these thresholds, we are   going to drop that model, and we're going to move  to a simpler model that we better understand."  That concept is called humble AI. It's this  sort of concept that we're going to have a   backup plan. We're going to have a fallback plan. The thing that's not occurring very often right  

now, Michael, is monitoring of these  assets. We did a survey with Corinium,   and we found that (across CIOs, CEOs, and chief ML  officers) that only 20% of leading organizations   actually monitor their models. Part of this would be, "Hey,   we'll take on some risk here, but we're going  to have this monitoring process in place.   If it crosses these thresholds, then we're going  to admit that it becomes too big of a risk for   us to take and we're going to drop down to a  simpler model." That's what we want to get to  

is that fallback but also a data-driven decision. We don't want it to be an academic conversation   (at the end of the day) because, yeah, you're  right. They're not going to – each of these   executives – have an ability to opine on  the relative risks and values of deep,   deep technology. But having those guardrails  in place and ensuring that if things are coming  

off the rails that you have a fallback plan is  critically important. I think that's also part of   the responsible AI piece that we haven't touched  on a lot, but this monitoring and what you do as   you have to pivot is a core part of remediating  or removing some of that risk in that decision   because you have a way to reverse that decision  or to adjust that decision if things are not going   the correct direction. We have another set of questions,   the same question from two different  people. This is from Eric Emin Wood and also  

from Lisbeth Shaw. Essentially, they both  want to know what is the responsible AI   framework at FICO and how is that working out? Ours is based on this concept of what we call   an analytic tracking document. This analytic  tracking document is kind of core. It's been   in place at FICO for 15 years or more. Essentially, it is describing all the  

steps of the modeling project, the objective, the  success criteria, the tests that have to be done,   who is doing the work, when is it going to get  done in each of these sprints, and then how many   are approved by myself and then regularly  reviewed as part of sprint reviews. We have   become very accustomed to running through that  responsible AI framework. It's working out well.  I have some of the very best data scientists  in the industry that are incredibly smart and   intuitive, but they appreciate the gravitas of  decisions they're making, and so they have found   ways to work within the frameworks, to improve  it, and to innovate around it. We don't see it as   a hindrance. We see it as a way to  ensure that we're operating in these   responsible swim lanes. And we're protected. A lot of our data scientists feel very well  

supported in terms of an ability to build  the models appropriately in the first place,   but also to flag issues throughout the  model development. More importantly,   once we establish what that looks like, we  run through it, and now it's like a well-oiled   machine – we generate hundreds and hundreds of  models with these processes – the scientists now   look for incremental ways to improve it. What I  really like about it is that they are making that   standard stronger and stronger each year with new  IP and new ideas around how to address some of the   challenges in the market. That allows us to build. I think one of the things that data scientists   really need to keep focusing on is that we  get stronger as an organization or as a group   of data scientists not based on an individual's  achievement but based on the group's achievement.   I think my team just see a lot of value in that  where, eventually, you get a framework around this   responsible AI development process  that, in itself, will superpower each   of them to achieve more in terms of better  incorporation of new intellectual property,   a venue to have conversations to challenge  what an ethical or responsible AI framework   looks like, but also to make sure that, when  the models go out, no one is losing sleep around   how they're performing and how we're monitoring  those to ensure that they're still on the rail. 

It sounds like you have built a culture  inside the company (and especially among   the data scientists) that prioritize responsible  AI and evaluating these models for bias, risk, and   all the other things that you've been describing. Correct. One of the things that even my strongest   research scientists will say is, "FICO is not  a cattle competition." We're not impressed with   small, incremental performance improvements. We've had enough time to understand that if you   see a little bit of a performance improvement,  that's probably not real. [Laughter] When it   goes into production, it probably won't  hold up. It's not going to be robust. 

We very much have gotten them, let's say,  off an academic track of performance is key   to one where robust and stability, continuity  of models are really, really important. You're   absolutely right. It's a different culture in  terms of what is the success criteria for a model.  Going back to what I said earlier where I said  explainability first and prediction second,   what are the success criteria? Prediction does not  have to be the number one. As soon as you start to   adjust that – transparency, interpretability,  constraining it to be palatability, ensuring   that you have controls in place that it can be  monitored and you can drop back to a different   technology – those are really, really good success  criteria where you're impacting human life. 

Saying that you have a particular genie  coefficient on a model and it's bigger than   the one next to it really doesn't  help anyone out. You need to ensure   the rest of these pieces are in place. Yeah, sure, we care about performance,   and we don't usually give up much on performance.  But we're not going to do that at the cost of   potentially doing things irresponsibly with  respect to models and harming people in that vein.  As we finish up, what advice do you have  on this whole topic for data scientists,   and what advice do you have for business leaders? For data scientists, I'd say this: Think a little   bit about your own analytic organizations.  Are they talking about this actively? Is  

there differences of opinion? Should you all get  together if you are kind of a city-state model?  Start to come up with your own AI governance  framework so that you can have a common   groupthink. You're not going to be able to  influence an organization on the executive   team if there are a hundred voices. There has  to be one consensus around how to do this.  There'll be some puts and some takes but have  those conversations, educate one another,   and see what that best practice is by getting  some of the leaders in your organization to   get together. Then make it a priority for them. For the business side, let's make sure that the   risks that are associated with AI and machine  learning is an active conversation that's   occurring at the executive team level and at the  board level. Then asking the question, "What are   we doing about it? What are those processes? Is  it written down?" We have these things codified   in so many other parts of businesses, but  not necessarily the machine learning piece.  If those two things occur at the data scientist  level from, let's say, the ground level up and   then at the top level (being aware of the concern  and the risk that organizations are increasingly   taking with these technologies), my hope would  be that they meet in the middle at some point and   they more effectively sort of corporately define  what that standard looks like. Both sides will get  

what they need in terms of ensuring that models  are built appropriately and the business functions   at a relatively low risk while they benefit  from machine learning and AI technology.  Okay. What a fascinating and very  quick discussion, quick 45 minutes.   I would like to thank Scott Zoldi,   Chief Analytics Officer from FICO, for taking  time and sharing your expertise with us. Scott,   thank you very much for being here today. Michael, it was a great pleasure. Thanks  

for giving me the venue to  discuss these ideas with you.  Thank you, a huge thank you to everybody who  watched and especially to those people who   asked such great questions. Now, before you go,   please subscribe to our YouTube channel and hit  the subscribe button at the top of our website   so we can send you our newsletter. Check  out We have incredible shows   coming up. We'll send you notifications. Thanks so much, everybody, and I hope you   have a great day. Keep the conversation going  on Twitter and LinkedIn. See you later. Bye-bye.

2021-10-28 10:24

Show Video

Other news