Good morning, good afternoon and good evening to you wherever you are in the world it's lovely to have you with us for today's webinar my name is Alex Nichols professor of social entrepreneurship at the Saïd Business School and Academic Director of our impact programmes here at the School today we'll be looking at technology and impact measurement and asking questions about whether it provides better data for impact investing and decision making I’m delighted to welcome Tom Adams a good friend of ours and a key player in this landscape Co-Founder of 60 decibels and today he'll be talking about his work and how it relates to some issues around technology and impact measurement so we'll be with you today for about an hour so Tom and I will speak we'll try to make it about 30 35 minutes to allow plenty of time for conversation we've got a fair number of things we want to cover uh in terms of housekeeping if you'd like to ask a question please use the Q and A function you'll see it on your screen put your questions in there of course you can use the chat function to talk to each other to network to introduce yourselves but we'll pick up questions for the second half of today from the Q and A scripting so please use that if you can so just a couple other housekeeping pieces I mentioned this is a webinar that relates to our program memes so particularly this is the second webinar supporting our Oxford Impact Finance Innovations programme a brand new programme that we're running later this year starting in November and really what we're talking about today reflects some of the content in that programme we'll talk more at the end of today about this programme and the other ones we offer in this portfolio but I should also uh name check my Co-Director of this programme a new programme Aunnie Patton Power whose book Adventure Finance I’m leaning heavily on in today's slides and I would like to caveat immediately that much of this is her work uh which were she with us today she'd be talking about with me but happily she can't be because she's on maternity leave so um uh congratulations to Aunnie from us all and we'll see her for the programme in November but adventure finance book available on amazon all good bookshops uh covers a number of these technological questions as well as others so I’d like to you know kind of acknowledge her presence in the room albeit uh virtually through the slides I’m looking at with you today so as I said I’ll speak first and then tom and I just want to want to kind of address the following questions with you really so there's a lot of talk around emerging technologies in our um in the modern world generally and so uh it's not surprising perhaps it's begun to be an important topic in the specific work I do around impact measurement and impact management and impact investing and social finance a range of topics so the questions I think we have to ask is uh and these are partly claims that are being made as I’ll show you in a minute is you know does it give us better data management better verification does it give a better data analysis apologies that should say ai not just a so artificial intelligence does it help us with better data collection and lean data models that tom will talk about later and above all does it improve our decision making so whilst all these technologies are very attractive in many ways and they seem to be offering us a whole new set of tools will they actually improve decision making for investors and enterprises in the impact space and to me that's the 64 000 dollar question and towards the end I’ll raise some limitations and potentially negative effects to think about as well what I would suggest throughout all this we ask ourselves the question how do we cut through the hype because there is hype around this any kind of new technology attracts a lot of interest attention noise uh many claims are made on how things are gaining changing I think we have to be clear-eyed in that regard when we think about some of these opportunities so unsurprisingly for something that's sexy and new there's a whole range of new consultancies emerging in this space these are just some of the ones I know but there could be many others kind of specific um houses and professional services firms looking at aspects of impact measurement through different types of technology all of these are interesting in their own ways but there are many others too probably you know about but these are some I have talked with in the context of another project we're doing around ai and the materiality and these are just kind of three random reports they're all interesting I think in different ways there's quite a lot more noise around reporting and publicly available analysis and data these are three that are quite interesting I think we could have others so there's this emerging infrastructure professional services consultancies publicly available analysis and information is beginning to support some of the claims or challenge some of the claims of this technology so I want to talk about two things really the verification issue and the data analysis issue I mentioned at the beginning so essentially these fall into two categories so for verification we might think about what distributed ledger technology may offer this of course is otherwise known as blockchain and associated with it different kinds of virtual currencies non-fungible tokens bitcoin of course so there are claims being made for that technology largely around verification I’ll give you examples in a minute the ai claims are much more around data analysis and data usage so how can we capture data use big data to provide data solutions quickly in real time through technology so these are the two broad camps of of emergent technologies and claims around them that are being made I think and the third would be sorry that I mentioned earlier would be the data collection technologies I mentioned the very end how do we use mobile and so forth to actually collect data but these are the first two I’ll talk about so the two or three case examples you might want to look at if you're interested in the verification piece and these we might argue are some somewhat hyperbolic here is IXO foundation or exercise it's called help to build a global digital immune system for humanity become a stakeholder in the IXO cosmos blockchain so this is a little language that you know we might associate with tech with uh silicon valley but what they're trying to do and you go on their website and find out more information is what they call the IXO pro protocol which has these five elements to it I won't read all the detail you'll be able to see these slides in the recording of the of the presentation that you'll all be sent uh it's about identification claims of impact verification to a third party what they call impact credentialing which is the kind of blockchain piece and then they even think they can create impact tokens which was this form of sort of bitcoin um currency that could be traded around verified impacts but I suppose there and each of these are dependent on other players so like as often as the case with blockchain models it's a kind of network model um for its success it will be reliance on other people playing their part in the verification or the collection of data parts of this protocol but this is the way they see it I suppose the step four where they're building blockchains is probably the big innovation they're trying to create in this uh in this model loads of products and services they're trying to sell you impact wallet is probably the most interesting and personalized you can see create your sovereign impact identity keep your personal credentials private this will allow you to to make transactions around impact tokens perhaps even use it to um to verify or to upload data impact data and have all kinds of wizard ai system systems built into it more practically this is what they claim and these might not claim because these are examples of six things they're doing with their technologies so each of these again on their website so we're going to all of them now you can check them out yourself give examples where um they are using all the technologies being used to in the first case the green hill wind farm to verify impact claims of climate then moving into other areas like what they call prediction oracles which are trying to use data to suggest future performance in some cases and in this case verify crop performance you can see interesting community uh currencies through their model impact tokens being this uh sort of bitcoin model cell sell-on model and the sustainable defies they call it development finance and the left-hand side the education bond will come to again with the next example is a way of trying to use their technology to create smart contracts in other words to have a system that verifies an impact which is predetermined as having a value with somebody who will pay for it and then you can create a smart contract whereby if the blockchain verifies an impact then it triggers an automatic payment in the uh in what's often called the impact bond model and this is an example an education bond they worked on or early stage all pretty small mostly pilots but you know activity is happening around their model and they've got a paper on their website that kind of sets out their sort of evangelizing piece about their positioning on the internet of impact okay a separate exam is Alice so Alice is another um sort of early stuff very early start-up um they're much more focused on that impact bond model so you can read here it says a decentralized platform for social finance and the idea is that they improve uh the efficiency and reduce transaction costs on impact bonds which essentially have these improvements that they suggest uh you can read them I won't read them all but the four key stakeholders in an impact bond which for those who don't know is a is a essentially a contractual arrangement between people who provide money up front people will pay for outcomes consequently they will pay back the original investors associated with service providers and others who create impact as a kind of three or four-way contract and their model helps to kind of think reduce transaction costs is the objective anyway so here's kind of how it works and you can see this is the standard kind of impact bond model with investors uh donors social organizers social organizers organizations sorry and validators to give some kind of assurance around achieved impacts when you create a payment model now this is actually you know brings in grant funding but you could equally have a government or investors sitting with the donors sitting in this picture so you can see that this allows them to to lock possible donations in their blockchain model to wait till projects then achieve milestones to then bring in the impact investors as well as capital to that have them also blocked in locked into the model and then have kind of smart contracts which claim and verify impacts and trigger um payments and returns in their model so it's a way of I think reducing potentially transaction costs on this model which typically is very costly cost intensive in terms of implementation and development this is a big growth model and here are some technological innovations that they're working on as well and again all these are on their website um some of these when you can read them are to do with um thinking about how you work in areas where um you have micro connectivity or you have you have um organizations which are have spotty internet connectivity but want to use mobile phones to uh to transact that's a tappy start-up um you can read them all there are different different models here that the summerly all tie into their assumption that you can create smart contracts and blockchain models to reduce transaction costs and to verify the authenticity of impact claims okay so uh another set of concrete examples and this is again one of one of any slides so must kind of totally acknowledge that which is a really concrete example I think it's interesting this is about establishing land rights so this is using blockchain which at least claims to have this um this quality of being incorruptible because once it's in the blockchain it can't be accessed and changed to use this to identify examples where land rights may be either disputed or unestablished and then when you get a land right um sort of agreement you can use it as a kind of blockchain piece that could have legal validity and then wouldn't be contestable and you could then also use it to transact property rights so if you sell something once you know who owns it then you can use a blockchain model to help transactions so these are different countries you'll see um in Brazil or in Africa united states Sweden these are different you can't really read the detail here but different examples of when this is being used to um to try to understand how we establish land right so again a verification model but also a potential transaction model so you know a lot of noise about there's a lot of interest actually I think a fairly few examples the ones I’ve shown you seem to be most of the textbook ones we can talk about right now and we'll talk about some of the limitations later of course we know some of the limitations around blockchain in terms of uh carbon for example and the use of servers but you know lots of interest in that okay second thing then is ai and this really is about data management there's some really interesting things going on here in in sectors like health so for example you can read these again in front of you the first one here essentially allows a kind of ai system to identify and analyse chemical compositions of drugs in real time with a handheld device to see if they're fakes so particularly if you're out in the field somewhere and you're trying to get drugs to people and you know supply chains are inconsistent or confused or corrupt this would be one way of checking quickly back against the database whether the composition of a of a drug is is fake or not pathology as you'll see on the bottom um in order to use ai technology to um to digitalize um scanning of blood and siemens samples and give you quick analysis of them and zipline is very famous example of using drones and ai geographic and geospatial technologies to get stuff out to um to remote places particularly blood supplies where you need to move things quickly in order for blood to get to the front line you can use technologies like that diagnostic technology is also using ai so the um what the waning ai system uses uh essentially mobile phones to uh to work out um early stage um infant um underweight uh um threats of being underweight for an infant so it can essentially take a snapshot of the size and shape of a infant and create a 3d construction which it matches against database to see it as likely to be a problem with um with a child being underweight and consequently possibly having health issues that need to be addressed early on its lifetime again in a remote space a remote this is a way of using using technology it's easy to use essentially non-mobile to uh to do a health diagnosis in places where it's hard to get say a new born baby to a hospital to be weighed and to be looked at a more kind of um a developed country problem or solution is the Queensland university app which allows you to cough into your phone if you have a chest disorder and it will diagnose from the sound of the cough whether you have any of those symptoms below so this is again example of whether the big data profiling ai can do will allow you to use this app to identify how ill you are and indeed if you have just a common cold or whether you have pneumonia against you literally coughing into the mouthpiece or the hand handset you see the woman doing there so um again very interesting kind of technologies another big area is education so online tutors gamification to teach and help language um thinking about how you create uh language platforms that work in areas of low wi-fi connectivity so you have to have stuff that can be held on the device or locally when wi-fi is spotty to allow um particularly African countries people to have access to easy uh easy ways of learning foreign languages so this is another one talking points is simultaneous translation so this is a focus particularly on families where for example if their English isn't their first language and they need to go to school as you can see in this picture they can use the phone to translate in real time what they're trying to say to the teacher and to translate back into their own language so you can imagine again the kind of machine learning the language analysis software you need for that is huge and it's a kind of classic kind of ai application to be able to handle complex knowledge quickly so here's what they do it's non-profit uh focused on family engagement for families of under-resourced multi-legal communities and it's designed to get away the that blockage which um which language can have uh around um communities and families settling into countries in which their first language isn't used commonly and these are all up and running right these are real examples many of the start-ups um here's an agricultural yield website this is another one that has masses of data geospatial data around ag which can be used a lot of it's free once you sign up and here's the kind of crop data sets they have and you can click on this on their website to see what information they have about crop yields and uh seasonal trends in these different countries around certain crops all use satellite imagery so there's satellite data big data a lot of which is free now being used to um to provide data sets which often can be used they want it to be open access so it's for the common good so you can sort of benchmark and see the patterns of crop performance in different countries and different areas through these big data sets which you can search so before we turn over to tom in a moment just to summarize before we talk about this slide you know the two big plays on this so the verification play which is about data robustness I guess having more robust reliable impact data and then the ai applications which allow us to create impact through um managing complex data in real time often on mobile or other portable technologies um which and I think it's the second that's probably more well advanced than the first in terms of as it were the market um both of which have clearly have great potential more touch limitations at the end so finally before I turn over to tom just the third category of technology emergence or what's going on that's exciting is data collection these are some of the things we've mentioned already satellite imagery is increasingly common to understand crops all kind of things um mobile of course is well understood for the mechanism of data collection uh the so-called internet of things sensor technology and farms or elsewhere and autonomous drones we've talked about the zipline and there may be others but these kind of the important thing about these technologies I think to grasp is how much cheaper they are now than historically so whereas many of these would not have been possible to use um in in rural contexts or by not-for-profits that is increasingly not the case but having got so excited about all that technology let's move to the real world and talk to uh to Tom Adams co-founder of 60 decibels which among other things I think lead are in the vanguard of thinking about how we use lean data technologies which are methods using some technology but also different kinds of on the ground activities to get really close to people's voices and to provide really useful and viable and usable data impact data to decision makers so welcome thanks so much Alex and thank you so much for the invitation to be with you uh for this session um I’m not uh just a caveat I’m not a technologist by any stretch of imagination I’m a social researcher bodybuilding social researcher um but I do get extremely excited by technology and sometimes um too excited my colleagues will tell me um and our company 60 Decibels does use a fair number of um technologies to improve the quality of our data collection and our data analysis and I can talk about those um things we've and I’ve brought up with the things we've done have been successful things we've done that have been less successful and why um as well for background we are a measurement company we work with um getting onto uh 500 social enterprises around the world running data collection exercises that tend to last 69 weeks where we create standardized data from director and beneficiaries so that we can produce assessments of of impact performance and critically benchmarks of impact performance against the areas of impact that beneficiaries are saying are most material to them so a lot of listening and we use mostly phones to do that Alex I think next slide um I’ve perhaps covered this we can essentially listen to anyone anywhere about their lived experience I think the next cycle is the numbers of people we've reached which I’ve also covered and actually this this is the four sort of building blocks of our work um standardized surveys uh you know what anyone's gonna get quite excited about impact management technologies my favourite technology is an old technology the survey and it's so useful because just listening to people you know I always think when you think about technologies um the benchmark as is how they perform against a survey and they can be more precise often technologies but are they as adaptive and surveys are so adaptable to so many so many situations and so many impacts are so complex and different just asking people about their experiences usually should be the benchmark by which we measure the quality of any technology does it do it more efficiently and effectively than simply asking people so that's what we do around the world um uh next item we do this through a network of people phoning people so the least sexy way to look at our technology is we're a global call centre social call centre and we have close to a thousand people now around the world you know the 60 countries calling people up all the time to listen to people's experience of impact next slide um we do have a lot of technology that particularly back end and we've tried to do more and more technology on front front-end launch SaaS product um but the technology on our back end in particular allows us to allow us to analyse data more and more effectively and efficiently um last slide I’ll go through some of those and and all this is about um our database that is critical we collect data in the same ways over and over again so we can find benchmarks and some of the technologies that have been most exciting to me are actually around how you store and analyse data which Alex touched upon and how we create better and better benchmarks of impact based on that next slides Alex so um mention a few of these things a lot of people will ask about the quality of phones as a technology and this is low-fi technology it's established technology and you use it and the answer I always give is obviously yes it's our technology of choice um and that I’ve spent a lot of time talking about you know phones as a social research tool for folks that are used to doing surveys in person um some of the important things about our work we spend a lot of time using technology as well to improve as our response rates um it's so important when you're looking at social issues that you have representative data um as well as significant data and asking making sure that people are included you reach enough people if you're going to make claims about impact it's sort of the foundation of of good data collection we have a response rate currently over 50 that reaches almost two-fifths of people we work with we will interview women our time to do this that data collection is short um none of this information by the way is our impact it's really important these are just vanity metrics that we show around to show that we're scaling but this isn't our impact in the slightest next slide please Alex so and I just thought I’d talk about I’ll take success I’ll take failures and where we're going to um you know phones I’ve been the benchmark of I’ve been a foundation of all the work we've done they're an incredibly useful tool as mobile phones have started to penetrate everywhere around the world it's dramatically changing the way in which we can collect data particularly in places where we do most of our work in the global south we might not previously be able to we love the phone call but we have also tried SMS SMS and IBR and other modes of data collection on phones and online surveys and they all have their pros and cons which I can tell you about um some of the most important things we've been able to do again this is the practicalities of collecting data anywhere and you know I get excited about whiz-banging technology and think oh yeah there's a great use case for it here okay how do I apply it over there and then the costs of repeating it are extremely difficult which is why I like phones and surveys because they it can be used anywhere um tomorrow um but some of the most important things that we've used are payment platforms some of them blockchain um I’m sorry had the distributed based um platforms for paying people anywhere in the world again this is about the practicalities of connecting data um you know flying a drone great once or twice how do you pay researchers anywhere in the world in a currency that they can they can use over and over again you know we run into we use western union to start it once or twice again we start to repeat that build business around that it starts to get very very very expensive um so there's a few platforms like VEEM that we use and also learning platforms a lot of the ways in which we train searches now are online or remote and we can measure and manage the quality of our data collection with a lot of remote technologies rather than an individual holding up webinar like this with 15 researchers to teach them how to do a new data collection exercise and of the more sexy technologies ai is definitely something we've had some success with particularly in improving the efficiency of our quality of assessments of impact when I say positive we ask people to describe their lived experience to say what things are most material to them often overlooked in impact management is the idea that you listen to people from the beginning you know you particularly your theory of change you say what's impactful or your investor says what's impactful your donor tells you what's impactful and the person actually experiencing that doesn't get a chance to tell you whether or not the things you thought were impactful were actually the things they experienced so we collect an awful lot of data and just ask people to describe their lived experience what things happen in their lives as a result of that and we use increasing amount of machine learning to look at and code the data that we collect and starting to do some sentiment analysis which might improve the way we analyse the data that we've collected tech failure next slide Alex you scroll down a little bit so that's got no text appearing yeah that's all I have okay all right so sorry so it's not coming through in the in the in the size I sent you but what it should say um should have four bullets there so you can imagine these um on tech failure so far um we've done a lot of work with sensors or we've done a bit of work since we've tried to do weather sensors um fantastic data we've put um uh sonars on the roofs of uh lavatories in in um to understand who was using those portals um and sonar's great measure height so you can measure heights of different family members using a portaloo um great when asking people about how they use uh laboratories is an incredibly unreliable way of collecting data people told you they use them um we even put a as a quote we put a solar solar lantern in the sensor so that people have light whilst they're going to the middle as well um fantastic but incredibly expensive and not beatable and we've tried to do some work around satellites and I wanted to mention that um most of that I still feel very positive about that but it's been interrupted by some solutions around pandemic and our inability to ground truth some of the data that we've got from satellites um which we've talked to people about blockchain I am really unclear fundamentally about what the use case here is um I get the verification might speed things up and might improve the quality of the data that it can't be fraudulently um fraudulently put forward but in the even in the use case that you're seeing Alice the verification can be done by a third-party social researcher with no incentive to provide dodgy data and I don't see that the use of a blockchain thus far as there's a compelling use to either improve the efficiency or the quality as yet it still feels to me a bit like a problem looking for a solution when it comes to impact management and we haven't got very far with putting all of our tools online for people to use themselves as a SaaS platform which I’m going to include in tech here mostly again it's behavioural problem people still want to feel comfortable talking to people about impact it's still an early market where people are still working out what it is that impact is and people feel more comfortable when they're talking to someone about their goals et cetera next slide Alex should be about taking the future um our work going forward so um I really think there's a lot of scope for more machine learning especially around quality data collection and sentiment analysis understanding what people are saying about impact from the sorts of language they use and even potentially the tone of their voice or their face as they as they speak you know I really think tonal sentiment analysis has a lot of scope and we can quite easily apply that to our work and again because phones are everywhere you can do it over and over again and similarly voice a voice to data translation that's an efficiency thing for us the ability for us to record data store data as voice rather than written words which it's improved the efficiencies of our work and potentially the quality of our data so that we don't lose things in translation as people are writing down what people have said rather than just recording and keeping that data um and I still feel like a SaaS platform for impact management will take over the world one day hopefully it's ours uh and people will get used to the idea that they can order social impact analysis online without having to talk to a consultant to draft from the theory of change and to show everyone that they've got a flow diagram but it's not there yet maybe one day I mean so before we go to questions just maybe a question from me when you you know we're talking about hype and it's been really interesting to hear you sort of have a sense of what's more usable or interesting to you than not but in your work is this is everybody talking about this is it a marginal conversation is it more to do with funders or corporates or people what's your sense of that conversation just from your perspective sitting in 16 smiles yeah I mean I think about three years ago I was really worried that as non-technologists social researchers there was going to be a wave of technology that was basically going to do us out the job and I was just launching 60 best girls in the company and thinking someone else is just going to eat our lunch here and there's going to be a better way of doing this than pulling people up and my experience is generally being that the flexibility of surveying people and the nature of social issues which are which are about people and their perceptions of the world mean that for a long time to come we're going to be asking people to describe that and so yes the sensor might get more precise data for one use case or another satellite mic for one use case or another it can't at scale the ability to just talk to people and listen to people so I feel more confident that there are maybe fewer of these sort of data collection technologies that will that will do us out of the job but I do but I do on the other side of the data analysis side of things I do think now that we collect so much data and we can store it in different forms I can store someone's record as they give me consent I can store their their conversation rather than they're all written down even in a database that we're going to be able to store data analyse it quickly analyse it with more at a higher level of quote-unquote truth around it won't just say Alex I thought your product was fantastic and we'll actually listen to the tone of your voice when you say that I’m going to make inferences about how much impact has happened because the way you speak about products or services and just finally before we open up on decision making which is where we kind of started clearly as you say these technologies have an advantage over humans in some ways and I guess big data is the obvious one but where would be the limitation I mean can you see a future world where you know everybody's armed with these multiple models they got a bit of ai over here they've got lots of grounded work over here skilled people are making judgments across the data sets or is the machine going to take over or is the machine gonna end up making help helping us make poor decisions because we get seduced by its elegance I don't I don't know I um I’m probably slightly out of my comfort here I’m I’m slightly sceptical that that but it will but at the end of the day this is people's experience as well but the clue is in the name it's social it's our experience and so that changes and so I so the idea that machines will predict what we feel about the world I I’m sceptical but I’m going to still be I’m generally a tech enthusiast so I still feel like even though we've had a problem on the road there's gonna be more and more technology playing a role in our measurement one question looking forward fantastic thank you right well let's turn it over to you ladies and gentlemen I think I can access the q and a box can you see it tom if not I’ll read it to you so we've got one straight for you from Shrutti how are you working on the interoperability of data question for 60 decibels um truthfully Shrutti not an awful lot um and deliberately so um I have really struggled to engage with other organizations in the cost of doing this and our focus is on building our own databases as quickly as possible and then coming maybe back back to this question but I’ve been engaged engaged with a lot of efforts to do this and they have been an awful lot of time and not not progressed that much um we're another new interesting conversation with one potential organization um about white labelling our data for them and so maybe this will change but it but so far it's been challenging so it's part of that just this is kind of proprietary stuff people want to build out data sets they want to own them they want to monetize them in part and it does seem a strange thing when it comes to data that even people that back private led organizations and are willing to fund them so that they build proprietary information come to 60 decibels and say what I mean what do you mean you're keeping data private this is outrageous you think well part of the reason we haven't got as far on data sets and benchmarks is probably that we always assume that everything should be public um and I actually think that organizations do need to build some private data sets lots of different questions on that of course how far do you take it and where does the line stop when it stops somewhere but we shouldn't always assume that all our data should be made publicly available or even fits data for doing good pretty fundamental question from Sarah Louise what's your definite definition of impact yeah an amazing question um whatever um a person who experiences that impact describes as material to them most of the time so uh the fudge the caveat most of the time is that some impacts um an individual are wider they have impacts you know excited the individual like when we work with can you know people who are buying clean cook stoves they might not often talk about reduction of carbon emissions and there's obviously impact there that's a wider impact community impact but most of the time in most places it's whatever a person tells you is impactful in their life this is a really important point about not only veracity but also over claiming I think there's a danger lots of people claim we've had this amazing effect on someone's life without actually asking them if you have or indeed if you make their life worse so it's a big you know a huge issue to do to be transparent and honest I mean just to that point how do you get people to tell you when something's failed so if you're talking to a community on the ground and they've you know had an intervention you know maybe something that's cost money has happened in their in where they live how do you get to be honest to say that was terrible that was the worst thing imaginable for us so I would describe this as perhaps one of our biggest challenges um because we are we have clients who are determining what things we can collect and so it is definitely a challenge for us unquestionably a client pays us to collect data for them um usually our technique is to try and get this in the back door so we will ask open-ended questions and about things that happen including my favourite question of all is there anything else you'd like to tell me and people might open up about their experiences and once we've heard enough about those things we might say look these things are material issues that we hear over and over again so a good example of that is um over indebtedness the financial inclusion sector we really struggled to get people to consider this to begin with kept asking questions about it and particularly in things like um pay as you go for so off with solar which was these knees and no one ever thought there could be a problem with people taking on essentially debt to buy solar we heard this enough times we'd say that this is a material issue and here's data about it and here's how these organizations who first brave enough to say yes we want that information have performed and you ought to collect data about that as well and you start to see something but it's harder it's definitely harder to get people focused on this so this is so I think Yash you asked a question about what's the one thing that's common across all impact ventures for example all the impact ventures uh improved self-reported happiness scores I mean I think I think you're answering it in the way I would which is to say there is no one thing but the crucial one thing is to listen so there'll be different responses to an intervention in different groups of different people but the kind of requirement of saying you're having impact is to listen and to allow them to tell you that whatever it is so it might be you know yes it might be a well-being question it might not be um what do you think is there one common measure response thing that's cuts across everything that we you look for don't quit except for me Alex or yes asking you tom yeah I mean I think the question of what things quality of questions that say please describe changes that happened in your life as a result of using x or y and you can tweak the question and then which of these are most material is what we've always always always asked we do have a kind of subjective impact score which is a bit like a net promoter score for us but I would take that data with a little pinch of salt um I think you need to ask people to describe authentic material and then collect the data indicators around those around those things and the limitation of that is that you can't compare impact of different of different across different sectors and everyone really wants to do that and different interventions that will really want some impact till the equivalent of a dollar and some people monetize impacts I think there are some real limitations to doing that and it only works in a narrow number of sectors right now effectively and I would rather collect I would rather of the limitations I’d rather stay with measuring and comparing organizations within them within sectors and saying I just simply cannot choose describe the impact value between healthcare and education and so I’m going to make a choice that healthcare is more valuable to me or if you're a donor or a funder or an organization then try and work out which is more impactful with somebody that's a big big question got a technical question maybe I’ll try and group these a bit Emma’s asking about um how do you transcribe voice recordings so how do you how do you I guess deal with that qualitative data let's if I can maybe break a few of these together maybe three so that's one about um voice recording uh wasa asked about in low-fi countries by which I guess you mean lack of access to wi-fi forgive me if I don't understand what you mean by low five um are the technical regulatory challenges so I guess that's a broader question about context um so voice surveys context and uh Ammar said about triangulation for robustness of data do you take any other sounding so kind of three whether you hold them on your head I’m sure you can three sort of yeah you should be asking some of these as well Alex you're not gonna let you off the hook without answering something but I’ll go I’ll go with uh um so lo-fi I mean that's why I like phones um you know they kind of spread everywhere and where we can't use phones we do in person-based collection I think some of the questions around um around regulation often more applied to research techniques and any human study that we have to be very careful of um big question kind of when you use um qualitative data how do you oh yeah you do get out of phones yeah this is this is back to where we will invest in technology and to your point around data analysis Alex some of the things you were talking about we are still um quite low tech here most of this work is done by people annotating those um but we are exploring more and more now that we have a highly people-based annotated um data set whether we can then choose machine learning but it's also put to your point that you were talking about early Alex says like when do the machines take over well if the machines if we at what stage are we comfortable enough that the annotations we've got are good enough that the machine can start learning from there and then what is the fundamentals of the social change and it's the machine telling us that but we're not listening again so we need to carry on listening and so there will always be this relationship between humans interpretation and social issues and then us and then us doing um using ai on top of that but I’m also really worried about how black boxes get I don't understand that well I understand when I can check and fact check all of our qualities of phase correction by individuals it's painful to do but we can do that and so we're also careful about you know does it get too black boxing do we not understand it don't we are we assuming that this has worked well because an algorithm has and so these are all really interesting questions for us around analysis and that so we're quite old-school right now but we are flirting with more and more sophisticated technologies in question so do you ever try testing you know the robustness of one set of responses against sort of um not very not much to be honest um I get a bit nervous about triangulation we've just written a paper about an SSIR around some triangulation techniques to impact and I think I’m typically more comfortable with the biases and inaccuracies that creep into people's perceptions of self-reported perceptions impact than the even and managing those through benchmarks in particular and I am the pretty wild um I see uh biases you can creep up into too much triangulation I I think that I haven't seen it done applied that well but other people could they have really great examples of it that I’m not aware of just talks ask you to talk a bit more about benchmarking because you mentioned it you want to just give an example of just how you do that in some of your work yeah it's sure it's really simple we ask people to describe what their lived experience is and if we work with um say 50 000 smallholder farmers this year across 35 countries should we believe more than that um about a range of different business models we would have asked all of those farmers to describe what big impacts in their life most material about inclusion in supply chain x except off take a y input z um and when we code those we say these things are our most material most of the time for most farmers for most products we will then put indicators forward for those things and then then we have our clients to let those we collect that data around those indicators over and over and over and over and over again for those clients and then we can start to compare impact um again between those clients one area of impacting cause which is which is the performance around particular indicators of outcomes that most people in that sector say are most important to them and we found that's the most interesting it just dramatically transforms how useful impact data is because rather than having even if you did some brilliant data collection with a satellite right or a drone or a sensor and you said outcome x is increased by y percent if you don't have anything to compare it to all you do is slap that on an infographic and pat yourself on the back and tell your donors that so as soon as we can provide some comparison even if even sometimes it's only to a few other organizations but sometimes to dozens of other similar organizations it transforms how people engage with the data and start to say right well this indicator I’m doing this one and this indicator I’m doing really well on how do I how do I do better on this in some other place thank you Alex asks how do you track behavioural change so I mean kind of multi-systemic complex changes in behaviour is that difficulties it's mostly something we don't do um you know we there are definite limitations to what we do we are more about once you have experienced something how do you describe less what could you how's your change what is your state of behaviour what what things you've done differently etc and we have done less around that we've done very long behavioural economics and I don't I don't as an area I don't actually understand it that well um so we stay in the lane that we do well if we hire one day a brilliant behaviour promise we might start to do more of it but it's but I haven't got great answers as well so a couple of questions more about investors from Michael and sarah I was talking about um sort of corporate disclosure and how you you know a firm might use some of this data in the financial reporting and so I was asking about how it could be used to inform decision making for investors how far you get into that world it's a long I mean this is definitely one that you should be asking actually you know much more about this than me um but we have been today it's a fairly agnostic organization about just collecting data we haven't aligned ourselves to a standard sense this is this is what you need to put into any kind of reporting et cetera and have been an onlooker as the standards have sort of coalesced and can come together we've worked a bit with the impact management project the next few years we're going to be very much more purposeful about saying which standards would we align our data to how do we expect our data to be used what is it what is what are what thresholds of good enough data look like uh but we won't we won't determine that we will be working and trying to work with organizations like SDG impacts I think really you know really kind of setting higher standards for everyone else to follow and say okay well let's work with some case studies of how our data would allow people to meet the demands of SDG impact and hope therefore that that's mutually beneficial for us and also see the impact of what would come to us so you might see sort of something like that powered like it's an side by sixty best versus data Alex you think about this much more well I you know I see a gap between practice and or what would be best practice and what's real practice I mean I’m a huge advocate of the work you do and the uh and the data you create as being more robust and valid than other kinds of outputs data but getting investors to understand that is crucially is incredibly difficult it's crucially important but very difficult often because it's messy data this is one of the attractions of technology if you've got a huge qualitative messy data set you can somehow clean it up put it through some magic algorithm and it gives you kind of statistical data on that that's what investors tend to want in the absence of that they'll go well just because someone says it was good for them I’m not really interested in that I mean that doesn't tell me much there's a real issue there I think which is improving massively and maybe people in the room have a different view from their perspective but I think there's a there's an obsession with numbers which is a question we have to ask all the time in impact measurement as opposed to richness of data and human I think one of the um I think I can probably say it's been recorded but I think one of the one of the efforts that was trumpeted itself as the solution to all this stuff the impact of multiple money which was basically a re-brand of SROI and lots of things that came before it yeah so it claimed that it could do this the simplicity you can just pump in a number we'll we'll scan every single research paper they get 600 and then you get your number at the end of it and I just think it was you know bono described it as having sold everything and it and it didn't and it was never going to and so I think we should always in general be accept that at the same time we should be making data collection easier and more costly it will remain complicated you know there will be a CFA equipment for impact measurement previously probably run out of oxygen USD um but we don't expect it to be simple we'll just expect it to be normalized so you know some of you will be aware that um the the discussion around standards of reporting for impact or let's call it non-financial data this is this world has moved incredibly fast recently and some of you may know the IFRS in the US is planning I think to uh to create a parallel sustainability standards board to its financials to that sanders board so this will be the beginnings of a potential you know regulatory structure for disclosing non-financial data in a consistent fashion um and that you know I hear what I hear is that may even be announced the cop 26 uh and if it were it will be in its own way a game changer the devil would be in the detail of what that actually says um but these yeah I think absent standards very hard to address these investor questions they're confused all the time about what uh what constitutes good impact um we're nearly winding up now but maybe um ken our old friend Ken Wilson who's been a great supporter of all of all our programmes has asked the question which I think you've answered but it's a little bit about simplicity versus complexity which is I guess one of the joys of lean data by definition of the term is that it it does it well and quickly and simply but focuses to get meaningful data but you put that in you know more complex uh context and maybe it's the same as the behavioural question how does that work I suppose maybe the question would be are there several lean data techniques that can work together it's not a triangulation question but it's some rich data I don't really know how to do it but um I suppose how do you ken's point is how do you not over simplify some linkages in the way systems affect people's everyday lives like I suppose yeah I don't know all the answers to that um I generally don't and I wouldn't say that um that we end with just talking to people in the scenes people um I would say we begin there but too often we don't begin there um and it is this simple thing and the value you get from doing so and what you uncover is just simply huge um so I think I think there will be other data points that we start to use that that don't mean to be dismissive triangulation earlier I don't think it should be and that will help us and when we when we do open up our data sets to interact with policy with others et cetera and it will become better um in the meantime what we will focus on is more often than not people being listened to about the impacts that they have and I and I think that sometimes you know some of those big questions which are absolutely right to ask if they we should say yes extremely important and are we doing the basics right now are we asking people enough about that experience is that there's a question from that hey although like how do you tell you know whether people aren't just telling you what they want to hear on the phone I mean um it's the same in person there's nothing in fact it's more likely or thinking that by the phone they'll tell you they'll be less social probably less social desirability and there are things for which we will never know whether people are telling us what they want to hear however if we ask enough people and the law of averages says we ask enough people enough times we're going to get closer to a to the truth even if that truth is structurally slightly biased and then we can compare performance of all kinds of organizations against it so even if every single farmer in the world tells you all the time live to some degree and that will be controlled by the fact that we've lost loads and loads and loads of other people and we'll be able to see the areas of high performance in the areas of low performance by listening to people over and over thank you almost the time so just two more things I want to do um uh Laila’s gonna have a quick word with you in a moment but I just wanted to give you my view on this so limitations and I won't go through these in great detail you can read them but we've covered some of this complexity high transaction costs we haven't really spoken about data protection security you know collecting lots of data from people on difficult issues like health and education and pumping it through ai systems we come into real issues around data protection and data monetization at what point if we collected data from people should they own it and sell it to us um I think that's a really important question particularly in areas where there's poverty and we might be taking people's data and monetizing it in our own consultancies or whatever there's a big issue about materiality challenges in this black box so the ai interface is a tech interface but you might have people either side of it who are social investors social enterprise people charities different language how do we translate that um and that's really lost in translation piece so let's not forget you know there's a big question about carbon now bitcoin it's worse than Estonia but better than Denmark in terms of carbon emissions that's just bitcoin so it's you know it's a massive amount of energy to run blockchain technologies there's you know stuff happening now about can we have sustainable energy to drive social bitcoin but there's a real question about how much does it add for the carbon footprint it probably creates um but then finally just the last two minutes I think lately I wanted to speak to you about our programmes I’m sorry to cut you a bit short not at all I don't need that long Alex but thank you Alex and tom extremely interesting and a taste of what is to come during the impact finance innovations programme we've had such interest in this programme and I’m sure you can all see why my details are on the screen and I look after four programmes in the social impact space I would be more than happy to discuss impact finance innovations or any of the programmes you see on screen I have put links in the chat throughout this session to our programmes a link to Aunnie’s book a link to 60 Decibels and my email address all registrants will get a recording of this session early next week and I hope to see you on one of our social impact programmes soon but thank you again all there was such wonderful engagement in the chat and the questions, we will try and get to everybody in the next few days or so thank you just to say just to emphasize that thank you for the team grace and uh I like have been involved in putting this together they've done a brilliant job thanks to tom for his you know his honesty and his support of all our programmes he's in many of our programmes and he's a great supporter of us and I love his work and of course thanks to all of you for giving us an hour of your time and I hope you found it useful and I wish you all the best for the rest of your day thank you very much bye bye
2021-09-16