2023 AutoSens | How Safe is the Waymo Driver?
it's pleasure to be here to see all these uh 3D people um everybody's beautiful today um I'm G to try to answer the question here that we're at a censor conference and I'm sure you're asking yourselves does does it work do we do we make an effect and that's what this topic is now I'll be trying to answer how safe is the wh driver so whma one is a is a service it's a shared elect autonomous Mobility service that's currently running in three cities Los Angeles Phoenix Arizona and San Francisco and I'm a scientist I I am director of Safety Research and best practices I I have a long background of publishing papers I have a Google Scholar account I have this citation index I've been Adjunct professor at two universities all this stuff my entire career I started out with driver monitoring systems so I want to talk today about this work and evidence and how how evidence and science makes a difference in what we're actually doing and within safety safety is a multifaceted um topic it's uh it's something that that we have an ecosystem of methodologies that's I think the best way of thinking of it but within that ecosystem people tend to think of one main metric the favorite metric is what is the crash rate how does the crash rate relate to Human Performance and that's what I'll talk about today but there are a mult like I said multifaceted there's two main perspectives in in safety one is an emotional safety emotional based an experience-based safety when you are in the vehicle do you feel fear do you feel uh safe there's a security aspect to that is it's it's part of it's a user experience and some people talk about safety as an experience in in an emotion but traditionally we've all worked with with national statistics Vision zero on fatalities and seriously injured requires a a a different way of working there's a national Road Safety strategy in the United States now focused on fatalities and seriously injured and I think that that one of the main insights is you need to work differently with the right part of this this diagram you need to understand that it is defined by human tolerance limits so kinetic energy passes the body's requirements on on kinetic energy and and and this is actually really important because we're actually in a phase where we need to maintain our focus on what actually safety is so I'll get into this um uh so we we've we've seen this bubble up now and we understand there are some concerns particularly in in San Francisco and we want to be good neighbors and I'll be uh showing our transparency our uh and how we would like to advance science so one way to think of it is that we are uh a company with a very unique um uh strategy where we're trying to differentiate ourselves by publishing papers by publishing data we've published 20 papers since I started in 2020 they're all available on our website now and what we're doing is trying to use the scientific method scientific method is publishing data and then we give our interpretation of that and you look at the data and you form your own interpretation of the data so it is actually really key to read the papers please do that and I think that uh so what happens is there's we provide data and observations our interpretation then there's rigorous skepticism that's the next phase and these these interpretations can be biased biased by either a predetermined pessimistic opinion or a predetermined optimistic opinion but ideally the discussion goes through a phase of of rigorous uh debate based on evidence not not on on opinions so that's what we're trying to achieve by by publishing these uh these assumptions they can distort interpretations so we believe that the data speaks for itself and um even though some of the debates are are are not driven by evidence and and even if if something is wrong if there's a is incorrect uh calculations that should be corrected and we we've seen unfortunately some instances of incorrect inflated or deflated Collision uh uh conclusions about safety uh uh for example in San Francisco SF MTA and CTA we've asked for retraction and correction of their letter towards CPU where the there was some incorrect conclusions made but moving on I I uh will summarize the key results that that we've we've um been able to amass and I think uh I'm really happy that we're amassing or a growing robust evidence a body of evidence that shows that we have an exceptional safety record so it doesn't feel like very long ago in the spring when we published our first million miles on rer own when there's nobody in the vehicle and in in that data it was a total collisions we looked at all contact events not just um collisions that are reported for um for requirements for nits or DMV we looked at all contact events even those without without injury or without uh property damage and we had no reported injuries no collisions with pedestrians and all of the vehicle to vehicle uh Collision we we saw in involved some degree of dangerous human driving so recently just a few weeks ago we released another study that we did together with Swiss re Swiss re is one of the world's largest Insurance firms reinsurance firms and their business model is to calculate risk so there's this whole process and of of uh standardizing how to do analysis that's been developed through 70 years of the or over 70 years in the insurance industry actuaries look at data this is important for their business model so this is what they did they looked at our 3.8 million miles with no human in the driver's seat and they we found together with them that there's the bodily injury claim frequency was reduced 100% And property damage was reduced 76% this is Major news this is massive effects and then in addition to that we've been doing other research previously where we've we've reconstructed all all of the fatalities in Chandler over 10 years this is an area which is part of our OD and we we we've we were able to show that we avoid all fat all these fatalities either avoid or mitigate all of the fatalities with with some form of collision avoidance except in those cases where you're hit from from behind so this is a one way of working with fatalities that we we are are using as part of our evidence and then we also are doing the more traditional thing which is uh scenario based testing we call that collision avoidance testing and in collision avoidance testing We compare ourselves with a nominal or a non-impaired attent of human the neon model is what we call it if if you run the the non-attentive unimpaired human through the the same scenarios what does that performance look like and we try we beat that so that's our our standard that we use when we do collision avoidance testing uh We've also shared uh all of our holistic Readiness methodologies in 2020 there's many there's many methodologies that we use we've also shared just this year the safety case approach our safety case is what determines risk and our and our our approach is quite uh uh detailed and I encourage you to use that as a toolkit for for for guidance but we also released um what what human speeding looks like in the cities we're operating in in Phoenix and and San Francisco and I thought that this was an amazing uh result because in the National Road Safety strategy and other Vision zero one of the pillars of that is to reduce speed and this is what we're we're able to see is that there's 27 to 4 7% of what we're what other drivers doing in the environments we're driving where speed is is 27 to 47% over speeding so that is eliminated with a with a um with an autonomous vehicle and I think it's really interesting to think of the effects when you start having more autonomous vehicles I mean there's been some re research showing that 10% would would create a traffic calming effect just with 10% of Whos So within safety impact we see that um we have uh to understand where it comes in this is our our our safety case and this is how we we see safety impact coming in is is a continuous refinement and evidence or or continuous confidence growth so we'll continue to do these This research we we had one million miles we had 3.8 million miles we'll continue to do this and build up our confidence as we go go forward looking at safety from a number of different angles so there's there's two primary ways that that you can approach this and one is what I've showed that we've we've worked in this partnership with Swiss 3 and this we chose that consciously because it has consistent reporting standards this is what really really important for my talk today there's a higher reporting frequency which means that there's more crashes that are reported to insurance companies than than in National crash databases and it also includes injuries that are also non-collision injuries but there's some other bonuses with Insurance data and that's what one of the bonuses is it looks at responsibility if you're looking at who is contributing to a crash who's responsible for crash we included all the cases where where we were in responsible or partially inv responsible so that effect is really I think what you think of with safety is causing crash but that effect does get get diluted when you look at the overall crash count an overall crash count includes when others are making mistakes and run into you and in many of these cases like standing still and getting hit from behind at an intersection is is an example of that where there's very little you can do if anything the other way which most people think of is to to use use crash comparison overall crash rates and that's I think where most people go and think what is the crash rate overall crash rate but with that I I I need to show you that this is an approach that's possible but you need to adjust for the all the under reported events so the gray part here in this you see that there's a lot of missing data in the reported crashes to governments what we were able to do in the insurance data and this is this is I think exceptionally interesting data is that we're actually significant and I me just see if this fancy tool works here ah yes there you can see the the writer only data that were uh in bodily injury sign significantly lower than the blue which is the human performance and then also on property damage we're significantly lower but in addition to that you can see see that we have testing operations let me see if this works the to testing operations that's 35 million miles and that exposure is Expos exposure to AV unique crashes or AV unique failures that's in integrated we've had that as part so if you combine those two 3.8 plus 35 million miles you you see a lot of miles that we've been exposed to uh cases where there there could be something going wrong as an initiator but the science behind safety impact is what I want to talk mostly about today it's a simple thing there's four numbers you just do crashes in the amount of miles on The Human Side against the autonomous side it's very simple anybody can do it but it's there's a lot of tricks that are going on right now and I want to talk about those tricks these things that that are are potentially uh people aren't aware of the mistakes that are being made and I want to rule out that it's from lack of knowledge that people are making mistakes so it could be that there's an intentional piece to it but I I want to make sure that uh anyone uh is not suffering from lack of knowledge so there's these challenges and misconceptions when calculating uh crash rates and I think you being Engineers this will be a fascinating topic for you hopefully um so there's there's four separate ones I'll talk about one and I think I you've maybe picked up on it so far that there's a difference in reporting thresholds when you compare an apple you need another Apple you don't do apples and oranges and this has to do with the definition of crashes what is a crash what do you count do a crash so right now in your mind you're thinking an injury or a property damage something that gets reported something when you tell somebody I've been in a crash they say oh my God because it's a it's a crash they don't say that oh it's a scratch or a dent on your car imagine if you went to a uh a rental car company and you forgot to inspect the car you come back and then someone claims that you have 20 crashes when you went to visit Grandma I don't think that's what you would agree with so what's going on right now is we have strict reporting conventions or requirements on us where we have a standing general order with with any physical imp impact with property damage any property damage and that's also from the DMV any collision with damage so what we reported when we did our one million miles is we reported all 20 we called them contact events and then off to the right you see contact events what what are those so we had nine contacts where there was no property damage for us or anybody else nine of them we had nine contacts where there's scratches or dents and this is what we're we're talking now about scratch is it or Dan or is it it's not fender benders so the term fender bender which was used previously as a minor crash that's not what we're talking about now we're talking about scratches and dents so we had two what we call C reportable it's a crash database in where which we it requires police reports and tows so we had two of those and both were involved with with a rear rear end cases so one of the things is we have a very high likelihood High recall of detecting events because we have the sensors on the vehicles and if you think of that yourself how would you find a comparison condition with humans how would that how would you find that and so this has been studied this is a Nicha Nitsa report where they're looking at um the Under reporting and what what they were finding uh for 2023 is that there's 60% of property damage only Collision that don't get reported to police so 60% and then the rest there you see this is the Mas scale where you have increasing injury and there's a proportion uh that averages out to 32% of injury collisions that are not reported to to police so obviously this is this matters and then on top of that you have to start thinking well how about all these scratches and dents and and minor collisions and that kind of thing how many of those have been uh been missing so or even events contact events where there was no no damage or no injuries how do you find those events so we're going to take a little um um estimation or this is this is illustrating the problem here with the data we we we've seen that's available the blue and the green comes from from Crash data bases so this is um has to do with the requirement of towway crashes and police reports and then this is a the the gray is a hypothetical you when you introduce the requirement to to compare with any property damage then you need to find something that that's comparable so it's either you create that you create that by doing an estimate and here's some examples it's really hard to look what is the property damage above $1,000 which is the reporting uh requirement for California and Arizona what is the property damage frequency below $11,000 how many cases of a $100 property damage are there out there someone has to try to answer that question on how many cases where there's no property damage or injuries out there and you get into this these minor cases and and it's either you correct it by creating an estimate or you have another approach and another approach would be for example to have a a reporting threshold like the edrs they have a a a triggering threshold for acceleration that's one of the one other way that could be possible the third example is a mathematical trap and you're engineer so you love this so what people do is they take the 3.1 trillion miles from Federal Highway and then that's the U denominator and they take 6.1 million reported crashes is crashes over miles but that's actually Incorrect and the reason it's incorrect is because you have two vehicles and there's an on average 1.8 Vehicles invol involved in
each crash so you need to correct that to you need to uh times it by 1.8 Vehicles you get 10.9 million and then you do the division so what what the difference here and why I'm pointing it out is because it really matters it's 75% difference between a two crashes per mile incorrect calculation and a 3.5 crashes per miles which is correct that's 75% difference and of course if you're if you're predetermined uh pessimistic you will let this fly under the the radar and you would want to use the lower number but I I would say a a a properly reviewed scientific uh paper they will catch this but I don't think your average person and there's a lot of people doing this journalists and others that are claiming things on crashes and this is this is a really important part that and if you don't believe me please look at this IHS study and uh study up about it the last one I'm going to talk about is OD specific matching so operational design domain The Benchmark you're using it must be calibrated for where you're driving and that's because not all streets are equally challenging I think this is something that's intuitive to you if you're driving in downtown San Francisco this it's not it's more challenging than when you're driving on a Rural Street a suburbian area and this is just comes out in the numbers there's an example in what I have with the blue we published a paper on this in 2020 and this is one of the controls that are typically done is you need to when you at least when you're you select a crash database does it have heavy vehicles to heavy heavy vehicle crashes in it oh if it does then you should take those out because we're not a heavy vehicle that's like at that level or that if you're driving at night only then you should be comparing to the right uh conditions of your OD or where where what streets you're working on Etc and there's a limit to that of course where where there's a filtering that is needed but you can't get as detailed as you'd like it'd be great to if we could get uh what is the risk of of a skateboarder running downhill on this specific Hill in San Francisco that would be that' be fabulous if we had that level of detail but there are limits to your your data sets but at least you should declare your limitations in these data sets I think this is kind of intuitive to many people but is often overlooked so I wanted to um uh conclude that we've we've taken a stance here we've we've shown or that we have incredible safety performance and I think this is part of what we we're calling a safety Paradox of new technology the Paradox is that when you introduce new technology which is later seen as as obviously improving safety and let's take some examples let's take the seat belt and please remember what happened if you're old enough in the 70s when there's a rigorous debate about the seat belt my uncles were all saying I'll be trapped if if I'm wearing this this seat Bel Etc where is that now this is the the number one safety technology that saved more lives than any other any other technology again you see that with with ESC stability control systems there's a rigorous debate about that you see it with Auto breing this is now mandatory in with the GSR there's laws on these these Technologies and during the in during the development of those there was a a phase where people were very negative towards these Technologies and this is the Paradox some people see safe technology as unsafe and I think that part of that has to do with emotions safety as an emotion so and we need to work with that and and develop the trust uh with that but but I I do think that that technology uh that is oriented deductively deductive reasoning says that an interior sensor occupant sensor that that um prevents a child from being locked in that's what you'd be using as your argu to save lives deductive reasoning you wouldn't put a requirement that says you need a billion miles before before I believe you but somehow that argument is being made against autonomous vehicles um and I'll get to this discussion about um fatalities so my main point today is that we must control the factors that that achieve a valid Apples to Apples comparison and if we're following the scientific method then that will be corrected because it's the evidence you're you're following if the evidence is presented in the correct manner but but uh okay so we should keep an eye on the bow so remember that curve with the exponential with the gray off to the left is that where we should have our Focus with the the most occurring the most frequent occurring things scratches and dents any contact event those are most most frequent I think most of us will agree it's the where the events with high energy potential kinematic energy that affect our bodies which is what's defining safety is our tolerance in our bodies this is what we need to be paying attention to we we need to be able to see which events matter we need to work with it in a different way because Vision Zero The Safe Systems approach um the National Road Safety strategy it's all directed at learnings for from how to work with fatalities and seriously injured and it's not it's it's not the same as working with with the more frequently occurring things because they are so rare so you need to work in a different way with deductive reasoning and other typically what you do is you look at the causation the crash causation you're developing a fatigue um detection system because fatigue is a is a f it's fatal that's the reason it's a connection between the causation and the and the and the sense and and the tech technology that you're developing for and that's part of the evidence speed is a well-known relationship many of the the treatments Road treatments that are developed are not based off of statistical evidence they're based off of a relationship a direct relationship between speed and fatalities so this is how we need to be be able to be mature enough to discuss these things and I think uh this is my my my last point so in closing I think that way we're doing our part by continuing to expand because if we continue to expand the public access we have access to not just the safety benefits but but to the other benefits like the the accessibility in being able to to move around independently to serve underserved communities did you know that we have electric fleets that are utilized four times more than a privately owned vehicle four times more so there's four times more zero Mission Miles by using a w and it's 100% renewable energy I think there's a lot of really really great uh uh reasons to choose AO and go go to San Francisco or or Phoenix or Los Angeles download the app and have a have yourself a first bride
2023-10-25 15:27
