Good evening welcome. To the, session. Tonight on, doubting, Science. And Technology. It's. Good to see all of you we, are. Going. To have a very exciting. And provocative. Presentation. And. Later, discussion. My, name is Li Jun and I, serve as a faculty, member in the Honors College the. Department, of psychology and. Also associated. With the African American and African Studies program. And. I want to greet you also on the behalf of the Dean of the Honors College dr.. Cynthia Jackson, Elmore who. Is away but. Is usually, at these, particular, events. I. Also would like to introduce. Associate. Professor, John Beck in the back he. Will be facilitating. The question, and answer period and ashore after. The presentation. And. Without, miss. Stephanie C pack in the back she. Does all the logistics. So we appreciate, the role that she plays. So. I want to thank you again for joining us tonight. Some of you are returnees. This. Also is being sponsored. By, the MSU. Alumni, Association, and. They are also live-streaming. It, the. Honor college continues to be proud to sponsor sharper. Focus wider, lens, it. Gives us a chance to showcase, faculty. Academic. And other staff here. In Michigan State University. And the, research that's going on I, would. Also maybe like to say to you if you have your cell phone to please put them on silent so vibrate, as we, go ahead, but. What the, faculty, does and what this allows us to do is to celebrate the work that happens, here in a way that lets the local community. Students. Faculty and staff exchange. Ideas about, big topics. We. Never answer all the questions, we. Actually come up in many of these sessions with more questions. And these, great faculty, members can go back and do, some additional research. So. But, we want to continue the dialogue. Beyond. Tonight, in terms, of the issues, that we raised. My. Role really is just to introduce. The. Presenters, and. To make sure that everything, goes smoothly, and the. Way that it's going to work is after I introduce, them they will each have 10 minutes and. After. That we will give them a few, minutes if they want to interact with each other raise, questions with, each other, I might. Summarize a few points, but maybe I want and if not but we want to give you plenty of time to engage. Them, in their discussion, so if you have questions, feel. Free to write them down keep. Them in your mind and then. We'll have a chance for you to engage. The panel. I'm. Going to introduce them, in the order in, which they, will be presenting. First. We have dr. Georgina. Montgomery. She. Is an associate, professor in the Lyman Briggs College in the Department, of History, her. Research focus on the history of field science, particularly. Development. Of field methods and, sites within. Primatology. And animal. Behavior studies. Dr.. Montgomery, teaches a range of courses on the history of field science, gender. And science, and the, history of primatology. And animal. Behavioral. Studies. Her. Publication. Include the book primates, in the real world, escaping. Primates. For law creating. Primate science, and various. Articles, such as one. In the journal for the history of ology and in endeavor book. Chapters. In, various. Books. And she's, also done, a chapter on teaching the. Animal, and a chapter on Darwin, and gender, a very, interesting, article. If. You haven't read it, she. Earned her doctorate from the University, of Minnesota. And. Then following her will be dr. Kevin. Elyot he's an associate, professor in Lyman Briggs College the, Department, of Fisher and Wildlife and the, Department, of Philosophy so, you can see he spanned several. Disciplinary. Areas. His, research lies at the intersection, of the philosophy, of science and practical. Ethics his. Books include is, a, little, pollution good for you. Incorporating. Societal, values, in environmental, research current. Controversies. And values, and science, exploring. Inductive, risks and a, tapestry, of values, and introduction. To values, in science, he. Earned his doctorate, from the University, of Notre Dame. Dr.. Rich walls is an. Associate, professor in the department of media and information, his. Works involve, understanding. How people think, about their interaction. And computers, with. Computers, and their interaction. With other people through computers, his. Research, has a particular, focus on security, and collaborative. Systems, a very, important, issue in our society, today he's. Currently the primary, investigator, on three, National. Science Foundation. Grants, including. A National, Science Foundation, early, faculty Career. Award and. He. Earned his doctorate, and we will hold this against, him because of what happened Saturday, his, doctorate, is from where the, University.
Of Michigan, no we, won't hold that against, it and. Finally. Last. But not least dr., Aaron that crypt. Macwrite. Is. The chairperson, and professor, in the Department, of Sociology. Employing. A range of Memphis and analytical, techniques, he. Explains, the structure, strategy tactics. And impact, of the us-based climate, change denial, counter. Movement, analyzing. Theoretical. Relevant, patterns, and trends. In citizen, climate, change views, and, investigating. Pratik. And predict. Of public, views of science and scientists, he. Is the author of, the. Risk, society. Revisited. Social, theory and governance, and community. And ecology, dynamics. Of place. Sustainability. And politics. He. Earned his doctorate, from Washington. State University, four, exciting panelists. And with. That I will turn it over to our first presenter dr.. Murray. Thank. You very much and. First of all I'd like to thank the Honors College and John back for this wonderful invitation. The. Talk I'm going to give today is an eight-minute, condensed, version, of an essay I wrote for, an edited volume by Michael, ruse that actually, I brought an issue just. To display on the table over there if you want to look at it later with the full citations, and more information from the, talk I'm going to give today. So. Today, the scientific, and cultural clout, of Charles, Darwin's name which, i beamed up on the screen there and his, works continue, to be interpreted, and utilized, in a myriad of ways to, promote diverse, political viewpoints. Issues. Of gender and evolution, continue. To be discussed in the field of modern evolutionary. Biology, and. Also. In the pages of popular scientific books magazines. On the, stages, of comedy clubs and on the small and large screen, in fact. If you teach general evolution, like I do you, even have an article in Playboy magazine for, the 1970s. And not many professors can say that so. Darwin engender truly, is everywhere. In. Terms. Of how Darwin is presented, it's, often in these kind of ways he's alone he, seems to be eternally, elderly with. A beard and sometimes. There's a little bit of humor a little bit of criticism like, this well-known cartoon, here where you have Darwin's, face on. Chimpanzee. Always. However he. Is displayed, as the lone genius he's, never seen in a team doesn't. Seem to have other scientists, around him, often, doesn't even need props because he has so much scientific authority he doesn't need props in early photography, to demonstrate, his, scientific. Authority. And. Of. Course Charles Darwin is probably best known for his 1859. Publication. The Origin of Species where. We set forth the theory of evolution by, natural selection, and keep, in mind that there were fairies of evolution before Darwin, so we need to distinguish that it's evolution, by natural selection. Perhaps. Less known are Darwin's later. Work slightly later Darwin's, descent of man in 1871. And his, expressions, of emotions, just one year later in 1872. Together. These three works can be seen as a trilogy or, one long argument, our. Pause here to note that even if well-known texts. Like the origin are not necessarily, well read indeed. Most of the students I teach have never read the Origin so. We start off by even looking at the contents page is a, new terrain I despite. The fact that it's clearly a household, name. These. Texts were published during a time period characterized. By a debate called the woman question or more accurately questions. What. Were, the moral intellectual, physical. Capabilities. And limitations of. Women in the 19th century what. Roles should be afforded, to women, in anglo-american. Society. From. Which arenas, should they be excluded. Where, should, they be included and, this. Political climate certainly. Shaped the contents, of Charles Darwin's descent of man in 1871. While. Also being significantly, impacted, by it. Anti. Feminists, and feminists, alike saw. The opportunity, to use the power of scientific, Authority, and specifically. The power of Darwin's name and his theory of sexual selection to. Promote what were often diverse, views, of women's, place in nature in society.
So. What does the dissent argue, well, one of the fairies it sets forth is the theory of sexual selection, male-male. Competition and, female, choice in non-human. Animals, you. Would assume this mechanism would function in human evolution, as it, did according to Darwin in non-human evolution. After. All, Darwin, quite Fame stated, that between human, and non-human animals. There was a difference, of degree not, kind this. Is a lot of similarity, between humans, and other animals. However. In the descent of man female, choice in any degree of agency, that went with it failed. To transfer, in. Darwin's, argument, to human evolution. Thus. According to Darwin, in modern. Civilized, society, the male had, become the chooser of his mate. Despite. Darwin, being commonly understood, as a great revolutionary. The. Human, male and female he described, in descent fulfilled, rather than challenged, the gender identities, and ideals. Of Victorian, culture. Darwin. Saw man as to, quote from descent, here the rival, of other men he delights, in competition, and this leads to ambition, which passes, easily, into selfishness. Despite. Man's failure, to be highly cooperative and altruistic or rather because, of these traits. He, had achieved what, Darwin, described as a higher eminence, in all, areas, when, compared, to the achievements, of women. In. Darwin's. View the, result of this apparent, superiority, were playing to see and, this is probably the most famous quote in. Terms of gender and evolution, from Darwin's work from descent if two. Lists were made of the most eminent men and women in poetry, painting, sculpture. Music. History. Science. Philosophy. The. List would not bear, comparison. We. May also in fear that, if men are capable of a decided, permanence, in other women in many subjects, the average, mental, power in man must. Be above that of woman okay, so he's saying that far, more men have achieved in all of these areas and women and therefore, men are mentally, superior to, women, and he's seeing this as a natural. Biological phenomena. Not something that's impacted. By cultural, circumstances. Certainly. Darwin's, words reflect an incredibly, limited view of womanhood and manhood. For that matter and one, that fully conformed, to his cultural context. Now. Here is a historian, my methodological. Approach must, include what's called historical, empathy, I must, situate. Darwin and his views in his time period. For. This reason, I will not label. Darwin sexist. Instead. His views reflect, the context of his times. Furthermore. For scholars such as Evelyn, Richards Darwin's. Conception, of men and women were founded on his rigorous application of, naturalistic, observation, rather, than sexism per se this. Is the quote from Richards is, not only historically, inaccurate to, impute an anti-feminist motive, on Darwin but unnecessary his. Conclusions, on the biological, and social evolution, of women were as much constrained, by his commitment, to a naturalistic, or scientific. Explanation, of human, mental and moral characteristics, as they, were by, his socially, derived assumptions, of the innate inferiority, and our Methodist II of women. In. Recent years Darwin scholars have increasingly turned their attention to Darwin's view of women as well.
As His relationship with women including. Family members amateur. Scientists, with whom Darwin corresponded. For. Example the work of a scholar called joy Harvey, and the Darwin and gender section, of the Darwin correspondence. Project a huge wonderful, online database, with Cambridge University, where you can read all the letters Darwin received and wrote has. Revealed that Darwin had a hundred and fifteen, female. Correspondence. Some. Of these women were family members are part of a social circle others. Were not known to him until they wrote in short. He was not the lone genius he. Was and still is often. Portrayed to be. In. Addition to women collaborators, Darwin attracted, women who simultaneously. Critiqued, his conclusions, while, seeing opportunity. In his data and fairies, for promoting gender equity. Charlotte. Perkins Gilman seen, here, is. One of those nineteenth-century, feminists. Who used the power of scientific, theory and terminology. To, speak out in defense of not simply met female, equality but. What she believed was superiority. Especially. In regards to cooperation. And altruism, in. Gilman's. Books some, of you may have heard of the Yellow Wallpaper, thank. You she. Weaved together her own inter rotation of Darwin and evolution and, socialism, to paint a vision of a world in which women, were freed from society's, shackles, in order, to fulfill their evolutionary. Potential, in. This. Case we see belief in scientific, authority seeing, opportunity. In science, while, simultaneously, doubting. And not trusting, aspects, of it often, because of being limited or restricted in terms of their ability to participate in, scientific. Work and. This, is a quote here it does not do to trust people too much it's from the Yellow Wallpaper. Interestingly. Another. Example, of a 19th century feminists, who engage an evolutionary, discourse, comes, from here the state of Michigan Eli, Zabar gamble a Michigan. Woman who pursued pursued, a career as a teacher she, contributed. To 19th century women's movements, in varied ways including, presenting, and writing about the women's suffrage movement and critiquing, Darwin's, descent of man, she. Wrote a book available, in Special Collections in the library here in 1894. Called the evolution of woman an inquiry, into the dogma of her inferiority, to man which. Is a fair analysis, of Darwin's failure, to see female, choice which.
Was So evident in animal, in the animal kingdom in human. Society, with. Elegant, and compelling, prose Gambel argued that Darwin's failure to follow through with his analogy between humans, and other animals a, foundational. Part of his evolutionary, argument, in the origin and expressions of emotions, was, a significant, flaw in his analysis, of human gender. And sexuality. Here. We see an exhibit between, which, was created as a partnership between Briggs, and the msu museum showcasing. The work of a former and a graduate of mine called daniela Beck Bex researched, a biography, of Gambell using archive materials, in Lansing and Ann Arbor to form the best published, biography, of Gambell out there I've had professional, scholars come to me at conferences and tell me Danielle Oh back, that's the article to read about gamble. And. Yet. This work doesn't make its way into mainstream histories. Of evolution, and I'm gonna end now because I've done my 10 minutes with four, questions, for you to keep in mind and I suspect these questions will resonate with the other speakers as well why. Do topics, of concerning, gender and evolution. Start to be part of scholarly discourse until, the 1970s. Why. Did different people see different opportunities, in the scientific, work of Charles Darwin in the 19th century how. Does the work of Charles Darwin and the authority, of his name and image continue. To be used today and, perhaps. Most importantly, for the young scholars in this room today what, can we learn from hit, this historical, story in terms, of the importance, of discussions, of diversity, and inclusion in stem today thank. You very much. All. Right, so. I'm gonna shift, gears a little bit from dr.. Montgomery's, historical, perspective, to looking a little bit more at a, contemporary. Situation, and I. Want to give you my sort, of three take-home messages, right off the bat so, first, I want to suggest and this really resonates, nicely with dr. Montgomery's, talk because she pointed out cases, where it can be fruitful, to be questioning, science, I want, to look a little bit more cases where it could be problematic, or can raise, challenges, for society so.
I First won't acknowledge that there are cases where doubting, science can have harmful consequences, but. Then my second, key point is that, um there's, this temptation to figure well if there's a problem with doubting science we ought to give, people scientific, facts and get them to trust and believe the science and. I want to suggest that that tends to be a limited solution, and as. I. Lay this out I want to acknowledge that the real social scientists, are down the table here so the, philosopher, can go ahead and you know throw these things out and then they can say what I say is wrong and you know what's right so, and then my third point is. That I think we need more creative, responses, to doubt if just, sort of harping on scientific, facts hitting people over the head with facts is not the best solution and, so I'd like to invite you to think with us during. The discussion, you know what might be more fruitful approaches. So. My first point I'll be very quick about this is just, we all know about cases, where doubting. Science, can have problematic. Effects on society if, we think, just throughout the 20th century the, ways in which doubt. Contributed, to delayed regulation, of tobacco and, secondhand. Smoke we've. Been thinking a lot in Michigan about the use of lead and, so. Promoting. Doubt about the science surrounding lead resulted, in a lot of problems obviously, using, lead piping, longer than it should have been LED. Paints longer leaded, gasoline and, so on we. Know about problems, with climate change, issues. Related to vaccines. Problematic. Responses, to AIDS I'm, not positive about the statistic, but one of the books that I have. Read suggested. That because, of denial of the, scientific, information about AIDS in South Africa, in the early part of the 21st. Century, possibly. Several hundred thousand people may have died who wouldn't, have had - and issues. Related to genetic, engineering and so on so there are lots of cases that we could look at so. My second, point though is that just, telling people look you've got the facts wrong let let us give you more. Correct scientific information is a really limited solution, and I, think that's the case for two reasons, I'm so, and probably, more but first. And, this is more the my philosophical, cup, of tea I think, it tends to promote misleading. Perceptions, about how science works making. It seem like science, is very straightforward, and we just need to give people the facts and they are correct for all time but then secondly, I don't, think it's very effective and, so again here I'll throw out some thoughts and like I said the social scientists, can give us more empirical, information, about this so. Let me just discuss these points in more detail am I worried that just. Telling people look you've got to get the facts right promotes. Misleading. Perceptions, about science, you, know science is always tentative, and it changes, over time and. Actually. You know we often think now because, there's so much discussion about the consensus. Around climate change it. Can give people the impression that you, know there's always a consensus. Around science and actually, it's somewhat rare and usually, it's only very, simple, basic, claims, like that he are contributing, to climate change, where you could actually get something like a consensus, and then. Often. When we're talking about doubting science, we're thinking about policy, relevant, areas of science and so, there we're especially, often pushing, the limits of science and so, it's even less likely that the science will be straightforward in a lot of cases again, there are a tremendous range of cases so you. Know one has to consider the differences, between different contexts, but in a lot of areas like my specialty. Is environmental, pollution and they're, often the regulators, and policy makers are trying to make decisions and the. Scientists, would be inclined to say well if you come back in 30 or 40 years maybe, we can get this figured out for you and really feel confident, but the policymakers, want to make decisions now on and so, in, most cases there's. Going to be reasonable, disagreement when it comes to policymaking in science and so, stifling, dissents, I think can be harmful to science and can, prevent, us from gaining, insights and we saw from dr. Montgomery's, talk.
That You know often, important, centers can challenge, really problematic, issues. In science so. I, also, would, suggest that just, harping, on scientific, facts and saying look you've got to get your your science straight here, may. Not be that effective and I. Had suggest that in a lot of the prominent, cases we look at, the. Doubting. Of the science, may, just be more of an excuse, for, being able to do things or hold, beliefs that one wanted to believe and it's, not really doubting, the science itself that would be driving, the. The concerns, so if we think about issues like climate change it, seems like and again professor macwrite can set, us straight on this but it seems like it's more a matter of fear of regulation, and big, government that's really driving the issue and. The. And that if you could address that there probably wouldn't be as much concern, about the science of climate change if one thinks about evolutionary. Theory again. I think that what's often really driving the issue is concern, about challenging. Religious, beliefs and if, that wasn't such a concern people probably wouldn't have so much concern about evolutionary, theory again. Genetic, modification or, genetic engineering. You. Know not, too that everybody, falls. In this category but a number the, opposition, may come from concerns, about the way our Agri business operates, in the way our industrial, food system operates. And, that may be driving a lot of the concerns and. You. Know I gave the example of South Africa, a lot of the concern there about the science, related, to aids may, relate to concerns, about Western, colonialism. And you, know imposition, of a, variety. Of aspects, of our culture upon, them and so thus their concerns about the science, so, my, final point and this is where I want to just throw out some ideas and invite your thoughts I'd love to have more discussion, about that, you. Might say well if we're not if you're telling us not to harp on the scientific facts how, should we address these controversies. If they can have important, social consequences, and so, I have two general, suggestions, so, first you, know I was just pointing out that it seems like they're often these deeper, issues at play I think, we need to explore, ways to relieve, those deeper, issues, and.
So I can offer a few examples, here my, second, suggestion is that maybe we need to find creative ways again. I'm thinking especially about policymaking to, fight less about the facts, and to. Sort, of find. Creative, ways to address, the issues. In. Alternative, ways so let me give some examples so with climate change I know you're gonna say this is easier said than done but, if we could make clean. Energy easy, I think, we would find that people would be fine I'll just you know I'm happy to obtain my energy in cleaner ways and if, that was easy to do then I think a lot of those stress about the science of climate change would kind of fade into the background now. If you think about an issue like, evolutionary. Theory I think. It's would be much more effective if one's interacting. With somebody. With skepticism, about that because, of religious concerns rather, than telling them look I know I'm gonna show you why evolutionary. Theories correct if you can show ways in which that religious, tradition, can accommodate, evolutionary, theory I think you're likely to have much more success and again, this is the philosopher speculating, we'll see what the social scientists, have to say I could. Say more about genetic, engineering we'll see if I have time here when I got my two-minute, warning but. I'd like to also talk about creative, ways to fight less about the facts so, this is an example that actually comes from a, colleague, of ours mark Largent, who's done a lot of looking at vaccines and, he, points out that in the state of Michigan, we really. Were able to increase vaccination. Rates. Just. By changing, our waiver policies, not it's just sort of forcing, people with the science or even just absolutely, requiring people to get vaccinated but, making it more complicated. Requiring. That people meet say with somebody from the public health department before, they could get their waiver signed so, it's just a creative way of addressing, the issue just making, it a little bit more complicated, to avoid getting vaccinated, could, probably be much more effective than trying to actually convince people with a bunch of facts in.
The Areas that I work in more environmental, pollution. You. There can be endless, debates about, the details, of whether a particular chemical ought to be regulated. Or banned and so. There have been some great examples of legislation, rather than trying to actually establish that, a chemical is definitely, harmful enough to be banned just. Taking, a list of chemicals that were worrisome, and saying, if industry, is going to use those chemicals, they, need to just report, how much they're using and also, report. On any alternatives. That might be available and just requiring, the chemicals, that are just requiring the companies to do that actually, they had some significant. Success in the companies sort, of feeling a little sheepish about using, so many worrisome, chemicals and finding. Out oh there are some decent alternatives. And thus. They were willing to shift rather, than fighting endlessly, over the details of the science and I would just point out my third point sometimes. People may wonder why do we have so much trouble addressing, climate change as opposed, to say, addressing. The ozone hole and I think part of that my understanding, is that there. Were alternatives, available that the companies could shift to and so again finding, creative, alternatives. Can enable us to get out of some of these conflicts, where we might just otherwise, endlessly, debating, the science if there aren't such good alternatives, available so. Maybe I'll stop there and, we, can talk further in the discussion but I love it if you all had further ideas along, these lines of coming up with creative solutions. So. I really like the idea that, dr., Elliott was just talking about about. Not. Debating, the facts because I work in technology and the facts are constantly. Changing in technology, and that's one of the things I want to talk about tonight is. Wait. Algorithms, have been, becoming. More and more common, in society, I mean right now if you just look around we have algorithms that drive. Cars that determine. What classes your takes in some cases determine, grades of classes.
Algorithms. Are becoming increasingly, common. In society. And, one of the things that's been really interesting in the last like two years or so is that there's, becoming, additional. Skepticism. About whether these algorithms that are being produced are trustworthy. As. Part of a initiative. From the National Science Foundation I recently convened. A workshop, where we tried to figure out what are some of the major challenges, and. Opportunities. For, making. Algorithms more trustworthy and why is this actually a really difficult, and hard problem, and I want to talk about some. Of the things. That we uncovered during, that work that. Illustrate. How algorithms. What, makes our rhythms so hard to be trustworthy. And why it's so difficult in society, so, I'm gonna start by there's. Been a lot of news talking, about different. Trustworthy. Problem problems, with algorithms this, is a new story about how uber is. Having. This problem where all the drivers will sign, off of uber at the same time, uber, will then think that there's not enough drivers and will raise the prices, then, all the drivers will sign back on and everyone is paying higher prices for their cars now. Another. Issue is YouTube, is, has. An algorithm that is trying to identify which videos, are safe for children and which ones are not and there's, actually become a kind, of cottage. Industry of creating, what people, are calling creepy, kid videos, that, are not, actually, safe for, children but passed through YouTube's, algorithms. And. Everyone's familiar with the, recent. Challenges, that Facebook has had related. To the election and how, many. Actors but Russia in particular have. Been. Able to manipulate Facebook's. Algorithms, to try to influence, what people know and think and talk about, so. I want to spend some time talking, about three or four specific examples. Of, of. Algorithms, that have had challenges, in the past and, use, those to illustrate, some of the big. Challenges. That make making, algorithms, trustworthy, really difficult. So. I'll start with a.
Colleague. Of mine Eric. Meyer at Case Western. He. Went through something that no one really should have to go through his five-year-old, daughter died. Of cancer she, spent. Months. Going. Through treatments, and she ended up dying in, June of 2014 and it was very sad it was a very difficult time for Eric he. Spent a lot of time taking care of her and dealing, with all of the doctors and all the grief. And all the challenges that come with losing, a child. Six. Months later in December, he's. Using Facebook, and. Facebook. Presents. This to him, it is, their, Year, in Review and, it says Eric. Here's. What your year looked like and it showed him a picture of his now, deceased. Five-year-old. Daughter surrounded. By people with party, balloons. This. Was a really. Challenging. Like, he, started you broke down crying when, he saw this using, Facebook this is it, brought back a lot of grief a lot of memories a lot of challenges. Facebook. Didn't. Know actually didn't, know how to deal with this this was really, challenging, this, picture, that they picked was his most commented, and liked photo from the year when. His daughter, died and so they didn't have a way of telling, likes. That were happy from likes that should not really be brought up again this. Personal. Social and cultural context. Is. Completely. Absent from the. Algorithms, that they used. They just looked at likes and comments, and shares to try to figure out what, was the most engaged. Content. This, is one of the biggest challenges, with algorithms. Is they require large, amounts of data but most of the data is very, similar, its likes and comments type of thing it's very similar to each other and it's very difficult. To take into account the, full, complexity, of personal. Situations, and social contexts, and. To include, enough data that you can take those into account Facebook. Is still struggling with this this is part of what led to the fake news issues it's, it's leading to a lot of other issues in many different contexts. Another. Context, comes up in my second example so. This is Eric, Loomis, Eric. Loomis is a convicted. Felon he was convicted of a felony in the state of Wisconsin, and, when. He went for sentencing, the. Judge in, his case, used. An algorithm called, compass, to. Decide. To influence, his sentencing, compass, tries. To predict how likely a person is to, reoffending. It another crime in the future in. Eric's. Case he was predicted, to be highly, likely to. Reoffending. Convicted. Sex. Offender before he before, this and. So they he, got a very long. Sentence. In prison, and. He challenged, this in court because this. Algorithm, is proprietary it's, not available to the public there's no way to inspect. It to figure out whether it's trustworthy or reliable, it. Went all the way to the Supreme Court of Wisconsin who. Ruled that it was legal that they do this as long as that's, not the only thing they use in sentencing, him. Partially. In response to this case in a couple of other cases, ProPublica. Went. Through and tried to analyze how. This algorithm works and, try to understand, and figure out how it works and they, came up with a really startling conclusion. When. They looked, and compared, African. American. Criminals. With white. Criminals, they, found that African, Americans, were very consistently, predicted, to be much. Higher likelihood. Riah fent, so. Then. This, was. Actually, predictably. Leading judges to give them much, longer, sentences. In prison for equivalent crimes. So. This was a big deal, the. Company that made this algorithm, called North used to be called North Point they changed their name after this but. The company used to be called North Point they. Actually, they, took. They got a lot of bad press about this but they actually had a really interesting reaction, and the reaction to me is more interesting than the algorithm their, reaction, was basically, yes, our algorithm, does this but. It's not biased, and they were trying to argue that their algorithm isn't biased, because in. Their data. African-americans. Are more likely to. Reoffended. Data that showed that the african-americans, that they were studying were more likely to reoffended, the whites that were in their data and therefore they. Were not actually. The, algorithm, was not biased against african-americans, but instead that. They. Were reflecting, what, was going on in the real world, turns. Out though by. Reoffended. Definition, of reoffending interesting. Because it wasn't, commit, another crime that's, really hard to measure they.
Measured How likely, they were to commit another crime by how likely they were to get arrested, again and, arrests. Are a human, process, that. Have, racial. Biases, in them and so. The, underlying data, was, what was biased, that they based the algorithm, on as opposed. To some, bias in the algorithm itself so, when they looked at their algorithm, they didn't see a biased, algorithm, it was accurately, reflecting, the biases, in the real world. That's. A really interesting challenge that's really, difficult to, deal, with in algorithms, is how, do you how. Do you design algorithms, that do. What we want them to do as opposed to reflect. The challenges. And problems that. Are already existent, in the real world. To. Compound these issues. That are coming up in algorithms, we're, seeing also. Another. Problem with algorithms with a lot of the modern machine learning based algorithms, that, is. Turns out is really, really difficult to stop so. This, is a photograph. Of a panda. Pandas. Are pretty simple and it turns out there. This this, photograph, in particular is a really common photograph, among machine learning researchers, who are designing algorithms. For, a computer vision that recognize. What. Is in a photograph and so it's really easy now we have a ton of algorithms, that can recognize that this is a photograph of a panda, it. Also turns out that one. Of the things that we can do is we can modify. This image, so this is an example of. We. Take the image and apply the algorithm and it says it's a panda, with 60% confidence, if. We take and add to that something, that looks like random noise but is actually carefully, chosen, the. Algorithm, will come out and say this, is a given type, of monkey with, 99%, confidence, other. People have turned this, photograph, into a llama, or a camel, or. A giraffe, or a couple other different things. The. Basic problem is that most, of our machine learning algorithms, have, this kind. Of fragility, the kind of fragile, and it, there are a lot of cases where if we modify them in a specific way then. What humans, perceive and what the algorithms perceive turn out to not be the same thing and so we can create situations where, the algorithm perceives, something different than what the humans perceive. This. Is also really, difficult to stop the. Way a lot of these algorithms work, it's. Really difficult to stop, this and to make robust, algorithms so it's really easy to find these and it's really hard to stop these and this is a this, is becoming more and more of a problem as we use algorithms for. In more situations, they're, also open to this type of gaming and this type of manipulation. We're. Also seeing a lot of really interesting cases, where people. Are fighting back against algorithms, that are then changing, the underlying data, so. Some of my favorite. Examples there's. A paper recently, about. How people are doing with our calling Voldemort, II they're, referring to something by. Another, name so instead of calling him folder more they say you know who or he who.
Must Not be named. Another. Example is. There's a really interesting recent. Art. Exhibit, in New York City an. Artist tried, to design, hair. And makeup styles that were resistant, to facial detection and, so these all, four of these pictures if you run them through most. Of the modern facial, detection algorithms, will, say that there is not a human being in these pictures and that's. Really interesting to try to think about how the, world is changing in response to these algorithms, thank. You. That. Shows you how, technologically. Adept I am. So. I guess. It's it's fitting, that our, first two scholars, tonight talked about science and the last two we're going to talk more about technology. Rather than than, science. Per se, so. I want to talk a little bit about. Values. In. Particular. I want. To talk about, the. Value, or values of. Cautionary. Tales. So. You. All have heard, cautionary, tales, growing. Up you've probably read Aesop's fables, and the Brothers Grimm some. Of you have made a hat have a had. A class in Greek mythology. But, basically, like your textbook definition, is a cautionary, tale is, some sort of folklore, or, fable, a parable. A proverb something, like that that's, meant to warn an audience, about some danger, about, a risk, so. It's usually a story where one of the characters typically, the main character. Does. Something wrong does. Something really wrong violates, a taboo or some, sort of prohibition. Against doing something, and then, they face a terrible, fate. So. Let's, take a deep dive into, some, famous cautionary. Tales. So. Let's. Start back with Icarus. Icarus. Of course is a son of Daedalus. Icarus, tries to escape Crete, by. Creating. Wings of, feather and wax. What. Happened to Icarus just. Shout it out. Yeah. You flew, too close to the Sun so, the Sun heats. The wax and, he comes tumbling, down and, dies in the sea. Cautionary. Tale. Another. Cautionary, tale, you might not know from the portrait. Here well I guess it's not a portrait but a painting. Prometheus, he steals fire, from. The gods on Mount Olympus, unmount. Olympus, and gives it to humanity to help bolster, foster. Progress. In civilization. What. Happens to prometheus. Yes. Doesn't. Sound like a very nice, fate. He's. Tied to a rock and every, day an eagle starts to eat away as at his liver. The. Next day it happens all over again and all over again for, eternity, another. Cautionary, tale. Now, we have, Pandora. Pandora. Of course, is. Earth's first woman, so. Zeus, gives, Pandora. To, Prometheus, brother, Epimetheus. Even. Though Prometheus, says, don't. Take any gifts, especially. Not from Zeus what. Happens to. Epimetheus. Or. What, happens with Pandora. Yeah. It's. Actually a jar we'll let that go right you know translation. And everything so. She gives up a me theus her, jar opens. It up and releases, all, the evils. Of humanity. That. Doesn't sound so good. Now, we have. Faust. So. Faust is this pretty, successful, but ultimately sort of like less, than fully satisfied, scholar he, wants at all he wants great, knowledge great success. What. Happens to Faust. Yeah. So. The, devil's, got too, much work going on probably down at Ann, Arbor and, he. Decides I, got, this helper, method, awfully I'm just gonna send him and. He, makes the bargain, with Faust saying I'll give you all that. But. For the price of your soul and then you'll be a slave in, hell for, all of eternity. Not. So good. And, then perhaps one, of the most famous ones especially in terms of science fiction and, our modern, concerns, about technology. Good. Ol Victor, Frankenstein. Creates. A monster, a hideous monster through. Vivisection. And then experimentation. What. Happens to Victor. Yeah. Someone, made that back. Here yep so. He, he not only dies, but. First though. His monster, his creation, kills his brother then. His childhood friend then his fiancee, and then Victor, dies chasin. His monster around the North Pole. So. All. Of these lend. Phrases. Catchphrases. Quotes.
Into. Modern. Culture. At. Least modern Western, culture we, often use, these over and over we talk about frankenfood. An. Opening, Pandora's. Box. You. May have used these like just in this past week to describe something, like oh don't don't, do that don't go that far. You're flying too close to the Sun or something like that. I'm. Really interested, in a lot of emerging technology. And. So, I've. Taught a class a couple times on different emerging, technologies, which. Is one of the reasons I think I was invited. To be. On this panel so, artificial. Intelligence, is in. A lot of our lives whether, we know it or not and, modern. Culture at, least going back several decades up to just a couple years ago is producing. Cautionary, tales about AI. Whether. They're, good movies or not so good movies I don't. Think there's a Michael Bay movie up, there I don't know I'm not sure. But. These each of these movies if you've seen them you'll realize it's some sort of warning, about going too far going too far too fast not. Being humble enough maybe getting too arrogant about ourselves and our inventions. So. I want to take a step back and put all of this in a little bit of context. And sort, of make, use of the anthropology. Minor I got 20-some. Years ago. So. If we talk about human, evolution I, guess, this goes back to to, dr. Montgomery's, first talk you. Might think at a very general level, that, human. Evolution is shaped by these opposing, forces so you have maybe individuals. Groups, organizations. Societies. That. Are. Willing to take big risks venture. Into the unknown. Advocate. For change. And. Maybe. They value. Innovation. And exploration, invention. Try out something new let's. Do something no one's ever done before. But. At the same time maybe in that society, there are. Individuals. Groups organizations. That. Are, risk-averse. And, they. Prefer order, and stability. Or, stasis, and they value looking, towards the past looking, at tradition. Practicing. Rituals, not looking, forward not running, into the unknown full, speed ahead.
But Trying to get back to, where we were in the past when, things were more stable, and known and comfortable. So. You could imagine this playing out, maybe not. Among, us well-dressed, folks tonight but all the way back in the caveman ages. Or cave-person, ages, these, just happen to be three dudes right, so, you could imagine there's, there's these three cavemen. They're, just walking, around the savanna. It's, like hey hear that in the bush let's. Go get it it might be food right. There's your risk seeker, he. Wants to run into the unknown because. It might be a good food source right. It's. Like brah you always, say that and it's. Usually something that can kill us so. Caution. Let's, slow down maybe, let's just go back to camp, go back to the cave right. So. Jumping, forward, to, the current day, at least in, last. Hundred, years or so. This. Ideology, of technological. Progressivism. Is pretty, widespread especially, in the United States so by technological. Progressivism, I mean, this idea or ideology. Or belief system the, strongly held. Worldview. That. Technology. Doesn't just produce, progress. That technology, itself, is progress, when. A new thing. Comes out be, it a phone a computer, a, robot, that, is good in and of itself it's good just because it's new it's progress. Manifested. In in this thing. Usually sometimes, it's a non. Tangible thing like a code. It's, a belief that technological, development, is sort of internally. Self-propelled. It's, always for the better and. By. Definition then, newer, versions. Of technologies, are inherently better than older versions and it. Seems self-evident a. Refrigerator. These days is, so, much better than a refrigerator 50, years ago. It's. Probably, better than 10 years ago, and. So. This ideology, is very strong in our culture and in other cultures, we Americans, it's probably the strongest. Among. Other societies, so. If. You, think of cultural, evolution of, societies, especially, those, societies, where scientists, and engineers are. Routinely, putting, out new discoveries. And new technologies. You, might think about now. The two groups might be captured by this ideology. Of technological. Progressivism, of. Almost. As a reflex, if you say the word technological. Half the people in here will say advancement. Or technological. Progress it just sort of comes as a reflex, like, we don't know what else to say after that but advancement, because that's what technology is and that, high tech is better than low tech and anyone. Who says, well, let's be cautious they're. Just chicken littles they're naysayers. They're, irrational, it's like you don't just you don't understand, the facts of this to, what dr.. Elliott was saying but. Then we have all these cautionary, tales, that. Are maybe. Preventing. Us from going, too fast, into the future or taking. On too many risks with, unknown, technologies, and so, I would basically argued that the the cautionary, tales in our society, that keep popping up are some. Sort of a check or a countervailing, force. To. Mitigate, or to slow down or to weaken the influence of this progressivism.
And I, am just going to skip, skip. And get to the the very end I know I'm a little over. So. I just wanted to show you in, this year 2018. The relevance, of these cautionary, tales so it. Was a 2000, anniversary, or 200th, anniversary of, Mary Shelley's book. Frankenstein. Or the modern prometheus. And. So the January 12th issue of science the cover story was Frankenstein, and all sorts of other stories, inside, talked, about the modern relevance, of Frankenstein, in talking. About science and technology, you've, probably seen lots of Boston. Dynamics robots. Recently. This was the dog that could open a door the. One that was doing, parkour. Just a couple weeks ago that terrifies. Me but, you could see in very small print, this was called the latest, latest Promethean, creation. Here's. A story about CRISPR, Castine gene editing an incircle. That's called perhaps are we opening, Pandora's, box. This, article is talking about how, adopting. Facebook, in Africa, might, be opening, up a Faustian bargain in, terms of all the information that's, that Facebook is gathering and big data analysis. Those, of you who've been around campus to see all the scooters and stuff well evidently, Charleston. Has, a problem with scooters and so they're, saying like are these bird, scooters flying, too close to the Sun that's, a sort of a lame headline, but you know you get the idea so I'll just leave what this. If. I think we need to talk about big questions I would throw these three out there so. Whenever we talk about a new technology, be it a big system, a specific, artifact, a way, of knowing, a way of doing, I would, say to, what extent, does. This new technology, actually facilitate. Real societal. Progress. In. Progress, is. Probably, another word we can define but we can leave that for later and then, to, what extent is control, of this. Technology, associated, with power it. Could help the, powerless, gain, control, it. Could also help very powerful, people monopolize. Or, or, shore, up their control, and exercise, their control over others and, then. Finally, how can we. Accurately. Perceive, and, manage. The risks associated with, these new technologies, at. The very least to. Prevent. The, most serious, high consequence, risks.
And. I'll just leave it at that. I'd, like to thank all four panelists. And just. Watching you as an audience you seem to be very. Engaged, unless. You were fooling me. But, we're gonna give you a chance to. Interact. With the panelists, but before I do that. Only. One of the presenters, had. A chance to hear. Was. This something you really wanted to say and didn't, have time to say, before. We go into our question, in advance okay. Just. One of the things that struck me how and, this was the first time we all heard each other's talks. I. Wrote in, a circle, here in my margin for all four talks power power, power power and I just think power dynamics, be. It talking about science, or being talking about technology, was that. Was for me the key thread that ran through the panel. Just. A quick comment would be that I think part of what makes this such an interesting topic is that there isn't a nice simple, answer, like that we should always be trusting, of Science and Technology or we should always be respecting. And trusting, those who are doubting it you, know I thought that you know professor macwrite stalk especially, brought that out the fact that you know we see the value of progress, and we see the value of questioning, and you. Know I was, kind of pointing that out early, on with our talks the, ways in which you know there can be problems, to being too respectful, of the scientific, ideas of the day but there can also be problems, to being. Overly, skeptical and it's very complicated to figure out how to address these issues so I think that's part of what makes this interesting. Okay. Okay. Well what what I'm gonna do is you. Know if I was sitting out there I would have a had a lot of questions, but I'm not out there the. Next phase of this will, have, professor. John Beck and, he is with the school of human resources, and labor relations, and. Here's a lot of credentials but, particularly. For this event. He's an expert in training, and development, and also, in workforce, development, and. He's, gonna facilitate a, question, and answer that's a nice way of saying that I'm gonna walk around with the microphone which. Is one. Way for chubby, older professors, to get exercise so thank you for that so do, raise your hands one thing I would lay out the. Power power power power is, a great, I. Think. Synthesis. But, beyond that think of the fact that naturally. Next week or the um very soon is Halloween and it's. Like no you. Know we always look at the horror films and go don't open. That door, you know it's just Aaron's, point so, raise your hands and I'll come around to. You who. Is the. Question okay I'm coming. Dr.. Wash you mentioned, the limitations, of the compass evaluation. In judging. And I worked with ex-offenders. And so. I'm aware of how the compass, information was gathered from the inmates and that also is somewhat flawed. But. My question is are you aware of how police agencies, are using something, an algorithm. No doubt called. Predictive, policing. Yeah. Would you compare that pretty much with a very flawed way a very biased way of saying that there's, always crimes in the Waverley Jolley area, of Lansing, and so we're gonna send a lot of police people over there because those, black people are always committing, black-on-black, crimes it drives me crazy. Yes. I think there's. That's. Actually another that's one of the things I almost talked about but I decided want to talk too much about policing.
But. There's a really interesting example. That, comes from a colleague of mine who works in Milwaukee and, he's been. Working with police agencies, to study how they're using predictive. Policing and. A really and I like I really like the way he the example he uses he went in and he sat. Down with with, two police officers who were they. Were actually weren't police officers they were recent. Criminology, graduates, from the University of Wisconsin. Milwaukee and, they. Just, gotten their master's degree they got hired because they had technology skills and they were given this algorithm. To do predictive policing and, they the algorithm was trying to create clusters, of wares. Which we send police and they, he asked them alright so part. Of the algorithm is you have to tell it how many clusters you are how did you choose to use five clusters and their answer was well, I went to my sergeant and he said that there's basically five different areas of the city and so I used five clusters and. In this case they're basically giving. Technology. Technological. Credibility. To, what, the, police were already doing, and. So that's one of the real challenges with, the as these algorithms become more common they also need, to be able to be usable by people who aren't PhDs, and. Be, able to be valuable in that way instead of if. Instead, of just kind of reinforcing, what we already have. Okay. Other. I'll, just follow up because I love, making connections, to movies so if we think about compass, and. Predictive. Policing I guess. Someone, might say that there's like, the the fatal flaw in and both, of those are the underlying data right. It's sort of the input, is flawed. If. You guys have ever seen the movie Minority, Report not, the not-so-great. TV, show that's. The essence, of the warning, in Minority, Report so. The. Main Precog. The. Female Precog, it was, the murder of her mother that. Facilitated. The continuation, of, pre-crime, that gave I. Can't, remember the the guy in charge, that gave him the power to, run, this pre-crime, unit so. At one level we're talking about a science fiction scenario, where. We're trying to predict who's gonna do what before they do it which. Is sort of like what some of these algorithms, are doing it either predicting. Recidivism, or predicting, crime to begin with and that's sort of captured, in, the. Movie Minority Report but. Also the. Same cautionary, tale of if, something, is flawed from the very beginning. No. Matter how good, your intentions, it, might still be flawed throughout, so. It's sort of like the the the lawyer term, of its its fruit, from the poison, tree it, could, never quite be fixed, it's always in there it's built into the system that's, left that flaw. For. Doctormick, right and dr., Eliot I just, sort of noticed the similarity, between they're, all the cautionary, tale and doubting.
Science, So I want, to know in your opinion. What. Is the role of the cautionary, tale is it harmful or do. You believe there is just a balance to be struck and. You. Guys sort of implied this. You. Would brought up dr., Elliot questioning. Or, harping. On facts, as harmful would, you argue that cause canary tales in the same way or sort of a cop-out, I'm in journalism, of just, another. Token phrase to use over, and over again. So. What, I end up thinking about when I hear the cautionary, tale because, of my work related to environmental pollution is the. Precautionary, principle which is an. Idea that comes up a lot with environmental. Policy, and regulation, which. Is the suggestion that you. Know even, if the science, isn't all settled even if you don't have all the you know a precise, sort of causal story laid out, that. It's still appropriate to take action, to prevent you know serious, threats, of various, sorts and what's. Really interesting again, as I was mentioning earlier it would be nice if we could come up with a simple. Solution and say yes we can always be precautionary, and that's the way to go in. Some ways I think it does it. Moves in the direction I was suggesting, not, getting lost in, a tangle, of facts and saying we've got to get all these facts figured out before we can do something so in that respect I think that this, sort of cautionary, or precautionary, approach is valuable, but. It. Would be I think too simple to say we, can solve our problems, with it because it raises all these other difficult, questions what, sorts of threats are serious. Enough that, we're willing to apply the precautionary principle, we probably can't just apply it to everything and how. Much and what kinds, of evidence would we want to see before we would trigger this kind of prakasha principle, and and what kinds, of actions would be appropriate, in response so it creates this tangle, of further questions. Which. I guess is kind of again fits with Professor McBride point that you know we've got this realization, that yeah, caution. Is is warranted, but it gets awfully, tricky to figure out you know sort of how to actually. Implement. It effectively, so, that would, be some initial thoughts. Other, guys I'm. I'm. Having an old man moment right here but but I'll say. One thing that I can remember, so, I just as well could have sat up here and given a talk about climate change denial, because I've been studying that for a while I, just, didn't want to because, I'm sort, of tired of talking about it even.
Though It's not going away I just don't know whatever what other new stuff to say about it. So. From. From what dr. Elliott brought up and. Dr., Montgomery, brought up with regards to science there's. Different. Reasons, why, we might or might, want to doubt. Or be, skeptical, of different, arguments. Or different pieces of evidence within science, I, have. Focused most. Of my attention. In. My career on sort. Of organized, climate, change denial whether it's the fossil fuels industry or, conservative, think tanks or, certain. Politicians, and. The. Motivations. For, that, skepticism. Or contrarian ism or denial. It's. Almost never because. The. People I just mentioned, are the groups I just mentioned want to get the science right, because. They have at the core of their, endeavor we, want to make sure that, we have the actual truth it's, usually. They. Are attacking. Science, or critiquing, science, or trying to poke holes in science, because they're. Concerned. About. The. Regulatory. Intervention into. The market some sort of regulation, or tax that, might result, once. Policymakers. Understand. The, nature, of the science so if science, says we're causing, this problem and it's big and. It's. Not something that the market, could readily solve, on its own and that's gonna require some, sort of governmental, intervention, and of course if it's climate change in its global that means not. Just a state, or a county or, a. Country, but many countries have to be involved, and. That. Really flies, in the face of. Some core values especially. For us conservatives. And us libertarians. And, so that's why you see them doubting. Or being, skeptical, or denying. Climate change or the, science, of climate change, not. Because they're in it to make, sure scientists, get it right and they're really concerned about you know peer, review and so on they're, just using. Attacks, on science, or they're just using science, as a proxy, for what they. Really want to talk about which is values. Their worldview, so, this. Solution is, antithetical, to the values. We hold about, private, property rights and limited government and. Free, market. And. It's. Maybe. Easier now, to, deny. The science, than, to have those battles, once, we've accepted that the problem exists, because. Then people want you, to do something and, that. Often means government, intervention, a. House. Divided, cannot, stand and. I, think that's a big, part of where, we seem to be in this country right now about 50 percent or so on, each side 40, 60. We. Seem to be talking past, each other and I. Think possibly. Part, of it might be that on the one side you have a very religious, oriented. Or, faith. Belief, type. Of approach. To life and the, other side is more fact-based. So. My question. Is, how do we bridge the gap and actually be able to communicate, where. We, can actually start talking, to each other and solving. The problems, that we have because, right, now. We. Just talk right past each other we're not even on the same frequency. That's. A big question. And. I wish I could solve it I can't I'll tell you one historical, antidote, that speaks to what you were just talking about and British.
You May have, noticed that I'm aware, US citizen I'm just going to say that right now. And. I will be voting on November. 6. So. I grew up reading a fairy tale called. The water of babies and, I. Actually collect, special. Issues of the water babies book now and, the water babies was a fairy tale is a fairy tale written, in the 19th century by Reverend, Charles Kingsley, he was friends with Charles Darwin and it. Was written it's a thick fairy tale much longer than those books we read kids nowadays. To. Teach children that they could believe in Darwin, and evolution and, Christianity, and, it. Had a figure. Called mother Carey who, was a female representation. Of God, and. She. Made the animals make themselves so she made the mechanism, of natural selection and, then. Sat, back and let that mechanism make all, the different organisms in the world rather than bothering to individually. Create individual, organisms, which is what, was believed at the time. And. I. One of the reasons I think I really still like to read that book I mean it's a beautiful illustrated. Book and to. Scholars piers Hale and John Beatty actually analyzing, their who storms our science they're writing a book about water, babies is I, think it's does a really beautiful job actually of bringing. Together those two sides of a debate that nowadays, is a. Schism for many people as dr., Elliott was talking about too so. Maybe there's something that we can learn from. It's. An example I think of meeting in the middle and talking, unfortunately. It's from the 19th century. So. I also don't have easy solutions, it's a really interesting question but, it does relate a little bit to something I was just thinking about before. You raised the question which. Is another connection between a lot of these talks is. Ways. In which values. Of area swords are. Intertwined. With the science, or the facts, so I was thinking about dr. Montgomery, pointing, out that Darwin was you. Know his thinking. He was doing this science, but he was very influenced. By the values. And perspectives, of that time and and. Then if we think about dr. washes discussion, about algorithms, you know we were seeing how you, know the. Algorithms, depend, on what's put in and, the values that go into how they decide what the right inputs, and so on are and of course you know dr. McCrae was you know emphasizing, these very valuating, perspectives, on. You. Know our views about technology. And I, would say I mean one of my themes in my research is the roles that values, play in science and especially in this policy relevant, science where the science isn't settled and so, so much depends, on, you. Know sort of various interpretive, judgments and so on so I don't know maybe one way to think a little bit about your ques
2018-12-22