Do the right thing: IT in the age of social responsibility - BRK2050

Oh. Thank. You howdy I'm Jeffrey sent over today. I'm gonna give a very, different, kind of talk than I usually give this. Is a very, interesting time, in the. Industry and, it's. Time for us to kind of step back and take a look look. At things and that's sort of one of the jobs of a technical fellow technical. Fellow has a the, ability to you know kind of not. Get. So focused in on the day-to-day we, have a bit of time to step back and look at the general trends now in general we look at the general technology, trends. And say hey there's, this great opportunity here hey, there's this great threat there and here's. What we need to do about it and. As myself. And a number of other technical fellows or you know having conversations. We realize hey there's this completely. Other threat, of things that. We need to be thinking about and discussing and that's that's what resulted, in this talk and, so this is you, know software development, in age, of social responsibility, so, just, to be clear this, is not a, technical, building session. Right skill building session this is a very, different type of session I hope, you will like it hang in there a bit you know it's, not your typical session. These, are my own views these are not the views of the company although there will be parts, that are where, I'll tell you what Microsoft, is doing in this area in fact it's one of the things I'm proudest, about at Microsoft, working for a company who's in the, lead on, some of these discussions, but just to be clear these are my own opinions. And. A. Lot. Of these will be forward-looking, points. Of view and. You. Know it's it's emerging, right so we'll see how this all all works out so, here's sort of the agenda, why. I'm, gonna make the case for why we all need to become better engineers, right to, deliver better services, deliver, Preta products. Indeed. To be more socially, conscious and what. Are the issues that we need to address I'll. Talk a little bit about what Microsoft, is doing because as I said, we've. Got a fantastic. I. Guess ease now the president. Brad Smith you, know Sacha of course he talks about this stuff he's great but also Brad Smith we got two great leaders who've been doing some leading some fantastic. Initiatives, that I want to talk about and, then all the stuff you hear and you're like yeah yeah yeah great. But. What do I do well, you know, me I'm, a very practical guy, I'm, always looking for that yep, that's where we want to go and here's. What I do tomorrow, and the next day and the next day and just to, set context. Here, what. We need to do is not, particularly, well defined, the, approach, for, what needs to get done is however, and you'll see that all right, it. Starts with the concept, of Marc Andreessen is paper, in 2011. Marc Andreessen wrote a paper said, software. Eats the world so. What does that mean in, that document, he really put, forth two, hypotheses. You. Know the of the screen if you do if you do that you, definitely need, to get the office, lens application. It is crazy, good, it, really is cuz you know you're sitting there and you get the skewed. Rectangle. And it's got it worked with MSR, to go and produce a normalized, rectangle, it's crazy, good office. Lens anyway. Back. To our regularly scheduled, event. Yeah. It's my view that's my view okay, so software, eats the world you really said there are two hypotheses, one software. Is eating traditional, businesses, okay, so think about bookstores, being really. Kind of devastated, by Amazon, ads. Being, taken over by Google music, by iTunes Spotify. Taxis. By uber, okay, that, software eats the world and for. Those businesses. That are not being really, threatened, by software, businesses. The. Value. Of the product, more. And more the value, of the product of the value chain of the product is being, delivered a control through software so, think, through a car the, average, car today this is crazy when I found this out average. Car today has somewhere between 10. Million and a hundred million lines, of code in your car how's that for a scary thought and. Of. Course the more luxury cars are the ones that have more code in it right, because software is delivering more value, to the product and the, software does stuff like engine. Control systems, safety, systems, navigation systems. Entertainment. Systems etc so. The software, is delivering, more and more the value of the product, so, those are the two hypotheses, software. Eats the world, every. Business needs to pay attention to this because it's either gonna replace you or the. Product that you deliver more, and more the value is gonna come from software, so, if you're not great at software you. Are in trouble, okay, so 2011. Interesting, hypothesis, true, not.

True, Let's. Take a look oh. And. In this his, bullet. Point was every. Company's in every, industry do not think that you are immune every. Industry needs. To assume that there that there is a software, revolution, coming. This. Is a list of the largest companies ranked. By market, value in. 2017. Okay this. Is worldwide list. The. Highlighted, ones are, those, companies, whose primary, value, is driven. By, software. Over. Half of the top companies in the world value. Is driven by software, 2008. 17, last year when. Andreessen wrote the article, in 2011. This, was the list of the, top, companies the. Highlighted, ones are. The ones whose business is primarily driven, through software, okay. So, 2011. He makes this hypothesis. Here, are the companies that are software companies. Six. Years later this, is the case I assert. His hypothesis. Is true. He. Said every, company needs to become a software company now, a number of the ones on this list JP. Morgan GE etc. They, were on the previous list and I did not highlight, them so. Even these companies, right JP, Morgan says hey they're not looking at the other banks, as as their exes ten threat they're, looking to Silicon, Valley for, the biggest disruptors, GE, undergoing. Digital, transformation. Seeing, that they're gonna drive their business through through, software, so even the ones that weren't highlighted, a number of, them are you, know focused, in on getting great at software. Okay. So software is important, so. What well. Let's do, some engineering here right as engineers. What do we do we look at the data we, have hypotheses. We draw extrapolate. From the data and and. Make, our best guesses, Ray. Kurzweil, talks about, the law, of accelerating. Returns okay, in this, diagram here he's basically showing, the. Let's. See what are we showing this is calculations. Per second, per, 1000, dollars and you've seen charts like this all the time but, the point is that here. We are in. 2015. And, and, basically 40, years from now I've picked 40 years because I'm.

About To come up on 40 years in the industry and it's, been about 40, years since the transistor, came out a little. Bit longer than that but, I think many. People entering, their careers, now will. Be in the industry for at least forty next forty years so. Let's think through what's things what things are gonna be like 40 years from now well. The law of accelerating, returns is, basically says and not-too-distant. Future. You, know the your ability, for, to spend a thousand, dollars you'll, be able to get the compute, power of all, the current human brains on the earth four, thousand dollars okay so you. Know that, great technology, thing continues. To, concern, to go, so. What does that actually translate. To okay. So where, would would be to in 40 years well let's talk about where we are today today. You, know we have Planetary, cloud infrastructure, that's, really how I describe as your, AWS. To a lesser degree but, really, it is planetary. Scale data centers everywhere, throughout the world, there. Were over 2.5, billion, cell, phones per, user there, are over six connected. Devices. Per. User in 2020. That's the expectation, we. Have smart homes we. Have smart power grids we, have smart chemical. Processing. Plants, and the. Trends, what are the trends we see trends around you, know information delivery. Information. And education. Delivery control. Over the physical world with things like IOT. Biological. Control systems Scott Hanselman superstar, Scott Hanselman has. An artificial, pancreas that, is controlled, through open-source. Software how, crazy is that so, literally, his life me get your pancreas wrong that's, like, Clifford life-or-death, stuff, so, literally software's, controlling, his, life. Crazy. As that sounds true, but, think of where we're gonna be in 40 years okay, now attention. Management. Surveillance. Professional. Cybercrime, nation state these are the cyber, attacks these are the trends. Let's. Go over that list again. Biological. Control systems. Surveillance. Systems, national. Nation-states, cyber attacks cyber crimes, control. Over the physical world right and think, of where this is gonna go over 40. Years when, $1,000. Gives you the ability to have the compute power of all the human brains in the, world okay so kind, of a big deal so. Really. This talk, came, out of that thinking right, doing, that analysis. And saying hey you, know what's going on here and the question was this if, software. Eats the world where, did that software, come from and. The answer is engineers. I. What. Could possibly, go wrong. I. Think. We know the answer. Okay. So. So. Really, it was this hypothesis. If. We. You. You. And, you and you and me if we are. The people building the fabric, of the future we. Need to step back and ask ourselves what, kind, of world do we want to build because.

Software Is going to define the world right. And software. Makes. Things happen, it. Allows, a set, of things to happen and it, prohibits, a set of things from acting happening. Okay. So we you, know software, embodies, policy. Software. Embodies, values, and we need to get that in focus and start to ask ourselves some hard questions, they. Might say well you. Know, hey. I'm. Just I just cold I'm told to write they somebody else makes that decisions, that's above, my pay grade, no. No. You. Are responsible. For both your actions, and the. Actions and, consequences of, your, code, your product, your services, you, are individually, responsible the. Fact that you're a member of team does, not diminish your moral, responsibility, for your participation, in something, so, you really do need to think through what it is that happening, and make, moral, decisions right. And you, have a voice look, there. Was a reddit, thread recently. Said what's the craziest. Slash immoralist. Immoral. Slash illegal, thing you've, ever done at work and there, was one of this crazy, stories, crazy, stories and there was this one and he said well the CEO, wanted. To like, hack into our customers accounts, to, find all this information and, they. Couldn't find anybody who, would do it and I was the young engineer, and so I did it for him ok so that was interesting and so I thought about that I said well ok well, wow. You, did that that's no no way no but. What he said was the. CEO couldn't. Get other, senior, people to do it and so. He found he came to me and basically intimidate, the. Senior people wouldn't, do it you have a voice in what you do you, know if you're being asked to do something that you feel is wrong you, don't have to do it okay. Now that has consequences of course but, also think about this you, really do have far more power, than you appreciate, right, you, have your choice of. Projects. And employers. And you. Get it you should decide whether, a company's, business aligns, with your values, when. You decide. To do it I will tell you this is stuff we think about as managers. And stuff right, like oh we want to do this will, people actually do that I mean we think about that and sometimes the answer is no they don't want to do that and. How, we do Microsoft's, obviously very keen on being, clear about our moral standing. You, know what our business, model is how, we make, money look, I'm very proud of that our business model I think is incredibly, noble our. Business model is not to take, you and your attention, and sell you to some advertisers. Our, model, is to, make you, and your companies. Successful. By achieving, more like. If you don't succeed we, don't succeed I think that's very noble and I think that it helps us attract. The best talent so. As you go and you decide what project, I'm am I gonna work for what companies I am I gonna work for think. Through is that something that you're gonna be proud, of does that you know Sasha says it's so well that guy's so good he, said, work. Is too important, you, invest too much of your life force for, it not to have meaning, and he said I don't even like that framing he said I don't like you thinking about working, for, Microsoft what. I want you to do is I want you to think about how you, can. Leverage this incredible. Platform we call Microsoft, to, bring about the change you want to see in the world, wow. What a crazy thought, it's great anyway you have far more power than you realize okay. So that's the why. I hopefully make the case for why we need to become better engineers. But really you know what's the issues is there really a problem here, and I'll, sort yeah there are some real issues we need to deal with right because. The future providing both, challenges. And opportunities. Now, there, has been this progression, of techno ecology right we, have had industrial. Revolutions, before, many, people believe and I do believe, that we are in the process of a fourth, Industrial. Revolution this, is pretty huge stuff right first, one was all steam power. Mechanical. Production the second one was, electricity. Third. One was the microprocessors, the.

Fourth One that we're in now is one, where data analytics, IOT. The cloud mobile, devices. Artificial. Intelligence, intelligent, cloud intelligent, edge this, really does bring about a different, model, of. Economics. A different class, of products, etc. Robotics. Genomics, crazy. Great stuff now each, one of these things was, precipitated, there, was a there was an invention a technology, which, allowed this industrial, revolution. To come about the, first one was the steam engine, the, second one electricity, the. Third the microprocessor. And it really is this intelligent. Cloud intelligent. Edge that, is precipitating. This new industrial, revolution. Well. Here's. The thing happy. Days powered well isn't it great. Datacenter. Is always blue. Calming. Ok so the, god years that technology. Disrupts, these, revolutions. They're called revolutions, for a reason they, disrupt, everything. Okay, now, stir. Up things there are opportunities, and there are threats they disrupt everything, there, will, be or, there is a risk of job, displacement, often. The jobs change, there's. Issues around community, safety, issues. Around income, inequality. Unequal. Access there's, a wide, list of things these, tend to be the ones that are getting a lot of airplay these, days so. Let's talk about that. Indeed. There are fantastic opportunities. Right. Someone, did an analysis, of what's, gonna happen to GDP, when, we really get this AI stuff, kicked in and and, going and so, this chart here shows technologies. Like AI can, increase, productivity. Absolutely. True and they, had this hypothesis, I don't know how they did this but it's it doesn't seem too far off basically. What, the baseline is in. Terms of GDP. Growth and, what, you can do from a mature state of artificial. Intelligence, and you see pretty large increases. Okay, but, there's also a challenge, that, challenges job displacement, someone. Did an analysis, of the job tasks. And said, hey these, types, of jobs from loan officers, receptionist's. And information. Assistants paralegals, etc. A lot of those job tasks. Can be replaced, through automation, so. As as Ford said hey, you, got to have somebody to buy the cars if you have a fully automated plant and you're not paying your workers anything, they, there are no people that have the money to buy your products, so, job displacement is a big issue right it's not just about you, know the. Billionaires, having the most money if they if there are no people to buy the products, that people don't have the money to buy the products, the engine, doesn't flow so this is an issue. Then. There, are opportunities, right millions. Of people have been lifted out of poverty, this is an incredible. Story it is one of the unsung great. Optimistic. Threads. Of the last 20 years the. Number of people that have been brought out of abject misery. Right, so as populations, of living, in poverty, is really just plummeting. But. Income. Inequality. Is, absolutely. Growing there's no doubt about that particularly, in the United States right so, this plots out the, percentage. Share of global wealth the. Top, one. Percent versus. The top 99, percent, and these, things lead to social. Unrest, in fact that's an interesting piece of history when you go look at it is, whenever we've had these technological. Revolutions, each, one of the technological, revolutions, was, in fact associated. With, a. Societal. Unrest. Okay. Large. Mild societal, unrest you might have heard of the Luddites idle. Luddite somebody who fears technology, that. Was a group, of people reacting. To the steam, the first industrial, Lucian's steam. Engines were taking their jobs and they went and attacked the machines okay, these things are real, now. Opportunities. Access. To the Internet. As. Increased. But. A corresponding. To that has, come cyber attacks and cyber crimes, the, ones that one specialist, said let's, see if I said it this way okay, you have the ability to reach, out to anyone, in the world and, you know crack their information steal. Their money in a, way that has little. Chance of being caught and if you are detected. Little, chance of prosecution. Says. What. Criminal, in their right mind wouldn't be getting into that business and, it turns out they, are okay. So. We see an increase in both cyber attacks, and cyber, crime it's, not just the. Criminals. Themselves it, is industrial, espionage. Etc. Bad. Stuff, now. As. I, said technology. Doesn't, just transform, business, transforms. Society, and has. New, challenges. Okay so in the past we. Had a group of engineers, who guess what they did they, move fast and broke things ever, heard that phrase. These. Industrial. Engineers. Who move fast and broke things killed. People, okay, steam engines, we're not safe to begin with there were lots of accidents there very dangerous things and a lot of people died and out, of that came. A. Set. Of reforms so.

It, Doesn't just transform, business, it transforms, society, and thus, raises, new questions okay. So, when we build software we. Need to stop just thinking about feature, function, right, so yes we. Do things for the sake of Technology am, I using the right tools for this am i reinventing, the wheel but. You also want to think about you know of course the business right is this the right business, strategy, is this, something that when people understand, what we're doing we're, embarrassed. And lose. Customers, or when. They understand, what we're doing they're, proud, they're. Glad. To be associated, with and they want to increase their business so, you want to think through that because guess, what, nothing, you do stays hidden everything. That you do gets exposed, at some point and that but you also want to think through what, is it, what. Are we doing for, society, because, it's not just and we one of the most evil things that ever really came about in my opinion in our society, was, this new, and, this is new it is a new notion that, came out of the Chicago School, of Economics that. Said that, the primary in fact the only responsibility. Of a corporation, is to, maximize, shareholder value. Now. A lot of people are into the mistaken notion that that's like a law or, something let, me be super clear, on this it, is, absolutely. Not a law and. It's, a terrible, terrible idea. Right, that the only goal of a corporation, is to maximize, shareholder value. Corporations. Are stakeholders. In a society there are lots of stakeholders in a corporation, yes. Shareholders, are very, very very important, stakeholder, the, employees, are stakeholders, well the, community those businesses, they exist, in our stakeholders. The. Society, they serve their customers their sake holders, and so, when we do things we need to think about all the stakeholders and, make sure everybody's, getting a good deal you know I worked for a great IBM executive, once and he said. Anybody. In my business I want to be super clear about this if you're doing a deal with somebody and they're. Not winning. Like, if they have the other partner, like we're winning but they don't win so, I don't want you to get that business, what. Really he says yeah he says if everybody. Doesn't win it's, not stable, the. Business the deal won't land they'll, figure a way out of that contract, so, you need. To take responsibility when. We enter a contract, with a partner, that they win. That it's a good, deal for them wow. What a brilliant guy so, the same thing was we develop products we got to play the long game yes, let's do a great product yes, let's make sure it makes a lot of money for the company and yes. Let's make sure it advances. All the stakeholders in that. We serve, our, employees. Our customers, our society. Now. A lot of talks been talked about oh well you increased diversity, and at, first it's like you know you know that sounds like a good idea I like that idea that sounds noble, etc. But, it turns out it's awesome also great for business, right, it's, been particularly round accessibility. So, think through here's a number, of use, cases right, so imagine, well how would you help deal. With people with physical disabilities say, someone, that only had one arm okay, what you'd say well you know we do this this and this and they you think but you don't can. We just be honest here how, many people are there in the world that only have one arm I mean yeah we want to do good by them but you know really and you might say, yeah, never mind but, now think about it well yes, you should do it for trying. To help everyone, but, there are a lot more people, who have arm injuries, right, you know that number of friends recently had had problems with their rotator cuff and they can't really give. These slings and helps. Those people or, situational. Environments, a mother, holding a child right. That. User interface that you built to help someone that only had one, arm can, help situational. Environments, like someone holding something, same. Thing happens with sight like, okay there's so many blind people and n number of blind people in the world but, guess what there's this huge bulge, of, baby boomers, who are about to get really old and have bad vision and. Have cataracts, right, the user interface, that you do to help the blind will, also help a huge, market. Segment. Called, old people, I'm. One of them are about to be one of them I want you to be kind to people like me. But. Also the young people, when. They're driving they, can't be distracted, so, the point is that a user interface that's good for the blind person can, also help distract. A driver or drivers, right because they can't be looking at something they gotta have their eyes on the road okay, so point.

I'm Making here is that, there. Was not a conflict, between being. A good member of society and doing. Things to be a good member of society and doing things to increase your your, business that, those two things go hand in hand they really do. Yeah. So software. Basically the net here is software, is too important, to forget about ethics. And if, as a profession. We misbehave. We will be controlled, and you, see that right how many people of these tech executives, are getting hauled up before Congress. Because. They didn't think about the consequences, of their actions, then think it through did, a set of things and now, they're up in front of Congress, changes. Will be made full, stop changes. Will be made the, companies will make those changes or the society will make those changes for them but, changes, will be made full, stop okay, so, as a, guiding, principle this. Turns, out to be great right when, you're doing something and you're in doubt. Optimize. And focus on solutions that, amplify, human dignity now. This sounds like a platitude, really. And it, is but. It turns out it's it works and it's right you, know I worked at digital, and digital digital. Equipment corporation Dec, and we. Had, a culture, there and it, was summed up in with single sentence, do. The right thing, do. The right thing and, when I first got there I think what a stupid, thing to say, do the right thing versus, why do the wrong thing what. Do you mean and, it turned out that there's, incredible, wisdom in this, simple, banal, statement. Do the right thing and what, it basically said was digital. Believed in the, moral, judgment of us as individuals, and empowered. Us to make, moral, choices. For. The business and. The. Number of times we get into a tough situation and they're, like huh and so, and say let's. Just do the right thing here and it resolved, the problem you know what you're right the right thing to do is this and we're gonna take a hit and we're gonna delay the schedule, or we're gonna take a hit we're gonna lose some money but, it's the right thing to do and, that simple, statement do the right thing had incredible, power, and and, and serve the company very well if you go study the search search and study the history of digital it was one of the most progressive companies.

In, The tech industry so. So, I encourage you to either do, the right thing or embrace, this one and I think you'll be surprised at how well this, phrase. Will serve you when. Endowed, focusing. On solutions that, amplify, human, dignity. Now. AI. Optimism. Lots. Of optimism, about AI let's. See if this works. I've. Got audio on the PC. Innovation. It's not, just a word it's an action, with, artificial, intelligence, we are not crawling or walking or. Running we are flying, today. Microsoft, AI, helps. An architect, bring history back, to life he, doesn't see data he, sees fragments. Of our past this, is now. Artificial. Intelligence, helps farmers grow more food with less resources. She's. Not collecting, the information, she's feeding them so. Here's the deal we're community, when I'm screwing up you got to tell me. Yeah. Oh sorry I gotta, pay attention. He's. Never, seen my PowerShell, unplugged talk I say PowerShell. Such an awesome tool because I'm a deeply flawed human, and, a lot of people think, it's, just making, up no. Innovation. It's. Not, just a word it's an action, with artificial, intelligence. We are not crawling or walking or. Running we are flying, today, Microsoft, AI, helps. An architect, bring history back, to life he, doesn't see data he, sees fragments. Of our past this, is now. Artificial. Intelligence, dubs farmers, grow more food with less resources. She's. Not collecting, information she's. Feeding a growing population, without. Wrecking. The planet this. Is real and, engineer. Explores, how I could, help the Deaf see, sound. She's. Not looking at obstacles, she's staring down opportunity. Innovation. Doesn't see the possibility. Of tomorrow. It, creates, tomorrow, and, are, you ready for the headline, tomorrow. Is, here, it's a day. All. Right amazing, stuff and it's true. And. It's true and. This, is true as well. Hey. You. Call uber a few. Months ago had a terrible, incident where, their AI they, had a driverless, vehicle.

Autonomous, Vehicle went, and crashed, and and killed someone so the. Optimism. Around AI is. Absolutely. Warranted, absolutely. Worth it it is fantastic. It can do amazing things, but. It, can also cause harm. We. Are not without our, own problems, in the space right, I mean we had Tay. Right within a matter of hours we, put an artificial. Chatbot, up there and within a few hours turned, into this horrible, racist. Nazi, bigot. I mean. Can. You just imagine how, such I mean just such a humanist, such a and then he's, like the public face of this horrible person oh it's, terrible anyway, so this, is not I want, what I'm trying to point out here is this is not a Boober bad problem, right. AI, is a problem with AI it is both a opportunity. And it. Is a challenge, so, one of the issues here. Let's. Get. To it in a second okay, so when. We build AI there. Is an emerging. Understanding, that. This, thing is a very. Very very sharp, knife and you. Need to treat that sharp knife appropriately. You need to put a handle, on the knife all. Right because if you have a sharp knife with no handle, like, it can still cut vegetables but it's gonna have, some consequences as, well right. So you gotta put a handle on that knife so, there's things around data set biases. Association. Biases, you AI, can, have biases, and it's important that we treat AI. Understand. This problem. And deal with it appropriately, for. Instance, facial. Recognition, turns, out facial recognition works, very good for. Me. Right. White. Guys works. Great. People. Have diverse diverse. People, people don't look like me not so good why is that, the. Technology, no because. When we were building the models for facial recognition they. Asked a bunch of people hey would you recognize would you do this then, guys like me went and and did it right, and they, didn't, recognize that hey my. Dataset is not, representative. Of. Everyone. And. So that's why it's tuned for for, people like me and so, this is the engine right the engines not gonna sit there and say hey I'm getting the wrong data it doesn't know anything it, only knows the data that you feed it okay, so, you need to realize, that this, case and you need to design your learning, model appropriate. Pick the right set of populations, so. That you get equal. A I to get out the data David, get out the biases, so imagine. Association, bias imagine, you have a set of pictures, and you, label. What. The people are and what their roles are and you train this data from, the 1950s. The 1960s the. 1970s and. 1980s and. Then, you see a picture of a. Woman as you, can say nurse. I say, doctor, a guy, say, doctor it's not gonna say nurse and you, look at that and you say well that's not right, you know we don't want to have that and you just need to be aware of these things the AI engine, let's let's put it this way I like. To think of the AI engine, as I don't like the term ai ai. Sort, of seems smart. Right, I think a better term is, psychopathic. A, psychopathic. Intelligence. No. Really psychopathic, intelligence, it turns, out that there's a group of people humans. That. You know every one of us in general have, moral.

Empathy, We, have empathy, but. There's a class of people that, don't have any empathy and those, people are called psychopaths. Okay, now Psychopaths not, necessarily. Milord they're not necessarily bad, not all psychopaths, are psycho killers, right in, fact, in CEOs, there's. A higher. Percentage, of psychopaths, it's as CEOs, than in the general population so it can be quite positive. You, know often a CEO. Managing. Through difficult, times you know lack of human empathy you may be, viewed as a as a positive thing or a useful thing I'm not entirely sure but. It's, not necessarily, bad it can be good well. AI are, a I engines, it is, definitionally. Psychopathic. It has, no empathy whatsoever. None. It, doesn't understand, a human it has, no empathy for a human it has no goodwill it has no ill-will, it, is, psychopathic. Now. That doesn't mean it can't be useful but, you need to understand this, thing could be a psychopath and therefore. I'm, gonna trust it with these things but I'm. Because it's a psychopath. Okay so, that's when you hear AI I think you were better served by thinking. Psychopathic. Intelligence. Versus, artificial, intelligence, because artificial, intelligence, just sort of seems smart, and anything, that's smart, you associate. With people. Who. Also you have a strong association of, being, people. Of goodwill people. With empathy people, with the moral judgment they, wouldn't do that you, know they're smart they wouldn't do that this, is smart and it, would do that because it has no moral empathy okay, so. No. I'm not saying the next CEO of Microsoft will be an AI I'm, innocent people have argued that a is well. We're. Not talking about Microsoft's, point of view on AI because, we have a very distinct. Point, of view on AI and I think it's the right one okay. So as I say AI, will cause harm, right the glitches will it happen as we advanced the technology these. Applications, are amoral and and. Know here's the problem here's the problem I sat down I thought about this is like wait a second wait a second let's think through the model for AI I have. A data set I take. This data set and I feed it to machine learning, machine. Learning goes through this statistical, analysis, blah blah blah produced this model, and then. That model gets, applied against, it in input. A set of data and decisions. Or judgments are made okay, now, this, AI this, data set there's. An initial data set and then you learn it and that. Model well, that's not static, it, gets done you update it generally. Frequently, and then, the data set comes in and you've got flaws, in the data set cameras. They've got our skewers. With dirt on them etc and, then, you make judgments, okay, any. Of these things can go very bad and at some point now somebody. Dies a, judgment. Was made an action, was taken somebody dies who's. To blame who's. To blame, right. Because each of those things was probably done by a different group of people right. The, provided. By a technology, company that data said often there are public datasets augmented. By private, datasets I've augmented by community. Datasets then, there are the input, sensors, the, data itself the cleaning, up of that data you, know you always at the clean data and then, the action so who is done and I assert, that you're not gonna be able to answer that question that it is, metaphysically. Impossible. To, answer that question because when you actually take a look at the AI engines, and how they work look. You, go talk to the AI researchers. They, will tell you we. Have no idea why it's doing it I mean it's working it's great we have no idea why the decisions. Are made why they're made it's just statistical. Decisions. Being made okay, now. The concern, here, is that this is going to cause backlash. And, that. In in you know in an age of liability, that this backlash, is gonna cause even more harm so, let's be concrete with an example.

40, Years from now right, so 40 years now a thousand, dollars buys you all humans, compute. Power today I, assert. 40 years from now autonomous. Vehicles, will, kill, people. .. I'm absolutely certain of that i could be wrong but i'm absolutely certain. So. Forty years are now a ton of C equals will kill people if, today, but, I will also say that when. Everyone's, using autonomous, vehicles forty years from now the, number, of people that die from car crashes will, be dramatically. Lower, than, it is today so. People will still die but, the number of people will be dramatically, lower if, today. We. Hold as a bar. You, cannot pursue, autonomous, vehicles, unless, we know with metaphysical. Certitude no one is going to die and. Therefore. Delay. Pursuing. Autonomous, vehicles more, people will die as a result, so. How do we deal with that problem okay, and I, assert that. We. Have a way forward and that, these two things the, underwriter, labs and the, US vaccine, courts, provide models, for how to deal with these moral. Quand. Ruiz so. First underwrite of labs I mentioned, to you these industrial, revolutions, during the electrical. Industrial, Revolution, young. Engineer, went to the United States are the Chicago, World's Fair where. It was the key thing was electrification. They. Were gonna Electrify, things lights were gonna be on everywhere wonder. Of the world and, he, looked at this and said oh my god people are gonna die here, he was just horrified, by the practices. Because. Electrification. Was, not, an engineering, art, right it was a bunch of people as a works works, craft a craftsman, thing. And wasn't even craftsmanship, people were just throwing stuff together and so, this young engineer, started the underwriters, labs then, the underwriters, labs was, about getting, people together coming, up with electrical standards, training. Standards. Certifications. Etc. Now, how, many of you have bought an electrical, device in the last year. All. Of you okay now how, many of you when, you're considering electrical. Vais a versus, electrical, device, D did. Some research to find out which one had a higher likelihood of electrocuting. You. None. None. Why. Because. Of the underwriters, labs you. Do not get to sell an electrical, component in America, unless it has an under lighters lab, certificate. It, is a set of tests, is a set of guidelines it has said as skills. For the people producing these things need. To hear to and that's, why you don't, think about it when you buy an electrical, device you know that it is safe okay. Underwriters. Labs really, is in the business of, engineering. Trust. Engineering. Trust and because, you trusted, that product, a and product, B would, not electrocute. Me electrocute. You you, then focused, in on other things and, then just bought them without any hesitation, we. Need the same thing with artificial, intelligence and technology. Now. There's, also the National Vaccine Court, okay in the, 1980s. There. Were a number of lawsuits filed against vaccine, manufacturers. Because people were harmed and the. Awards. Were very, very high the. Result of this is that the vaccine manufacturers, all but one of them exited. The, vaccine, business, said you know what I don't make any money on this we're, doing it as a service, now, we just got whacked we're not we're not in we're out and, the, United States Congress looked at this and viewed it as a national, security threat right. The, herd immunity of the nation was gonna was that threat and could, be devastating, and so, what they did was they set up the, the. National. Childhood vaccine, injury, Act and, what this did was it established a, court, of law that if. A vaccine, manufacturer. Produced, a vaccine. According. To best practices. Then. If, and when a caused a harm occurred. It, would be they, could not be sued, that. The claim had to be brought to the National vaccine, Court the National vaccine court had a different, standard of evidence and it, was very heavily, tilted toward, the plaintiffs. But. There, was a maximum, award, his. Cap of am action award and so, what this did was look, a lot of people will tell you that there is no harm, whatsoever with, vaccines, but, the reality, is this reality, is is that biological.

Systems Are very, very complex, and we understand, them we don't understand, them very well and so even something that might be fine, for ten. Million people with. One person it might cause harm, and so, the, again saying. Well that, person, can then sue. You. Know the. Company did everything according to best practices, ten million a hundred million people are just fine using this product but, one person had it and had a severe reaction and, they. Sue and we're able to have high rewards, and put that business out of out, of put, that company out of business that, does not serve, society. So. Society said we recognize that there's inherent risk, in here and we, need to move forward because a societal, benefit, of this technology vaccines. Is so, great that we're. Gonna help the people for which there's a problem and we're, gonna let the businesses, move forward and, think this is the right model for, artificial. Intelligence because AI will, cause harm, now. The trick then is to say what, are those best practices and, therein, lies a challenge. Now. I want to shift gears and talk a little bit about what Microsoft, is doing I think Microsoft is is leading, the way on a number, of these issues and. So I like to talk about a few of them one, a general, program that it won't get into the details and then one that I find, particularly, important. So. Starts. With this the. Mission. Statement, right, Bill Gates said. The. Microsoft, mission was to put a computer, on every, desk and, in, every home running. Microsoft, software this. Mission statement changed, the world may. Bill Gates one of the richest guys in the world and, is, responsible, for a lot of you sitting in the audience today, okay. But. We, have a new mission statement. By. Way notice. This mission statement is what. It's. Technology, focused. And, it's. Microsoft, focused. This. Is our new mission statement to, empower. Every. Person, and every, organization on. The. Planet to, achieve more. This. Mission statement is you. Focused, and its, achievement, focused, I love. This mission statement, okay. So. We. Have initiated we have rolled, out something, we call a cloud, for global, good we, recognize, that this new industrial era, of intelligent. Cloud enabled, by the intelligent, cloud intelligent, edge has, intrinsic. Risks, associated with it great, opportunities. But, great risks, as well and we, have on, our own signed, up for a set of initiatives and this, is the one where we're on now we continue to look for new ways to, meet our. Responsibilities. As a stakeholder, of society, and we things like the trusted, cloud so. We are the ones leading, the way on protecting, personal, privacy when. The US government came, to us and said give, us this information, we, said no, you'd, not have wheat, Byway let me first start off with we. Respect, the legal right of the United States government, and legal, jurisdictions. To, do proper. They have well-defined, responsibilities. And well-defined ways to. Summon. Information. From us we. Felt that the United States in court, the, United States was asking, for information that, exceeded, their responsibilities. And we took them to court that went all the way to the Supreme, Court we, fight, to, protect your, data and that resulted. In the cloud Act changed. Legislation. To address, this. We. Took, the initiative to start a, conversation. Around how do, societies.

And Companies. Deal with something you know deal, with cyber, attacks and cyber warfare, and, we, set up a conversation, we call they trying. To create a digital kanaeva, Geneva. Convention. Set, up the rules of the road look there is a, legitimate. Place for. Nation-state. Espionage. Might. Not like it but it's gonna happen and it's legitimate, but. There are need to be rules associated, with that and so we want to establish what, those rules are and get nations to comply, we, have a responsible. Cloud right we are a zero, footprint. A zero mission committed. To zero emission clouds. Are, a carbon neutral clouds I forget the term of art but basically we try and get as much software, energy. As we can from carbon neutral sources, we, buy solar. Power credits, etc, we, have an inclusive cloud, focus, in on people with disabilities and. I, want to drill into this one providing. Affordable and, ubiquitous, broadband. Access. Okay. So one of the challenges, if you live in America you realize that there is a. There. Is societal. Turmoil. Going on and as, I mentioned whenever there's a technology revolution, there's often societal, turmoil, many. People, and I subscribe, to this point of view say, that one. One, of the, root causes of, this is the. Disconnect, between, the, people doing well, and the people who feel like they have been left behind, sometimes. These are called the, flyover. States is, reality that people aren't either into the coast are doing far better than the people in the middle have and there's. A lot of resentment and that has led to political. Turmoil. Well. Microsoft, has taken initiative. To. Help. Address some, of the digital access because we believe that, access, to, digital technology. Can, help lift all boats but. That doesn't help if you don't have access to it so, today. 34. Million Americans, don't have access to broadband. Twenty. Three point four million. People reside in rural communities and, here's. A map of broadband. Access by. County, and you see, towards the middle no. Bueno. Okay. So. We have a very clear goal and, I think we're about a year a year and a half into this we, want to close that rural. Broadband. Gap within five years so notice. Very. Ambitious goal, and a, time frame. And. We call this the rural, airband, initiative. Where, it's at. Excuse. Me I'm right mentioned the flawed thing. There. Is somewhere, oh, good. Lord I. Got. A video. That's. A good. Just, because you live in a rural area doesn't mean you shouldn't have the opportunity, to be connected. I. Was. About I see, ninth grade year is, when I began, taking college, classes through dual enrollment, and that. Requires, that you upload papers and assignments to. The college, as well as to the high school I was. Like the internet was terrible, at home so I have to do, my summits really, early like a whole month early he had to submit, a lot of his paperwork it's just gonna sit down just spin and spin and spin you. Know that the deadlines come in and you know you need more research you know it's not enough it's. A struggle. In. The beginning, of the semester one of the questions I asked the student said just. A show of hands who who, has internet access and. Typically. Just. A few. Right, there that that tells me that I have, to be a little limited, in any, kind of homework assignment, I give, it's. A handicap, for teachers and in, a 21st century and, here we are hands. Are kind of tied. The. Homework gap is a major problem for our region, and the exciting. Part is that Charlotte, and Halifax County are two pilot areas, are no longer going to be left behind in this digital divide. TV. Right space is a new technology, that the, FCC, has allowed vendors, to transmit. Broadband connections, wirelessly, over previously, unused airwaves, rural. Markets specifically, it is critical. To have access, to that lower, beam spectrum, that allows internet, providers to serve that last mile, affordable.

The. Goal is by fall to have about 250, connected, and we're hoping to have a thousand, families connected, soon after that it's. Gonna open, the doors to, economic development and, allow the, people in our area, to. Have access, to a world-class education. It. Was a dramatic increase, in, productiveness. And efficiency. Dylan, has a small antenna at his home which is connected, to a TV white space device turns, on his laptop connects. Via Wi-Fi I, don't know it takes that all I know is it works. Now. That I have TV y-space internet I can use, the cloud and my, greed actually, shot up going, forward with TV white space I got high, bees and AIDS, this. Is where I grew up and I love it I'm a country girl just. Train our kids so they can stay home and have that knowledge and then they can bring it back here so that our community, can grow tvy. Space held up very well with my college application. Once. I graduate, I will be going to attend. Old, Dominion University in, the fall studying. Computer. Science. All. Right so hopefully I made the case for why you, know what some of the issues are what. Microsoft's, doing now, the hard, part, what. Do we do about it I think it's very easy to sit there and say you know we, must change we, must do a better job but. And hopefully. Gave you some ideas. Of what Microsoft's, doing but you might say well yeah, it's easy you know Brad Smith president of companies such an Adela CEO, of the company they can do things like that what can I do what, can I do is that person what can I do as a project. Honestly. Much harder but. I have a way forward and let. Me be clear it's, a way forward it's not a solution, so. Number. One, gotta. Acknowledge, we have demolished that this is an evolving dilemma, versus, a problem I want to be clear this is not a problem. Problems. Have solutions. There. Is no problem here is the dilemma it must be managed. We. Have responsibility, for the moral actions, we've got to acknowledge that we have more, responsibilities. For ourselves and, our actions and we, really need to shift this focus. From, just, shareholders. To, stakeholders. And. Then lean, in right. Educate. Yourself on the issues there's a lot of conversations. A lot of resources, these days talking, about this problem so that's great and. Then participate, in these things get educated, lean in but. Then do, what, we do we're, group of people with the set of skills how. Did they. Say it in taken we, use. Them yes. At, lifetime acquiring, a certain set of particular set, of skills. Our skills, our engineering. Skills, so, how do we solve problems we, engineer. Them away or, how do we solve how do we solve, dilemmas, how do we address dilemmas, we had engineer, them away so. Basically, incremental. ISM right don't, try and just solve. This problem it can't be solved, it can, be improved, so, incrementally, improve yourself, your team your product and then, build systems. Which, help you find, and address the problems, build. Systems. That help you find an address, the problems, that's the heart of engineering, Robert. Gates Robert, Gates wrote in his book Robert Gates as defense secretary for. The Bush administration. He, was asked by President Obama to stay on and be his defense secretary and, when, all the secretaries. Got together about hey was, in this grave what are we gonna do they, asked him mr., Gates would you please tell us you, know what things you know have any advice for us, he, was very sobering, he said listen. At. This very moment one. Or more people in each of my colleagues departments. Were doing something that was illegal, or, improper. And. He told them the key was to have mechanisms to, find such people before, they did much harm, so. Notice it was not about hey, make sure this doesn't happen, it. Was, acknowledged. That it will happen and then. Put, a mechanism. In place to, find the people before. It gets a problem before. Does, much harm so, it's not you. You want to have focus in on building, a system to, detect and resolve the, issue versus. Pretend. It's not going to happen and the other great point about this was you, know don't, try, and hide it when. You discover it it's, not you, know the, worst thing you can do is deny it or try, and hide it have a mechanism deal, with it in a no-drama, way, engineer. Your way out of this now, the heart of all this is. Process. Engineering. Well, guess what we're sort of on the path to get process, engineering, already. And that's called DevOps. So. Let's take a little bit of a divergence, and I'm going to talk about DevOps, if you're not already for me but we've shown, hands how many people are familiar with DevOps, okay. So everybody in that case I'm gonna go pretty, quickly through this because you probably know most of it but, I'm gonna talk about how. DevOps. Can help be, the solution, or be a foundation. Or provide, a framework, that. We can then attach, some. Of the problem-solving that we need to do for. These issues, okay so last twenty years you had development.

Yet, The wall you. Tossed coat over the wall to operations, operations. Failed. They tossed the bugs back, this was the chasm. Of despair, I hate. That. DevOps, is really about merging, these two teams, so, that they jointly work on the entire lifecycle, and, it's all about these learning, loops learning. Loops learning. Loops now, here is where the process engineering, comes through because. The heart of DevOps really, I thought was encapsulated by superstar. Sam Guggenheim, er if you haven't seen his toxie's, he gives some great talks, cuz he's such a smart guy and he talked about this thing called the law of thirds, and, basically, it was anytime, you have a hypothesis. Right you have some issue some. Metric, and you want that metric to go up or you want to go down well, how do you do that and, the answer is you have a hypothesis. I bet, you if we do this it will go up and, so we'll. Have thirds you execute, that hypothesis. And one of three things are gonna happen the, metric goes up yay, the, metric goes down boo, or, the metric doesn't stay so. If it goes up yay. Keep doing that if it goes down undo. That thing and if it stays the same yeah. So. We said was. What. You want to do is you want to have very small you've heard this before, DevOps. Is all about doing work in small batches very, frequently, because, each small batch is associated, with a hypothesis. And if the hypothesis. Is wrong you, don't want to invest a whole lot of time in it you want to throw, it away quickly, and if it's right you want the gain, to happen quickly so, it's all about being, able to do things quickly to you, leverage this law of thirds, ok. By. The way why, do you want to do this because, failures. There's two types of failures have you heard this one two, types of failures anybody okay okay two types of failures hey. Devops, you hear a lot of you, need to get comfortable with failure, anybody. Hear comfortable, with failure. Usually. There's a couple yeah usually a couple I'm not so comfortable failure, some. People love to succeed, other people hate to fail I'm always hate to fail people but, what I found out one, way to get comfortable with failure, is to, do the old architects. Trick I like to say that architecture, is the art. Of deciding. When two things should, be won and one, thing should be - okay. True. Turns, out failure. Is not, one thing it's, two things, failure. Number one, you. Come to me and you say hey Jeffrey you. Know that thing I've been working on for the last two years yeah. I said I failed. Okay. You. Come to me you say hey Jeffrey you know that thing I've been working on for the last two hours yeah. It, failed Oh, for. You I'm gonna say great, let's grab some lunch tell me what you learned, you. Gonna, say what's. Your next act right. So, there's two types of failures. And. The way you get comfortable with failures, is to make them small and learn if. You, spend a couple years and then fail that's, that's, sort of unrecoverable, so that's why you want to do things if these have this hypothesis.

And Fail. Fast right. Because then it's a small failure okay, so really. DevOps is about process engineering. Process, engineer oddly, enough a little little trivia fact I started, my career as, a process. Engineer so, this is sort of in my wheelhouse as, a process, engineer it's, not that complex, if you're, if you learn. Science. Guess what you're a process, engineer because. What is process engineering, we call it the scientific. Method, scientific. Method one define. A set of repeatable, steps and measure, the outcomes, we, should call these experiments. Right you do an experiment you don't change the steps between, experiments. You run the things exactly, the same way clean, your test tubes run this make sure you get the temperature right then, vary this what, did you see, do it again exactly. The same and, vary that what did you see okay, so repeatability. Absolutely. Clear then. Make, those steps smaller, now for DevOps make, those steps smaller, and smaller and do them faster, and faster. And. Automate. The steps degree, to which there are humans involved in these steps that agrees to which you get variability. What, you're going after is consistency. And speed, when, you have a human do a step takes, longer, and they. Might vary things you want to be consistent, and, then what you do now here's the key then you do you love the thirds you're, doing something in a repeatable way that's fast and you're, measuring the outcome, you, change one of the steps and you see what happens to the outcome if you like the change, keep, doing it if you don't like it undo the step okay, so, you modify, the steps, to, optimize, your outcomes process, engineering, 101 pretty, straightforward, now, here's where this, comes to help us. When. You have new problems, you, produce new measurements, and new. Modified, steps. Okay. New and modified steps so. Basically you, know oh we're seeing this in the field oh that, means that there's something that we didn't measure let's, measure that thing etc so. These issues around, social, responsibility. Justice, AI. Equal. Opportunity. Job. Replacement, they are they're, just additional, measurements, right and so, what, you want to do is like. If anybody, here produces, commercial. Products, in. Your engineering process, you have a step that says wait. Stop. Before. We release this we got to do a patent, review because. Once you release it then, your ability to file a patent, is greatly. Diminished, so there are times where you have to just stop in your process, evaluate. Whether you have anything you want to patent and go patent, it before it gets released and so, what I'm asserting, is in the, same way that you have a process and, there's. Some process for, features and there are other processes, for concerns. Concerns. Like patents, you, inject, in in the same way, concerns. For things like security. Accessibility. Ethical. Issues, privacy, social, justice, okay. So hey we're about to release this hey do we get someone. That doesn't have as physically, impaired someone. That's blind someone, that doesn't have hearing some that doesn't have great mobility to. Try out this product no, oh well. Let's do that okay. So it's pretty straightforward. Have the process, injected, in the process, measure. The outcome, etc. Let me give you a personal. Microsoft. Case. Yeah. We've been pretty public about this okay, so, I. Will tell you that Microsoft. Has a diversity, problem. I. Will. Tell you that everyone. I can tell be totally, honest about this everyone. Had. Believed, that they were being completely fair, in their, head in their heart right we completely, fair we don't bias towards men we don't bias towards white guys here everyone's, completely fair but. When, we stopped and looked at the data it. Was clear that something was wrong that. Something was wrong. And so we stepped back and said hey this, is a problem that we need to address. And. So when we analyzed well what's actually going on here so first we, were doing something and without, any measurements, we thought everything was fine but, we were getting some signals like hey you. Got some problems here and so then we measured, that and, we measured it and looked at ourselves and, said hey you know what we got some issues that we need to address as, we drilled into it what we realized was as we, evaluate leaders. We. Value, a certain, set of leadership. Characteristics, that. Were narrow in fact.

Leaders, Come in lots of different types, and we, were evaluating, the leaders that looked like Bill. Gates and Steve Ballmer, right, sort, of butt heads red. Drive, drive drive you, know really you know roll over people get the job done people. You. Know lots, of damage in the in the in the way in the in the wake but. There are different types of leaders and we realized hey we are not adequately. Valuing. A diversity, of leaders, and so, it turns out that when you change those things, the. Scope of who gets. Promoted gets, changed, and so that's what we've been doing and we've, then been measuring it right, and we've, been measuring it both in our promotions, and are recruiting. Etc, and we're making improvements but they're not enough so we continue, to do that so, that's an example of a. Stakeholder. Issue that. We addressed, through process, engineering, right, so one of things we deal with in, in terms of process engineering, is hey, when you're hiring these. Job. Positions, there, has to be someone. From a diverse pool that, participates, in it and does not mean that you have to hire that person but, you have to consider that and then, that's helped us like, oh yeah and, has. Increased. Our, diversity. Numbers we got a lot more to go but I thought that was a great example of this process. Engineering. You. Made aware of an issue. Translate. That into a measurement. Look. At the process, tweak. The process, measure, something, still, not there tweak, the process again measure. It iterate. That's, how I think we're gonna address some of these big hard problems. Microsoft. Has, what we call the ethical, AI program. Here. You know this is led from the very top right Sachin. Adela Brad Smith you, know we need to be thoughtful about how we address the societal, issues. That. AI brings. About. We've. Been very clear about our position, on, AI. It's. Different than other people, we, believe that AI, is, all about augmenting. Human, ingenuity. Ai. Helps. People, and organizations. Accomplish, more does. Not replace people that's not our vision our vision is not to use AI to.

Get You to buy more soap, detergent I it's, to help you. Augmenting. To achieve. I'm. Sure that makes any sense anyway so we want to be clear about accountability. Transparency, we. Pay attention to fairness. Reliability. Safety. Privacy. Security inclusiveness. Boy. Sounds, like a great slide right but, what do you do with that like, really what do you do with that right, like, yay saluted. But what do you actually do and we, translated, this into a set of actions, right, you gotta translate, it into a set of actions or the world doesn't change right. We, said we wanted to be diverse that, doesn't change anything what change is something is when you change your actions. Change your. Actions, we measure something we change your processes, then you get changes and results, so being saluting, the flag on, we should be good AI citizens, isn't enough so, what we said was hey we're gonna have a framework for this what, we've done is we've created a pool of experts. In ethical, AI and, right. Now we think this will probably evolve over time but right now the model, is when, you find yourself with these situations, go. Seek help of professionals, by the way it turns out we do the same thing with security like, we do not work we have great security, we do not require everyone. To be great at security, what, we do is we educate. Them about what, the security, issues are and when. They. Encounter, something when, they should bring in an expert and then they bring in the expert and the expert helps, them solve that issue. And. So, then our, guidance to everybody has learn our principles understand. This right, align, yourself, with our, moral. Position, on AI, ask. Yourself, these three questions about, when you need help and go. Seek help when you you, need that, there's. A great example the. Industries, now there's, lots of people engaged in this conversation this. Is one that I've recently become, aware of I think they've got a very nice framework for, dealing with the issues they. Call it the ethical, OS, and, it's ethical, OS orga, encourage you to go check out their framework I think it's particularly, good but. How. Do I move forward whenever. You want to make a change, whenever. You want to make a change, whether it's a change in an engineering. Or a change in personal, habits. Anything what you find is often, like I should do X I shoul

2018-10-04 11:15

Other news