This. Is IOT. 204. Connected. Robots IOT, in the warehouse, Paul Cap'n Eddie I'm an IOT solutions engineer, with Google hi, I'm Christopher Cacioppo, I'm the CTO and, one of the cofounders of 6 river systems, 9. Joseph Hughes director of DevOps at 6 river systems. We'll. Be talking about collaborative, robotics. And how it helps companies meet the ever-changing needs, of today's consumers, so. Quick. Agenda I'm, gonna take some time initially, to try and frame some of the problems, that warehouses, are trying to solve today I'll, also go through through, some of the cloud solutions, that help I'll. Then hand over to Chris Cacioppo, here for, a. Double. Click on the business case for collaborative, robotics, and a little more information about six river systems, and then. Joe will finish up with all of the tech behind six river systems, and the products, that they offer. So. There. Is a Dory for today's session and I'm. Told if you go to the app and go to our, session IOT, 204, you, can see a link down below Dory is an internal, tool built at Google that's being made available at next you, can go here to post your questions for us while we speak you, can also see the questions that others are posting, you. Can upvote and download, those questions, as you see fit, the. Idea is at the end of the converse at the end of the presentation. We'll have a bit of a stack ranking, of what the audience wants to hear from us we're, gonna aim to add and. 6, minutes 7 minutes early, to try and get, as many of these questions answered as possible, and afterward. We'll be able to answer questions questions, offline. So, a. Quick, survey so I know who's in the audience today how many people out there by show of hands are developers, architects. People, that are building systems, you. Have quite, a bit that's great there's, definitely, content, for you how. Many people are in the warehouse space, working with warehouse, management systems, show, of hands, got. Good. We got a bunch how. About robotics. Working. With Ross, great. There's, a bunch of those too how, many people have ever bought a physical, product on the Internet. All. Right good, captive. Audience so in, all seriousness this, is the audience, that we built the talk for hopefully you find some of this useful. So. What. Are the needs of the warehouse today and as an IOT developer, that's been working on a lot of different connected, products working. With a lot of different IOT cloud solutions, the, first thing that strikes me is the sheer amount of data available. So. There's. Products. There's product locations, there's lidar, points, there's all sorts of things that. Without. The technology, you had and now adding robots you have things like batteries, robot statuses, many. Many many different types of data and as a, person, that's managing. A warehouse I want to know that there's a way of obtaining, this, data there's a way of reacting, to it formatting. It and. Google. Cloud has you covered if you're. Able to connect, your edge devices, and your sensors to cloud IOT, core you. Can quickly, securely.
And Easily get that data to, the cloud. From. IOT, core you. Get your data into pub/sub. Pub/sub, can act as a kind, of central point in your cloud architecture, from there you'd, be able to trigger, cloud functions, to react to your data or, data. Flows data, procs to be able to try. Form your data into an optimal, format, and then, ultimately sync, it to bigquery, and I, say bigquery, there's a lot of different data, persistence, mechanisms, available on data cloud but bigquery, I believe, is uniquely, well-suited to the problems, warehouses, are faced with today. Bigquery. Is Google's. Server. List data warehouse in the cloud and it's built to excel with analytics, workloads. So. Let's. Say you have your, data, publishing. To the cloud and you have it formatted. You have it synced to bigquery you're. Quickly faced with the next daunting, task, there's. So. Many things that you could potentially change in, your warehouse to try and optimize, your pick rates or find. Bottlenecks, or just, overall. Get, your throughput up, you. Could potentially, be spending, man. Months on a part of your system that even if done purple, perfectly. Would, only shave a small amount of time off your average pick rate where there, may be a bottleneck elsewhere, that, if you've spent a couple man-hours, you, would unlock. A whole bunch more throughput. The. Quickest way to get to this insight, is since. Your data is in bigquery you. Can hook it up to data studio or many. Of the other business intelligence, suites out there to, visualize, your data and, understand. Where the time is being spent. Armed. With these visualizations, you'd. Know that spending, man. Months trying to make your user interface on your robot more effective, for the picker might. Not be as impactful as maybe. Increasing. The speed of your robot, or your object avoidance for your your. Slotting, of your products, because move time Dwarfs, the amount of time that it takes to pick a product out of a box. So, there's. A certain level of insight you could get to with dashboards, but. To potentially, move beyond you, can try, and employ a machine learning. Machine. Learning might uncover, different. Patterns in the data that might not be readily apparent with. A dashboard. Or visualization. It. Also may. Help. In not only leading you to the opportunities, for improvement, but be the improvement, itself, you, can potentially, create models who can deploy on, a robot, and have, that robot be more effective, in, its day to day job. One. Of the best things about Google cloud and machine, learning on Google cloud is you, don't need an army of data scientists, or tensorflow. Developers, to be able to start to take advantage of it, since. Your data's in bigquery you, can use bigquery auto ml to write a sequel statement, that's a returns. Your pick times and all of the different facets, that might, affect that pick time and then. With two extra statement, two extra lines of code you can generate a tensor, flow model, that forecast, snap pick time when, you change some of the other parameters, around it. And just like that you. Might know the, change, the. Effect, of the change that you intend to make under your system you. Can take it same. Data and bigquery and return. Robot. Failures, and come up with a model that gives you predictive, maintenance you. Can use none of that data and use, a pre-canned model that would give a robot the ability, to see whether an. Obstacle in its way is human or not and for. Those of you that aren't, in the robotics space this.
Is Valuable because a robot moving, very, fast in close proximity to, a human, would, make that human very uncomfortable. Very. Understandably, so but. If. A robot can't deduce whether or not a human. Obstacle, is human it would have to slow down for every road cone every, piece, of box. Or anything, in its way and that, could dramatically. Impact. It in the pic race. So, let's, say you have your data it's, in the cloud it's in bigquery you, have your visualizations and, you started to pick things that you want to change about your system to, optimize it now. You're faced with another problem because, to, be able to get the data you need to. Validate those changes, are in fact having a positive impact, on your system, could, be costly you, can't change. The size of your warehouse very easily or scale. Up your robot. Fleet for one day two, three four or five X it's. Pretty, cost prohibitive, to do things like that. So. This. Is where Google, kubernetes, engine can help. You. Can spool up simulated, robots and simulated, warehouses, and fuel, bigquery, with. The same types of data you would get if those robots, were real, that. Then fuels. The same dashboards. That you used to, pick what optimizations. Who wanted to take. So. This, is how you can quickly validate, your change and iterate, on to the next. Now. Before i hand over to chris i wanted to quickly introduce six, river systems, six, river was founded, on the idea that the, technological advantages. That are seen in the the, large million, square-foot warehouses. By the e-commerce, giants, should be available to all so. Not only should these robotic systems, scale up very well they, should be able to scale down. Traditionally. To get a fleet of robots, in five, hundred thousand. Square-foot warehouse you're. Talking about six, months, to twelve, months you're. Talking about many, many years before the ROI pans, out, these. Types of upfront costs, don't work for the 50,000, square-foot warehouse. So. Six. River has a system, that can be installed and up and running in weeks and, fluctuate. The size of your fleet for peak times and, get, ROI a lot quicker, what. They're trying to do is in effect democratize, robotics, and level, the playing field with, that. I'll. Hand it over to Chris thank, you Paul, so. Why did we decide to use collaborative robotics, to solve a warehouse problem, and at. The end of day I wanna give you a quick idea of the problem space and then we'll talk about that. Up. Until about ten years ago. Distribution. Centers were places where manufacturers. Bought a lot, of cases, of products they were then separated out and then shipped off to stores. To then fill their socks so there was pretty much moving cases, from place to place which.
Is Reasonably. Easy to automate there nice touch. And square and, it doesn't require a ton of labor but. In the last ten years things. Have changed. When. You guys order, things online you never, want to case a toothbrushes, you want a toothbrush, and a toothpaste maybe, an iPad and some diapers right they're all differ for everybody and so it requires a little break me open cases taking, things apart dealing with odd shaped items and so. The warehouse does look more like this. So. There, are people working on automatic, hands, and arms and you've probably seen picking challenges, and they always took off nice clean pick faces but, it's clearly. For us at least five or ten years off before they have robotic arms and go in take things apart pull. Things together. Gift, wrap all that south requires laborious there's a tremendous labor impact, for. The foreseeable, future in this area. To. Make matters worse when. I was a kid I. It. Was my birthday my mom will you order something the Sears catalog and we've mail it in and it would take six to eight weeks to show up and I would have forgotten what it was but I'm sure was excited to get it anyway, well. In the 90s early 2000s. Shipping. Time for about two weeks he ordered something online for early e-commerce they show up in two weeks everyone, was happy it was great, but. Today's day and age people, expect. Things entry-level, as two day shipping then. It's like one day and like, it'd be nice to be sure there's an afternoon and so the pressure that puts on to fulfill centers to get things out the door is tremendous. So like. Many things as latency. Goes down the efficiencies, get, worse and so there's. Even more pressure on labor these. Fulfillment, centers and distribution centers are located in, places where land is inexpensive and before, there was didn't need a lot of labor but they've kind of sucked up all the available labor I was. With a customer about two weeks ago and they said it's not so much I care about the price of labor they'll, try to do its that I literally, can't get people to reliably, show up anymore I could get enough people to do so, what we've done is to solve this problem with, with, Chuck and Chuck. Do I come over here. So. Chuck is is dressed. Up in its Google best, Chuck. It's a collaborative robotic, robot. And. The. Thing I want to point out here is this, makes the the. Associates. Lives much easier but. Everyone. In our company is super excited about robots, and, I. Think most people in this talk would think robots are pretty cool some. Cameras out there so that's pretty cool but people that, are often associated working warehouses robots, can be scary for a number of reasons and so, one of the things we did is when, I started this I didn't want a little circle or rectangle or, square in a warehouse that wasn't. Obviously was gonna do so I put, on my in Dussel, an. Additional. Engineering hat and said okay we got a good designer and said I want, this to be obvious so this is Cleavon modeled after a car it's got Ted, headlights and taillights it, looks like a car and when you look at it you, know in here things could go that way it's gonna go that way and if it backs up which it does very rarely it's, gonna beep and look like a truck so we want to do is get people comfortable with the concept so. That they're comfortable if, people are not comfortable work with your collaborative robotics, they're not going to use it and nothing's gonna work well so, we've had very good luck and everyone loves using our robots. So, the. Way a typical warehouse. Works for car pick operations, an associate. Will, walk over to an inductor area with a big heavy cart and they'll put a whole bunch of boxes on it and get a whole bunch of paper tickets and walk, over to their pickin racks it, to look like Costco. Or a backing from Ikea or Sam's Club except much more dense they're then gonna walk up and down Tahoma tiles for filling the order so these paper tickets and finding everything where they remember and they're, gonna bring it over to a takeoff area or possibly.
Multiple Ones and there's, a big loop okay, at. Six River we decided what's most important, that the associates. Can do is that much better than anyone else it's the grass in the hand so we want to try to alleviate, as much, of the other stuff, to doing as possible, and these aren't small places as we talked about these are 500k. 50k, to 500k, or more warehouse is there tremendous, amount of movement. So. What our system, does is there's, a person who sits in an indepth area on top and they, put work on the robot and then, the robot takes off okay. The robot will then meet someone in the picking area they. Will then walk, with them through the area showing the pictures taken from place to place and the best method possible and then. Where they're done that. Robot will take off and deliver it to multiple places essentially, and another, what we waiting for them so we'll keep them localized, in doing, what they do best in the, picking aisles. What. We can do is improve, pick rates two to three x over traditional, car pick okay. And we do this by number ways we reduce the long walk some people to walk across giant warehouses we, have very effectively, pick orders that go together in a nice clean way so they walk as little as possible, we. Pick them awfully from place to place and we, also have, very clean you eyes which lets them get the data fast and they can pick faster they can lower error rates and. They can ruse training time from weeks to hours, and this. Has been hugely positive, our customers. The. Next thing about Chuck that's very interesting is it's a disruptive technology so, there's, a number of technologies, out there that. Allow car. Picking, that. Enhance. Carpet, or replace carpeting, so, but, most of these operations, have three to five years ROI and are. Bolted, in the floor and I don't know about anyone in this audience but I have, no idea my cup is going to be in three to five years like technology changed so fast it is a tremendous, investment to, do to. Make matters worse. There's. Generally. In this industry there's a huge. Seasonal, change so right around holiday, season most. Of our customers see a three to 7x change in their operation, right tremendous. Changes so not only do you have to predict. Potentially. Five or more years in the future but you also have to scale up so that your operation, is running pretty, much at peak, to peak time and then eighty percent of your time it's significantly. Under utilized. So. What we, allow it happen is we have a two to one year ROI for our products. They. Can instead of being installed a lot of opera most automation. Is six months to a year of installation time we, get up and running in one or two weeks okay, it, also allows you to scale your operation, if your operations, getting more and more going up and up you, just buy more robots we just deliver them they work as is if your, operations, feelin's scaling. Down well, if you've another DC or another fulfillment, center just move the robots, if, you. Have seasonality, just. Instead of building our system from six River that works for your peak for the entire year just build for your average for the air and we'll leash you more robots for your peak time so we allow a tremendous, amount of flexibility, that it's just not possible anywhere else in the tree.
So. From, the first day I sat down on the my, dining room table of my two other co-founders to. The day we had a small fleet of collaborative robots picking, live order to blues toys and. A customer warehouse was nine months which is looking. Back on it felts much longer but it's, an amazing feat and I don't think this is possible, ten years ago even with the the best group of people you could ask and the reason is I think there's three key technologies, for the last. 10. Years that have made a tremendous change. One. Is ubiquitous, computing power, so we've invested invidious. Jetsons, tx2 platform. Until. That gives a tremendous amount of processing power at very, low energy consumption and a reasonable pipelines and so, we've done a lot of work with them but it's not just that it's all the centers and all the things that allow robotics, to work have come down tremendously, in price it, wasn't that long ago that the. Spinning, lighters you see on top of the Google cars they, cost like 50 grams but, and now we can have one to the robot that cost less than a thousand, so there's a huge change of technology, that allows that's, enabling, things that just weren't possible for. The. Second one is open source so I'm sure everyone here uses it's. Gotten better and better over the last ten years I want, to call up Ross Ross, is Colt is a robot operating system for. Your non robot assists it, allows you to bootstrap, robots really fast so you can focus on the things that your business, cares about and not focusing, on the plumbing and infrastructure, so we could very quickly get a robot up and working and then, pile. More stuff on topic to optimize. It for our environments, and. Last. But not least is, ubiquitous cloud platforms, and. We've embraced Google cloud from day one. This. Allows you to like scale. The operations, leverage. Your engineers we have do. A lot of testing stuff we have a lot of security things it just allowed us to go so much faster than, we, would have otherwise, so. Given the audience I think what's been the rest of the time talking about the Google cloud. So. One thing that's very important to me personally is data-driven decisions, pretty. Much this entire conference. Is about data right, and so collecting data and everyone knows data is good but, how do you use data how do you make it effective for you, so. Let me go through some of the ways and we use it and. We have a lot of data both from a logistics side and also from the robot side if you have a workers robots robots, generate a tremendous amount of data and make lots of decisions and it's very hard to turn. Through all the data otherwise. The. First thing we do is, operational. Intelligence we. Have a ton, of data on all of our customers operation, and we can tell them how, much work they're doing how much time they have left the rates they do things a whole, bunch up we can say which which of Associates, imploring better and worse but, even, more than that instead, of saying Chris, is underperforming I could say Chris, seems we doing pretty well he's, just really slow at scanning, maybe he doesn't have that section works and to associate can come to me and help coach me through something that I'm weak act as opposed. To just, dealing with it or deciding they should get to my other employee if that's even possible. The. Next thing is visualizing, data this. Is one of my favorite things so this, up here is a picture of, a. Robot. Go, through the warehouse all the time they report back wireless.
Signal Strength. Constantly. So every second or so and so this is a map of the wireless signal strength at various ports the warehouse so, people, do wireless surveys, and they're sort of ad hoc and occasionally happen we, are constantly real-time. Showing, a wireless survey of the warehouse so you, can tell them this warehouse is a big area in the center that's probably pretty weak and if the customer cares they should probably update that. But. I think what's, more important is how we use it for our internal engineering, personally, so. When you're starting to build a robot you get a robot and, you get wave to you first you have to make it move, check, it moves and then. We don't just want to move we want to move from A to B so we want to move over there excellent. Check, we have to have a to be and then, you want it not to hit things all, right check and you go through this whole list of things whether it's robots or any product, you guys do until. You get an MVP and it's in front of the customer at. That. Point it's, great we get some feedback and we work on the most important things next. How. Do we choose the most important things in most. Operations, it's done at one of three ways one. The, manager, or whoever is in charge decides, to. The loudest engineer decides or three, if you're doing things probably correctly you, have a consensus, of engineers that decide together but. The problem is even in the even a latter case we have a consensus, it doesn't mean you're working on the right thing or most important thing first you just all agree you, could all agree on the wrong thing so. This. Is a graph up here about. The time that, the associates, are spending on our product which is important of, of labor management and labor savings, so, the big blue box here is movement. Says the, large amount of time about 44% is spent on the robot moving from point A to point B with, the associate in intohe. There. Are other things down here too just scanning right, and so. One. Of our interns may have a great idea we can just remove scanning, entirely and that sounds like a great idea we can just skip that step and be awesome let's work on that and someone else have an idea it says we can reduce movement 20%.
That, Doesn't sound is good but, if you look at the numbers it's. Way better to spend time reducing, move it by 20% and is removing, a stage even though it doesn't intuitively, feel like just removing something isn't just a better idea so. I'm. Gonna give an example that. This. Is another customer warehouse and instead of plotting, Wi-Fi. Signal, strength we're putting average movement, speed of the robots, so. You can see right a slower green. Is faster. So. If you look at some of this and analyze a little bit at the end of aisles things are slowing down it makes sense there's there's cross, traffic there's blind turn so you gotta be a little more careful around that but. They're slower is in the middle of the aisles but doesn't make a lot of sense right robots should be humming along they're to be fine. So. We can do two things here and we do both them one is we can push back into customers and say hey you guys should clean this up the robots move faster and your rates to go up but. Really. It's better of like why are we slowing down like, so because, everybody, wants a robot suit faster so I think obviously, the, next thing we do is just make the robots top speed faster, or. Is it. So. Silver engineers Chuck's over robots and put it in a test track and. Instead. Of just reporting the maximum, speed in robots they, report, the, internal, choices, of robots, and essentially, crash, course in robotics, here we have a bunch of different. Tasks, that determine how fast a robot can go and the. Slowest one wins because we believe in safety and everything else. So whatever chooses the slowest, and that's the one that we go at, so. If you look up here red, says I'm near an obstacle, yellow. Says, go top speed so, if you look up there and we increase our top speed, really. Wouldn't have had a giant impact on our robot right now because mostly, we're having around objects and we're slowing ourselves down because we're being extra careful.
So. The engineers took this data in tackling. That problem instead, of trying to make the robot respond faster to go faster. And, analyze. That problem and after a couple weeks they did the test track again and, this. Is what they got you'll, notice instead of being like 85%, red, it's, not like 40% red and there's a lot of it yellow which is the top speed the, robots just cruising along at top speed so, there, was a huge impact of diving into this problem and Ulta. About a 15% movement speed increase, and. Now that the interiors can look at this and say okay, should we keep, pushing on that problem should we look at that at the teal over here which is a pathing issue should, we think about raising top speed but, it allows a data-driven decision, when obviously - everyone says we should just make the robots move faster and that was actually the wrong decision, so. That's. My part of the talk I'm gonna hand off to Joe who makes all the magic happen here, thanks. Chris. So. You. Just got hired. At the next IOT robotics, company and, you. Wake up the next day you're feeling good but then you think to yourself I need. A robot infrastructure. Lucky. For, you you're here Google, cloud next. And we're gonna help you. Understand. How we've used Google cloud to address some of the most challenging aspects of this problem. The. First and most obvious problem, is you, have all this data you, have hundreds, of robots in the field you, have robots, that are in your test environments. And your customer, environments, you, have to get all this data and you have to make some sort of sense out of it. So. We're gonna talk about this slide for the next 30. Minutes all. Right you guys didn't run away we're not gonna do that but I think, here there's an important thing that I want to point out about this architecture, we don't just use cloud IOT, core for everything we also use Google Storage for a couple of things and. The reason we do that is that there's some data that is really important to have in real time but, there's other data that we may need later that. Is really, dense and and, Chris talked, a little bit about that data that's your, lidar scans, your point clouds the. Robot Isis if, you're not a roboticist that's really kind of what the robot. Is seeing and, vlogging. All of that and we need to put that into Google storage because those files are big they're dense. Then we need to run it and break it down so. We. Make, heavy, use of pub/sub, and cloud functions, to massage that data get it into the format that we can consume, some. Of that data comes all the way back through to, a, web. You know a web page where people can consume it some. Of that data goes to bigquery and that's. What drives some of those interesting, graphs that Chris showed you before. What. We're going to be doing next is using that data in bigquery as, as Paul alluded earlier to drive some machine learning models so we can get better and think more about the, data that we're collecting. But. This this infrastructure, for us this. Design has. Survived. The test of time and scaled, with us with no intervention, engineering. Intervention, at all. So. The so. One of the things that you want from this data is real-time location you, may think to yourself well what that seems important, but I'm gonna explain to you why it's very, important, if, you're managing a million, square-foot warehouse, and. You. Can't have, line-of-sight to everything so, you need a bird's eye view of what's going on you, have 25 trucks in the field you might have 50 employees, in the field and you, really need that real-time feedback. So. Most. Obviously you just need to know where Chuck's are if you can only see 10. Trucks at a time you need to know where the other 10 are and make sure they're doing the things that you expect that they should be doing in your warehouse. You. Also want to see if chucks in trouble you'll, see two trucks here blinking red if a. Warehouse. Manager, was to click on one of those they could see that there's. An obstacle in Chuck's way or that, there's some other unexpected. Event maybe some sort of maintenance, that needs to be done to chuck in, any case it, really calls out very, clearly, that there's a problem it needs to be resolved, they can get on a radio talk.
To Someone and ask them what the problem is or they, can make the decision to walk over but, if, you're constantly walking, around a million square foot, warehouse. Looking, for problems most. Likely you're not going to be able to find them. Lastly. You want to understand the operation so. If you see Chuck's queueing up in a weird way or you see that. Chucks, and your your operators, are not in the right part of the warehouse for where they need to be at that point of the day then, you can correct that and make sure that your, rates you, still hit your rates and that, means that everyone in the audience gets their shoes. On time or their toothbrushes, on time or whatever. You might be ordering off the internet. The. Next big thing that you want from all this data is real-time monitoring our, customers run, 24/7. The Internet's up 24/7, people are ordering 24/7. And you expect your stuff in two days so. We. Have a tech support team that also runs 24/7, and, when. Something comes in they need to really quickly respond now, for, our cloud software that's running in the cloud stackdriver. Was a clear choice out. Of the box gave us alerting, it gave us all the monitoring that we could possibly want and we. Didn't have to invest any engineering, time into instrumenting, which came with the platform. We. Use Google, core IOT, to feed data into influx DB for time series database and then also drive the dets dashboards, like you see behind me this. Gives us our CPU metrics. Our memory, metrics, and our temperature, metrics some, of these warehouses get up to over 100 degrees so the temperature, of their CPU becomes super. Important your fan fails and your, CPUs overheating, you're, gonna have a very unhappy robot, and eventually, you'll have a very unhappy customer. But. This really all drives preventative, maintenance if we, can take a chuck out of service before it fails and clogs. Up the works then, we can save our customer, time, they may be running a degraded, system, for a little bit but hopefully, we can pull the chuck out of service replace. Apart, from a spare part kit and get, Chuck back in and doing, its job. Lastly. When, we touched on this little earlier but enabling, technical. Support super, important they're, really the front line of defense and they need tools to help, the customer resolve their problems quickly and not, interrupt engineering, if, engineers, are spending all their time solving. Customer, issues in the field then. They're, not going to be able to build the next great feature and make Chuck better and we're. Constantly doing, the experiments, like Chris showed you earlier to make Chuck faster, and to continually, raise our customers rates through, just software, and better engineering, practices. So. This is the second, type of data that we, were talking about in that architecture, slide this, is data that's not really valuable in real time from every single Chuck because. It's very dense and it's very it. Gives you the whole picture of what Chuck sees in the world but. You, may want this data. Because. You want to replay Chuck travel you, get a something, from the field and say hey, Chuck, is moving strangely, around this shelf.
Or Chuck. Can't see this, particular, thing, on the ground or something, of that nature and that comes to our engineering, team and we. Need to be able to see. That and so, we have two options we, can get on a plane we, can fly, over the warehouse go. Take a look at Chuck and say oh well. I guess I can hook in and take a look at the sensors or maybe. It's not even doing that anymore by the time you get there but planes. Are expensive, and engineering time is also expensive or. You can replay this truck Chuck, travel from Google Cloud you pull that down and you, can replay in real time so, what you're seeing on the right of this is you can actually see Chuck seeing other chucks on, the left you can see shelfs that's, what's called a point cloud you can see it's made up of a lot of points from. This our, engineers, can deduce a lot of interesting, things there's. Your. Test environment what, you have in your warehouse where you do all your tests and you try really hard to make Chuck move the right, way all the time inevitably. Though you, get out into the field and there's things you didn't expect. These. You, know we call the movement issues they're, caused by some interesting, things some. Customers shine their floors so. Or they have skylights and that feeds back into the sensors and makes, this point cloud much, different than it does look, at our, headquarters. On, our test track. Help. Improve efficiency, Chris. Kind of alluded to this earlier but. If there's a specific specific. Place in the warehouse where the Chuck's just always move slow sometimes. What we can do is clearly just see that, the. Wide R or the camera is seeing something that it shouldn't and with, a little bit of adjustment by the customer or a little bit of adjustment in the, map that we use to navigate we, can alleviate, those problems. So. You, have all your data but. Now you have a hundred a thousand, a million robots in the fields and you, need to upgrade them at some point, so. How do you do that. Traditional. Software management, ansible. Saltstack. Puppet. They're. Great tools for managing servers, you have an always great. Internet connection, and you, always and you and, you can kind of rely on either a connection to you, know make, things good for you. The. Great white whale of Chuck is reliable, Wi-Fi, warehouses. Just, don't have reliable Wi-Fi there's, a lot of racks there's a lot of interference and so what we end up with is half applied States so. Almost, worse than an upgrade just failing is a half upgrade, an upgrade, that looks like it went good but it didn't go all the way and we, battled a lot of these problems early. On so. We, said to ourself there. Has to be a better way you have to be able to reduce the complexity, of these things we, have to be able to be. Be, able to minimize than the amount of data that's being pulled down and do it more deterministic. So. We. Shoehorned. Everything into docker docker. Containers, are pretty, great at solving this problem if you have a container it either downloads, or it doesn't if, you need to roll back your roll back to a different container you, have a very good view of the world of. Where. Your software, is updating, and so, if you need to update a thousand, of these things you. All you need is a, good internet connection to, download a single file, versus. What, you would have to do in the other situation which, is a bit, of a back-and-forth between a lot of different services. We. Really leverage Google, cloud platform to, make this possible and, don't. Get scared it's another architecture, slide but we're gonna go for it quickly. Google. Has to go a lot of great tools for building containers, cloud builds, scales with our developers, as they need to make containers, and we're rushing to finish releases, then, you, know we can have 20 developers doing 20 different builds and Google Cloud builder scales up we, push all those to the container registry and. Then. Then. We have our custom, solution, to this problem but. Really, Google has, a great, toolkit with kubernetes engine that you can run software, that. That. Just consumes, a lot of this stuff, and finally.
This Software allows us to push things out through IOT, core IOT. Core can't just collect data but it can actually manage, the configuration, of the robots so, essentially the message goes out to one. A thousand, a million robots saying, we, need you to download this container from container registry and then, they go into action, so. I'm gonna show you how that works. Here's. An actual upgrade. So. Let's just sped up a little bit but these containers, download the. The, robot gets the command gives us the does. It the signal, that it downloaded and then we, can apply that upgrade, and so. You'll see we do three at a time here they. Download, and they apply and so we're taking three trucks out of service we're downloading a container that container gets applied they go back into service but. We know exactly if it if something didn't work if it did work and the, nice thing is going, from build, X to build Y it's. Just as easy as going from build wide back to build X because, I think. All of us know that sometimes software, upgrades don't go well so almost. As important, as your upgrade path is your downgrade path but. With docker. Kubernetes. To, drive this to, drive this software and IOT, core we, can really manage. This problem very well. So. We. Have, all. The data we have. Thousands. Of robots and. Now. Our, customer, comes and says to us hey we. Think we need to rearrange our entire warehouse we, have 100 trucks and we. Think that this design might work better and you're. Like ok well. So we're back into kind of a problem here so. We can either fly out to the warehouse bring. Another 100 trucks. Get. Another warehouse set it up the way they want test and make sure that it works or. We have to do something else and I, think we all kind of realized we, pretty much just have to do something else. So. Going. Back to our docker. Adventure. Here we. We. Were really pleased when we got to this point when we said okay we already have everything in docker so, all, we really need to do is change, a few things in this container and we'll, actually have a docker container, that can be run in a simulated, world. So. We have the robot operating, system running completely inside. Of the docker container. All. We do is simulate the, sensor data and so, what's, really powerful about this is that, the sensor data coming in is going through the same pipeline that, it would go through in the real world so. As we make adjustments, we can be very confident. That those adjustments are going to behave the way that we expect them to in the real world and. That. Really just gives us that extra bit here where things. Are things, when we go to deploy them behave, they're the way that we want. So. Why would we want to test with hundreds of robots the warehouse layout change is one thing, but. Another thing is algorithm, changes you, know we have a lot of logic. And algorithms, as Chris alluded to earlier that, batch, orders, and to, try to gain more efficiencies, in the warehouse and so, when we change those algorithms, it's, not really, great for us to test them out on a customer because. You can imagine what a customer's gonna call us and say they're. Gonna call us and say hey Joe. That software upgrade you gave me now my rates just fell by 50% and. That's. Not a really great conversation to have and that's not what we want to do for our customers and so. With, this sort of infrastructure, we, can prove algorithm, changes before we deploy them and we can confidently tell a customer when, you take this release you're gonna get X number of X, percent of improvement, you're Pickers will be this much more efficient your warehouse will be able to get this many orders out the door and be. Very confident, that's an accurate estimate and.
Lastly. And I think which is kind of an interesting way to use the system is that we have cloud systems, as well and we, can do synthetic load, testing in those cloud systems and throw you, know fubar data at it or we. Can run real, robots, again real simulated, robots against that system and do load tests for. Warehouses. That are even impractical to build in the physical world so, we can really put, our other software, and the cloud through the paces, using. A very, large deployment. Of simulated, chucks. So. Scaling. With kubernetes engine is super. Easy auto, scaling works great, and that, is really important, for this problem you don't want to have to call up DevOps. Engineer and say hey I need a cluster with 100 simulated. Chucks you just want to tell your software, I need 107, simulated, chucks and then minutes later you have a hundred simulated, chucks. Stackdriver. Monitoring. Is built in and so, you can see your, simulated, chucks and see trends in their CPU, usage, their, memory, usage and see. All of that and feed it into bigquery as Paul alluded earlier and make sure that your dashboards, perform the way that they should and that you're seeing the algorithm, improvements, that you should. You. Can scale up and down quickly scaling. Up is super quick scaling. Down is also super quick because this is expensive, resource. And you don't want to be spending money with, a simulation, cluster, that someone forgot to shut down, for. Like a weekend, and your, Google, Cloud bill all of a sudden is a little bit higher than you expect it. So. Here's a real simulation this. Is a, pretend. Warehouse in the cloud we, have pretend, robots running around doing, robot, things they're picking orders they're inducting, orders they're, even they even go to charge they. Even have simulated batteries, so, because charging, can actually, take out of the efficiency, of the system so, we simulate all of that and we. Can put all, the things I talked about into this and. So. Very. Very, flexible system. Another. Thing that happens is that. When, you start simulating things if. You spawn 20 robots, in the corner of a warehouse they kind of form a mosh pit and they. Don't really do very well and so, these are some of the fun things that you start to discover when you build. A simulation platform is like oh wait a second they all can't just spawn in one place they, need to, actually. Spawn at randoms points and we need to kind of figure out how we scale this up and down but, yeah that's a fun little video of a robot, mosh pit if you ever wanted to see one. So. The, last thing we need to talk about is. Integrating. With your customers, you, didn't just build this robotic, IOT, platform for, yourself you built it because you wanted to bring it to the world and you want people to use it and companies, to use it and so.
You Have to kind, of discover. Who your customers, are discover. What kind of technology, they have and and you have to integrate with it. So. In traditional in, warehouse, man warehouse, management systems, are the systems that handle, all the orders handle, all the inventory they, kind of determine, like what that day's picking, is going to be and they. Give us our work and so, we have to innovate with them. Our. Customers, I have, older, systems, there, in the logistics, business, they're not in the IT or, the tech business and, so when they find a solution that works for them they, stay with that solution and and. We want we don't want to go into our customers and say well. We wish we could work with you but, you have to update all of your IT systems, to adhere. To you know rest, standards, or into our api's, we, have to do something different. So. We adapt, artists, our rest api's to our customers, technology. They. Use socket. Communication with, fixed with messages. Are. You soap. They, use SFTP, I mean, they use everything. And so, that means we have to build everything and we have to adapt our api's to everything and. So that's a big challenge but, luckily, when you're using Google cloud platform you, have a lot of tools, at your fingertips to help you with that challenge. We. Reach for cloud functions, for the simplest of integrations. And. Kubernetes. Also, makes super, quick with, things like hey native and other tools that are coming to the kubernetes platform, well we can build these little micro services, like their, only purpose in life is to translate. Our API messages, so that our customers can understand, them there's. A lot of challenges still, with, but, with these tools, without. These tools we wouldn't be able to do what we do today at the, pace and at the scale we do it today. Google. Kubernetes engine on prim is the next great thing that six our river systems is really looking at and, some. Of our customers, have, such. Tight regulation, you can think government, and those. Type of things where they just don't have the option to have anything, outside of their four walls and so. With, these type of deployments. We're. Really excited to experiment. And to use Google.
Cooper Nazy on Prem to solve these problems. So. You know these these regulation, and concerns are definitely, one part of it another. Part it's of like bringing the sensitive sent latency, sensitive things, that need to happen to, the edge and the. Edge being the, customers, own, infrastructure. And so, if we have to validate something every single time we scan it and we, can't afford the, trip up to our cloud back, to their data center back, to chuck then, some. System. Like this makes things a lot easier for, the, customer and for us to, get that integration done and to. Really have a, great customer, experience. From. Our point of view it's. So. Great, that we can have the. Same deployment, tooling that manages, all of our normal cloud deployments. These on-prem deployments, as well and so. That can reduce, our overhead for you know DevOps, tooling, and for, and to keep the. Interface, that we use for our software engineers and for our tech, our. Tech support engineers the same.
2019-04-12