Programming your home automation with the Google Assistant (Behind the Actions, Ep. 3)

Hey. I'm natto developer, advocate on the actions on Google team and I'm here today for one more behind, the actions episode and today we have a new guest hey they know how you doing I'm doing well how are you and that, is my teammate so what are you doing our team so so I'm a developer advocate here at Google specifically. On the Google assistant, and actions on Google, team where. I work a lot on various, smart home in IOT related, aspects so you can see he has cool suit he do is a cool industry, so different kidding so okay, but what is this weird. Stuff here what what they're gonna present us today absolutely, so this is a Raspberry, Pi it's, actually, a Google aiy, vision, kit, where, we, will be modifying. This, off-the-shelf. Device, to. Enable, Google, assistant, controlling, where. You can give it various smart. Washing, commands like start my start, my cycle. Run cycle turn, it on and off things, like that so if for example I'm, running behind I'm, running late. To get to catch my bus and I forgot to start my my. Washer machine I could send some comment and leave it from my shadow say hey forgot, start my laundry, is exactly. Exactly you know what part of the reason why I chose smart. Washing machine today is because it's, it's a little bit more complicated than just a smart light bulb right we still have the on/off you would have with a light bulb but you also have things like modes, and various other traits that, that, you get exposed to when. Developing, a smart washer yeah but much, more it's. A good fit for our behind, the actual right I'm gonna show some deep, dive on this DC structure, but yeah before we start, can you show us how hard work absolutely. So I, have, this already set up with my account I have it set up with my Google account all, my everything has already been deployed and. So what I'm gonna do is actually talk to Google assistant, hey. Google, turn. On my washer. Well. You can see here the led okay great so it blue means it's on now if I start it it'll actually start, start flashing various. Colors hey, Google, start. My washer. Starting. The washer so now it's like the washing, cycles running right exactly, so so that's that's a various. Cycle. That you can start on a washing machine now there's also other toggles. There's another type of trait, that, you can have on a smart device okay, now, if I turn that on hey, Google turn. On turbo for my washer. And. So, now turbos. On on the washer and and. So that's a very städel, that you can control through, voice as well and. And. Ultimately you can turn it off as well turn. Off my washer. Oh, cool. So. What's. Happening here is a Google assistant, is, handling, all the fingers head calling. An API at. Some point and sending, the comment so, here right exactly. Okay it's a very high-level. Overview. Can you explain some details how these these all these stuff. Is going back and forth and working definitely, so at. A high level it's, you can think of it as cloud. To cloud communication. Okay so the, first element is just me the user saying, a command turn, on my washer. Google. Assistant, takes that audio, transcribes. It into text and then, applies natural, language understanding and. Ultimately. Outputs, a, structured. JSON, response to. Your. Cloud service, cool, so think of like Philip's, you for example right if you buy a Philips you light bulb that, will, have its own application. Its own cloud service, that you can use to control those lights and you have to log if you're on account to make all the process to assure that you are not turning of the air neighborhood, lamp so something like that right exactly and so in this, case Google, assistant. Communicates. With, their cloud service right so if you're turning on and off a Philips you light bulb Google, assistant, communicates. With Philips you okay, so, see my that we have for the, regular. Action, what you call the web hook we, can manage that the same high. Level tetris so i have web, hook where, google will call this way we hope to send the comments right exactly, right exactly right and so what. I've developed here today. Is my. Own cloud service that accepts. These structured, JSON responses, and. Then, communicates. Those, tie that information. Down to the device okay, in this case I'm using firebase. As. My. Transport. Down to the device and this is looking at various database, states that. Is subscribed to okay, cool, so basically. What you need is like each you set up your web. Hook your environment, your cloud. Service. Or whatever you are using and, enable. These to receive comments, through JSON, from the Google home right for the Google systems and, there. Is one word you said like, home, graph yeah.

What Is home graph what does it mean yeah so the home graph is a, very critical element, with. Smart home okay it houses all, of the contextual information about, you your devices. In your home okay so think, of you know you might have five. Or six different smart light bulbs in your house you might have a smart washer you might have a fan, you might have microwave, microwave or, or thermometer, somewhere right and thermostat. All. Of, that information is stored in the home graph so, when you say a command, turn. On all the lights in my kitchen, Google. Assistant, is querying, the home graph to, get a list of all the, lights that are actually in your kitchen, right if. You say for example, dim, the lights just a little bit right it's going to query and see what the current state of the of the dimming is and then, dim it just a little bit and this is important, because for example at my house I have me. And my wife so and, I for example I I was. The only one that configured. Elides and set it up on my on my assistant, but, she's also example, to turn on and turn off because since. She belongs to the same home in, my configuration she, has access to the same home graph right, as. Long as you share the, UK share of course you have to authorize right. Exactly, exactly and you know it's important to note too this is a very, critical aspect is that a, smart. Home action, that uses, the home graph is distinctly. Different from a conversational, action yeah this is my next question, and and, so with a conversational. Action you, have to provide various training phrases and you can create any custom intent, that, you want for, your an agent you don't need to do this that's exactly right so for smart home we have a fixed set of intense. Their-their. Which include things like, on sync, on query. On execute. And on disconnect okay these, are the various commands, that you can give, to a cloud service to. Do various, things with devices, when, the commands you said was turbo, right and, it's. Because you're washing your washer. Supports, the turbo mode if. I said change, another program, I don't need to train my. Map, my asthma. Action my my home automation action should know all my different modes not because you understand, the all the phrase and you send the parameters, that then you handle these parameters in your backhand right that's exactly, right so when that's. One of the beautiful aspects, of smart home is that you. As the developer, define. The capabilities. Of the device right. So if we take the example of a smart light bulb right most. Smart light bulbs that I know you can turn on and off right some, you, condemn and others. You, can also change the color to color yeah right so these are three distinct, different capabilities of, a. Light bulb now how do you define that so, that Google assistant, can control those capabilities we. Have a thing called device. Traits device. We have a huge. Collection. Of different device traits that you can use and. To, define the capabilities, of your let's say okay, nice nice, so okay you, said developer, as a developer, so I love developer, I'm excited, why don't you build something to, my, home brew system and. What. Are the requirements what what, what, what do I need to start developing, for, for, the for home, automation of with Google systems yeah so my. Personal, favorite. Route to go is our, smart, home code. Lab that we have online um, the. Biggest thing that you need is just a computer, you, don't even have to have a physical device to, get started you, can actually do it all virtually, through the code lab if you, want you can definitely have a raspberry, hi or any other type of device school this boards are cool exactly, and and, my, suggestion, is to take a look at the code lab the, code lab walks you through every, individual. Step of creating your, first smart. Device that you control through assistant okay so, go let's, start holding, this and actually see how the parts, fit together and, how the. Codes organize it yeah let's let's show the code people are shaking. Without so.

Many Time without code let's see perfect, so the. Way I've set this up today is, using. Firebase. Cloud functions, cool so I have here, in, and, my my, code here, various. Functions, that, I've, deployed. That. We use throughout. The. Controlling, aspect of this, IOT, device okay, and you are using the no J's client library right see exactly have we already have a client lab that supports, all the interaction. Of the home automation right yes so we have a actions, on Google client library, for nodejs, you. Can see here i'm also i also have imported, firebase, functions, and, you, can see also that i'm using firebase, admin for my persistence, layer to. Store the various states of my device okay cool and so. The, biggest things that we need when, developing. For smart home as, we stated earlier are those for fixed, intents the on sync on query. On execute. And on disconnect okay and so what, i have here is the on sync what, this does is it, provides, a, list, of all, the devices for. A particular, user on your service, it's like google querying for the status, of your, of. Your, stuff exactly, you're connected stuff right and so when, you initially. Go in and say you just bought, a, new light bulb and you. Want it you want to control it through assistant, you have to go through this initial linking process without, linking yes right when I own on, hub hub home, hub when I just, swipe, to left and CLD the. My. Home, stuff, connected, like light Bob thermostat. It's, query, all my devices calling, on sync method for for all sync API for all my stuff connected, and returning, me they settled right so something like that right right okay and so so we see here that I have one. Device called. A washer and, we have a specific, device type of a. Washer as well okay. Quick question you know these types are define. It in our, documentation, right so you can't just come up with metal. Device I can come up a new device and just add device, there it, won't work right because we won't have the traits, exactly. So you can mix and match traits, on different device types but. You are correct that we have a fixed, set of device types okay um the reason why we have these device, types is that you. Know for example when, you when you actually say a command, we. Show an icon of a washing machine for example okay right we use the context, of the actual, type of device in various. Ways in the home graph yeah. And. So, you can see here I have five different traits, or five different capabilities of.

This Device so, it can turn on and off I can start and stop the, various cycles I. Have a run cycle I have modes, and I have toggles. And. So, those. Are the capabilities of device and I can mix and match if I don't want it to be able to have the. Turbo turbo. Toggle, for example then I can remove that and you can no longer use turbo for example so some feature you don't want to expose through the home automation you can just don't have the trait so exactly. Yeah like many, door lock door door, lockers they don't have for example lock, option, right because someone could say hey, and lock the door and can open your and go into your house so you can hide some trades that you think it's critical, for example right that's, it exactly cool so you, can see here a little bit more about the attributes so okay when we look at what, modes right so when you look at modes you're gonna have a set of modes okay, in, our, case I've I've, set up two, modes there, is a small, load and a large load for a washing machine right, and so, you this is what it looks like in our structured, JSON response. On. How that's defined, for. Giving. It back to Google okay cool, so, if I'm for example. I'm defining, my I'm, a specific. Brand and I have specific, more, modes, that I want you to expose. That but. Then I launched a new feature along, a new. Washer. Then, that can add new your, traits for example new modes I can go here update, my my, home sync. Method and then no need to deploy anything to the Google CS and it's like I will be live available. Should test these new modes for example right so. Yeah as long as you're you are developing. In. A private state you can continually. Update, your various, traits, and your device types and like that on your, devices, and. And. It'll be able to reflect, you, know what on the device used, you would still need to have some type of on device, logic, to, be able to know what you know in ingest, stars that doesn't mean you need to receive this at some point the your. How, can I say your server must communicate if your stuff. At home and send the commentry or so the washer must. Have like a Wi-Fi, board and you we, must have gone through the setup if your washer, couldn't, make sure you are Q your Wi-Fi, and this kind of thing so yeah I imagine. That we are talking about D the home, stuff, right they they assist and stuff but, yes you just keep in mind if you are develop something you should still. Work on the harder side we got we don't we are not giving everything to you yet but yeah, we need to set up the part that your own machine at your own your, own device make sense yeah and we do that on purpose because we. We. Know that developers, want flexibility, we, don't define, a a fixed. Approach on how your, cloud service, communicates, and control mixes device yeah that make sense and cool. So this is the the own so yeah, and I said for. Some ties no question can say these are the that are is Lockheed something like that the or just you see they stands when I open the the on.

My Assistant, and see this tells of my stuff it's. Hitting, this part right this, methyl what, in what other situations. These methods use it for example um so. The, sink. The actual response, that's returned, is also. Used, when. You request to see so okay so say, can. Request explicitly, it can request a sink for example you can request one from your, cloud service so if a. User already has five, lights and they go out and purchase another, light they're, gonna have a now six light you. Want and if they want to be able to control. That sixth, light in addition then, when they register. It on your, cloud service it's, ideal that your cloud service then alerts Google hey they've added another device will, you sync with me cool and what. I say Konoe said turn on the washer, what how it, happened, so. What, happens, there is it, actually goes through on, to, the on execute. Intent execute, okay and so this. Here. Is the, execution. Intent of. Going through and looking. At the various inputs of. The. Command, itself okay, and, so, I look for what. Exact. Command. That it's doing so on-off start stop pause. Unpause, set, modes or set toggles and equipment, comas that's word you find it in your own your own sink for example when you define the the when. You define device to define their the what you are waiting for right exactly. And so when it, all of these you can see match, to. The. Traits. That we've defined earlier like you said so the on off matches. So this is devices. Dot commands on-off, matches. That, of, traits. On-off okay, and so and, so each different trait has different, commands that are associated with it okay cool. And so another important, aspect when. Looking, at this code is the on query right I can ask Google, assistant, is my washer, running right now right another, great question is if you have a refrigerator you can actually ask Google assistant if your refrigerators, running. And. So, yeah, and so, on. Query is though, is the function. That actually returns, that state okay, nice, and so you can see here it's it's getting which, devices, the users asking, about the state for okay, and then it's able to get. Information, and return it. Lastly. Though, the the final and forth, intent. That I want to go over is the. On. Disconnect. So. For, on disconnect, the. Disconnect. Intent is the one that that shows. When. A user, unlinks. Their account with Google assistant, ok so think of, this. Light, bulbs for example right they, remove a light bulb and they maybe you know or they they, no longer have, their washing machine, and. Maybe. They sell, it and they don't longer have it ok. And so, that's, what, the on disconnect is used for ok. And so. These the second part that I want to go over the. Front, for this is the on device, logic, that I have yeah cool cool so this. Right now for example go, can you go to the for, example and execute, yeah and execute cold I'm, seeing here that you are ever in this case you are doing like four, bays and you, are changing. Updating. A field right with all and, this. Is your logic right and then I imagine, I suppose, as a. Amateur. Maker so. You are have something, in. Your device that is connected to this database that, will receive an update that this value, change it and then can do the the hardware. Logic, is something, like that right, that's. That's. That's correct okay so you. Can see here my. Firebase. Reference. Here, I have. A specific, field of the device ID that, represents, my washing machine you don't want to turn off your neighborhood, machine washer, machine and then I have a child called, on/off that, I update, the value of to, the, parameter, that was given to me in this, request and, question. This was define. It by you for example right these you are the hardware maker you are the the, owner of this process, and you defined the structure so if, the our, viewers, are ok but I don't use firebase, I use. MQTT. To send messages. From the function, no, problem right no problem at all it's here. Capturing. This call from here and. Changing. The status on the actual machine on the actual hardware there, is no. Locking. No dependence, of like, having, to use a fire base for this case right correct. So we even, though can be like external, API that, the only other API and another service that you want to make ezel aid from the home stuff, it can have another call for your API right, exactly.

Okay And so all. Of this you, you can abstract away to any, type of persistence, layer like you say you. Can use anything from GCP for example, to. Store, this information and. Query later yeah because sometimes for example I'm, just imagined, right of course use. Firebase it's a great tool but, maybe I already have something I meant that they already have an a stretcher and like infrastructure, for these, hardware's, and they. Want, to implement, so you don't need to migrate to firebase. First to integrate your harder so if I already have not hired if you, are looking to integrate to home Google system, automation. You, can do it you don't need to use firebase all this structure so this is I think this is the message right right right. So. When, we look at how, the device. Ingests. These, these, various. Database, this. Database, information okay let's, take a look at that sure so, this, is the information that this is the file that I have that. It. Runs on the Raspberry Pi on, this aiy kit and you can see here a few things one I'm. Using firebase, admin to as, the, way that I subscribe. To various. Elements. And look this is not no. Js' I'm, seeing, correct, yeah this this is Python so I don't need even to make this use, the same language the same because, the it's not runny on it's. Not running on the, cloud function right correct, so this code would be running in the, device that's. It that's right and so so. This is everything that's running on device. You. Know so you could have anything you could, be running Java you could be running nodejs, you. Know any number of other languages yeah yeah and, all all you, know doing, the same thing of. Listening. For different data Changez. So here you have the logic, for your washer, machine so this is the point. Here and part of this logic of course is read they, state from. The home automation right, right, right so, the things that I'm controlling here, is, one that the color of the, light on. The button of this washing machine okay and, then the various initial. States that I set of you know it's currently off it's. Not on turbo it's not running and it's and it's currently. Possible. And. And. So what. We have here is, the. Various. Listeners, and. What I do and inside each of these listeners I actually set up the listeners here so I'm I'm referencing. Various. Elements within, my database yeah yeah so, a good, a good, thing to take a look at here actually is let's let's hop, over here and, look, at the structure, of my. Database very cool so, this is how you organized, the data for each. Each. Device, you have right this is these we reflect each washer. So if I imagined, I'm this is the database that. The brand. Owns, right. And we'll have like a 1 registry, per person. Per linkage, of the account exactly, right, of. Course they can't change the octet or they can have different for, a technician but these, register, on the washer, it's specific, for one one, harder right yes. So because. This is for a code lamp this. Is there's. Some hard coding, direction, to just one washer, and one, user right, so this this kind of goes the as, an MVP, bare minimum, of our showing an example, of, how this would work exactly and, so I have here, you know the I've expanded, each element, here to show what's. Inside, and, so when I have, from. Here a reference. Of washer, on, off on what's. That what that's actually looking for is. Changing. Any changes, to this field, here, let's. Let's show this can you do they ask against you know but no here see the command on your voice their phone yeah to turn. On let's see change, here the database of the so they can see how it's connected, each one and so what you'll see here is you'll, see this all of, this change you'll see there you know some colors changing and, the value of on/off to become true you'll.

See This also, flip to be on yeah quick question what is this so, this is, sink. Of. Philips. Hue website, or the Philips, you mobile phone app this, is the. The. Codelab front-end. For, controlling, your washing machine even if you don't have your, device physical, device you can check the status for these front from this webpage, that's. Right and even if you're in a testing, mode you can write these in your for, your own hardware, and your own in your own end, point, and check these status through these so don't need to actually have a higher other time with you right right this is exactly yeah because this right it's reading the same data right so cool, and so, you'll see you'll see this change here you'll see this, change here and then you'll also see, in this terminal the. Actual device. Have. A an. Output, of when it received the command ok it's like your debug window for the device right so this is I'm SSH. Into my, Raspberry, Pi here cool so. Let's let's watch it happen to the show. Turn. On my washer. Okay. Oh. I see there it's it's true here, this dual go there is on and. It. Showed that running, on so, all. Three elements there and, near-real-time. Is event based to wear based, on your voice it's doing, all of these things so. It's good because you said the common, consistent. Part of the comeand understood. That was. Something. To your washer, check. Your home graph say yeah yeah he has a washer, then called. His web hook and, with. The on execute, send. A tray there what the driver you updated, the database and, they and was like immediately like strainers. That's, the beauty of all. Of this going it's, all it's all event based it happened so fast and. That's. That's why I like doing IOT yes man, it's cool yeah this is personally. It's something I really like I spend I able to spend a weekend. Connecting wires and trying to, run random. Stuff, so. Cool, I think we cover a lot of stuff here right can, you just like recap, some stabs that they have, to keep in mind when developing this for sure so from, a high level the big thing to remember is, that this is cloud to cloud communication. Where. The. Developer. Responsibility. Is taking. In a structured, JSON response from Google. Applying. Some logic to that of what, state the device should come you would should do and. Then actually, controlling, the device oh and. So, so from, our perspective here, the way we've set this up is that we're using firebase. Cloud functions for, the various intents, that we're listening for okay and, then we're using, firebase. As the, persistence, layer to. Store. The device state and then. The Python, application, that's running on the, Raspberry Pi, listens.

For Those various states cool. Yeah, this is so cool I'm very excited because I like this this stuff, I like IOT. Stuff so yes. Thanks. For being here thanks for sharing this is amazing, I think, if they want to find us they can reach us through our teams. To either handle write actions, on Google and if you are sharing something with us also, use the hash tag Aug, dabs so, you can see your cool stuff or if you're making question, to us it's it'll be great and also we, have the reddit, actions. On Google so you can ask questions meet, other people from the community right, and, if you have a question with where they can go for technical, questions Stack, Overflow Stack Overflow so, action zone Google will be great right, and thanks. For watching please give your comment, your feedback, what do you think what the next episodes you are waiting for and we'll, be glad to produce for you okay see you next time Thanks. You.

2019-05-22 12:28

Other news