Local Technologies for the Smart Home (Google I/O'19)
I. Hope. Today you were able to listen to some of the great talks we've had and explore. The different sandboxes, and if you haven't seen the smart homes section in the assistance sandbox I highly, encourage you to do so, thank. You we have a great team there and some really cool demos, including. Showing, you how the smart home API can, help you grow a better garden, on. Behalf of our team were, really excited, to share with you some, of the smart home technologies, that we've been building since, we last spoke to you at I 18. My. Name is Carl Vogel, I'm a product manager on the smart home team my, name is Bennett I am a software engineer I am, guru I'm a solutions engineer and, smart home, I'd. Like to begin with a story. One. Of my friends recently purchased some smart home devices in. Particular. Light, bulbs from one company and smart, plugs from a different one and he, put them in an area that he calls the downstairs. And, when he gets ready for bed he walks upstairs and he, says hey, G turn, off the lights, well. He was telling me that often times they don't respond, together one. Set of lights often, responds, a half second or a second, faster than the other or. Sometimes they just take a really long time to respond in general or. They, don't respond at all as. Google. On our developer, community work, together to, grow smart, home we, need to work together to, solve these challenges. We. Believe one method, is to shift processing. From the cloud to, the local, environment, we've. Taken our first step in this direction to. Improve the experience, for users like him and the. Millions of other users, that use our products, on a daily basis, we're, happy and to, introduce that, that first step is the. Local, home, SDK. The. Local home SDK, enables. You to locally. Process, and fulfill smart home commands, received, from the Google assistant, we. Do this by inviting you to build and run your smart home business, logic locally. On Google, home speakers, and Google. Nest displays. Then. We. Securely, give you access to the lower level radios to, communicate over the local area network with. Your smart devices, through. This we. Can together deliver, substantial, improvements. In latency. And reliability. But. First before I talk about the local home SDK, let. Me give you an intro to the smart home API and then, show you how the local home SDK, layers, on top. The. Smart home API is the foundation. Of our smart home program, let's. Begin with how devices, are defined, and, integrated. Into the assistant, first. Developers. Specified, the vice tete type, this. Is really the what is it factor is that, a light bulb is, that our microwave, a, camera. Device. Type is our method to classify, the, overall essence, of the, device in a, user's home. Second. Developers, specify. A device, trait, and this, really describes, the overall functionality. Of, the device what. Can it do how. Can users control it, devices. Oftentimes have multiple traits for, example a light bulb may have the traits on/off brightness. And color, setting. Once. The device type and trade are specified, we, can then bring these devices, into the assistant, and specifically. Into our home graph using. The sync intent, home. Graph is our database, that'll, enable us to build a topology, of the users devices in their, home now.
Hints, To the Google, home to, start looking for a device and, we'll, use the information in this field to help duplicate. A device that we find locally to, a device that you told us about via the cloud to cloud integration. Here's. A sample of what that looks like you'll. Notice that this other device IDs field, appears. At the, device level so, you as the developer can, choose which devices, you want to be locally controlled or not. So. Now we're going to jump into the diaper skip app and this is the app that has the business logic that. Can control, your devices, it is. A simple but it's a critical component in this whole process and just a quick reminder we need to handle too intense. Let's. Quickly put these intents in perspective, so. For those who are familiar with the Smart Hub API we, have sync query, and execute, intent that Google, servers sent to your cloud services. For. Local we are adding two new intents, identify. Which. Is fired. By platform, when we've. Scanned a device that belongs to you and second. Is the reachable devices intent which is optional but, required for if we have scanned a bridge or a hub. Let's. Get started with the typescript app but. Before we do that let's, look at the SDK, interface, and. When. This launches, in June you will be able to download a sample and a boilerplate code from github. So. The interface export is, exposed, by the SDK is pretty straightforward. It has two main classes first, the, device manager class the. Device manager class provides, methods to communicate, with your devices and like, Method mentioned earlier it could be TCP UDP or HTTP, HTTPS the. Second is the, app class, this. Provides the methods to attach the intent handlers. So. Let's look at the Taipings for device manager class is. The, send which. Takes in an input for, command, request type object, and returns. A promise which, is resolved, when command, has completed and like. I said earlier it could be HTTP, TCP or UDP. Let's. Look at the typing for the app class, so. For those who are familiar with the actions of Google node library that you use for the cloud site integration, you. Will realize that there. We have on execute. On sync and on query as the handlers you can attach to here. We have unidentified. Unreachable. Devices, and on execute, methods which. You can call to attach, the handlers for your app after. Your app has attached the handlers, we call the listen API, and that's. An indicator to the SDK that the app is now ready to process these intents and notice. That these methods, are chainable, finally. When you are ready to communicate with your device you, will call the get device manager, API to get the singleton, object for device manager, and use, the send API. Let's. Put this interface into perspective by looking at the skeleton of a sample app so. In the sample app I have like identify. Handler and execute Handler and in. The constructor, for this class I create. The instantiation of a local home app get. The device manager object, attached. The two handlers, and call. The listen API. Now. Let's start. Looking at the events that, happen at runtime, and what, your app does to, handle those events. I'm. A visual learner so let's take a look at this in pictures. You've. Already updated, scan config via actions at Google console you've, updated the sync response and now, that information has been set down to a Google home the. Google home starts. A state machine where, we look, for local devices. Then. This process repeats, so whenever, user, plugs in a new device will find that too when. The smart device responds, to one of our scans the, local home platform, generates, an intent called identifying, like Gaurav has been mentioning and we, then call, your apps identify, handler, so.
To The Google home app or Google assistant, app to link re-entering. Those credentials, it's, it's not easy for users. One, of the other things we heard is that and scenes that users have, a lot of apps on their phone to manage their smart home and for. All use smart home enthusiasts, out there you'll recognize, this phone on this screen behind me that, you need a folder, to actually manage your Markham. So. We took our first step towards solving this device setup problem, with GE Lighting and developing. A seamless, setup experience, that. We delivered, first in the Google smart light starter kit we. Gave users the ability to, natively. Set up C by GE smart, lights in the Google home app without. Needing to download, any additional apps, and, instead. Of this 10 plus step process, we, reduced it to three steps and about. 30 seconds, let's. Take a look and see what it looks like, so. When a google home device discovers, a. C by GE light we. Prompt the user. Would. You like to setup your smart light, then. We go ahead and connect to the bulb and discover services, at, which point then the bulb will begin to blink and this, will let the user know which bulb they're setting up they, click setup and choose which room they want it to go in and give it a name at, this point we're provisioning, the bulb to the local network and registering, it with home graph and in. Just about one second, you'll see that this the smart light is now, set, up and at. Which point it can go ahead and start taking Google, home commands. And, so. 30 seconds is, really incredible for our users and we've. Heard really great feedback so. Far. And. We. Accomplished this by allowing GE to run their code on Google home devices, and yes. As you may have guessed they, also used, the local home SDK. However. To do seamless. Set up there's more intense. Too right than just the identify, and execute we talked about today including. Indicate, provision. On provision, etc, and those are all part of the SDK and, also. This SDK, can. Be used for more than just the Wi-Fi radio as part. Of early access we, allow this SDK to also, leverage, the Bluetooth, radios, for, direct connection. To ble, devices, and, through. This seamless setup experience, with ble devices we. Use the, Google home as a hub, so, you don't need to go out and buy an additional ble. Or Gateway. We're. Growing the seamless setup program now I'm focusing. On ble, devices, in the near term so, if you're interested, let us know by, visiting the link on the screen behind me. Next. I want to talk about existent, connect which, is something that you may have heard about at CES and we've been continuing to invest in since it leverages. The same local home platform, that minute talked about to extend the reach of the Google assistant, which. We call assistant.
Extensions, And we've, classified these into two categories, the. First is input extensions, which, enable a simple method for user to activate, the assistant, to, do everything from simple queries such as what's, the weather to, triggering advanced. Smart home routines, here. We have a simple programmable. Button that, instead of always requiring, users, to say hey, gee they. Can go ahead and just push the button it's, really great for some of those frequent, queries in, addition. We also have output, extensions, that, enable devices, to show assistant, responses, such, as what's the weather or their, schedule, from, their Google Calendar so. This, is currently in early access right, now and throughout, 2019, we're, going we have a really, busy year ahead of us we, have some product, launches coming up later this year so stay tuned and our, teams are finalizing, the reference design and preparing, the assistant connect SDK, for public release later, this year and by 2020, we, expect developers, to have self-service. Access and the ability to even more easily and deeply, integrate, the assistant, into their products. So. To recap. We. Believe that driving, logic from cloud to on device is central, to, our strategy, to create even better experiences. For users and we. Believe that by going local, it we can also invite developers, to integrate more, deeply, with Google the. Secondly. The. Developer experience is key to, building a great smart home ecosystem. Our. Ecosystem. Is only as strong as our developer, community you. All we've. Taken many sape steps to make onboarding, as simple as possible, for example, by, not requiring firmware, updates, to garner, the benefits, of local execution and. We always welcome feedback on how we can further improve so, definitely, let us know and, lastly. A big. Focus for us in 2019. Is reducing, friction and making device setup and linking more seamless, I encourage. You to explore our programs and learn more. So. With that I realize, a 5:30. P.m. talk your for going happy hours so we thank you for coming here today and listening to learn more about local technologies. If. You have any additional questions check out the links on the slide behind me visit, us in the sandbox or swing by our office hours and with, that enjoy the rest of your i/o thank. You.