Born in the cloud: Lessons learned while building a successful IoT startup in Azure - BRK2410
I appreciate. Everyone coming out I hope. They had fun at the. Celebration. Last night I know. It's a little bit early after. Unlimited. Free booze all night so I really. Appreciate you guys making it here so, today we're gonna talk a little bit about some, lessons learned while building a startup. Specifically. An IOT startup in Azure, so. My name is Joey lauric I am, a Technical, Architect at the Microsoft Technology Center in Denver but, I've only been in that role for about six months I recently. Moved here so, recently. Moved from here. Working. At a company called uni key technologies, with, my former. Boss Alan, Blythe I. Want you were to do a quick introduction, interview. Absolutely, so, I'm the chief, architect, at uni key I've been there from the, very beginning, when, we first had a product, in, concept, I'm, so I was involved. In everything. From hiring the team to building our first API. And back-end systems. Technology. Vision security. Small. Company lots of hats, also. A father of three and a runner. Big. Running nerd yeah bigger running. So. Let's talk a little bit about what we're gonna talk about today. So. In this session we're gonna cover really, just getting started on the right foot some, ways to help launch a startup also. Designing to scale so. Unique II was a pretty, media. Recognized, company which we'll get into in a little bit and we, knew we were gonna have to scale really fast so, being able to plan for that and then, as we grew moving. To micro services there, was a lot of challenges that we faced that helped, us go. Down a path that we. Knew that we needed to be able to split some of these services up and splitting, those services up we knew we had to embrace DevOps, methodologies, otherwise we weren't gonna be able to be successful we. Weren't gonna be able to move fast enough and then. My. Last part that I'm going to talk about is some. Of the pitfalls of IOT because. Suddenly. You've got ten twenty a hundred thousand, devices out there that you run the code on and you can do some really interesting and some really dangerous things doing that, and then, we'll end up with where, you nikki is today and a little bit about where they're going, in the future so. Let me hand this off to Alan to talk about a little bit about you Nikki as a company and a. Little. Bit about the early history, yeah, thanks Joey, so. Everyone. Says startup it means so many different things right, I don't, know if I could really call you Nikki a start-up today we've. Been, with. The. Current technologies, and teams and systems we've, been in production since 2013. And, we started building in 2012. We, very, much are a tech company we're somewhat. Of a odd breed here in Orlando, being. A more startup style tech company it's not a very common thing here yet. So, it's, you, know cool office environment. Although it's not exactly, where we started we had pretty humble beginnings. To. Describe, really, what we do were an access, control. Platform. We're a platform strategy, and we really focus on, providing. Either. IP and, thought leadership or we, provide software as a service, offerings, and those, are the two ways that we go to market our, primary, strategy is that from. Day number one is we didn't actually want to take, on the. Basically. All the labor and all the capital, to put, together a, smartlock. It's. Estimated, that you need to raise ten to twenty million dollars to fill supply and distribution lines. So. The beginning of our company, is we focused, on all the technologies, and then we partner, with. Industry, leaders our. First partner was corset. Locks at the time owned by Black & Decker and then later owned by spectrum brands. Just. Random factoids, same company that owns George Foreman grills and rayovac batteries. Somewhat. An interesting environment for me up until this point my career most of the places I had worked it was, mostly.
The Same discipline, of engineering, it, was a bunch. Of back-end developers, database developers, things along those lines. Munich. II is a lot different even in the beginning when we had a very small team of just six engineers, it didn't, go very far. When. You have mobile. Engineers, embedded, engineers back, in devs. Electrical. Engineering, it's, many different disciplines under one roof so, it's. Pretty interesting the way that products, get delivered because they're all individual, products to make one large, platform. Today. We have, products. In two different verticals, in production, so the. Quick-set Kiva was the first product that's. In the residential space consumer. Electronics, and. We also recently launched, into, the commercial. Space so now I can go to work walk. Up to the door touch the reader instead of using a card and. Use my phone as a key there as well. Some, of the some of the challenges that we really faced in the early days is in 2012. Bluetooth. Low Energy it, was somewhat new at the time, trying. To find engineers that have experienced, with it number one and then the rapid changed to the technology, at the time was somewhat challenging. The. The product that we ended up building probably wasn't. Even possible. With Bluetooth, classic and, that's one of the big constraints, that we had is from day number one we had to build a smart lock, that. Could run on four. Doublea's for an entire year and, when, you get down to the nitty-gritty of how much current draw that is it's a very meager. So. We, also had a, pretty. Interesting, start. This. Is really what propelled the company to the platform, that we have today our founder, Phil Dumas was on Shark Tank it. Was the season, finale of season three and it, was the first time that all five sharks wanted, to invest, in a company so. They shook hands on TV, our. Our, founder, Phil did an amazing job with. The. Presentation on, Shark Tank and, when. You watch something on TV everything's. Been edited everything looks perfect, but the.
The. Smart lock that was used for that demo on shark tank actually had a bunch of wires all strung together, a, hand, soldered, inside, the there's. Like, a wooden. Stand that holds the lock and, it. Was a total, rat's nest inside, of that thing and it had problems where it would reboot and you'd have to pull the batteries and put the batteries back in. But. Luck shone on us that day, and it, actually worked perfectly, the first time the, first time that they taped, so. That was very interesting. So. After, that the. Contracts, with, the, Sharks, they really are sharks it was pretty horrible. Phil. Decided, it wasn't something that he wanted to do after. The show aired there was tons of private, equity interest, in the company so, the company took, money from some private investors, our, first investor, was FF EC out of New York. And. From. There the, company started hiring the first employees, and that's where I come into the picture I was, the first, software engineer with the company is the fourth, employee back, in 2012. So, it was right around that time in 2012, right after the show aired the the final contract with Black & Decker was signed and we started building the product, and. Like. I said earlier that first product that we were building was the the, quick sake though which you can find in all 5,000. Retail outlets that corks that is sold in. Now. This. Is one of the very interesting things is. Now. We have money we've. Hired a couple people what. Are we going to build to, launch a national product, in 12 months. Pretty. Challenging problem, but very fun at the same time. Set, the stage for you guys technology. Wise at the time and this is where. Ezra comes in at. The time there weren't really any platform. As a service offerings. For. In. Any of the competitors in this space so, when. I first, sat down with the rest of the guys, at the company talked, about hey so let's, go towards Azure and I'm actually sitting down with a Linux guy a Mac guy and a, business guy right so it's not like I'm having conversation. Where Microsoft technologies. Are just broadly being used so. Explaining, kind of some of the differences other partner. Offerings versus, what Azure had at the time. As. Your cloud services were pretty groundbreaking. Platform-as-a-service. And. Basically the sales pitch was hey if we, put cloud services, in place I think with one developer, we can actually get all the way to production with these systems so. From, there the rest was history and we started building. So. I wanted to give everyone a quick, idea, of. The, technologies, that we use this. Purposely. Is the blended, talk of some. Technology, in some business because we really wanted to give it a good holistic, picture so we're not going to drill into all, the nitty gritty we're, purposely going to stay at a fairly, high level but, you can see where we started was cloud services, also. Using. Service bus one of the very important, things. We, also implemented Redis, which we'll talk, a little bit later so, that's where we start and then where things go to going next is we. Were kind of growing with Azure right so, as.
As, You're introduced, document, DB which eventually, became, cosmos, DB that, came part of our repertoire, and. Then Traffic Manager, is a critical, piece for us as well which Joey will talk a little bit about later so. What. Came next after, that Joey's. Gonna get into the service fabric, and micro, services a little bit later but, service fabric was the the next thing that we started building a lot of our services on, remember. We're a platform strategy, so we endeavor to build services, want once. Andriese multiple, times and. Then. What's coming next for us this, certainly isn't inclusive. Of everything, but we are looking at service fabric mesh do, you get back to, more. Of a platform as a service, style, offering, if any. Of you guys aren't familiar with, service, fabric there's. A heavy, infrastructure. As a service, component to where we did have to start building VMs, again and for. Our culture, of moving. Quickly that was a little bit difficult for us so, we're excited about mass, and. Then since, kubernetes, is all the buzz these days we're also thinking about that as well and. There are many different parts to our smartlock plat, right. To our access control, platform so, there's, probably a place for many different technologies. So. With that let me pass it over to Joe and he'll talk about kind, of some of the pitfalls. Of startups and how to get started yeah so getting started with a start-up is really, difficult. 75%. Of all venture-backed, startups, fail just. Outright. Was. The quote from Harvard Business School half of all new businesses fail within the first five years and 75%. Of all new businesses fail within the first ten that's, a really, risky, ecosystem. To be getting into so. Being able to do it successfully is is. Hard, it's a big challenge so. One, of the first things that we knew that we had to do is. You. Know just be prepared to jump, into this ecosystem and starting, a new businesses is risky, because they all fail it's, not just a chance, of the business, failing but, anybody that's a part of that business has risk, you're. Taking a huge, you know there's a huge opportunity cost, just to participate in the startup just, coming in and working you're probably taking a pay cut maybe. You're taking a lot of equity but. Equity. And what you. Know you're taking equity in something you haven't built yet you're really based on hope. And. Especially being in tech you know these the ecosystems, are always changing you could be working on something that somebody else is working on and launches first and. Then, lastly start starting a new business is really expensive so. Typical, series a round is two to ten million dollars and that's, Series A that's after you've already gotten a product that you can show that you know you want to market further, it's. Really, difficult but, there's, some really great ways to help so. What did what did we do as you, know key and I'm gonna I'm gonna say we a bunch here because it was, we, now I'm with Microsoft, but it's, a lot easier and just have it so.
Unique Arrays 1.1 million dollars as a seed round in June of 2012, that was kind of after I think, Phil says a lot it's like the. Shark. Tank episode aired and then his phone just started ringing off the hook the next day and, that was enough to kind of hire, some people hire, Allen say let's build something, we. Knew we needed to get something out the door but. Really that money that's not a lot that's not enough to pay people for a long time so he raised another 1.5, million dollars in May of 2013 and then. Based, on just, that alone so 2.6, million dollars that, was enough to launch the first generation of kevo kevo. Which is kwikset's smartlock product come up touch to open it's in all Home Depot and Lowe's and Best Buy. That, was launched with what like five people five, or six people. So. Six people in. A year on a. Little under 2 million dollars that's really, impressive and using. Cloud services using, platform, as a service leveraging, startup programs is a way that we got their, unique. He actually raised its first big round in the series a round in April 2015 which is when I joined the company which. Was ten million dollars so if you look at it 22 months between. The first seed round and Series A that's, really only enough to hire like 10, engineers, below. Market salary, for Orlando, that's. Really, really tight, so. One of the things that we did we had a tiny, office. Esteban's. Been there, see. See you out in the crowd nothing, glamorous at all it was actually just, in like a little strip mall kind of area but. One of the most important things is to hire self driven people. Who really valued, what they're doing people, who would live and breathe what your idea was, starting. A startup if you just hire any, developer, off the street there just wants to come in and do their nine-to-five and leave it's gonna fail you, have to hire people that really believe in the vision that you have and will. Work I don't, want to say work extra but work very hard to. Make it happen and then. The one of the next most important things is to leverage a network of connections to get into programs to help you find people who have done this film. Made some connections from shark tank Allen, had some great partner connections that we leveraged and just, you know reach out to anyone that you can that can bring advice because chances, are almost, everybody in the room is going to be doing a lot of this stuff for the first time because, it's a startup and it's something new. So. Why did we succeed kind. Of what I talked about we hired a lot of driven excited people myself, and Alan included, but, I think, when I came on I was employing an employee, like 15 or 18 and me, and we were crammed in that little room I remember hiring somebody, that I hired on my team we had to sit him at a ping-pong table for the first couple weeks of this job because we didn't have enough four desks it.
Was, It. Was kind of an interesting environment but it was actually really fun and really exciting, I look, back at those times even though it was super cramped really nostalgic Lee but, then. Take advantage of programs so, there's a lot of programs for startups we went through Microsoft, Bismarck Microsoft this part plus which is now Microsoft for startups a fantastic. Program if you're looking to start a new business Microsoft, for startups Microsoft. Is investing, five hundred million dollars over the next two years in startups, that's a lot of money and with, this, part or what, is now as you're sorry, Microsoft, for start-up you can get up to 120 thousand dollars a year in Azure credit that's, a lot and it basically comes with ms the equivalent of MSDN subscriptions, to so you get Visual Studio Enterprise. You. Get off access, to the whole office, 365, business premium licensing, you, get access to dynamics and then you also get access to a bunch of different resources so. I work at the at the MTC out in Denver we, typically only work with enterprise customers, we work with like you, know Fortune. 1000, companies that, come in where we're usually working with like VP and c-level, people but, we also work with people in, Microsoft for startups programs we've had them come in several times these new companies that we have a an. Interest, in and we see that have a strong trajectory, so, you can get resources that, you. Know by being a part of these program that you wouldn't be able to get any other way and then, there's also exclusive, events that we have for startups specifically, to help network between the startups make. Sure that you, know you get all the right connections that you can be successful because by giving you all this either credit we're you. Know we're not asking for anything back but, we're in investing. In the company's going forward because we want to see them be successful and, Microsoft. As a company is very willing to help support, even. Through our personal experience, when we graduated, from Bismarck and finally ran out they came Microsoft, filmed some commercials, at our office we. Always had a great back-and-forth, with both partners and our kind, of regional support staff that would come through, so. I want to give a quick shout out to nebia technology. Esteban's, here and he, owns, and runs Nebbia and this, is this was really great for, us and it's actually how I got into, uni key at. The. Time I was a running. Some projects, doing Ruby on Rails development, and I. Got, some random, message on our local Orlando, devis Lac saying hey we're for side work I just bought a house I started picking up I don't know those like 15 20 hours a week something pretty small, pre-series. A and then, I wound up getting a job offer from uni key post series a when they had enough money to hire more people but. It. Was a really great partnership because, so, estaban and allen used to work together and they. Both have tons of experience and being able to leverage that experience being, able to leverage experience, of a partner who's already very connected in the ecosystem, for, better. Connections, that's part of how we got in to. Bismarck that's how we got to know our local Microsoft, reps that's how we built close connections, I actually said that's probably the, the close relationships. That we, had with Microsoft, in general as a start-up is why I have my job now so incredibly.
Important, And then also leveraging. Local partners expertise is really important because as a start-up you don't have a lot of money and hiring. Someone is really expensive but. Sometimes you, need to get more done than you have the headcount for sometimes you need to be able to get a project out the door and being, able to reach out to a partner, just. Say hey can you help me build this project you. Know can I can I get someone for two weeks to do staff augmentation or, can you go off and build this on its own there's been a ton of stuff we've been able to hand off to partners and then. Just have it come back and be done without, the cost of you know full, overhead for someone new. So. Overall with this the you know the Microsoft ecosystem is huge but the circles are a lot smaller than you think so making those good connections, a clear question really important yeah, you have a question oh. We. The question was were we part of Microsoft reactor no. We weren't we just got, into the Bismark program so there's no investment from Microsoft in us other than this park but, we made a lot, of connections. So. Next I want to talk or Alan will talk a little bit about designing, to scale. Yeah. So that this is a pretty interesting topic. I've. Been. Doing this for about 20 years when I first started, it was order, the servers from HP, build. A bunch of stuff for like six months and then ship it, kick. Stuff over the fence to the QA team right, and. We. Tried to take an entire different, approach to things you. Know being a small start-up with limited resources we, we. Had to be as nimble as possible, and, ability, is one of those things for me out. Of everything that I do it's actually one of the things that I'm actually most, passionate, about I love to talk about it I love to talk about the trade-offs of, consistency. And scalability, and, this. Is, really. One of the things in the beginning is how do you decide when you know a product, is launching nationally. On. Day one it's, going to be in all these different retail, outlets how much scalability. Is enough, so. It was actually, something that we strongly considered. So. Some. Of the some of the things, that. Were important. To us is. We. Only had one API developer, at the time, unless. We were using outside. Resources. Through partnerships, and that sort of thing. And. Then we. Also. It's. Pretty difficult to, decide exactly what, the product is that you're going to build especially, when it's a partnership about between. Multiple companies and sometimes the trade-offs that you talk about end up leading to changes. In the product even, so. It's one of the things that we really talked about heavily, in the beginning is do we try to ship, a full-featured. Product, or do we try to ship MVPs. So. We tried to make sure that we are always focused on, just what our core competencies. Were and what, was the true minimum viable product. We. Talked about past a little bit earlier it was actually, absolutely, pivotal to our success, in our early days, before. We are in production we were deploying at. Least four or five times a day. We, basically had all the development, team that was working on the product co-located. In the same room sitting across the table from each other having. The conversations. Of ok well I. I. Went ahead and called API and I got this result why, was this thenn why didn't this thing worked and then we would actually jointly. Debug problems like that that was absolutely. Pivotal for us as well so. Scalability, one of the things I love to say is it's it's not our technical, concern only. It, often involves, some. Consideration. Of the product or the business as well. So, scalings. Multi, multi, faceted, the. First product, that we endeavored, to build it was pretty tough decision, of what to build it, was the type of environment, that there, were quite a few features that we built and then decided, not to ship. So. That meant that if we were going to be really, building, something that, was perfect, technically, it, might not be something that ended up shipping anyways, because a product, like this that's very tactile, and. Very, user focused, we had, to actually be able to use it and see it. So. Very, important for us that the, technology, solution, what, we settled on to make sure that we could get, out in, production. In time was to focus on things that we knew at. The time marketing services was being talked about but we knew it might be difficult to actually understand, the domain well since, the product was actively changing while we were building it so, we actually decided to build an anterior system, I describe it as a distributed, monolith where, we heavily focused, on caching, techniques and we, also.
Focused. On asynchronous. Processing, of data. One, of the most important, tenets that we kind of set forth architectural, II in the beginning is we. Would never be, subject. To third-party performance. Impacting, our customers, so. Absolutely, every external call from our system whether, it was using another service to send emails or another, system for renting templates, whatever. It may be that was something that was always kicked asynchronously. Through service bus and then picked up by a background, worker, for. Those of you guys that have used cloud services. And. Then the last last, point is. Absolutely. Have, to embrace distributing. Computing concepts, this. First product that we launched the. Quick set key though it's not internet enabled it's, only boe it wasn't actually until two or three years. Later that we launched the kevo plus which. Is basically an Ethernet to, Bluetooth bridge. So. When, we talk about what is the trade-off of the usability of a product one not internet-connected. It. Wasn't actually just for scalability. To. Embrace. These distributed, concepts, it was actually for the usefulness of the product as well. So. One. Of the very first things is we didn't want to make sure from day number one that we were scalable, so one. Of the thing one of the big concerns that we talked about back and forth had trouble deciding is, we. Did ship with cloud services, and sequel, Azure and. We did ship to production with a shorter, database, and, that was it. It wasn't a system that was highly scaled when when we first launched it but we wanted to have enough of the technology, pieces in place to. Be able to get there and. To. Make things easier for ourselves as an example we implemented, a unit of work pattern within our code to. Make sure that sharding, wasn't something that complicated, development, of the product as well. Like. I said earlier we we did designed a distributed, monolith. It's. Actually. Often said by some industry. Experts that. Developing. A monolith first it might be the best approach if. You don't have good understanding of the business domain it, helps you figure it out and then. Later you can start to decompose, into. Micro services at a later point a good. Example with uni key is today we, are entering the third business vertical, automotive, so. We already have two verticals, under our belt residential, and commercial so. Now we can actually see, the, paradigms, across all these different verticals, and really extract what's common among them a good. Example is we have a custom, in-house security. Model that's. Used across all of them it's all based on PKI and it's how we achieve, security, across. All components basically we have a rule if, any device, interacts, with our system, it has to be PKI BarNone. It's. Helped us be, protected, from some things like heartbleed. When, it came about we. Weren't very, susceptible to any of the issues that happened during that time because. That consideration. Another, good example bewdley vulnerabilities. If you use the built-in security, we, weren't vulnerable, to those either since we actually have our own custom protocol over Bewdley.
Mentioned. A synchronous, earlier it was one of the pivotal things, for us absolutely. Important. So. With that let's get into a few, details about, what our microservices, journey look like, absolutely. So you. Talking about security too reminds, me so a few years ago one. Of unis keys best marketing, pitches ever it was at DEFCON 20. 2016. Yeah. I, don't remember what the exact number was but pretty much every, major smart lock it, was a market it was 13 13 of them had. Critical, vulnerabilities, released except. Akibo so. We've never had any. Released. Incidents, at all, knocking. On wood up here guys you've caught in QA in the past before but, yeah. It's the only smart lock out there that has has not been been, broken, in some way some. Of the ways that the locks are broken too was pretty, crazy there's a really good take, that as a challenge by the way, yeah. It always, makes me laugh cuz it's like a doors, usually right next to a window and a window is the easy way to go in but so, let's talk about moving to micro services so 2013. 14, 15, the. Word micro, services kind of almost became a buzz word I don't, know when it was actually coined but, it's been around for a while but. Everybody kept, saying you know oh you have to move to micro services start in micro services even today people they, hear kubernetes and they think okay that means containers, and micro services and. That's that's, not really, a good, way to think, there. Are certainly reasons, to go to it it's a great pattern, but. As allen talked on you know good design. Transcends. Your implementation. And I. Really love that quote because, you. Can have good, design within a monolith and make it easy to move to micro services micro, services might force you into some good patterns, but. This was really the next step in our journey driven, by business, use cases so. For us we were moving to support multiple verticals, at this, time you know we launched in residential we, had a lot of talk about hospitality, which we haven't done too, much and yet but we, were moving full force into commercial and suddenly when you go from one to two that's. Really going, from one to two is almost the same as going on one to ten because. You start to see what's going to be reusable, and this, is really where we started to gain clarity around what it what are we doing what are we actually building what is our platform because. Before our platform was just the set of features that quick-set, wanted but. Suddenly, starting. To be with you know we have two or three different partners on the residential market and the commercial market now, we can really see what what, is our what is our real value what is our platform offering that we're gonna do so, embracing, residential, commercial hospitality. And more even now I mean there's a partnership released, with Fox automotive so getting into the car space, you. Know that's that's pretty great and our knowledge that we gained from, all our previous years helped us decide. What. What, should our services, be and this was really a big driver towards. Moving to micro services. Supporting. Multiple partners, multiple, verticals and then we, also had a really dynamic team, so, I joined his employee number like 17 or 18 at. The time we were moving to micro services we were probably at about 40, so the company doubled over two years and had. A fast, trajectory, and because we were always.
Changing. Who we were focused on so different partners we couldn't, have dedicated, teams for every single partner and every single platform we, knew that we had to have shared resources people are always bouncing around, we. Needed a very flexible, deployment. Model so that we could continue to. Keep up with the pace, and. Then, one, of the last real, kickers for us was deployment started to get slow you, know building a monolith can, be great from, getting. Something out the door but. When you have all your code and kind of one basket. And you're deploying to you. Know in the case it was cloud services which is platform offering but still standing up VMs rolling, out code running, integration, tests and that's something pretty important is we had lots, of integration, test setup which was critical, to us not breaking. Anything in the future especially with the monolith but, our, deployment started to be really long and we started our release cycles started getting long because, now it wasn't just changes for residential, going into one place we had changes for residential and commercial and. All these other concerns, that are starting to bleed through to where our QA process, got rough our deployment, process was long and where, we were releasing early on we would push the production four or five times a day because it was easy we. Got to a point where our deployments. Were like once, every couple months and for, a small startup that's that's, absolutely too, slow, you, have to be able to be moving fast because the ecosystem, that we were in was changing fast. So. We decided to move to micro-services. So. What do we do. So. This is my favorite slide. I. Think. This is very indicative of a, lot, of modern application, architecture when people first moved to micro surfaces in my my apologies, to, anyone. Who likes Mongo I did not make this graph this. Is from micro services the right way by danielwoods which. Is a good read to of, talking, through how this is how where. A lot of people wind up and this is actually really easy to do. Especially. Given some of the frameworks that make micro services so easy, suddenly. You've got you, know you see you hear the word micro and you're like okay so I'm gonna write this little crud service around all my different entities and they're all gonna talk to each other and I've got these business services that kind of stitch them together and maybe I've got some web front-ends, that expose them in different ways and now, all of a sudden you've got every service is just calling out to every other service and you've built this just mesh, where. You. Can't change anything even. If it's pretty well versioned, it's still really hard, to. Have all these dependencies, we. We wound up building almost a monolith, out. Of micro services meaning we got the worst of both worlds which. Was really really rough and then you wind up with these long dependency, chains where if you have a service to call the service to call the service to call the service if any, of those fail, and. How the whole thing's failed with a monolith you've got one context, one database transaction, if something fails the whole thing fails but, now I've got five different services, going to five different databases with. Five different transactions, and. I don't know if any of you have dealt with doing, distributed, transaction, coordination, but it's something, you certainly want to avoid if you can. So. This isn't really the. Ideal way to do it this actually caused to cause, a lot of problems for us and it took a while for us to untangle, it yeah. I think an important, note too one of the things I mentioned earlier, is you have to understand, the domain well at the time that we're attacking. The the commercial space. When. We were first going. Down the path of microservices. We didn't, have a great, understanding of, exactly what, the commonalities, were over. Toward a residential, system that we had, already built so. It caused some, of this path that we went down like Joey said yeah absolutely so one of the first things is what is micro, some. People here micro services and think oh they should be less than 100 line so they should be less than 10 lines or they should be and there's. Not really a definition. For this, to. Me if. You, think about micro. The the smallest thing that it should be is something that accomplishes, some piece of business functionality, so. Thinking about your code from a code perspective or, from an engineering perspective isn't, really the right way to go about it if your Microsoft account less, um thing from a business perspective it's probably.
Too Small. And. Then, be conscious of Conway's law so Conway's law if you're not familiar with it says, an. Organization. That builds services, will, likely build services they're shaped like the organization so. If you've got three teams you're likely to get three separate deployments, of three separate services so, just be conscious of that because we started to run into some situations, where if you divide a team up and they start working separately they're probably not working on the same products, you. Can almost intentionally organize, your company around how you want your services to be and. It actually works really well because you have a team that's working on a micro service and for me micro services is all really about deployment, independence. If, they're working on things, that are independently, deployable then, then it's great because, you, know separate teams they can work fast, they're not dependent on each other so. Just. Be conscious of that and then consider. Domain driven design so. Domain driven design is, is really just the concept, of breaking. Up your business functionality finding, bounded context, between these. Different functions and that, if you, dive into it it can become a good guideline for how to build your services and this is what really helped us you, know doing some exercises, around it figure out how, should we lay things out what. Parts of our services need to be dependent what parts of our services can be dependent, but asynchronous. Where. Maybe it makes a change and then some message goes off and we don't care about what, happens on the other end from the service perspective but. To accomplish the entire business task there's this whole workflow that's happening. Another. Thing that I think is really important is you don't have to use containers, there's. This thought today that micro services means containers, but, it doesn't maybe. If you're doing containers you're doing micro services but you can deploy a monolith inside okay. All these tools for container orchestration, are fantastic, but that doesn't mean that you're doing micro-services, so. You. Know just, don't. Jump into it right away obviously. You're gonna use something like kubernetes, you're gonna need to use containers but, service, fabric has a great platform for orchestrating, micro services that are just running, executables, they, can be your own processes, that you're running or guest executables, that you just want. To run and have it orchestrate, so. As I mentioned on the last slide be aware of dependency chains I think.
At One point we had a cyclical, dependency, between micro services did, which, was pretty, terrible let's say you make one call and then it would just infinitely call and call and call call thankfully, caught in QA before we we got out into production but if. You don't have a good pattern for, how you want your services to aggregate some. Of these things are possible, and actually really easy and at a start-up when you're moving really fast, it's. Easy to not notice it because you're not spending probably as much time in code review or unit testing as you should because there's you, know somebody from a partner that's saying hey you really need to get this out the door and there's. Just two people working on it and that's it there's not time. And. Then lastly choose an orchestration method that really fits your company so we. Started using service fabric right when it came into preview. I think, kubernetes was around but, not as well known service. Eirick had been used internally by Microsoft for several, years before then so we knew it was well proven and. From. A web perspective, from a web team perspective we were very much a Microsoft, shop the rest of the company wasn't but. I've had the chance to ask the, service fibre team why. Service fabric versus kubernetes and the career day seam why service, I versus, kubernetes and a lot of the a lot of the answer comes down to culture, how, do you want to work what are you familiar with if you're. Used to going to the community for support if you're used to waiting for community updates or submitting, your own updates and you're used to that release cycle yeah. Kubernetes, fits great if you have to be in containers yeah that, is an option service, fabric is also open source and also supports containers now so fantastic. But, if you're used to coming to conferences like this and getting support from Microsoft. You, know Microsoft's, not gonna be able to give you support when you're aks. Deployment, fails unless it's some kind of infrastructure, thing to fail but. We certainly can give you support, on service Oracle because we wrote it we support it so. It's. Not even. Down to a, feature, versus feature although there are some differences but for most these cases they do a lot of similar things both really well but, a lot of it comes down to company culture and so in our case we chose service fabric as our hosting model. So. Here is, it's. Really tiny on here so I'm gonna try and look at the big screen here's, a, simplified. Version, of what. We settled on I. Will say that this is a cleaner. Picture than I'm sure what's actually implemented. So, a little bit of this is aspirational, but this is what we were looking to head towards and we. Actually got a lot of this implemented, where, you. Can see at the left side there's. All of our front-end, applications, so, in this case we've got administrator. Portals from the commercial side where dealers, can go in and set up their customers and say who has keys each, customer, has an application because the whole, process that unique he uses is based around somebody having a mobile device with them that, has certificates, that they can validate, keys, that.
Certificates, That they can present to the lock that block and validate to know that they're supposed to be able to open the door there's. Also gateways, so we have these residential gateways that, sit in people's houses that have, access to open the door they have certificates as well and people can interact with them through the mobile app so I can open up my phone hit a button my lock on my door unlocks at home and then. There's applications. From our own side administrating. Things customer, service customer support, for, internal, to yoona key and external, so, real though it's simply you know oh it's just the kibo app there's that was that alone there was four or five different applications, that were running there's websites mobile apps that are all connecting indifferently and, they all speak a little differently and there's, a there's, a great article that I found from Netflix they used to have a one-size-fits-all API. One. API for, every device and they're kind of the because they're on a lot of TVs, and computers, and mobile devices they're on like I don't, know like 70, or 80 different, platforms. And everyone. Has a little bit different need and having a one-size-fits-all, API is is, really. Hard because sometimes you don't want to expose everything and for, us like we had a public, API we exposed to partners, that people can actually go and consume and send commands to their locks and see what they are and then, we had our own API for our mobile, apps which had. Or rights and if, they were all together on the same one it becomes more, difficult so. What we started doing is putting together kind of this API, gateway pattern of you. Know put together an, API now you could use API, management for. Some of this and actually that's, the future where we'll get to in a little bit but. Just building up app services that provide only what's needed for that exact, client and that, allows us to do aggregation. Because a mobile app you know maybe these internal services are very restful we, don't know exactly how they're gonna be used by the front-end or maybe there's two or three services, that need to be aggregated, together to give the, mobile app what's most efficient for it instead, of the mobile app making all those hops back and forth to a one-size-fits-all, API they can hit one of these API gateways that, does the service calls for them and it, provides an interface where the application. Can, give the mobile app exactly what it wants and one, of the things that we started to the path that we started the blaze down is doing all these in asp.net. Core, so anybody could work on them we, have all these mobile developers, that are in you. Know Mac, Linux, I would, say outside of our team everybody was on Mac or Linux and. Now. Any, asp.net, core they can help, you. Know they can help define the API they could even build out, prototypes. And publish. Them of these, app gateways that aren't calling out into our back-end services, yet but, they can basically mock, out what they want and continue, forward with development and we, could really focus on what's going on in our cluster we, could figure out how we wanted to do our micro services how we wanted to do to employment for those and and really focus as a web team on this web platform, so. We use service product pretty heavily almost everything, was stateless. Services we. Definitely do have some actors in place which. Is a pretty. Phenomenal pattern, but if you look at it how this is laid out a request. That comes in these kind of API gateways can handle some aggregation but, each service, has its, own data store now I have these represented, separately, but really a lot of emerges separate schemas on the same database because that was easier but. That also makes it easy to move out if they need to scale separately, and, each of these accomplish, some kind of business functionality within themselves so identity.
And This, is actually something now with Azure, Active, Directory. B2c. We could potentially, completely, abstract and with some other partners coming up with. Open ID Connect identity could be completely, moved out but the concept of registering a user finding. Out you. Know setting up the credentials, finding out if they're authorized, granting access tokens that can, all be abstracted, out, and. Some of these other services can do things like locks, the. Lock service you know locks or more residential, concept, readers are more, of a commercial concept in, readers. I say I, you know for, a for a person I give you a key to my house but I don't necessarily give you a key to my building I give you a card as. Someone. In the commercial space that, card grants you access to all kinds of different doors it's not a one-to-one reader so, building security groups and granting access to a single group that. Service can encapsulate who's. Supposed to have access to what but, then it doesn't actually issue the keys a lot of our keys, that's. An asynchronous, process, that can happen that, can happen off in the background doing certificate, generation, and delivery down to the mobile apps so, it can say yeah here's the state that we know we want to get you in and then send off a message to our service to generate out all the keys. So. This works really well, and. I wanna I want to do a shout out to eShop, on containers, if. You've never seen it it's a repo that Microsoft puts out it's. On github and it. Basically shows hey, here's here's, an asp.net micro, services application. All, nicely, defined and, let's. Take the exact same application. In the exact same project, and let's host it in service driver and. Without. Any other changes let's take the exact same project, and hosted in kubernetes and, see. How it. Works in both of those systems and it's, laid out very much like this with, front-end, applications, API, gateway layers, or. In this case they have services that just do this that are hosted within the same cluster which, is totally. An option you, know these independent, micro services each with their own data store all. Communicating. Asynchronously, over serves of us. It's. A great pattern to follow there's. Actually, a whole PDF, published, about how he shop on containers works they, even have a shop for modernizing applications, where they take old legacy apps and run, them on is, on windows containers and put them into kubernetes, which is pretty cool too so. Definitely. Check it out. But. This is a this. Is really my favorite pattern for for implementing microservices and this has let us go to a point where now somebody. Can work on identity and deploy, an update to identity without affecting anyone else same, thing with locks or readers we can actually work on these business concepts and launch new features, without. Having, to interrupt, anyone else because these, aren't dependent services.
If They, were tightly coupled services, as soon as I'm updating one I'm probably breaking something else so, this gives a lot more flexibility, to work it. Lets people move fast and in a startup that's something absolutely critical so in, the line of moving fast let's talk a little bit about embracing DevOps kick, it over down. So. Like Jerry said, moving. Fast actually that's one of the company Creed moving, fast we say I. Want. To make sure that we're always, working. As efficiently, as, possible in delivering, results, so. When we talked, about moving to micro. Services the. Idea of having all these independently. Deployed, things. Having. To, have. Maybe. A DevOps guy or configuration, manager, manually. Deploy all these things it sounded, crazy. Where. We actually, started, on our residential, platform. Was not actually a strong, QA focus, for our api's. And back-end we, actually have a testing, suite that covers probably. Guess 99%. Of all functionality. Of what our platform, provides, so. After, we, go to our staging, environment we can run those tests, and have, good confidence that something hasn't been broken as changes. Were introduced, so. We want when we were talking about DevOps what does it mean for micro services, well we wanted to retain some of that so some blended approaches, to using unit testing, and still using some integration, tests. At. The time we, also had. To decide well how do we automate these things so it just seemed naturally, that we fully embrace. As. Your DevOps formally, bsts. At. The time when, unity. Is somewhat of a security company right because. The nature of us allowing, doors to be unlocked. Where, we actually started, is running a bunch of that last scene tools internally, a server, self-hosted. We're a cloud first company, everything's. In the cloud but, we did have source. Control important, documents, things like that and our internal networks. So. We actually decided. To take the plunge at this point to actually move to the, vs TS or Azure DevOps hosted, get, source control, to. Make sure that we could fully leverage, the CI CD capabilities. Built, into Azure develops. And. Like we've talked about many times through this talk resource, constraints, are huge, up. Until at. This point that we're talking about the company's history we actually never had any one dedicated to helping us with deployments, monitoring, systems, it, was just the development team that was doing it it, was actually only recently, about the half-year that, we've started into an employee DevOps manager. One. Of the things that has also been our Creed from day number one I often. Like to say that we're on the bleeding edge we. Have been on preview, of moat. I would say most of the azure services that, we use and. I'm not saying that we risk our production, systems and we're mmediately, in production, on it but, we're absolutely willing. To make informed decisions to, leverage the services, that our partners provide, right nazzer is a huge one for us we. Have to stay focused on our core competencies, which is building access control, platform and. We see Azure DevOps as the same thing so we at. The time and, and fully embraced it, using. Service fabric for our micro services like Joey said. Started. Getting all the arm templates, in place as part of our build and release pipelines. Getting. The. Deployments, as part of the building released pipelines. As well and. Then testing, of course. This. Is we don't have a large team still so this is always piece by piece we. Kind of set out with a goal in mind and it might take this week might take us four months but. It's something that we're always continually, chipping away in. The. Culture, is a huge thing for us it I. Worked. For larger organizations in the past, how difficult it be it can be to get like an Operations, team to understand, just pushing code all the time and the, reluctance or a QA team that says no we have to regress the whole platform every time you push anything. We. Have to be willing to change those mindsets, that unique, it's pretty easy for us since we are more of that tech, company and startup feel we. Are still a fairly small team with 45, employees, so. It was pretty easy for us to have those conversations, it, doesn't mean that we don't have to work towards it though. Of. Course with micro services one of the main benefits is deploying, often deploying. Small changes. One. Of the one of the things I mentioned earlier, is that we're a platform that's, actually comply comprised. Of many different products, we have several different mobile, applications, we have all the API is which are actually, you. Know five to ten different applications. Actually in the backend with. All these different moving parks we need to make sure things are treated.
As Separate products, deployed separately, in version dwell and that's, where deployment. Flags come in for us always. Had their pros from day number one whenever we're, adding new features it's, either, based. On the understanding of user by beta and that sort of thing or we. Actually have two deployed systems, we're always concerned with forward and backward compatibility, so, we can actually release things at different times and. Then change basically, some feature flags and our platform to let, those features go live and this helps ease the relationships, with our partners to to, make sure that we can time. Marketing. Announcements, and that that, sort of thing as well that's been absolutely pibil for us, Traffic, Manager was mentioned earlier it's one of the things that we heavily use to manage traffic whenever, we do deployments, we do site a and site B deployments, as an example. So we'll actually start to bleed just 1%, of the traffic over to the new release that's going out, monitor. That so this is a big piece of DevOps as well is you, have to make sure you have continual, feedback of how is your application performing, so, monitoring. That release as it's being pushed off we see a problem we bleed the traffic back to the old deployment, one. Of the amazing things I like to brag about we that we have actually never had, a, maintenance. Window on. That unit key since 2013, that's. Really a testament, to the good design that we have in place and a. Lot of the the way that we manage customers. On the platform. So. With that I'll pass it over to Joey so he can talk about some of the fun times we had yeah, we're. Out a maintenance window maybe. Some other windows we'll talk about in a second so. Pitfalls. Of IOT. Deploying. Your application on, a mobile device is, is one thing because. It's something that you can easily update to point to a web server is actually super easy because you can just push updates whenever you want but. IOT is is different. It's. Getting better but it's still pretty, new and the technology, sets actually really limited so. We were facing some challenges, so. We launched, I forget what year was maybe, 2016, 4 kiba + 2015. 15. Yeah. Again around october/november. Or something like that and. So. What it was as I explained is it was a it was a little box that connected, to the lock over Bluetooth and then Ethernet, to your router I we, always called it the rpu 4 router plug in unit and it. Would basically just make your LOC online so you could send it to command it would reach out to your lock it would report back it reports back battery status things like that now different events when someone comes in it actually let us build an entire integrations, platform off of it because now we knew real.
Time When you came in and out of your door so. We could send things off to like nest, and Honeywell to say hey turn down your thermostat or. Go into if this than that and make all these you know basically anything that you want happen from there because that's a pretty great integration platform or. That also opened up the ability for us to work with like Alexa, so, you can say hey, Alexa asked keep it open my door and then we'll go open your front door. But. We had these. Things everywhere. They were selling really fast we had we were handling tens to hundreds of thousands, of persistent. Connections at. Any, time we were using WebSockets, which. Was really necessary for a real time feel of an IOT product when I have a door lock and I open the app and I click a button that says open the door I don't, care that it's going up to a server in you. Know the east u.s. data center and then, bouncing, 2 3 or 4 back end the Chandler's and processors, and to another server that has a persistent connection down and then the lock is doing a secure negotiation, with the you. Know with the rpu and handling certificates, and verifying everything's good and sending it off I just want, to push the button and know that my door opened and so, there's a really really quick time. That's required they say is about 300 milliseconds, is that you know what your target is that's really the, fastest somebody's really gonna notice if they click it and it happens within 300 milliseconds, it seems instant, and having. A persistent, connection was the only way we actually built this with a polling, model early on and it was like 30 seconds to open a door yeah first implementation, was EDP yeah, and. We I, mean that was one of the first things that we did over a hackathon of like can we do this with WebSockets and we. You, know hacked together a demo 24, hours demo. To like 5:00 in the morning and showed it working in you, know under a second to unlock the door I was like wow this is amazing this is the right path to go. But. We're. Dealing with manufacturers. Kwikset's. A company, that's been around for a really really long time and knows, that it wants to make lots of these and so there's this negotiation, that happens whenever you're gonna produce a new product of hey, here's the chip I want to use now. It's too expensive all, right what about this one no. We got to go over and so. We wind up with a lot of these embedded products we're on the lock yeah. Of course we want the most low powered chip that we can I mean they did some really, amazing, work with the original key block to make it work on for ss4 a year and of that for double is 90 percent of that's driving the motor back and forth, actually. That's a good point we didn't mention earlier that the first quartz at Kiva that we launched actually runs at 80 51 which is a 8k. Ram. Processor. With, eight, megahertz. If. You can believe and we're actually doing PKI on it yeah, so accomplish regular, crypto algorithms, on these tiny tiny machines, that involve some handwritten. Assembly, and stuff but so we still circa, 1985, technology, I think we, got a much, faster processor, for the kibo plus but still really really. Really underpowered. For compared. To like a modern. IOT platform that, you would get off-the-shelf you, know if you go by you. Know any of the products, of support like Windows core IOT you're talking that's going to be thousands. Of times faster. And. Because we were on these embedded devices they're all real-time operating systems, they don't have all these things built, for you so, we're talking about TCP a tcp/ip. Stacks that were pretty custom, there. Was a lot of things like tcp/ip, was mostly there but HTTP, wasn't and so they.
Had Custom libraries for doing HTTP retries, SSL. Was a third party we were using wolf us to sell, because. It wasn't even built in it's not something that as an. App dev that I even usually think about it just comes for free with almost anything it's mandatory. And. Then we had such a low space for memory that. Doing. Things like DNS, lookups we could only do once. So. What they embedded. Team and I know but it's not the best choice but, to get things out of the box they would do DNS when you first plugged it in and then kill all that code and not look again. Firmware. Updates we're really slow and they, were pretty risky you're dealing with updating. Firmware, there. Was a lot of effort put in to make sure that the boot loaders were still good and there was always a factory image so the user if it did get corrupted could go manually reset it but, these are really slow devices and doing firmware updates was was a slow process and then also there's no WebSocket, libraries written for any of these things there, were no WebSocket, libraries in C++, for a real time operating system they just didn't exist we had to write them from the ground up, so. We were way way, ahead of the curve at the time and you know we're talking this is a number of years ago now you can grab something off the shelf that makes this a lot easier so. This gets into my favorite part of the talk based. On all this stuff how, do you do, s yourself. This. Is actually, the. The part of what. I this, this started this presentation I've wanted to give this talk for a while because. This is really funny to me and, sad, but funny so. A distributed, denial-of-service attack, is basically, when you've got people. That are just reaching out constantly, hammering, your servers. From. All over the place so not one computer. Hitting you a lot but lots of devices, hitting. You at the same time and, they don't necessarily have to be hitting it really really fast or really really hard but, coming from a giant, number of devices at the same time can have the same effect and, because, it's distributed, it's it's actually really hard to stop because it's not like you can just go ban an IP and so. We had been selling these kibo plus devices we only sold them in the US. And, within three. Weeks they were in like sixty countries we actually built a map of where people were shipping them all over the world let. Me open to this whole there's this whole shipping industry, around like I'm in a country that doesn't ship to it so I'm gonna, ship, it to this place that'll drop ship it out to me that, I had no idea would even exist some of our even I think there's one New Zealand has an address, that you can mail things to or those forward them to your house it's.
Pretty Interesting. So. A lot of things let. Into this and I'm sorry for this slide being so, tiny but I'm gonna try, and read it so. DNS lookups only happening on startups during Duda memory constraints, the. Next one a partly custom tcp/ip, stack including retry logic at one point we found that we, were handling HTTP. Failures and TCP, failures with the same code, when. They're drastically, different things so. If the sockets just not open and you can't even rewrite it yeah you want to retry but if you get a 500, back or a 503, saying. Hey service is unavailable please slow down it, would just instantly retry at the same as if it was just disconnected, an. Embedded. Development is difficult, you're talking, C++. And C++, not on, Linux. You don't have a lot of the normal system. Calls, that you have available, it's very custom so what they would do is if, they got into a problem the default was let's. Just restart start. Clean it, reboots in like a second so just let. It start over and. Then. Firmware updates were slow and risky so as I said before you know it's we, roll out firmware updates. You. Don't know if something is gonna be corrupted. Without. A great, platform, to handle for more updates which actually we did a lot of effort on to get it to be really really reliable, but. There's still there's a little bit of risk every time you want to roll out an update and if, something did fail a user could go manually reset it but we don't want that to have to happen so. We would have a huge QA process, around every single firmware update so our time between firmware, releases that, you can't just go roll it back if you break it so, our time between firmware, releases was was pretty long we. Had a pretty pretty, rigorous, process to go through and, then, something that that's an interesting thing is users are unpredictable. And not conforming, the user can come in and use your website whenever but. If you're writing code for IOT devices they are perfectly, predictable, because they do exactly what you tell them to do which, sounds awesome. Until. You realize they do everything. In unison. So. One. Of the other things that led into this Azure cloud service deployments, we were using deployment, slots where it stands up one set of VMs runs, your code for you when, you want to deploy you deploy to a new slot stands up another set of VMs and. When you say yeah I've run my tests against this everyth
2018-10-05 12:12
