A Developer's View Into Blockchain Network Architecture
I'm. A constantine, rector I'm the emcee, host of. The night I'm. The CEO and co-founder of block Damon and. We're. Here today with a, fairly eclectic group of people who have, a, lot of experience, in, computation. Formation, blockchain. Infrastructure. Protocol, management, etc etc and really the purpose of today is to talk. A little bit more about, the. Current state. Of blockchain. The infrastructure. Challenges. And the tooling that. Is available to actually you, know build real things and one of the things we've tried to do is to, blend, some. People who have a lot, of experience in building cloud. Solutions. With people who build amazing blockchain. Technology, and so we. Had brought demon and maybe also just a quick introduction I, don't want to shell our services, but we're a venture-backed, company. We're. Described. Ourselves, as the Heroku, for brock chain if you know what to Roku is are, you in the right room this is a software centric discussion but it's a cloud configuration, a tool and heavy. Bit actually was started by the founder, of Heroku, who's, also an investor and we're part of the heavy bit developer. Community and accelerator, and so this is why we're here today we've. Also had a couple of interesting announcements. Today as block daemon we've launched. Our stellar, deployment. Which went live today. And. As. Well as forum but we're very happy to have Jed here, so. He's you know the founder of Stella and a few. Other things and so I'm, quickly gonna you know so everybody. Gets a chance to introduce themselves but, I want to say a couple of words about everyone so Jed is one, of the people that you know inspired me, a lot as a you know true developer, who probably. Is one of the most, influential, people in the space that you know the least about um he, you. Know started and founded ripple he worked with mount galaxy had a bunch of different experiences, and spin in the space for a long time so we're super happy to have him here to, my left Brian who was the executive director of hyper ledger you, know a man who has a lustrous, background, in the open source, unity with, Apache and a bunch of other very. Exciting projects, I've been Avicii you, know been, in the space for, a long time and then at the end. Jesse. Robbins the. Myth the, man, anyway. But Jesse's a partner at heavy bid their investor so and. So jesse. Is also, the founder of chef he. Was instrumental in, actually, setting up the AWS, cloud infrastructure, business as well as a bunch of other things and Jake, here is one. Of the lead engineers, at a small blockchain, startup called coinbase. They're. Doing. Pretty well and so you. Know the reason why he's here is because these, guys actually have to manage infrastructure, and nodes where you know if things don't work. Breaks you know that's and so, I kind, of feel. We have a good section here I I don't wanna this. Isn't too like kind of sort of scripted so the goal is to have a sort of a little, bit of a you know I want, to start sort of reminiscing, a little bit about the open source sort of community and how you all guys started, you. Know becoming developers. And engineers in, the space and and what. Got you excited about really open source systems, and. So, maybe we start that way and we go just sort of down the line while you introduce yourself so say what you who you are if, I've missed anything substantial, and and. Then talk a little bit about what inspired you about open source and and and you know what you currently do in blockchain and then we'll riff. On that sure. So once, again brian behlendorf executive. Director of hyper ledger I kind of call myself the nerd diplomat, though because, we. Kind of add a hybrid, or part. Of the Linux Foundation so. Our job is try to pull together two, kinds of communities, communities. Of developers, building, underlying, technology, for, distributed. Ledger's and smart contract systems the, second kind of community are the companies, like block daemon like IBM. Like Baidu. And $0.10 like, all these startups around the world who are building and want to build products, and services on top of these technologies, or just incorporated, into their products and so, I that, you know the Linux Foundation is kind of pioneered a certain model about around, how to get kind, of companies, to work together as, kind. Of a consortium. If you will not, so much a charity but recognizing, the fact that mo most open-source development actually is done by people working for a living like working, to fix, the bug or to add a feature for their employer and, how do you really harness, that how do you channel it how do you try to keep the amount of unnecessary, disputes.
Between The, companies, or between developers, to a minimum right by kind, of providing, air-traffic control. I don't, actually employ any developers, nor do I write any open-source, code these days and the world is much better off for that but. I got into it when I was. Did. You got I got onto the Internet in 1991, as a freshman at UC Berkeley and I, it. Was amazing, to me that this thing had been built. This internet it felt like a secret universe, right you know this portal to a place where you could send messages to other people on the to people on the other side of the planet for free right, you could access FTP, sites or talk to people over IRC, or. Read. Usenet, or mailing lists right, all of this seeming. Infrastructure. Seeming to have emerged, out of out of wood out of people writing protocols, and white papers and and. Standards. But not only writing the standards but writing and giving away software, that implemented those standards right, and it was software you could inspect you could like crawl inside and see how it worked and if if, you tried running it yourself and I quickly got a job as a sysadmin for, the, business. School at Berkeley like again, just kind of goofing off but it gave me a chance to like crawl, inside this code and when you find a bug and if you can come, up with a fix maybe submit it maybe it's the right fix maybe it's completely, the wrong fix but it provokes, somebody else into actually doing the right fix right. And so that culture, of people. Sharing code long before the term open source was invented which was roughly, 1998. Long. Before Apache got started which I kind of fell into again, because it felt like we were continuing. This tradition of people working together to build common technologies. Long. Before all that this is how we got Internet software and and I dare say it probably appreciated. Proprietary. Software in, the earliest days of how people kind, of figured out how to make computers do what they needed and. So continuing, that tradition when, the. Web started to grow up there, a bunch of us running websites. Built. On top of the pre-existing, NCSA, server this is the same team, of student, intern, developers, working at a University, of Illinois who built the mosaic, some, of whom went off to start I don't know VC firms like a andreessen. Horowitz right but. The server side of that the students all left we all of us using that software got a note one day that said good news all these students went off and got a job at this new startup company called Netscape, and we were like congrats but. We're still using this old software. And. So because we had been exchanging, patches with each other anyways, we just decided to keep doing that and called, ourselves Apache, and and eventually started a non-profit around, it but that was it was felt like the most natural thing in the world it's like you riding a bicycle, and you just keep riding that bicycle so, how. Many people in this room have. Contributed, to or or built, a product, on an, Apache licensed, product. Okay. Pretty. And almost, all of you regardless, of whether you use a Mac or the, Internet, or, pretty, much any or an Apple product or a Google product depend, on the. Fact that Apache Apache Software Foundation, exists. Produced a license. Model that, allowed us to write and share code freely, to deal with issues like patents, and other things all really, hard and complicated, now has solved problems, at least in.
One General category, and. It's interesting because the. History that Brian just shared is. Often lost when when. You think about like how things get built now how easy it is to build things, now we depend, on a foundation that. That, many men and women built, and that, they. Built over, a period of many years that, came from the, same energy that's starting to bring you into this room and I'll give due credit to the free software foundation which, actually started in 84 85, something, like that so that also helped lay a lot of the the tracks, that we were able to drive on but I think with right, around when Apache got started in 95, there was this recognition, it wasn't, about charity, necessarily, wasn't even necessarily about ideology, about, the, world has to be free software, it was instead a very pragmatic sense, that by you, know this underlying, infrastructure, stuff we all have to use that hopefully. We can all have the luxury of ignoring you. Know you don't worry, about the plumbing in this building, for example right maybe we should I don't know but. You. Don't have to work out, like like somebody, else is dealing with it well software. Is some software is infrastructural, some software is in your face the more underlying. Infrastructure, software we can develop, as communities. The, more time, and money we can all spend on the truly differentiating, stuff that, gets lost sometimes in the world of I SEOs and the world of everyone building like really, bespoke blockchain. Infrastructures, that there's a lot of commonality, between these technologies, and we should be working more and so that's that's kind of what hyper ledger really at the end of the day is about. Jake. You want to say hi and yeah, I'll try to follow that. Internet. And. You use Apache based softwares among other things I'm not sure yeah and so yeah my first sort of foray. Into sort of open source was when I had started doing rails development and, the fact that this sort of framework existed. In this space and it was you know open source and you could see what's going on and you could see just, this huge community around that, space and all the stuff they provided, that I could just go in and and. Use like, I I worked, primarily in consulting, before joining coinbase and so I've worked with a lot of different companies and most of them that we worked with are using rails and able to really spend up these things very fast and have all these features they want really easily by just adding this gem here and there and being able to do that and then.
It's. Just a really really cool thing and that's what got me interested in it and then from there one, of my first like open source contributions that, have I think some some value was. With actually the ember CLI project, and so, there was a really cool thing that happened there where, I was just working on ember.js, applications. And I, was building this thing to integrate with this thing called Cordova, which I think was Apache right is now. And, so we were using cord we're using Cordova and we. Needed some tooling that integrated better with ember CLI and so, I was building this separate library and then would happen was like Stef and Penner and from. The you know aetherium core team saw the repo and noticed, and said hey like how about you add this as like an add-on or something, how about you like add some hooks and never scale I so you can actually do this your like yourself with the tool chain and so it was just this like reaching out of him on this repo he just opened an issue that. Led to being able to actually contribute to ember CLI and sort of put in the first, steps of this sort of add-on system that they have today that's. A really powerful thing and so the fact that I was, just able to just somebody, do you know what this small little company. In Houston be, able to contribute in this big way and then now the software is used by you know thousands, and thousands of people around the world I think, that's really powerful and the fact that you can see this and you can learn from it I think learning was huge being able to work with these other developers, who knew more than I did and be able to learn and see how they did things the trade-offs they made and bail have those conversations with them and it just sort of be this community, together that pushes sort of the world forward and just shares this software and says hey you can use this and you don't need to worry about this part of it and you can just build the sort of thing that you want and that your customers need, great. Oh good. Yeah, let's see, my. Relation I the. The my, first big open-source projects were definitely in the blockchain space prior to that we would always have this decision, like whether we should open source or not and like the, one of the cool things I like about the watching spaces there's no there's no decision I you you definitely have to it has to be open source because like, Brian is pointing out it's like this open, infrastructure, that that is this, kind of this underlying thing and people want to be able to see it and it's used by many, many parties and so, everyone needs to have access to it essentially and so that's been really great and one of the other cool things I think may that. About, this space in general is that what you're building is not just software but it's these these, open platforms, and so it's, not just people contribute, to the code but they're contributing to the actual ecosystem, so there's just a lot of different players that are involved you, know contributing, all kinds of ways which has been, pretty exciting, because it's kind of like you're on a big team not with your own just organization, but with organizations, at large so that's been pretty inspiring and fun so yeah. Cool. Jesse, do you want to tell. Us what you would you know what, is your your, role. Well let me so. So. I I am, motivated by a belief, that developer, technologies, and tools have the power to make a better world and, so. In high school my start I became assisted men and, I wrote a paper in in, high school which was before. Open. Source was a thing it. Was I want to work with a big group of people that are gonna make the Internet technologies, and protocols, that for, this thing called to quote the internet and. And. I really, described, a vision. For wanting. To participate and, help build organizations. That make, things that make things better. My. Role at heavy bed I'm one of the part-time partners, so I have the good fortune of now helping, other founders, and teams, avoid. Many of the the problems, and challenges that, I learned is a first-time CEO. At chef and, often. Provide a lot of the experience, that, that. Comes from building a you, know what is now a big company and having worked on building, big infrastructure, and other environments, so that's. That's what I do why, I'm hoping that everyone is here is to. Learn, how to embrace, the next wave that is coming and either, build on in, around, or, contribute. To a, set. Of tools and platforms that, have a potential, to change the way and improve lives for the next, decade great. And so. Thanks. Guys and so maybe one, way to. Talk. A little more about open, source is the, concept, of decentralization, and so I'm curious, what that means so we've got two guys who you, know kind of build protocols, that are open source and and, our, block chains where the the. Word decentralization. Is supposedly, meaning it, seems like block chains either can be built for speed or consensus, mechanisms, and there's this trade-off in, between, so I'm like kind of curious, what you think about the, you.
Know The sharing of infrastructure, that's needed in order to actually have. A. Decentralized. Network and how important, that is for you and. And. What that means for the security of the network and then, actually, I'd be really curious to hear Jesse. What you think about you know the the sort of degree to how. That translates, into you know cloud formation, and that world that we came from and. So I'd be curious to hear about that from both, you guys. I'll. Start sure what. Does decentralization. What is the centralization I mean that's a pretty contentious, argument, now and there are people who take fairly, binary, kind of points of view on this so you're not decentralized. Right. There's. A lot of kind of like tribal, is I'm a single power failure. Do. You think a lot of people have felt that the last 15, years on the net there's been less. Decentralization. Of the Internet as a whole right there are there's, fewer places where you can get an email account a fewer places to go get caught up on news or to build you know I you, know a group. For your family you know to share messages that sort of thing like it feels like a lot of this has started to centralize. Or, fall, into a small number of parties right and a, lot of very smart. Very active people who for 10 years have been trying to build ways to read assent relies the net and I think a lot of people read in Satoshi's original, white paper Wow here's a way that helps. Counter, some of the scale advantages that, you get from centralization. Right because, centralized, systems, actually are incredibly, cheap incredibly, scalable like, we know now thanks to a lot of the hard work of open-source developers, how to build, central. Infrastructures, to support things at the scale of a Facebook or a Twitter right it's actually harder, to build more decentralized message, systems or decentralized, databases, those sorts of things and so in. That vein a lot of people been trying to ask how do we tease this apart and, build, more decentralized systems, right and I think some of the cryptocurrency community. Had taken the the perspective, of if you go all the way to decentralized, and, say you know that the only source of validity in, verifying. Transactions. Is the proof of amount of CPU time that, you've been able to successfully burn, on a trivial problem then. That's, perhaps the most fair way that we could think of to build. A broad. Network without anybody in, the center but, I think they forget that there's still a governance, involved, in deciding what software to run on the nodes wouldn't saw how that software is built the, debate, for example in blocked size or. In the etherium, community between bailing. Out the the, DAO hack and not bailing out the parity hack right is. It is basically, a sign that there is still governance at work and I think you can either recognize. That up front and have a structure, for having, decision-making, inside of a community or you can try to pretend that it doesn't exist and you end up with Lord of the Flies and how do you do it for fabric well so so, there's that side of the world and then there's another side of the blockchain world which has been about saying okay let's say you have a formal recognition, for, the role of a governing entity in in, a, network right where, I it the metaphor I've been using is it's kind of like having a referee on a football field alright or a soccer, field somebody, who basically. Grants. Certificates. To participants, to, join a network and then once you're on the network you can participate in the consensus, you get a full copy of the ledger that's being created, you can write transactions. Frankly, that ledger could also be public read/write but. You have these trustworthy, nodes that are basically able to submit these transactions, to a common pool I think, that's still decentralized. If you. Have the ability to fire, the referee if they start making bad calls right, if the referee you, know can basically make, up rules as they go and they can throw yellow flags because they don't like you or, they can charge you a hundred dollars as a player to get on the field that's a bad referee and you should be able to fire them, so.
I Think my, hope is that we can use blockchain, technologies, to take a lot of business. Networks out there that today are very centralized, in operation. That where you do have to trust some party at the center who charges a 40% fee for the service by the way and. They keep all of the systems of accounts clear and you have to trust them and. They are the ones kind of at the end of the day and the point of authority and, instead tease those apart so that were much more more, decentralized, more. Individual. Competing, players in a field but still able to play a game and, hold, that governing, entity to check and I'll end with that's very much like the degree of decentralization. You have in the open source community open, source projects, the one defining characteristic, of an open source license, is something called the right to fork which is the right to say hey there's here's a team building code I disagree. With them I've got a better idea they. Might disagree with me that's fine I can take this code and go in another direction right, or if you and I have a dispute right we can part ways and that's actually a way of lowering the risk of working. Together on on. A piece of code because, it means one of us doesn't have to throw away the, investment that we've made and. I think blockchain, networks certainly, the public chains work the same way that's how we have Bitcoin. Classic and or theorem classic, and and all these others right, but. I yeah. I think I think we'll see that phenomenon, on both, the permissioned and the information Ledger's said. I'm curious because Europe, is the you've been, instrumental in starting ripple ripple. Is one of those protocols. That gets a lot of stick about is it a blockchain is a decentralized, you've. I, think, really took kind of the open-source components of ripple are created Stella that actually was an enhancement, of that what, was the thinking there and what's your view on on, decentralization. And what's a blockchain and not sure yeah I mean I think I, mean obviously there's a spectrum of decentralization. You, know I don't think it's, not binary for sure that there's like and not only is there a spectrum but there's lots of different axes you know there's like like. Are, the people contribute as a code is that a set, right, group, of people are the people running the servers is that a set group of people there's like the governance, like there's a lot there's lots of different. Aspects. To it and and you can vary on all of those and it's it's really really hard to be like fully, decentralized where, I don't, even know what it means to have like nobody control the thing like because we're you know there's ultimately a theme of the day there are people running this software it is software right so somebody has to run it so.
Yeah, So I mean it's always kind of a trade-off and it also depends on like what your what your goal, of your of your system is like in, many cases decentralized just doesn't even make sense like you're, at the end of the day if you're having to trust like like, if you want to issue an equity in this decentralized system at the end of the day you're trusting some company, to redeem that equity and like and. Like turn. That equity into something so there's a party that you're trusting so does it really make sense for it to be a fully decentralized system, so, you. Know there's a lot of weirdnesses, like that that you see in the blockchain world where people are actually like trying. To make something way more decentralized and it will be at the end of the day so yeah. And let's. See what else would say yeah I mean I. Mean. That's it I mean I guess and then like in terms of like what's a blockchain what's not a blockchain to me like what the the defining characteristic. Is is there, some like public, record out there that that people can see but but you can't change arbitrarily, right so it's the it becomes this like it. Becomes essentially this trusted, third party warily. Like what you hear in the dough heck yeah, well so that's like that's like a that's. A problem, so yeah. But. Yeah what it means is like you know if there's some database like like Google is running a database for all their email right Google, can arbitrarily go and change your email right they probably don't but they have that power whereas. It you know with like Bitcoin, no. One can just go arbitrarily change like your balance because because. It's requires, all this proof of work and all and that there's this chain of hashes yada, yada right so that's, what I mean like there's like this public record that people can see but, but there are certain rules for how it gets permutated, right and once. You start varying those rules that, then you, kind of deviate from from. What. It means to be like this boxing I think, that's the important characteristic, rather than the, actual underlying consensus. Mechanism, so, yeah. How about Stella so what was the thinking there in terms of even. Enhancements, around ripple like what was its a foundation, it's like there's structurally, some, more decentralization. In yeah I mean so the idea is that I mean, when, I first learned about Bitcoin I was super excited about it I think it was like I never thought this was possible to solve this problem before but. We, want to things that's always bothered me is just like how much. How. Wasteful it is and so I'm just trying to think of other ways to solve this consensus problem and again make this thing where there is this this, public record that people can see but it's, not just arbitrary to change and so that was kind of the idea is how do you how do you come up with another system that can solve that same problem but maybe in a more efficient way or way that's tailored for a different, kind of different use case and. Then, along with that is like once you realize you have this there's, other a lot of other problems you can solve besides being just a new internet, currency you can do. Lots of different it's. Dollars trying to do so yeah thanks. So, Jesse what do you feel like as a sort, of guy, who comes from cloud. Services. Enterprise, to laying, you. Know chef like what, do you think about all this stuff these decentralized. All, this stuff like you know two, three words the Internet's built on a on a distributed. System called, BGP. And. It's, a routing, protocol, that's how packets, find each other and, it's, a consensus, oriented, model it requires global scale it requires also the participation of multi, parties and act and actors, there's hostile, actors from time to time that, will change Internet routing tables, this is how, how. You, know a packet gets from point A to point B is a complex, thing that it requires humans. And machines in, the loop doing, stuff in, order to agree, on how we we, send data around. And, another. Version of this is DNS the way that we find what a hostname is that turns into an IP address that routes through the BGP routing tables, and, all of these things are our. Systems, that were, born from, the from, a decentralized. Early model they have tended, towards, centralization. As I, find. Ironically, that humans tend to centralize we, are anti and tropic, whereas. Where. As you know the rest of the world tends to you, know work the way the rest of the world works so we, seem, to be organizing, machines, um there's. A couple of things that I over. The years have noticed and. Trends, that I see one, thing to know is that I am a firefighter EMT. By, training that, has been a thing I've done throughout, my career and, speaking.
As A firefighter, in this room I want to talk about some things that have emerged over time the fire safety issues yeah, so. So, here, we are and we have assembled in this room and this is a converted, warehouse space and if you look up above you you will see that there are sprinklers. And if you look over. There you will see that big red hydrant. Stand that comes out now, that is part of a decentralized, fire. Suppression, system that, has emerged through a system of codes that. That. Are required, for buildings, that have occupancies. With a certain type of risk now, the fire department, did not install, this there is no central, install. Installation agency. For, this sprinkler system this is installed because. There are safety codes that, have emerged from, one thing and one thing only which, is a loss of life no part of our fire safety code has, emerged, that, and been widely adopted like, sprinklers. Without, a significant, loss of life event. That then causes people to go we need a regulation, that says if there's this many people in a space with this type, of construction you got to have sprinklers, you, got to have exit doors that'll that work in a particular way that is a part, of how occupancy, models, work and as, this as the systems engineer and a person that has built a lot of infrastructure, I have observed over the years and in fact been responsible, very early my title at Amazon was master of disaster and I owned, website availability, for every property that bore the Amazon name so I I know, about, this stuff and the. The big thing to understand, is that as we start to build these these models, in a distributed mode we. See the evolution of what Brian described, which. Is you, know you have groups, of people coming together with. Rough, consensus, and running code and, over. A period of time we build systems that become things, that people depend on and then, every once in a while we have events, that cause a loss, a sigma, that and the greater the loss the. More likely it is that you'll see calls, for regulation you'll, see calls for these other types of changes, so, one, of the things that I look to is, thinking. About the world often like we think about fire codes, sprinklers. Are expensive, and that means that buildings, that you know if you want to open an artist space, for instance in Oakland and you, want to be hackers who have form a collective and you, get, together and you build all kind of beautiful artwork and that other things and you do that in a building that doesn't have sprinklers, and then there's a big fire that results, in in, tragedy, right and so.
That's, Why, when we go oh well some, things need to be regulated some things need to have safety. Standards, emerge, when. You're kind of thinking about it with your hacker mindset, you. Begin to understand, that the more people depend. On you the more that people depend, on a system, working, in a particular way, the. Less they can they can really be responsible for their own safety in depending, on that system so, what I expect, will happen over time is that, we will see the emergence, of safety, standards safety protocols, that become common, whether we're using a ledger for. Storing, and sharing data, publicly, as you're describing or. Whether. We're using it as part of a financial system or a transactional, model what, I believe is that we will see a distributed. System emerge, very similar, to, the way that that BGP routing has. Emerged. And standardized, the way that DNS has emergent standardized unfortunately, it is now centralized, far. More than BGP has and. That as cloud standards, and then the. Watching, related. Standards start, to emerge they will be based around the idea of just simple. Concepts. Like oh this, is how you can audit this is you can see this, is how we can make sure that systems. Work that they have. Minimum. Understandable. Risk models that are mapped appropriately. To the work and the value that they are doing if you're staying in a tent you don't need a sprinkler, system right where you don't need sprinklers when you're camping you need them when when there's a concentration, of risk and. The people or the things that are exposed to that risk don't have a method of evaluating, it for themselves so that's what I that's my my, sort, of longitudinal, perspective on. On, how we. Will see see. Both. Standardization. And centralization. Occur. Over. A period of time right, great. I want to stir, us briefly. And we're gonna open up for questions and you know we'll have time to dive into like different subject areas that we've touched on but, let's talk briefly a, little bit about the current sort of technological performance. Of the, infrastructure, and where we actually currently are instead of blockchain land. Jake. We were speaking a, little bit about the the you know Columbus obviously runs notes there's, a lot of you know stuff happening on coin base. And. So, I assume and. So you know the question is really you're. Probably you, know you probably have the first-hand experience, of the most the. Edge case of running, notes and and keeping a ledger up-to-date and running queries and and and, how does that infrastructure, look and and how, much do you lean on centralized. Tools or you know existing developer tools to help with your, current, infrastructure. That I don't know do you run a few have nodes for each currency that you rhond, do you outsource any of that like what does that look like yeah, yeah we'll talk about that so yeah.
I Mean we run all their own nodes we, build them from source we don't rely on external releases, we you know check. Hashes and all that kind of stuff we, use docker so, docker power is basically everything we do so it's a very you know core, open source component to our stack and, allows us to sort of build these in a really good way and get them them shipped for. Actually they're running of the nose themselves there is still tooling we have to build on top of just running a node like running a node isn't it can be just downloading a single node and then starting it up and then you know letting it sink the, problem with that is in, our system you that sometimes need more nodes at some times you need less nodes and so you've got to be able to start those up and you can't wait you know, a week, or a day or whatever it may take to get the note up you've gotta be able to get nose up in a reasonable amount of time be able to handle the sort of increased amounts. Of traffic you may see and so, for that you have to build in these systems that will take you, know the entire data base of the node which like an aetherium sort of you know worst, case to be fair of this sort of archive know that stores all the state history that's. Right now around like 1.2, terabytes. Or so and so. Sort of storing that and then shipping that around like, your cloud platform. For. Each instance, you need to do is a lot of a lot of work it's a lot of data transfer it's a lot of storage that's out do you use by the way I'm just creative yes it oh yes how, much you pay a month. I'd. Say it's a lot and probably, I mean at least unlike the crypto side of things probably a lot for just transferring, backups, from one to the other any time we spend up a notice like all right transfer, a terabyte over s3 or something we've. Got some new things in place to that, we're working on and make that a little bit better so it's not as a just, this like raw transfer but using some like really nice Amazon, features like the. Elastic like lock storage like EBS stuff which. Are you really cool but that's like it one of the core things that like delays getting nodes up and getting them available the. Second thing that I think is really critical in sort of our world is we want sort of redundancy, we don't just run a single node and expect that to work the whole time because if it doesn't then that causes problem people can't send their payments, anywhere and they just they can't receive them there's a lot of problems so we need redundancy, and for, that okay you throw a load balancer in front of it but now all these nodes are completely isolated from each other they may be in different regions and maybe in different lazy's. And so because of that you have this sort of eventual consistency problem. And this. Is a really interesting thing because we have to take all the data from the blockchain from the nodes and basically. Process, it ourselves because we have a much bigger set of data to look at we need to take you know hundreds. Of millions of addresses, and take, a look at those and say alright do any of these addresses this is getting sent to are they ours and the nodes aren't built to do that so we have to sort of do that ourself so running query is across that, network is, how.
Do You do that like so yeah so rather than running queries for most things we actually just kind of pull all the blocks so in, the like all of our knows right now we're all like block base you can take every block pull all its transactions, and just inspect them ourselves so we're sort of doing duplicate work here like the node has already done this and sort of updated, zone systems but we now we need to take that employee into our system because the nodes are not meant to handle hundreds, of millions of addresses like, they're. Not built for that they're not built for our use case and they aren't built for centralization. Like, you are essentially you are the central part of a decentralized system. My. Laptop doesn't have 1.3. Terabytes, of storage you need a new laptop. To. Get cheaper I guess yeah, II. Know yeah. I know so, that's like a big thing for us is like being able to pull this data and it's all like separate and they're yeah they're not meant to run in these centralized, worlds but I think, the future is there's gonna be a lot that are running in these worlds there's gonna be a lot of companies that get. Big and that they need to run these nodes and they need to run them in a way that's efficient. And the. I think there will be specialized, sort of software for this that's built for this case and I think we always want to have this sort of base case where you can't run it on your laptop where you can't spin up your own thing when you can take, like what y'all are doing and like run it on your own AWS, world I, think that's a really important thing to maintain but, I do think that for this sort of scale like it just seems inefficient, the, way that it works in a centralized, system and being able to process that and manage that for people hmm, Darren Oh makes sense and we're talking, earlier we're part of a, skype, group of a hundred, xx. And, of. Exchanges. And wallets where you know we talked exclusively around, if you're in get notes not syncing and stuff and so not not do you know that's not like how do I not, get added to that group yeah. You. Love what it's. A but it's a you know it's fascinating fast, as a sort of solution provider in that space to also learn that you. Know the most larger. Centralized agents obviously don't want outsource any of that right so they they they'd like to maintain the. Centralization, there and that creates you know an enormous amount of cost and stuff how many people work in the infrastructure, sort of on that side in co-invest, you know like I'm just curious, probably. Just you. There. Is a team we, have a full infrastructure, team dedicated to sort of keeping infrastructure, stuff healthy as a reteam for that kind of stuff but outside of that it's mostly the crypto team which i think is 10 to 15 people or so right. Now and we kind of keep an eye on that and as we grow we are sort of figuring out how to sort of reorganize the teams to have like specialized sort of, where you know we might have this, world where you know we have our sort of internal team for for spending up nodes and that we can do that efficiently and they're able to maintain that and provide really good sort of metrics that's another thing that's I don't think ideal, in the current state of node software, is sort, of getting insight into what's happening within the node we, might see some weird case that seems odd to us that doesn't match our understanding, of what the blockchain is supposed to do or the way and so we, don't really know what's happening we might log data out and we're gonna log things from the nodes but what. The node logs and what they mean aren't, usually documented, they don't usually they. Might be vague. They. Also just log straight to standard out so doesn't make it as searchable there's no sort of formatting you can do on the, software that I've seen at least the software we run and, so there's a lot there and sort of insight into how the node is performing what's happening there sort of metrics and being able to actually have a clearer picture because for the most parts of black box and we we trust these things to. You know power our entire business, really and, without.
Knowing That it makes it harder to run these things and know, like when things are wrong and know what's wrong other than you know trying to see like oh hey it's not sticking blocks why. Here's. Some logs. Yeah. We. Hope somebody can can help out with that I'm chuckling, over here when, it was we were getting, ready for the panel I was talking to Brian and so basically you. Know that so heavy heavy bit exists to help developers, build developer tool sets right and that means, that you have to walk through whenever, there is a major change and how infrastructure, works and the tools, that we rely on there's. Going to be a bunch of technologies that or companies, or both, that, emerge that have to instrument, the entire stack right here you are describing your orchestration, layer and, like oh we use these docker containers, in order to you know soap that sounds familiar and we do that on AWS okay that sounds familiar and then oh it's really hard to instrument, and understand and inspect what's actually going on and if only we had and so what what's interesting is is that there's this perfect repetition. Of, every. Cycle. That we go through we're you know pretty soon they'll be like the Splunk for blockchain and the bla for blocked in the Heroku. For right, we talked to. And. So, what what's interesting and important, so. The problems that that, coinbase has at scale, which, you. Know are the problems, that happen, when a platform, wins are. Different, than the problems that many, of you when you're starting and building a first, project or maybe in an entirely different space, are gonna face and. There and so you know the advice so like I can give people and you can give people about operating, at large scale systems are interesting, but, there's there's this other side of it which is like how. Do you think about the the space as a whole so, one of the things I'm curious about is kind of as people think about the tool chain like what, are the problems, that that, you see that that you can take from systems, management and you can apply to decentralized. Systems management, like what is the tool chain that you're looking for what's where's the white space where you're like if only I had, bla I would not have to do I would, I would get way more sleep, right, what's what's what's waking you up at 2 o'clock in the morning on your on-call rotation oh yeah.
Currently. Ya. Know it's it's being a little weird for us but, no I, mean, I think that stuff that block team is doing is pretty cool because, like you, mentioned you know what if you are you know starting some new business and you need and you're building it on the blockchain and you need notes like, yeah you can just go manually, configure, 8 of us to do something and like, hope it stays up and hope you don't have to restart and resync the blockchain or like change some configuration, and get a new version and, you need to auto scale it up and so you need to willing in place for that and I, think like y'all are doing stuff to make that better and there's I think there's probably a lot of tooling that can be built to how to automate, that sort of solution and to do it for, you by just you know wrap your you know process and this thing and it will automatically go take the data dear and like pull that off somewhere else do you build your own metrics do, you have your own metric stack or you how do you do your monitoring how do you like, once you have deployments, up, how. Do you how do you operate, the the platform that you're running yeah, these days these. Days most logging, and sort of metrics, logging. And all that goes into cabana elasticsearch, all that kind of world and, then we also use data dog for specific. Metrics and and tracking that and then we do do some of our own custom, instrumenting. From what we can via like our pcs on these nodes to, take them and say you know what is the block height so that we can graph that and see oh it is like freezing at this period of time, you know what's the memories what's the dis space how, many you know peers does it have sort of the, information we can pull from their PCs we can take out and try and use that you know do some monitoring and alerting and such on top of it is is, that something that you, think of as being proprietary. To coinbase or is that and let, me know if this is out of scope, the. I mean, I'm interested, because everyone in this room hopefully will be building technologies, that operate at your scale, right and so, there's there's a certain amount of operating experience, and exposure that that you you uniquely. Have in this room and so. The question is like do you think that is. That toolchain itself part of kind of the value of building a blockchain, business. Separate. From the monetary like. The fact that you're running and exchange the way that you are but. Like is that is that sort of table stakes or is that something special I think, it's an important thing to build a business unless you want to sort of reinvent the wheel and I think there's a lot of sort of a lot, of this stuff can be built in sort of an open source world and do it and I think like we would love to do more open source things as as needed. And it's, one of things like yeah it's you're gonna need it if you want to run your notes the other alternative, is to like, use a company using fira to run your notes and I think aetherium you know majority, of it's probably powered on in fira it's a very centralized, entity and like they haven't done anything bad yet but they could and so if you want this sort of decentralized world, you, need to you.
Know Be able to run your own nodes and so, right. Now it's really hard and it's. It's not that easy to do and it's not easy to do reliably, and like if your system goes down and you're not tracking that your nodes not sticking blocks anymore for some reason um it has no peers like nobody's gonna tell you that the notes don't have things built-in to tell you that so like you have to track that yourself and so, you can totally, build tools on top of this not. Do that kind of stuff for you and just wrap it all up and it doesn't have to necessarily live in the node there. Are some things like like the logging kind of stack and all that that I think are something that could be improved, I'm just help like better analysis, of, sort. Of issues when they come up rather. Than just like raw. Standard out strings. Logging. It's out of this all the fost Ella what was your thinking there how if you develop that architecture, how do you how, do you monitor or the yeah, work you, have an exchange as well on. Stella, right well yeah I mean it's built into the network but yeah I mean I think I think a lot of issues are because, the this software is not designed for this use case it's not designed for to. Be run at scale like we were saying before and. The people that made it was more just like someone's gonna put, this on their laptop it's like it's like a descendant of of Bitcoin. D basically all these most of these projects are descendants of Bitcoin D and that's you. Know just going to you the first Bitcoin right node slash client and honestly. It just wasn't, really written with. This. Kind of use in mind like it's combining the API with, the, validation and all the stuff is just doing all these kind of weird things where it's not like separating, the stuff so basically, when we design, stellar we we kind of tried to keep that more in mind like basically, our API server this whole of software. You. Know stellar core just writes out to a database that that, now then the API can like access, so it's much more easy to like you know put load balancers in front of it and like and like scale, it out and then we also like emit all these metrics stellar core does so you can kind of see what's going on like what parts are like slow and all this kind of stuff so we, tried, to keep, the. The idea. That eventually, you. Know it would be cool if everyone runs this on their laptop but that's just really not the way the, internet works like anyone, can run a mail server but in practice only a few companies do and I think it'll be the same is true and then we used, to run mail sir right.
Run. Your own DNS server but you don't like you let other people do it so you kind of need to make this like your, core software, be able to be run by enterprises essentially, so yeah. Cool. What are you most excited what's next for Stella also just Fry's, look yeah what are you what are you excited about in it like what's your where, do you spend your day yeah, technically well we're hiring about right now that's where I spend my day unfortunately but but but, basically for, us like we're really focused on scaling, right now I mean I think all of these, all. Of these blockchain technologies, none of them are really none, of them could fulfill their total promise yet right like obviously there's huge scalability, issues with aetherium right now and like will, eventually hit them I mean we have a lot of room to run but I mean I think. Not. Like they're all kind of aspiring to be this like global, distributed. System, and and none of them could really handle that load at this point so that's like our number one concern is like how, to actually make it get there so that's what we're. Time on we take a kind of a different I mean, hyper-alert, even before I joined kind, of staked its a flag, in the space, that said maybe, it's not one, giant, ledger that everybody kind of lives within and writes transactions. To write maybe it's not one global, computer right maybe. It's actually a patchwork of, a lot of different Ledger's that exist out there that, are fired up easily right between a set of parties you realize they have a common set of use cases that. I might. Have a one different one, ledger might differ quite a bit from the other and choice, of technology, but also very, prosaic things, like how frequently do you upgrade, right you might have one set of customers. And one set of applications, where they want just ultimate, stability they don't want things to change you know more than once a year and be heavily tested others you want to update every week when there's a new release right one. Who wants full a turing-complete. Programmable. Environment, for there are smart contracts, others who say no there's just three kinds of contracts to have on here and we'll audit. The the. Code. To that you know thoroughly and we'll just lock those down right and. So this idea I would wager is actually more decentralized, than a notion that says everybody is on one set, of rails and I think it's, you kind of do need that when you want payments, because you want payments to have as large an audience as possible and be as portable as possible but. So you really do want things that operate at the scale of a. Bitcoin aetherium a Steller's those sorts of things, but for a lot of these kinds of other applications. Out there for blockchain technology, and supply chains in, in. Healthcare in all, sorts of bespoke financial, markets like, it's perfectly, fine to have a blockchain comprised, of 20, to 100 nodes right, processing. A couple of hundred transactions, per second that's, the masked, majority of use cases out there would be more than satisfied by those as long as you have ways to tie transactions.
Across Ledger's, to do discovery, of which Ledger's are out there and I'd say as long as it's always easy for that, 21st Bank to join a network for that one, hundred and first healthcare. Organization, to join a network so. There's there's a mix of tech and politics, in those kinds of like questions but. I think we're gonna end up with most of the blockchain world looking, like these kinds of easy to spin up more. Manageably sized kind, of ball chain networks that might have lifespans you know they might even be rolled up at the end and some net settlement, and become, basic, a historic. Thing and then restarted, again partly, to address some, of these scalability, concerns great. Well let's roll. This panel up and. You. Know but. I do want to open up for questions see me in a database right. Like, what we have is like I'm a I'm a fortune, 500 company and what I want to do is like write some data and do some work on data and have, that have high integrity interests but it's. A database well to write. So that there'll be clients that run under each of these different you. Know companies, that are on a shared ledger the, interface, between that and the rest of the organization, will be web services interfaces, that'll be a local SDKs, it'll be stuff, that's familiar to the communities, there they'll still be an integration, challenge, and I think the biggest question is migration because, you're actually come, to for any of these block chains to be useful they have to become the system of record all right they can't be an echo, of a transaction, record happening, in some other system, they have to if there's a dispute they have to be, the, final the final like arbiter right and, so migrating. From, a bespoke, in-house, kind of ancient database. System or accounting system to use something like that as the reference could. Be a big leap both technologically. And politically, inside of many large companies yeah, the, database is definitely a big thing having, a database that you can actually query and do analysis on is, huge, and I don't know much about how, stellar stuff works it but I know you support something with Postgres and you, may be able to actually like query that and do things this is a problem we have today a corn base is, our data team wants to look at the blockchain data and like match, that up with our sort of internal accounting, and make sure everything's like kosher and, we. Can't, without building custom tooling that just takes all the data from, the blockchain and pulls it into like a sequel database that they can actually use and. Then on top of that with like, like. Sort. Of moving everything over I think another big thing is sort of transaction, fees right, you have to be able to trust that this thing is going to be constant, when, you're in the world of aetherium or Bitcoin or really any of this sort of popular chains today the, fees may change from out from under you with no, fault of your own somebody comes in with the new ICO they come in with the new DAP and all of a sudden the fees are you know 10 40 times more expensive and that's. Not something you can use in this sort of public world and so maybe private blockchains are one of the ways that you can do that to make that more of a consistent experience but. That's another I think huge barrier to sort of adoption, is you don't know what it's going to look like tomorrow and. You have really no assurance, of that so just, to be clear though a blockchain that, has a sequel interface to it is a database, and, it's, just a cool database, right it's.
Really Cool and it has a bunch of interesting benefits, it because a shared multi master database resilient. To hostile actors this is the novel part that's fantastic, all, these are data I mean like bitcoins a database if you want to like go that far yeah. I mean that that's. What a fortune 500 company when, they're not looking at it for fight for financial, reasons but are instead are looking at well what are the benefits to a system like this and to, a shared one to a centralized, the, private model they really are looking at what, are the benefits of having an enhanced ledger where you, get away from the many restrictions, of a relational, database that you can't understand, and audit and do other things too but. But, fundamentally, if you're in this room and you're trying to build a startup that's building for fortune, 500 companies and, you're trying, to think how can I help them the. Way that they're gonna approach you initially and actually pay you money is they're, looking, for something that's like a database but has a bunch of these benefits, so. You, know if that's you in this room just remember, that like this is a chance to make, a better thing that. People understand, a little bit and. Do, so in a way that they really understand, a lot in, five years how I fit in a day of my life I don't know I feel like it'll be like robot, Holocaust I don't know a. Positive. Sentiment, right yeah, yeah. Probably some fancy diet. Yeah. But one, thing to that though what. We found interesting when we started. To deploy, notes and networks and trying to iterate around the, infrastructure components, and and and. Orchestration. Services around blockchain is how, early, we still are I mean like there's just so, many components missing, in order to build. Anything remotely. Consumer-centric. Or you know anything, that hits scale, on a level that you're used to that. You. Know this is I mean from my perspective is. Not something it'll, be five years before you see something, even remotely relevant it's, still very very early from. From that level and that was interesting for us when we start block Damon last year is because, we started just before the sort of boom and you know token prices, and valuations where blockchain suddenly became the hottest subject, because the valuations, were so crazy and we. Started raising money with like well you know infrastructure, scaling issues all that type of stuff which people, really didn't want to hear at the time and now it's a very popular subject so. We're doing better but, the. But, it's been really interesting to kind of sort of also that dawn of understanding, on how complex decentralized, systems really are and how. Much work it takes in order to Shepherd, open. Source community is to you, know have agree on consensus, morals and build tools that you. Know can live on a sort, of a blockchain that with time becomes more and more and more complex you know so an interesting question would be you know how large do we think Bitcoin or a theorem or any of those changes in five years you know like how do you do that you. Know I don't know yeah, I have, a phrase, that I wrote. Once. Mentor. Both Brian and mine Tim O'Reilly I said, to him you become what you disrupt, and Tim. Likes to throw that back in my face every once in a while because you, know I've I've, done some industry disruption, and then suddenly you find yourself in that model, and so I feel like at, the moment where we are is if we, succeed, and suddenly we see you, know mass adoption be in in finance, in a in a mainstream way if we see mass, adoption, by that by the, you know major enterprises. And they're using blockchain, at, what you're gonna find is is that with you you look like the entities that you disrupted, and so you, know if, you've built something that looks. Like a database congratulations. You run a big database company, and you. Know and oh it's got these other advantages, cool so maybe you disrupted, Amazon or Microsoft or, Google and.
There With our cloud environment, so you just look just like them and so. The the interesting thing is is that. Decentralization. Has a potential. To change, to. Change that fabric, in the same way that it did the first time when we launched, him and and began to commercialize, the Internet and, so what I what I hope is is that we see a wave of innovation, and. New services that emerge that five years from look, like what, we started to see in the internet in you know 99. 2000, where it was like a whole new true, amazing, world and. So. That's what I'm looking, for but I do believe you become what you disrupt the, who said it better, say, hello to the new boss same as the old way yeah exactly, right yeah. And so a great. Question and so on the public note fast it's more of a sort. Of aficionado, product, I do. Urge people I like Bitcoin like I'm not you know nearly, as savvy as in you know these guys but I do like the concept of easy note, deployment, and making it easy for notes to be deployed and run and I think that's very important for Bitcoin and theorem. And any of the larger. Public. Networks. The, use case very often is you know connect your own wallet toward have a complete. History of the ledger you, can run some curry you know pull some data out of it but. It's not something you need to do you can you know go on coin base and have him worry about it you know so I think the and. So, the, the. Use case on the private, stuff is is the way we productize, it it's really a network management, tool and what that means is any protocol. And project, that has a native token that meets outside nodes to validate transactions, needs. The system in order to make, it easy for people to deploy. A node and connect it because it is actually fairly tedious nodes, are a little bit the gremlins of software they tend to be all a little, idiosyncrasy. Ston the protocol and configuration, and. Then so you know just because you know how to do an a theorem note doesn't mean that you know how to do a sellin out or or, you know any other type of node so they're all specific, and there's so many of them now that as a developer you have no time educating, yourself and all these different you. Know kind of serve very specific, use cases you need in order to connect to a network so customers of ours are companies, like a on for example which is a more popular protocol where, you. Know the gold network which is a quorum a version that we've just launched actually renounced it today these, guys need outside, people to run a transaction validation. Nodes that, are banks and commodity, traders and they have no idea how to deploy a complex form node and so, we have a wide label product that they can use they sent these guys an invite it takes them to a back-end where there's, a big old, button and you just click that and it deploys a node and you can select the infrastructure, of your choice for example and so that's sort the thing where we make the most money with, ultimately. I would urge everyone to you, know I do feel that a node is sort of kind of your voice, in the consensus, mechanism, and I, think on the larger networks you know it's kind of fun to run them but you know obviously I'm biased, and I like, that stuff but yeah, and, you'll bootstrap, and network on. A locally. Owned block diamond with the presumption that at some point the network will grow and they'll be nodes on that network on other infrastructure, providers absolutely yeah totally and and like we've worked with everyone like we're trying to be very open we're. Trying to kind of open source our version of it where individual. Protocols. Can actually add their own nodes, to our service for people to deploy so we don't even have to do that anymore I'll be my dream I mean.
I Don't know how to answer like because it I mean good you know it r
2018-09-01 01:59
