Rafał Pocztarski - Fullstack is not enough in the era of DevOps, Docker, Kubernetes, Terraform [EN]
The Server, when. A file from this special directory, was requested. Instead of serving the files from the disk it was actually executed, them. So. It was executing, them and. They got, the, all of the data via. Environment, variables, and. Standard, input, and they, just printed, the html, on standard output, together with the headers and this was just. Just the response. And every, every time you hit the same end point. Uh the program, was like restarted, from the beginning so it was stateless. Uh, it. Acted pretty much like today's, slump the functions, in in the modern. Clouds. In in that way. Uh. Next. Uh. I would like to briefly talk about. Where we just started. Those, how could we uh. How could we, actually serve those files, so, the, really. First era was just that people were mostly running, uh, our own hardware. So we had to. Buy a hardware. Buy some computer. Uh get some. Network, connection. To that computer. Manage the network, manage the routers, manage, the some hubs or, some other. Less, complicated. Network. Devices. Because. Routers, were expensive. And hubs were not but they were not as performant. And we had to, manage our own hardware. Uh install. From the beginning the operating, system. Uh some people use bsd, sun os, linux. Um. I mostly used linux, on my own servers, and i worked at the time on sunos, and solaris, servers. But all of them, were basically, having. Apache, httpd. Perl. Somewhere, using php. For for the back end, runtimes. Of course the apache, was, uh. Was optimized with some mod pearl and mode php, modules, to make it faster. And. No longer, making it really stateless. So, it had some advantages, and disadvantages. But. You just had to do everything, yourself. From the network connection. From from the cable, from your. Network, provider. To the hardware, you had to you know, listen to the, funds. On the cpu, if they are not, spinning fast enough to to replace, them before the cpu. Overheats. And. The. Disk drives, if they sounded. Funny then you you had to be prepared, that they are dying, and things like that so it was a lot of uh, things on your mind other than just creating. The software. You had to manage, a lot of things by yourself. Then, the next era was shared hosting. And people were just putting, those html, files and php, files. Via hd via ftp. On some other people's servers, and, those were mostly not, secured. At all, it was just. One if you if you wanted to, in your own. Uh in your own backend code you could just read anything, on the server, usually. Because it had to be word, readable, so the, apache, could, uh read it and execute, and apache, was usually. Run by a nobody, or http. User, in the in the operating, system, a very low privileged, user so, that any exploit. Would, would not you know be very. Um. Very devastating, to the server but it meant that. The apache, had to be able to read, your. Uh, your own, back-end, code and usually you you had their, for example. Uh, database, passwords. So it was, very, hard to get secure, and uh. Users were not really. Uh. It was. The case that on the linux system or bsd, or, solaris. Uh. Those system had pretty good, separation, of the users but you had to make your scripts, readable, by the server, and if. Everyone, had to. Make their scripts readable. By the server, it meant that. Your scripts could read other scripts really. Another. Idea, was. Using. So so this was like the first, idea, of. Of cheap hosting. Of your own, websites, and it was pretty good for static, websites. Not very good for dynamic, websites with some backends because, uh the server. Anyone, could. Could like bring. Bring bring down the server and your website was not functioning because you had to share it with a lot of people to make it cost effective. If you were serious, then. Going from your own servers, uh. You were going for collocated, servers so that instead, of. Keeping your own hardware. In your own office, and, getting, a very good network, connection. You were just. Bringing, your own hardware, to those. Collocated. Data centers, where, they were just renting you a rack for your for your server, and, the network. And you were putting actually your own hardware, there so, you you had to you know pass some. Um. Some security, and. You know show that this is your server, and they they were secured. And. Um. And, um. You could work on your server usually paying, a lot of money and i know about it because not because i was in such a collocated. A place but i was when i was working. In the. Late 90s, on, remote, servers. From poland in the united, states then, the client, told me that, and actually i was, uh, i had the task of. Updating. The. Ssh. Server, on the server, the sshd1. Sshd. Uh. By which i was connecting to the server so i knew that if i do it wrongly then i would never be able to connect, it to it again to fix it so. I was just basically you know cutting, the. The branch of the tree that i was sitting on, uh and they told me to not, you know not screw it up because.
In That case someone would have to take a plane, to, to, to fly to the co-located. A place, where the server was was, uh. Um. It was running, and a physically. You know reinstall, the operating, system, sitting there and paying a lot of money for the trip and for the time and for. Even being in the, uh in the data center so. It was. Better, that. At least you didn't have to have your own great internet connection, or like, ups. Uh you know some redundant. Power. But you still. Were buying your own servers, and this was very difficult, to you know to predict how you have to scale because. It was not like, you could you know change the idea, of uh how many servers you have in in a minute, you have to really buy those servers and they were really expensive. So the next, idea, was that their servers were actually rented so that you didn't put your own server, on the rack but you just actually, was, renting, one of those that were there already. So, um, it was much easier than buying your own, hardware but you you had to do all of the other. Work, uh the pretty much the same you had to install your own operating, system you have to keep it updated. Uh you have to you know patch the system, to, install, and provision, every software that was there and so the work was really the same but, it was easier to rent it than to buy it but it was still your own and no one else was using those. Uh those particular. Servers that you have rented. So another. Good idea, and this was actually. A very good idea with virtual, private servers, where when they came out, it was shocking, how cheap they were because, you were not renting, the entire machine but you were renting a little. Part of it. And the real, machine was running, some virtualization. Software. Uh, that was. Presenting. Uh, as. Multiple. Little servers, so one big server were, running like a lot of virtual, private servers and you could. Rent them very cheaply. But you still had to manage everything, yourself you had to install. The operating, system the installation, itself was usually pretty easy about but you have to you had to install all of the software yourself. Some. Time. Later it was, made, easier, but still. You were just the administrator. Of the system. And at that point. Um. Usually. Um. Going to the to now to the presentation. Uh title, of full stack for what what the full stack really meant, at that point, uh usually if you had to manage your own server then you had to be like a system administrator. And a front-end, developer, even if you didn't have a lot of, back-end heavy stuff on your server then. You couldn't really. Work on a virtual, private servers not knowing anything, about. Usually linux system administration, or some other unix like system. So, pretty much everyone, had to just know it to to just put a website, somewhere. Another. Another, good. Good idea, was some provisioning, software that came out like chef puppet, ansible, salt stack. That made it much easier, to provision, software, on uh on multiple, servers. Like if you have like. I don't know 10, servers, of the same. Software, installations. Then it was much easier to just script. Those installations, than to, um. Than to do it manually. But this was, from, one point of view, the tools were very. Uh, very good, working but the idea i was actually, doing pretty much the same in the 90s, by just writing bar scripts. To use. You know how to get install, on debian, or things like that so, the idea was maybe not very good, very new but uh but the um implementation. Was really convenient. Uh, and then came, what i call a, heroku, era. Which was, actually, pretty, amazing, time because, now, everyone. Uh was able to very quickly create a back-end for their, for their, server, or, to serve the front-end. Um, and what would i uh. Not only, heroku, but also. Um. But also. Uh, netlify.
Compose For databases, mail gun for, transactional, emails, cloudinari. For uh. Handling, images. Uh those are those are excellent single purpose self services, like you know unix, type that one tool does one. One, uh one job but very well. And what i call it is, the arduino, of computing, so you can just like with if you want to do some electronics, then if you get, arduino, then you can start pretty much, in five minutes and do something. Nice and very exciting. Um, but. It's not the case that. Real. Real world electronics. Is. Being built from arduino. So, at the same time. What was really not very, um. Nice to work with was, was, the amazon web services, azure, and. Google cloud platform. And. What i call it is that while heroku and netify. Are like adafruit, industries, which is, a website, to to get, very, nice to work electronics. Like, arduino, and stuff that can interact with it, and hwas. And azure, and. Google platform. Google cloud platform are more like digikey, electronics, where. It's not a very nice website, it's not. Very intuitive, the hardware there is not very nice to. To work with but. It just. Has every, hardware. That you can buy for for electronics. And for like real, electronics. Where, you know if you if you do some, i don't know a microwave, oven then you are probably. Uh, getting. Your parts from somewhere like this and you are not, actually putting arduino, in there. Not that it wouldn't work but it would be much more expensive, and, you couldn't source your parts, as well as, uh as using uh you know uh, real, uh distributor. Of, all of the pretty much all of the parts in the world. And, so this is what what i really call aws. And azure and. Gcp, it's like, uh. It's like real, world. Building, factories. It's not easy it's not fun it's not nice, it's com, very complicated. It's it's not created, to make it, uh exciting. But you can, pretty much do everything there. Um, then. There will, lambda functions which is also a very nice idea, but it is actually, quite a heavy abstraction, and only supported, runtimes, can. It can be used there so you cannot use a language that is not supported. But then um. Which is, right now very popular, docker came out. Which is nothing new if you know solaris, but very new if you, only know you in linux. Uh, and, now you can, not only, package, your, own back end very conveniently, but you can also, put your own runtime there and you, no longer need to care if the server supports your runtime or your language or version of, whatever, it is like node.js. Or dino, or. Or. I don't know or. Um. A java virtual machine. Version that works with your with with your code or anything like that you just put your you containerize. Your. Back end and it works everywhere where docker works.
But. And also, the, vagrant were. Much more popular back then but now maybe less. Usually for, development. But some people also, use it for. Deployment, on the servers. Docker compose. Which is an abstraction, of using multiple. Docker containers. To to run them. In sync when you use. Especially microservices. But also in one docker container, you can have your backend in another, database, in another, queue. Or anything like that mshq. Um. Now there is kubernetes. And. Also this is a very. Very. Like. It, it is a revolution. In like. Orchestration, of the container, so, you can now build another, abstraction. Over those containers, so you, are looking at much higher scale, and. You can build a very, big systems, that, interact with multiple, services, and microservices. Architectures. And external, services, and databases. And, message queues and anything. Also there is terraform, that you can use, uh. To create infrastructure. As code so instead of manually. Provisioning, services. Uh, you are actually. Writing, code, that. Describes. Your infrastructure. And you can version it you can put it on github you can make a pull request, to change. The number of servers, or things like that it's very. It's, quite complicated. It's very, very powerful. Idea. There are also custom solutions to do that, like, aws, cloud formation, or azure resource manager, or google cloud deployment manager. Um. But. As i see a lot of people are using. Not. Maybe. Are preferring. Things like, uh terraform. Because then you can change, the cloud that you are you are not, longer, you no longer, um. Very. Dependent, on that. On the cloud that you happen to use first, and you can, change your provider. The future, is. Is what i see it would be wazi. Fastly. Is it's it's a. A very nice company that is experimenting, with those things. Bytecode, alliance, is something that i would, uh keep an eye on because, i think that. Even lighter. Um, abstraction. Than, docker containers. Is needed, and it will. Really change how we can deploy, things and uh, how performant, they can be. Um. So, this is basically, what i presented, maybe at the end i will just say that, uh, how it, uh, relates, to the original. Idea, of, that, full stack is not enough because, it today, i think that people, calling themselves. Themselves. Full stack developers, are. Usually. Uh. Front-end. Developers. Who also, do, back-end, development. But only, like writing. Node.js, code or maybe python or, other. Or other language and runtime. But. Usually they are not concerned, about things like, like. Terraform. Kubernetes. Docker. Uh. The native, cloud, solutions. Or azure. Deployments. Or, hws. Deployments. Or ibm. Um. Cloud. Cloud platform, and. But. It's easy today to create. I don't know a website, using, uh. With a simple backend on heroku. And deploy, it in two clicks, from github. But. Usually what i see that very large, uh. Organizations. Rarely use. Heroku, for deployment, of their production, a code. So at some point, if you really. Want to be full stack in a way that you just like. Um, or what i sometimes call a, even fuller, stuck. Um. Is, if you just. Want, to be able to work on everything. From your front end, back. On every level of abstraction. Uh. Using, uh, some back-end apis, that your front end directly connects to but also, what, you do with those back-end apis, how you deploy them, uh how you can deploy them on, on, hw. S how you can deploy it on azure, and how you can, create microservices. Architectures. That, connect with some message queues like i don't know rabbitmq. Or. Or other. Or other, cues and. Uh how to. Manage, multiple. Uh multiple. Uh, instances, of the same servers to scale them how to scale it up how to scale it down how to you know the. Difference between scaling, out. Um, and. There is so many things to keep. An eye on right now also because, the traffic, that you sometimes have to handle is is so great. Uh. Most of the. Big, deployments, right now i are using microservices. Architectures. And this is not something, that you usually, start with when you create some simple node backend on heroku. Um. But. This is something that i would. I would, suggest. Especially, front-end, developers. Who are, aspiring, to be full stack developers. To keep an eye on because. I've seen some time ago i i did some research and, a lot of full stack jobs, currently, require, terraform. And, um, and kubernetes. And docker knowledge. And, native cloud knowledge. Uh so. Uh. So so this is something that i think is is good to keep an eye on and, just to go like fuller stuck than only the next level from the front end because usually people. Say they are full stack developer because they are doing front end and also the backend api but not not anything.
Below That but i would suggest. Going below and seeing how it all works. Uh this is today is very cheap to do because, you can just you you pay usually by the hour. Ins. Fractions, of cents. For services, on hws. And. And other clouds, so. This is my presentation. Uh if you have any, questions, then, uh, then this is the time for it thank you very much. Thank you very much rafael, thank you for this awesome presentation. I. Appreciate. That. You, asked, uh. Uh. Yeah the. Shared all of this information. Important. Important. Information, i can see that there is a joke, on youtube. Uh, i think, uh that's, uh no it's not it's it's a. Right answer i don't know how do you think. It's from yourself. I will not, say, exactly what is the right answer because i encourage everyone to just try it because it's. It's looks like something that you know you map over three the same value so you, should at least get the same. Some that's three the same. Values on the result but this is not as obvious as it sounds so, and let i would let everyone, run it themselves and trying to figure out, how, it works and why. Okay. We have a question from arthur. So i will read that. Do you think, it is a good career, bet, to become, a fuller stack. Some people, say, they don't believe in full stack since. This is just too much to handle, by one person. Yes it's a very good question i. I myself. Am. Focusing, on the back-end development. And i sometimes, call myself, a half-stack, developer. Because of that. And. Basically, because i think that. Being a very good front-end developer, is like a full-time job and this is. A lot, is, going on and you you know you cannot. Just you know focus on only one framework, like angular, you need to you, know react, view. Some newer. Things like svelte. Or, marco, and, you know, that there are things that you. You cannot just. Learn, once, and. And. And use for many years. But, i think that if you are going full stack just one level down then it would be. Very good to just know. More stuff, below that. Because. Not to be even an expert with that but just to be able to understand, what is a cloud formation. Uh, what, is uh, what is terraform. How to, create. Some. Simple, kubernetes. Uh configuration. For your own uh. Containers, how to continue. Containerize. Your application. Uh, how to run you know docker compost, to quickly. Um. Check it before deployment. So, uh, not to be an expert, devops, or backend. Developer, but to, not only see the one level below the front end being the usual, you know a backend api. But just. Basically, understanding, what is going on below that i don't think that anyone could be an expert. Of everything. You can be you know the jack of all trades and master of known. But uh. But i think that. A basic, idea of every level of of the stack that you are working on is is very useful. Uh so and, some front-end developers are going deeply down and for example, become. Devops engineers, so it's. It is, a, rare knowledge, i think, not a lot of people are doing that because it's not. You know it's it's not nice to look at if you create the front-end, you can create beautiful, interface. And. Configuring, docker, is you know it's not very. Rewarding, but, but it is a very useful knowledge, and, uh some companies, i see are hiring, front-end, developers. With the full stack knowledge going down to the devops level and, and really kubernetes, docker and.
Cloud Formation, yeah yeah i'm i'm happy that you're you're, saying that, uh especially, uh that docker and kubernetes. Which seems to be. Now quite universal. Between, uh gcp. Aws. And azure it's all, probably, going to be aligned to that even the less popular cloud so, so yeah that's uh, well, still, it's a it's a long subject, so i hope we can talk, on that uh. On, during the networking. So. So i hope to see. You.