7. Evolution of Tech

foreign [Music] set of Technologies um we will actually be spending time on what are the different facets of the cloud which makes it extremely love productive for Big Data practitioners such as yourselves the ability to guide programs at very large scale is immensely helped by certain facets certain features of the cloud you will look at how they evolved but even more importantly we will look at how Cloud itself came to be right and why as programmers who go into a world which is cloud first you have to be aware of this foundational architectural program of software which is called as Cloud native um so by the end of this week you would not only be familiar with what cloud is but also be familiar with how to write Cloud native programs and where exactly data and data engineering and data science fits into this landscape of cloud native software so let's start um let's go to the history of it as we declared Map Reviews and big data we would spend a bit of time talking about the context of um context of the cloud and how it evolved so imagine as though you've had time on the x-axis right that's the bottom half of the screen and reading up to where we are today logically uh there's been a bunch of different you know Evolutions that has happened to the way it's and software has been created so back in the day when um when specialized software as our first built I'm talking 16 2016s 1970s the kind of Hardware that was required for you know running those software were special especially in the sense that you needed to engineer that hardware and needed to build that Hardware in the sense of being able to run only those kinds of programs and available so special so that happened at that time using um what came to be known as appliances and the kind of problem statements those appliances were solving for with specialized software that was built internally within those appliances were catering to specific business problem statements right they were not general purpose they were highly specialized places of equipment so the best example that comes to mind come back in the day was um ATM machines so in fact most of us today probably do not even use ATM machines uh regularly because we just use like exact UTI but back in the day when ATM machines were revolutionary um the reasons why they were revolutionary is because you don't you didn't need to go to a branch of a bank anymore to withdraw cash but even more interestingly you had a machine that could reason about um cash right a good reason about what notes were provided by you as part of a deposit and a certain that those notes were not throttling and then in return if you needed to get lots out if you needed to get cash out it had mechanisms to understand what denominations how many numbers were necessary to meet your requirement and then be able to you know service that requirement by a disbursing cash and this dispersal also required certain mechanical levers to be pulled some certain boxes to be opened right so that that allowed you to put your hand in and draw the cash Etc and it also had mechanisms for authenticating who you were as a person as an identity so a lot of things went went into creation of those ATM machines and those were very specially designed because you need to take care of security hacking um you needed to sub cater to scenarios where he ATMs were deployed in very far-flung places which didn't have frequent term human presence you have to assume certain things about the network connectivity in fact in most cases the ATM machines are wired up to the bank banks on network using dedicated cable right dedicated network connections not the wide area network not the public internet that we see today okay that's one example the other example of an appliance that I want to pick up as an analogy is the world of Home Appliances right nobody thinks of Homer plexus as running software but you've seen so many advertisements of late of smart machines smart refrigerator smart ACS and things like that they're smart only because they have some programmable software that's available within them right it's not that they are genuinely smart and thinking about problem statements like you do but even before um the connotation of smart came into you know marketing the home appliances back in the day home appliances were indeed um equally smart right if not having the Slick marketing budgets they were programmable they were having the ability to take inputs from users and they had ability to regulate themselves in terms of functioning so one of the best examples here was a thermostat okay the double stack was a piece of equipment which you installed in a home um especially in colder climates and you could control the temperature that you desired using a setting but then the thermostat itself regulated um how much it how much heat and how much cold it dispersed to the entire ground of the entire house and sometimes it got it approximately right but it was a case of a specialized Appliance um which had its own mechanism to you know control for the I mean read the temperature and then have some kind of a programming to control how much heat or how much coal to provide in the household usually Heat and okay so two examples of appliances that we saw now what happened over a period of time was that um the cost of the appliances which means the cost of producing the appliance the cost of maintaining and servicing them was significantly high right and over a period of time um as the processing capability became very powerful and became cheaper right mainly due to the Advent of the Pentium processor right and even before this the x86 processor from Intel basically the Advent of Silicon Valley right the words the word Silicon Valley came from that time um other time what what ended up happening was servers which packaged a bunch of CPU memory and hard disk together into boxes they started to get shifted they started to become cheaper to assemble and cheaper to um to procure and the service for general purpose because the chipsets were general purpose you could program them to run anything right not just very specialized pieces of code and as a result um new breeds of software needed to be built to operate on top of your servers and because the chipsets were general purpose the programming interface that the servers provided but also general purpose and that's when C C plus plus became popular it became easier to write general purpose programs which could operate on variety of different types of servers as long as all those servers supported the same chipsets right and that form factor resulted in the array in the rise of commodity Hardware manufacturers like Dell as an example HP as an example right so a revolution opened up because of the way that the chips were standardized and ready purpose and became really cheap to procure while at the same time the processing capability just shot off through the roof in fact this there's an unofficial rule of thumb which came to be known as the Moore's Law which effectively stated at that time that the uh the ability to process doubled um you know in a standard server for every 18 months or so right every two years or some such thing right um so so much and Revolution was actually happening in the world of server-based computing what this meant was the Monopoly the appliances manufacturers had was breaking so now you could write software independent of the servers you could ship software independent of servers and in fact you could replace software as and then you you needed to right you could ship this out server separately you could get the software as there but then you could replace the software software with newer forms of software or even newer software some newer competitors you were no longer tied to a specific competitors rather you know who provided software in a specific frequency you could you could ship software more you could ship your forms you could direct some newer players and this meant that business problems could now be solved by completely different types of companies right no longer only those companies which had huge amounts of money which could maintain these appliances for you now this resulted in an explosion of different types of software and as a result some of those software started to produce telemetry what I mean by Telemetry is some of the software started to produce data about its own functioning so think of telemetry as exhaust right just like an automobile like a vehicle or a two wheeler or a four wheeler runs and as part of its execution as part of its running it consumes fuel right fuel is basically the analogy is servers here right it computes it consumes Fuel and then it burns fuel with a mix of oxygen from from the atmosphere in an engine and then it propels that particular machine forward so the analysis it propels the business forward the software propens the business forward has exhaust the engine producers uh gases right and those gases themselves get released back out right and those gases are effectively exhaust from that combustion process right likewise the software where it produces outcomes right that's necessary for the business to continue to be propelled forward at the same time it also can be made to release data as exhaust which captures what is the current state of what is happening in that business like just like you can analyze the exhaust of an automobile to understand how what is the health of the engine what is the extent of fuel mix what is the current uh where and there that's actually happening inside likewise you could analyze this data exhaust to understand what is happening inside that particular business is that software meeting the business needs if not we should we have to change it etcetera Etc so what started off as data exhaust from software very quickly became critical data that's necessary to analyze and in order to analyze the data you want to start actually storing it at the moment you have to store data you have to actually have mechanisms of storing data which was structured and that's how um relational databases were created for the first time rdbmss were created by the likes of Oracle and IBM and then later by teradata and then became open sourced over a period of time using MySQL Sun Microsystems was one of the first uh to open source databases they created MySQL for the first time available for Mass Market uh etc etc then postgres came about as a way to as an alternative Philosophy for open source databases so all that Revolution started uh basically the data Revolution started around the 1980s as a result of being able to look at how to you know optimize software how to you know separate data as an exhaust from Mission critical data that's necessary for analyzing how the business is performing so notice that the need for data comes from the fact that um software which is built for a specific business is already outdated the moment the business changes right so as businesses change you have to go back and update the software but it's um it's hard to understand what is the current version of the software without analyzing the data that comes from the software so data gets analyzed um it gets compared against what is the current state of the business what is necessary for the business to change is then propagated back to the software um as the output of data analysis right so that's how data became Mission critical then that cycle of analysis starts to become bigger and bigger stored in databases now while this was happening what became interesting was servers which which were essentially assembly of Hardware of CTU memory Network and disk they became more and more centralized into these large players who became very good at it like I'm talking about the likes of Dell and HP IBM right so over a period of time it became a prohibitively expensive to own these racks of servers at a particular point of time because you not only had the procure servers for every single thing that you wanted to run as software be it if you want to set up a web interface you have to have a web server if you are to have a business facing software you have to get database offers database servers you have to get software application servers you also have to get servers for running your networking servers for running your um those are dedicated servers effectively like routers and switches you also have to get servers for essentially interfacing with third parties etc etc so you have to start procuring racks of servers together and not only that you have to start renting space from data centers because tracks cannot be set up in any you know any office so to speak because those racks ended up creating a very high footprint on the environment uh if you didn't know how to maintain it properly you have to pay for power you have to pay for air conditioning cooling or you have to worry about networking those servers together keeping all those lights on if something goes wrong then somebody had to worry about constantly going back to the to the rack and keeping the power keeping the lights on or you have to pay quite a quite a large amount of money in order to get somebody else to do it for you as a managed service so so the operation of servers became something that was quickly becoming expensive um so what ended up happening was startups which were starting their businesses for the first time and they were very small very limited size who were wanting to stand up very quickly like a web-based business like I'm talking uh the heights of the.com Boom back in the year 2000 uh just leading up to it you have these startups which are coming um and creating like like fantastic innovation in very quick periods of time but they had to then go and procure these racks of servers somebody had to go set up those racks in the data center and ordering the machines from the likes of Dell and HP took like months together in terms of time so agility in terms of in terms of servers in terms of setting it up became a problem compared to the speed at which the startups were necessary with creating value right the speed at which the business wanted to change and couple that with the cost of actually operating that that resulted in a market opportunity that was capitalized by the Advent of cloud right and the Advent of cloud was created rather the concept of cloud was created by um at that time Salesforce one of the up and coming startups at that time um which played to the problem statement of how do I make um sales more effective what how do I make sales of software more efficient right of sales of service products whatever it is whatever it is that you're selling if you can how can I make that more efficient Salesforce started you know as that kind of a company providing software that made the selling process more effective but then very quickly they realized that um uh people did not need to buy another machine just to run my sales software I could run it on my own I could run it on my data sector but I love people to rent my software um package together along with my Hardware along with my cost of maintaining the rack and the power the cooling Etc you know all of that package together into one offering but what the user would see is one price point right so they don't have to worry about any of this and in return I could keep charging them rent for this right instead of selling them software at once and keep keeping on maintaining it for a small fee I could just charge them rent um just like a building owner charges rent for this building to his tenant right I could charge rent for this entire thing and keep charging rent forever right as long as the user kept coming back for the software I'm good right I could just keep charging that so this resulted in a very different shift in the way that business is operated but it also resulted in a very different Financial modeling so now a company like Salesforce could reject into the future that I'll keep getting rent like this for as long as I'm the only provider of this kind of software or this kind of a rental software for this sales application right so if you kept innovating if you kept making differentiated software like this I could just keep charging rent over um like you know over long periods of time and that resulted in a guaranteed cash flow and that became very attractive for certain kinds of investors to invest in but more importantly for the users of that software it became very attractive because you no longer needed to rent um or procure Hardware anymore right you could just rent it directly from these cloud-based companies and then not have to worry about it at all right you don't have to worry about maintenance you don't have to worry about keeping the network cables plugged in etc etc so um Salesforce started this concept like I said but Amazon it like took it to a very large degree and made it super attractive for startups to engage with right so startups became their first clients I remember dealing with them back in the day when I was part of a startup beginning very easy to set up my own machines because quite complex from a developer standpoint compared to where we are right now in terms of cloud but nevertheless it was still easier than you know having to learn networking having to learn how to develop machines together etc etc great so right around early 2000s Cloud started to become popular and in fact it became so popular that servers increasingly started to become replaced by Cloud so today if you see um it's very rarely a case that a startup would go by servers most often than not the startups will be born on the cloud over a period of time as you become bigger there is still a case to be made that you could look at what's called as repatriation which means bringing um certain types of workloads back from Cloud to servers because it's cheaper to do so right it gives you more control it gives you more flexibility but even that is going to be rather today with today's economics in place it's probably cheaper to do so if you're very large scale even at that larger scale there will be some part of your footprint on the cloud so um uh All Things Considered cloud is here to stay and cloud is probably your first goal of Port of Call if you want to start a business if you want to you know just you know start to consume a piece of software that's of interest for you great now let's look at what's happened today though right from a piece of from the perspective of data and software more and more popular what ended up happening is a lot of the common things that you associate with various parts of the cloud became commoditized meaning I could get whatever type of resource I want at my fingertips right if I wanted ctus of a certain quality I could get that if I wanted ctus of a certain performance I could get that if I wanted memory of 15 GB that's easy if it's just 128 GB it's just a like a simple click and point right rather point and click if I wanted gpus right because I wanted very heavy processing needs simple right so all of that became super easy but then what happened with software though right um what was earlier designed to run on servers now suddenly needed to be run on cloud and Cloud had different form factors so if you needed to be able to sweep completely a set of hardware and replace it with some other set of Hardware the way that software was defined earlier was tightly coupled with the kind of Hardware that was running within the servers even though the chipsets were general purpose at the end of the day it was still controlled by us a handful of chipset manufacturers effectively it was Intel which was doing x86 chipsets and AMD which was actually doing its own chipsets at that time so a piece of software that was designed for x86 could not really operate on AMD servers and vice versa and if you generalize this a bit more a piece of software that was assuming for certain kinds of hard disk or certain ways of connecting artists together like raid the hard disks as an example yeah it couldn't really work with other kinds of network attack storage easily right um so what ended up happening was the software has had interfaces which were tightly coupled to the kinds of resources that were operating underneath but with Cloud coming into play and Cloud becoming popular software had to be designed that would allow plugability of different types of Hardware underneath right based on what the cloud was providing and even more so um as and when the user changed right um the kind of Hardware the software had to immediately work right not to it it shouldn't be down for any period of time it should not be requiring elaborate testing Cycles to certify that it's that it's able to work on this new hardware that you've provided as an option uh whereas the expectation from the user is I'm changing the hardware immediately within one hour within half an hour within 15 minutes and sometimes even in real time the soft fetch should just work right so in order to cater to those kinds of requirements the software had to be changed but notice that it's not the functionality of the software that's required to be changed but it's the interfaces that it was assuming uh was necessary for working with the cloud in the back end so that those interfaces became what's known as non-functional part of the software what I mean by non-functional here is it does not affect the functionality of the software it only affects the execution of the software where is the software executing what is it assuming about the underlying Hardware so those non-functional cases started to be engineered separately from the functional pieces of the software right and those non-functional cases became attuned to what was assumed as available from the from the cloud right or more importantly uh what was not assumed so um with the cloud coming becoming popular um the ways in which you could engineer software to be general purpose and applicable to variety of different Cloud options um became possible through an enabling technology called as a hypervisor one of the first companies to popularize this was VMware but even otherwise there were other implementations of this that became extremely popular but those were enabling Technologies meaning if you built your software and if you constructed your non-functional interfaces to work with hypervisors then you could take it and run it on other any Cloud which actually supported hypervisor Technologies over a period of time um this became even more simpler with the Advent of what's called as containers containers are loved very lightweight execution of units of software without having to go through even hypervisors hypervisors had to actually go through extra layers of translation in order to convert your instruction to operate on a disk to the specific underlying disc that the cloud provider was providing but with containers those became even more lightweight and became super easy to transport and you know have portable software working on different types of Hardware so long story short a lot of work that happened in creating non-functional interfaces catalog software to be truly portable across different settings uh so this ability to write software in a way that actually transforms um you know the ability to execute across multiple different types of cloud together uh created this Paradigm of software called as Cloud native so effectively Cloud native is an uh think of it as an architectural pattern which called for creating software which had some very important properties and those properties as long as you created software that obeyed those properties you know for a fact that you're going to be able to have software that will operate very quickly on different types of cloud vendors different types of cloud environments can have resilience built in can have scaled built-in can have all the good things related to Performance without having to read write software later on right so this became the foundational building block for startups which wanted to grow at very very large scale so imagine you were a startup back in like mid 2000s like somebody like a Flipkart somebody like a myntra you started off small you had only let's say a thousand users and you you had software that would do your e-commerce billing right or e-commerce shopping exercise but then the moment you uh figured out that um you could attract a lot of eyeballs and thousand users became a million became became 10 million then you cannot offer to go back and keep writing new software just because the scale has increased right um that's where Cloud native software became really really powerful as a as a architectural pattern as long as you create a road software which would be Cloud native you could run it on 1000 run it for a base of 1000 users but you could easily scale it up to a base of million users without changing the software right it would just seamlessly operate on bigger resources bigger Hardware footprint and nothing needs to change in terms of the software that kind of a guarantee is possible if you engineered the software with Cloud native architectural principles now notice one thing right um data that's getting produced by software is actually being produced by both the functional side as well as the non-functional side the non-functional side of the data is pertaining to things like how much memory did I utilize for the specific operation what is the current utilization of my cluster what is the current IO pattern on my disk Etc account right that's all non-functional and that's useful for optimizing right that's useful for the cloud native side of that software to figure out what bigger cluster do I need Etc but the functional side of the software is also producing data that data is actually being analyzed by the business and that another analysis is producing newer changes right to the software that is required like for instance let's assume I am IRCTC I have a specific way in which I do discounting and um as a result of the last one month of uh ticket booking that's happened I've collected the data about the tickets so basically my software is producing data about tickets and that data is getting analyzed by my business and the business comes back and says yeah it's Diwali time let's do your time um I'm going to change the way that discounts are operating so I'm going to make discounts um very specific to individual regions right so for urban uh population I'm not going to give discounts for brutal population I'm going to give discounts so I give I come back and and say that's my change so then some software engineering is going to go back and look at the software which is on the functional side and do the changes that the business is demanding for the discounts to operate right so that cycle of production of data out by the functional side of the software analysis by business and then generating the changes that the business wants to create uh newer versions of that function software that cycle will continue that cycle continues over uh that's been continuing for decades now so with Cloud native what has happened is the non-functional side of the software and its dependence on the cloud and its independence of the cloud right let's put it that way that is automated so now if you write non-functional if you write non-functional part of the software using cloud data principles you could automatically take it and run it at Large Scale run it on AWS run it on Microsoft does not matter right it just works out of the box or the Google for the matter doesn't matter but what um what has happened this on the data side um as the scale has become bigger on on the on the infrastructure on the non-fructure side of the software has just kept uh kept pace with that scale but the functional side has just produced more and more data and that more and more data has now required Big Data Tools in order to analyze so that the business can take some action on top of it right so in a nutshell Cloud infra and Cloud native apps have made scale of operations super easy so if you know a startup you can start small and scale it up super super easily and in fact the expectation is that you would go from like 1000 to a million like within within a week and nothing will change about how you run your business right so that's as powerful As It Gets data on the other hand though it's still being analyzed by business it's still being analyzed by data scientists data analysts Etc using tools which were traditionally built for small data sets but ask a scale kept growing up it's necessary to now start looking at Big Data right Big Data tools because that's when you you'll be able to truly analyze the data that's being produced by say data that software that that's running at scale and then that is what the business needs to look at in order to truly do accurate changes that are necessary for it to continue to make progress so with that we'll stop with um today's with the video on what does it mean to um do Cloud native software in the next class we look at specifics of what the cloud native definition is and then what are various services that cloud provides you know in order for you to build cloud data software

2025-01-03 11:42

Other news