IT Masters: Alright, good evening everybody to the third week of our master class series on comparative cloud technologies. IT Masters: The wonderful TIM Jones is going to take it away in just a moment talking largely about storage and database technology, welcome to anybody who has not yet been in one of these webinars you can find the other, the previous two weeks. IT Masters: Class materials@learn.it masters.edu.au and I believe that the two previous webinars have been uploaded to YouTube as well. IT Masters: So you can find those if you've missed out thanks very much everybody for coming along just a reminder as well that technical questions we really encourage you know, participation and you know you know audience participation chatting amongst yourself. IT Masters: amongst yourselves, the technical questions about the course materials can be directed to the Q amp a section on zoom. IT Masters: And we will try and answer as many of those questions as possible and then any other comments questions or particularly questions about the kind of administrative side of the course.
IT Masters: can be directed to the chat section we've got kit and Hannah sending by doing sort of the tech support slash administration efforts for us this evening, thank you very much, get an handle. IT Masters: And they will also be able to answer any questions that you have about that kind of administrative side of things in the chat section. IT Masters: As well as being accessible via learn it masters.edu a you for for the questions tonight and at any other time throughout the short course. IT Masters: My name is jack Stuart i'm a coach consultant with it masters and I am now going to hand over to the illustrious TIM Jones who's gonna tell us a lot more about cloud technologies thanks very much TIM.
Tim Jones: Thanks very much jack. Tim Jones: Welcome to. Tim Jones: Week three everyone of the cloud masterclass so to get in touch. Tim Jones: Alright, so we're on week three of week four so if anyone hasn't. Tim Jones: didn't join, one of the previous classes as Jeff mentioned they're all available up on YouTube so you can catch up, you can do the tests and you can get the certification still you don't need to.
Tim Jones: join these live, even if you're joining this one without having done the previous ones, this week, where it's all about. Tim Jones: file storage databases so we're we're going into a bit of detail about these sort of technologies and. Tim Jones: In between we're also going to have a bit of an in depth view with jack around what it looks like to become an IT master student moving forward as well. Tim Jones: So, to start with jumping into file storage so. Tim Jones: First of all, what it looks like from the perspective of the main cloud platforms, so, as usual, looking at the main players that we see in the marketplace aws as your GDP, of course. Tim Jones: there's IBM there's di data there's other companies as well, but these are, by and large, the biggest plays, especially in enterprise companies.
Tim Jones: And so what we see from aws as a technology called elastic file services, we see it as your file storage and we say fall store from JC LP and, as I mentioned previously. Tim Jones: Very much all of these technologies that we talked through this entire course are synonymous. Tim Jones: Mostly between each of the different cloud service providers there's some edge cases and some slight differences in the way they implement these major technologies but, for the most part, you can think of them as very much the same.
Tim Jones: So, first of all, what do we mean by file storage, so this is files that we access over a network share so this could be something that you. Tim Jones: Access from your home computer accessing directly into a cloud environment from your corporate office from your laptop connecting through a direct connect or a. Tim Jones: vpn connection into your cloud interface or for application news within the cloud or outside. Tim Jones: And this is very much in line with what we have in data centers as well, so this is a technology that was. Tim Jones: that's called network attached storage or you might have heard it as nance and there's companies such as neta which made most of their business out of selling network attached storage for enterprise class applications and the ability to share that only files, but to have.
Tim Jones: That basically the way that you run applications run of this shared network storage as well in the home environment, you might have Kunas or a different device that offers the same sort of capability as well, so. Tim Jones: Good for user or application use cases, as mentioned but we've got some limited capability with our stores around how we can search them it's a hierarchical structure, as you can see from the diagram on the right hand side. Tim Jones: If we want to search for an individual file, we need to search directory by directory right, so we don't have a global view of all the files.
Tim Jones: And it takes some time to go through and find it especially when we start to get up to very large numbers of files so say, for example as especially as we move into the world of. Tim Jones: automated systems iot etc, when you have a large number of images or video or other things that need to be searched through when there's thousands or millions of events per day. Tim Jones: You need specialized technology sitting on top of it to go through the file system itself doesn't scale to that many objects itself but from a cloud perspective. Tim Jones: The files that we have within the environment quite your standard sort of SLA we have across the three platforms we're talking about.
Tim Jones: Is a 99.99% sli of uptime availability that you'll be able to utilize the environment so ronnie talking about a small amount of time per year, that you would expect, not to be able to access your funds so it's a it's a good technology, but then we come across to. Tim Jones: Block storage so with. Tim Jones: What storage again across the main three platforms we've got EBS, which is the main Amazon platform for block storage. Tim Jones: As your page blobs or premium storage many discs or persistent disk with GDP now what we're talking about with this is synonymous with the idea of. Tim Jones: Your hard disk in your laptop whether it's actual spinning disk or an ssd Whatever the case may be, it's it's some storage, that you can directly access and utilize and is. Tim Jones: is something that you can run an actual operating system on So if you have a file store, for example, we talked about before you can boot.
Tim Jones: An operating system off that specific file stored in the cloud environment, simply because you need to have an operating system before you can actually access the network file system, so the way that the. Tim Jones: We talked about the osi model and the way that different capabilities come into play, over time, the running an operating system requires a. Tim Jones: certain amount of capability of stuff in other situations, you can do things like pixie boots for your own as environments, but that's not the way things work in a cloud environment so. Tim Jones: Within cloud to boot, we need a block based system to boot off right and.
Tim Jones: Within that we write anywhere on the disk that it takes care of where it's writing to and the rights of done in certain block sizes so whenever we do. Tim Jones: An update to a disk the update is done so we rewrite the entirety of the particular block size that we're looking at so that might be okay, maybe 64 and maybe 256 K. Tim Jones: And the difference in block size means that if we're making a very small change like just a couple of bites to something that has a 256 kilobytes block size. Tim Jones: We need to rewrite the entire block back to disk again or back to sst tokens writing things down, but if we have a very small block size, it can impact. Tim Jones: Read time because we need to request a blockbuster block requested block right, so we end up with latency or lag issues, so this is around technically what we're doing with delivering performance and capability from a disk perspective. Tim Jones: So.
Tim Jones: The real advantage of it, though, is that, again, we can write anywhere allows us to have an operating system sitting on there, and can be very fast for reads and writes, because we can just keep on. Tim Jones: Writing strings of data data to disk basically in a in a sequential row along a track within the environment, which means that. Tim Jones: We don't need to if we're using physical disks we don't need to move the head of the disk that much if we're using SMS days it can just basically take the information and and check it in there straightaway. Tim Jones: So same with reads if we have sequential writes, we can add sequential reads which can be very fast as well, so. Tim Jones: We have very little capability with this, though, being able to do search we don't have any metadata that's associated. Tim Jones: And we don't have any built in recover ability to a standard disk so we have a disk failure we have an outage.
Tim Jones: For that particular environment so any capability for search metadata or recover ability needs to be built into a higher level of abstraction for what we're doing on a layer above what we're doing with the actual hard disk system itself. Tim Jones: One of the other things that we have within the cloud environment is what's termed as a femoral discs discs that. Tim Jones: aren't designed to stay around after we have system shutdown, for example, or if we have an outage so these have the advantage that. Tim Jones: The block storage, we were talking about before is access from across the network so there's central storage within the cloud arena that. Tim Jones: We have all of the storage actually sitting on and it comes to the compute nodes.
Tim Jones: For them to be able to access it so it means that any rate or rightness to traverse the network, which means that there's. Tim Jones: A bit of latency a bit of time for it to go across the network and a bit of additional slowness of all of that waiting, especially if you've got a very chatty application that keeps writing and reading all the time. Tim Jones: The if we have a femoral discs though these are discs that are built into the actual chassis is or racks that we have within the cloud Center data Center itself so part of the. Tim Jones: system deployment model, and we have fast ssd disks that are available locally now because they are local to a particular rack they can't be guaranteed that they're going to.
Tim Jones: survive any major outage, which is why we've got central storage elsewhere and. Tim Jones: The way that they architect these systems is that you don't get your data back so they're very useful for when we've got elastic application architectures where we don't expect to see any long term storage and we've got something else in place to manage long term storage capability. Tim Jones: And we also have very low latency and high throughput from this so great performance out of it very low latency and but we don't have any durability expectations for what we said. Tim Jones: But just a question before we move on to the next section just a question from REX should all files stored in cloud be encrypted just for the known person to access is there are other ways to protect and secure the files so. Tim Jones: by default, all of these technologies have a layer of encryption within all of the cloud arenas so any data at rest. Tim Jones: So when we're talking about data at rest it's when we actually write something to disk or in the old days you're writing to tape or something else.
Tim Jones: All of that within the cloud arena is is encrypted right so by default we utilizing the aws keys, and the encryption capability from aws or from as your of J CP to deliver that capability so. Tim Jones: If someone within aws or as your ICP that had enough admin rights they could potentially read the information because they may have access to the keys in some other way. Tim Jones: If we wanted higher level of security, we can use our own keys or an hsm.
Tim Jones: hardware security module it's available from the cloud service providers, or we can provide our own keys so there's different layers of security can provide and that security can be applied at encryption for data at rest as well. Tim Jones: Right. Tim Jones: Now we move on to object storage to similar format here we've got amazon's s3 simple storage service we've got as your blob storage storage or we've got cloud storage available from Jason. Tim Jones: And here it's a flat storage model so all of the files, if you think of it in terms of a file system, our regular old file system, the objects, are all stored in the root directory essentially so they're all. Tim Jones: At the same hierarchical level as each other from an object perspective, the way that we deal with things slightly different here is that we have the. Tim Jones: We have the systems available for us to have a great deal of metadata associated with each of the objects and part of that metadata is.
Tim Jones: pretending that the objects are actually within a hierarchy, so we have a folder structure or a quasi folder structure that we. Tim Jones: tied to the object and that metadata is is basically the information to show where it is and to allow us to browse information or keep it slightly separate. Tim Jones: Now, because we have metadata all of the information about time and other information that was stored at the time that the object came about. Tim Jones: It becomes much more readily searchable, and so it works really well with applications that might be time based so we want to pull back certain. Tim Jones: Certain objects for a particular period of time in a file system or if we had a file environment that was across many, many folders because we had millions of images being saved to disk all the time, it could take a very long time to search through that type of environment. Tim Jones: In this type of system, we can do a search of the metadata and get information back really quickly.
Tim Jones: And this is really designed for how we utilize non structured data right, so we can have some component of structured data in here as well, so we can have. Tim Jones: Some components like csv files or other dumps of file objects that come in here and. Tim Jones: This is where we start to look at things like data science and the capability to do data mining through.
Tim Jones: object based stores, where we can correlate information across multiple areas so, for example, being able to correlate information from security events to. Tim Jones: To weather information to time of day to when particular staff were in the office by pulling together multiple different streams of information and stitching them together through data science and data analysis. Tim Jones: So we go through it one of the things which is pretty cool is the the floors website, so this is something that I suggest, everyone has a look at because it not only goes into a.
Tim Jones: fair bit of detail on the aws side of. Tim Jones: How you access the file storage, but it also gives you a great idea of all the different ways that people can hack into a file store that's web accessible so it's a great site really well put together so recommended so have a look at that, after the course. Tim Jones: Right now to just have a quick look at a DEMO, so this is just a fresh instance that i've loaded up from the course that everyone's got access to. Tim Jones: And from within, here we can go into s3 and have a look at what that looks like now, if we want to go through and create a storage area within the Amazon environment that's called creating a bucket so if we go in, create a bucket and if we go through, and just call it my bucket. Tim Jones: One of the things that comes back with his bucket with the same name already exists now What this means is the the bucket names within aws need to be.
Tim Jones: Globally, unique we can't have a bucket name that anyone has used but anytime before so. Tim Jones: Kim. IT Masters: Just really quickly, are you able to zoom in once again yep perfect thanks so much. Yes. Tim Jones: No one else has before So if I go.
Tim Jones: To masters. Tim Jones: Just they shouldn't be anything that's been quite a lot that before and. Tim Jones: Go create okay.
Tim Jones: we've now got a. Tim Jones: pocket available for us, so this means that we've got very simple information available that we can go and put. Tim Jones: Data doddering so if.
Tim Jones: We can drag and drop a couple of files. Tim Jones: over here. Tim Jones: Maybe it's time to let me. Tim Jones: There we go. Tim Jones: So here saying that we're uploading a couple of files. Tim Jones: Pretty small.
Tim Jones: Now that we've got the in place. Tim Jones: can say that they've been successfully. Tim Jones: come up. Tim Jones: Within here one of the things that we can do with any of these is, we can make them web accessible as well, so we can essentially make a very simple website super quick, so we can enable steady coasting. Tim Jones: and give the names that we've already created. Tim Jones: Within here.
Tim Jones: and safe enough. Tim Jones: So now we have. Tim Jones: These website available we've got objects that we can see if we select it, we can go through, and say open. Tim Jones: And then we'll go through an open completely fine now what i'm doing here is i'm not accessing directly through the Internet, this is an Internet accessible view.
Tim Jones: Of the file, this is an internal view of what the file looks like if I go through and go to the website so anyone else can access this as well if you say it masters dash TIM Jones. Tim Jones: dash dot s three dash website dash us, etc, you can hit the same website and you'll see a range what problem to begin and so. Tim Jones: by default, one of the things I mentioned in the security section is by default aws as your GDP, they build their systems to be as secure as they can to stop people from running into problems, so we can come through and change the. Tim Jones: backup settings. Tim Jones: So that we're saving.
Tim Jones: To allow public access. Tim Jones: And we can also go through and create what's called a bucket policy. Tim Jones: And I can copy and paste one of my credit earlier. Tim Jones: And from here, this allows us to.
Tim Jones: save it off. Tim Jones: Those and. Tim Jones: we've now got the bucket policy in place if we come back over here, this should allow us to go through an exit if we go through and put in. Tim Jones: Something that doesn't exist will end up. Tim Jones: should end up with an error message right, so this is an error message that was part of the fun, so this is all.
Tim Jones: easily accessible, you can see, you got a website up and running super fast, but it's easy to do i've used it myself it works in with the DNS that's available and things like that as well. Tim Jones: So. Tim Jones: Now that i've gone through this one of the before we launch into any questions around the file area i've got a few words from stop sharing and jack is going to come on board with a.
Tim Jones: bit to talk about the it masters offerings so you're. IT Masters: Good jack I am I am one moment please allow me to share my screen. IT Masters: Alright, so thank you everybody i'm going to just talk really briefly about some of the opportunities that it masters and CSU. IT Masters: You know, can provide to study full length post Grad I take courses with us.
IT Masters: So, as I mentioned in the first week's webinar one of the reasons that we present these short courses is because. IT Masters: We believe that the content that we provide really speaks for itself so it's a really good opportunity to show anyone who's interested in pursuing higher education in the field. IT Masters: To kind of get a bit of a taste of what our courses are like and what they can expect from our subjects, so the short courses that we present the content is taken quite directly from subjects that we offer so, for example, Tim has taken a lot of the content in this short course.
IT Masters: Oh, is this. IT Masters: i'm from the subject that he teaches architect in cloud solutions which is available in several of our courses. IT Masters: Which are cloud computing and virtualization cyber security and networking and systems administration. IT Masters: So there are obviously many people who are taking the short course today purely out of interest or for enrichment. IT Masters: You know just to deepen their own kind of professional or you know interest based knowledge, some people might be looking for further sort of study opportunities. IT Masters: There are some people for whom the kind of industry certification pathway in the in the industry i'm going to say the word industry too many times I apologize.
IT Masters: Is is a little bit more appropriate some people who are looking for more of a traditional kind of university. IT Masters: style formal educational opportunity, and some people in the middle so who might want to kind of combine those approaches and things like that to increasing their kind of level of qualifications. IT Masters: In the it field, so you know, for example, one of the reasons that it masters essentially was founded. IT Masters: it's kind of bridge the gap between the industry and the academic spaces when we're talking about it education so as i'm sure people would be aware, there are. IT Masters: You know, there have been a lot of developments it's an extremely fast moving field, sometimes the traditional university system hasn't always kept pace with developments in the industry.
IT Masters: And there are a huge number of vendors, certificates and other kinds of qualifications that are available to people within this industry. IT Masters: which can teach you just as much as, if not sometimes more than a university based qualifications there's a lot of on the job learning that a lot of people do as well. IT Masters: Which is incredibly valuable and provides you know a lot of experience that can't necessarily be quantified in a traditional kind of educational model.
IT Masters: You know, we we speak to people who are going for jobs, highly technical roles, perhaps in non it focus organizations and they're struggling to get a foot in the door, because HR type professionals. IT Masters: don't really know what, as you know, CCA CCA SP sorry CSP or aws certification actually entails and the level of expertise that that requires. IT Masters: But they do know what a graduate certificate or a master's degree means and so that can sometimes be a little bit of a stepping stone for people into the industry kind of positions that they're looking to get to. IT Masters: So.
IT Masters: Why study with IBM and CSU i'm glad you asked, so the full length courses that we offer we offer graduate certificate and master's degrees. IT Masters: And I think one of the one of the biggest benefits of it masters being involved is. IT Masters: That we can take our understanding of the course and the industry, as well as your goals and objectives and tailor a really sustainable and suitable study plan for you. IT Masters: So we really prioritize offering students flexibility and a broad range of options across our courses. IT Masters: In terms of the subjects available to study the courses that are available and then also trying to find the best option for each student within the course that they choose.
IT Masters: So we really want to make sure that each student has the opportunity to capitalize on their strengths and enhance the skills that they are looking to enhance learn the material that they don't already know. IT Masters: and avoid making people who knows sit through courses, where they already know the material possibly you know well enough to even teach it just for the sake of kind of taking a box. IT Masters: So talk a little bit more about eligibility in a second, but this approach is made us market leaders in it and we're really proud of the way we've been able to kind of achieve this for a really broad range of people. IT Masters: So some of the major benefits are the different pathways to entry so real our entry without a degree. IT Masters: into a post graduate level course if you've got sufficient industry experience or industry certifications. IT Masters: And then you can continue into a Masters even without a bachelor's degree to start with, if you have any bachelor degree, you can start in a masters.
IT Masters: And then people who don't even have a tech background it's not necessarily a handicap. IT Masters: We start people without a tech background with these short courses to build up the basics, we want to try and set people up for success and we you know have flexible options, you can study between one and four subjects at a time. IT Masters: Depending on what works for you the subject structure and things like that it's all really thought out to be as flexible and as accessible as possible for as many people as it can be. IT Masters: And we have a variety of courses, including one that's actually specifically tailored it's called the graduate certificate in computing Korea transition specifically tailored for people without that specific it or tech background to transition into this kind of Korea.
IT Masters: there's also the opportunity, if you are an Australian citizen or permanent resident all of our courses are available on fee help. IT Masters: So that's that's a huge benefit to people rather than needing to kind of pay up front and also if you're a citizen or permanent resident at the moment for 2022 that is. IT Masters: A combo of supported place scheme available for our graduate certificates in cyber security cloud computing and virtualization networking and systems administration and the graduate certificate in computing career transition.
IT Masters: subjects so there's a lot of those pathways, as you can see, on screen we offer six different intakes per year as well again part of that kind of flexibility and accessibility, that we really strive to provide for all of our students. IT Masters: So there are a variety of options, you know the pathways are pretty endless and i'm very happy to speak with people about the kind of pathways they're looking for whether that is. IT Masters: You know, studying now and then kind of applying for jobs or kind of applying for jobs and then learning on the go.
IT Masters: it's very desirable for a lot of people, particularly those with existing kind of tech experience. IT Masters: There is a huge demand at the moment, so particularly relevant to this short course huge demand for people with experience, skills, expertise and qualifications. IT Masters: In fields like cloud computing and cyber security in particular and employers are sometimes willing to subsidize study that's relevant to the job. IT Masters: As well as that demand right now being really, really high, to the point where people are being hired while still studying the course purely because the employer knows that they will have that qualification very soon so. IT Masters: As I mentioned earlier, with eligibility we really like to recognize the effort that students have already made so whether that's been to certification.
IT Masters: work experience if you're certified by the Australian computing society which you can find out more about just on the ACS website. IT Masters: You can get some course credits as well, so that's being credited for a subject so you can lessen the number of subjects that you need to complete in order to achieve a graduate certificate or a Masters by up to 50%. IT Masters: If you've got the right vendor certifications in the past, if you can demonstrate your experience to the ACS all your work experience were able to kind of assess that as well. IT Masters: The short courses, so these short courses, such as this one, if you complete and pass three of those we will award you one subject credit towards any of our post Grad level. IT Masters: Courses as well, so that can take that's up to you know 25% of a graduate certificate just because of completing three short courses.
IT Masters: Which is always a nice little kind of sweetener makes it a little bit more appealing. IT Masters: as well to kind of be able to try it out decide that you would like to study and then you're already one quarter of the way there. IT Masters: And if you've got previous post graduate level education will also assess that for relevance and can potentially award you recognition of prior learning about as well. IT Masters: So what now feel free to put any questions about eligibility or course pathways in the Q amp a section, and I can answer them at the end of the webinar.
IT Masters: A large part of my role at it masters is assessing the eligibility for study and for a recognition of prior learning so my email address is that and I will put it in the chat. IT Masters: Of the zoom once we finish presenting this as well as the address for the eligibility assessment and very, very happy to discuss any of you potential course pathways of any of the courses that you might be interested in. IT Masters: and check out our website other short courses and have a bit of a reflection on which pathway might suit you best if this is something that would interest you in the future so. IT Masters: Thanks, very much for for having a little bit of a listen about the.
IT Masters: Potential postgraduate study options for you and i'll hand you back over to TIM now thanks. Tim Jones: The thanks jack. Tim Jones: So just before we get into the question stem off the back of what jack was saying. Tim Jones: To give you my experience, I did not manage to for a number of various reasons didn't manage to complete an undergraduate degree, but. Tim Jones: In 2007 I managed to get 50% credit for the entire Masters Course and did a speed run through my masters in one year, so this year it's.
Tim Jones: A hairy time to head through and and get through all of that, but it's it gave me a well regarded tertiary qualification in quite a quick time and it's been quite value to my valuable to my career, since so yeah I recommended. Tim Jones: Okay, so just before we get into databases those few questions that came through about files that i've marked here so. Tim Jones: First of all, what is the difference between file storage and block storage so when we talk about file storage with it's around the technology that we're using to access the objects themselves so we using a technology like sips for accessing Microsoft based type sharing September.
Tim Jones: or nt based file sharing is through a secure protocol, or we might be through an nfl protocol, so what we're doing is it's pretty cold based it's requesting data. Tim Jones: On based upon an actual file itself or looking at file information from a system so. Tim Jones: Protocol, a high level order that we're accessing information by when we're doing block storage we're requesting specific blocks we're saying we want.
Tim Jones: You know if we're looking at an old hard disk with we want sector five track hundred and 26 on played a three right so and we want to pull that information off there so it's a very different way of accessing information but and using different technologies to do so. Tim Jones: The question from Leo would accessing the storage from micro services, the hype similar in a similar way. Tim Jones: as would a vm to the cloud storage side certainly possible when we're talking about micro services most often we're talking about accessing through.
Tim Jones: Containers Cuban 80s, or through function as a service or service technology, so we can grant all of the cloud service providers have. Tim Jones: Really robust identity and access management systems that allows us to not only give rights to individual users, but to give roles and access to. Tim Jones: applications and other services that we run within the environment. Tim Jones: So we can create, for example, a file share and give rights to that file share to a particular container or to a function, or something else, so we can absolutely use these within micro services.
Tim Jones: But it requires just a bit more planning than what you might have in a virtual machine environment where you can connect up and test things from a command line so. Tim Jones: The short answer is yes, but you just need to have a bit of thought about. Tim Jones: Another question from Leo is where is block storage used mainly night so it's absolutely always used as our. Tim Jones: In the cloud environment for our food system now operating system that we have on our environment so it's some whether it's an ephemeral disk or whether it's. Tim Jones: A different type of block storage, the other space is it's often used for when we need to have high speed access to data. Tim Jones: And so, if we're running a database for example we're running a our own particular type of database, the speed with which we can make updates and the access that we have to make sure we don't run into problems.
Tim Jones: Like file locks that we might have in a file system means that running a database, we would need to run it with a block. Tim Jones: Time disk rather than a file system and object store we're pulling down the entire object every time. Tim Jones: We want to read it and we're writing back the entire object, each time so an object store isn't appropriate for something like a database unless we structure it completely differently so that each element is a individual object, but then that's that's very different in itself. Tim Jones: Question from Derek other block sizes dynamic or predetermine during initialization, so this is this is set when we create the disk right so when we basically do a format command or something like that it's predefined how that disk segment or that this petition is actually accessed. Tim Jones: From grant Stevens TIM do other to the major CSP is offer similar functionality to send technology that is network file copy.
Tim Jones: That is file transfer which doesn't involve processing resources from a server so there's some capability for this there's the built in capability into database systems to provider, we can have. Tim Jones: distributed systems, especially for file systems that allow us to have something like a distributed file system that you might have from the Microsoft world. Tim Jones: If we want to do it with with block level resources for virtual machines will be having the cloud. Tim Jones: We can go through, and have some copies of that we can have some backup information as well, but it starts to get a little bit more complex. Tim Jones: than what we might have in the general enterprise environment so certainly the technology is only you can do all of the things that you.
Tim Jones: could do in a an enterprise data Center but often there are better cloud based technologies than the old ways that we would do things across a product Dr site so. Tim Jones: Looking at things through a different lens from the way that the cloud technology offers to do things can give you a better outcome. Tim Jones: They can the operating system be run on from an ephemeral disk yes yeah so that's that's a very common way that we run things because when we start up a virtual machine in the cloud environment we can give it. Tim Jones: static information so can go and download all of the programs that it needs to run from a central store or something like that, and it can be up and running right, so this is especially useful when we. Tim Jones: want to have a scalable environment we don't want to to pull all of that information and manually install applications, we want to have them install automatically or we can run and a system from a.
Tim Jones: Basically, a cloned environment we can create a template of one of our virtual machines and have that brought up on an ephemeral disk as well, so the only thing that we're missing out on in that situation is again. Tim Jones: If we turn it off or it shuts down unexpectedly, we lose the data, so the whole thing is, we don't store any important data on the local disk. Tim Jones: From anonymous we have so having metadata available to an environment, this would affect speed in retrieving the data so not the speed of retrieving the data itself. Tim Jones: This is the speed in which we can search for the data that we're after so once we know the object or file or anything like that that we're looking for. Tim Jones: Getting it for an object store or a fast or isn't going to make too much difference, the metadata provides us the capability to find the information we're looking for rather than retrieve it. Tim Jones: And last one i'll do before heading into the next section is from priya.
Tim Jones: How about backup method for all cloud storage ephemeral block object storage, maybe we can back it up offline or online or maybe something some type of storage and cloud. Tim Jones: So absolutely, so this is something which is very common that we have in the planning branch So yes, we can create snapshots of. Tim Jones: Our have any about discs that we have if we're using EBS volumes, so I don't think you can do it for the ephemeral volumes, but you can see EBS volumes or file. Tim Jones: or objects and the storage location and all of that goes back to is the object store so in the aws world that's talking about amazon's s3 platform is the backup space that we put all of these things to now.
Tim Jones: One of the things which a lot of companies doing a migrating to the cloud. Tim Jones: And the migration to the cloud is basically taking their infrastructure, as is their virtual machines and moving it from their data Center to the cloud space. Tim Jones: And so they're not taking advantage of a lot of cloud capabilities and as such they don't work in so well with the backup or other capabilities and.
Tim Jones: They may also have existing data centers that they're continuing to backup as well, so. Tim Jones: that's where like the company that I work in my day job they've got a tool called data domain as an example and there's plenty of others from other companies where. Tim Jones: All of those companies have integration with cloud capabilities now as well, so that you can have all of your backups from your data centers getting backed up to. Tim Jones: cloud providers, we can back up our virtual machines in the cloud to the same space and we can have a central console for how we manage the.
Tim Jones: backup and recovery and being able to, for example, restore our systems from backup directly into the cloud into our data Center or anywhere using smarter tools so. Tim Jones: there's there's quite a big market now for third party tools that can provide a much more into when view of data security and management, especially when you're looking at multiple data centers and multiple cloud providers. Tim Jones: Okay, so we'll go and launch into databases now. Tim Jones: But so just to start with. Tim Jones: What are we thinking about when we're looking at databases, so the first one is the scalability of the database right so will the solution be able to scale up.
Tim Jones: Then we're looking at the storage requirements, how much stuff will be checking into the database. Tim Jones: The object size and time, so what sort of information is it just some text is it some numbers, is it a whole bunch of files or what we called. Tim Jones: blobs, which is an acronym for binary large objects, so if you see blobs anyway, and so it's just basically talking about files that you would you would store somewhere. Tim Jones: or finally we're talking about, as well as your ability and so do we need to make sure that all of the data is available all the time. Tim Jones: How do we deal with recover ability and do we need to deal with things like consistency, so if we're having read from two different clients do they have to have exactly the same information.
Tim Jones: And so, so and when we're talking about database for anyone that's not familiar with it, what we're talking about is essentially a store of data in a. Tim Jones: Somewhat organized way, so if you think of an excel spreadsheet and the columns that you have on there that can be thought of as a type of database of where we can have columns and rows of data that has a whole bunch of information in there. Tim Jones: So we have two separate key types of database across the board, so the first one is relational databases, these are now what we would term as the traditional database types. Tim Jones: Right so Microsoft sequel server Oracle my sql db to progress sql Maria db a whole bunch of. Tim Jones: Other database technologies that are based upon the idea of having multiple tables and having what's termed as relations that work between them, so you might have a client number and the client information in one table and you might have order information in another way, you have. Tim Jones: Product numbers tied to customer number or something like that, and you build applications and information and capability through these relationships and it ties very closely with business needs and how the business actually operates and processes data and information.
Tim Jones: And non relational is something that's come about much more recently right so. Tim Jones: The Non relational databases what we would call primary key databases and what they provide is the ability to scale absolutely massively so. Tim Jones: When we have relational databases, they can really crank out with their capability to keep up with the number of users or the amount of information in the database, when you get above a certain level. Tim Jones: Non relational databases and made specifically for millions or billions of objects to be within the database and very rapidly be able to find the key information that you're looking for so examples of this are mongo db Cassandra read us, and so the main. Tim Jones: databases that we're looking at today from a non sql or no sql side are based, have a Cassandra sort of.
Tim Jones: backbone but but that's not really important. Tim Jones: So one of the other things, to think about with the. Tim Jones: database technology and and the requirements that we have now, we have acid versus base now they. Tim Jones: What we talked about when we talk about acid we're saying that it's atomic we're looking at individual components or individual cells within.
Tim Jones: The database itself it's consistent, we expect every time we read from any client we're going to get the same information back. Tim Jones: it's isolated it's not it doesn't have any requirements on other systems, give the data and it's durable, we expect that the data is going to be there when we wanted, so this is. Tim Jones: What was really tied closely to relational databases going back some time, so if you think about something like an ATM transaction. Tim Jones: If you write information that someone's withdrawn $20 from an ATM cash machine.
Tim Jones: To a database, you want to make sure that every other cash machine has that same information available immediately you don't want to have something that's. Tim Jones: Eventually, consistent, or that the information might just disappear or that it might be hard to look up or something else. Tim Jones: Right, so we expect that type of transaction a transaction to be an acid transaction the other type is base so something that's basically available it's where we're pretty sure it's going to be available for us it's eventually consistent so will.
Tim Jones: When a client asks for whether that clients got 20 bucks in their account then eventually they'll get the right information back and it's a soft stone, so that we we don't have a truly. Tim Jones: crystallized version of the database at any time because we might have multiple nodes and lots of people accessing it at the same time, so. Tim Jones: What All of this means is that no sql sort of generally ties towards the base side of the spectrum of having something which is a little bit.
Tim Jones: less rigid in the way that you can access the information and that the data is controlled, but in the case of no sql it's incredibly scalable. Tim Jones: On the acid side, this is more towards the relational database side now this isn't to say that you can't have no sql. Tim Jones: Where we're deploying something which meets the acid requirements or by servers, so this is just in terms that you're coming across working with databases, but that they help. Tim Jones: Really line up some of the things that look at from the architecture of how you deploy applications and the expectations of the business.
Tim Jones: Right so. Tim Jones: just want to check was there any questions that came up jack off the back on just the introduction to databases. Tim Jones: and IT Masters: I think you've actually gotten on top of them all that is extremely efficient. IT Masters: People have not been answering many answering asking new questions about these so i'm going to go ahead and assume that that means that you've been extremely comprehensive. Tim Jones: Go actually i've seen this. Tim Jones: One here that i'll answer so just a there's a question here does aws support sybase and do we have a choice to select db version on.
Tim Jones: aws for compatibility or portability purposes right so so know that so there's no support for sybase directly so there's. Tim Jones: we'll go into this in a second for the relational databases, but you can deploy an application in a managed environment, which is what we'll be talking about mainly. Tim Jones: Or you can deploy a database on to a virtual machine So if you have a sybase requirement, then you can just put it on to an easy to instance or another compute node within aws and but you can't run it as a managed environment as such. Tim Jones: And the second part, we have a choice to select the db version, yes to a degree, so you can't go back way in time two versions that have been superseded, but they support multiple versions. Tim Jones: Right so launching into relational databases. Tim Jones: So here we have.
Tim Jones: Support for across as you're an aws of pretty much all of the major database types, so we can create any sort of application and. Tim Jones: What we're looking at here is essentially as a platform as a service, and so what that means is that. Tim Jones: The cloud service provider looks after all of the software and the management of. Tim Jones: The instance for us make sure that software is working in, and we need to manage much less than we would in the traditional world, so you still need to have some. Tim Jones: database administration, you need to have the application integration and things like that, but you don't need to do the database infrastructure management.
Tim Jones: So the benefits we have had a relational databases so they're easy to use and they're quite understandable again, as I was talking about earlier. Tim Jones: If you think of a table within an excel spreadsheet that can be thought of as tight as a table within a relational database we've got fixed rows and columns of information and. Tim Jones: All of that is set to a specific schema so that we whenever we're writing and reading information from the database, we know exactly what type of information that we expect. Tim Jones: Every time we make an order, we need to provide customer number, you need to provide order number, we need to provide part numbers and.
Tim Jones: The number of individual components that you're ordering as an example of how we make an order and if we don't have all of that information, the application might work. Tim Jones: The we've got a common language that's used some sort of structured query query language across all of the major systems there's different ways of. Tim Jones: reading and writing it slightly between the different things, but mostly the way that you access databases and similar and. Tim Jones: As I was mentioning before the relational databases ideal if we've got a need for acid compliance. Tim Jones: If we've got strict schema roles, as I was mentioning around that order ordering based system or we need to make sure that we've got our data quality up to snuff that we want to make sure that all of the data coming in, meets some rigid criteria.
Tim Jones: So we don't have the capability to scale again to massive web scale applications, so when you're looking at a system like instagram. Tim Jones: Then that style of application, where you've got hundreds of millions of active users, right now, the front end for finding user data would be using a no sql database and not a relational database. Tim Jones: So. Tim Jones: If we let the cloud service provider manage our environment right so it's a bit different to what we would have in the past, so, first of all. Tim Jones: We have the host databases on OPS. Tim Jones: As databases on premise, this is where you manage everything so you've got people looking after the racking and stacking of your servers you server maintenance your ios installation looking after your patching your database your database management backups etc.
Tim Jones: If we've got it loaded when we're talking before about having sybase if we wanted to load sybase into the environment, then we need to deal. Tim Jones: With just everything above the operating system, so we still need to deal with patching operating system. Tim Jones: But we need to install the software itself and manage all of that, but we can make it a bit easier by migrating our sybase environment to the cloud, rather than having it in our own data Center. Tim Jones: Finally, if we have an application, where the version number that we want to run is supported by the cloud environment we can go through and use an RDS version for the. Tim Jones: aws side, and all we need to worry about is the application level and indexes and and data management within the environment, so a fair bit simpler to deploy manage and control, then.
Tim Jones: Putting everything else in the plan. Tim Jones: Alright, so just looking at the questions of the back of relational databases so just a question from Stephen description of database. Tim Jones: Like excel spreadsheet with columns and rows seems like structured data databases used for unstructured data, so the. Tim Jones: The relational databases are all structured data, so we have to have for relational databases, we have a schema that tells you what all of the.
Tim Jones: column names are essentially that you would have an excel spreadsheet So if you again had customer information, you might have their address their phone number. Tim Jones: The name of the contact and some other information that that might be in there, if you need to go and add additional information so, for example, it's. Tim Jones: You need to add someone's instagram handle from the company because that's a new, important piece of information that you require for the way your business runs. Tim Jones: And you need to have that, as part of the database changing the schema in a relational database is often a massive piece of work. Tim Jones: Because it can change the way that the databases work.
Tim Jones: And it can change the way that the applications interact with the system as well because again when we're writing information to the database every time we're writing we're expecting to give. Tim Jones: All of the information to fill out all of the columns as part of a row, so if we're creating a user and we have an update command from an old system. Tim Jones: That doesn't include that instagram piece, we might have an error in our application and the whole thing might die.
Tim Jones: So it can take a fair bit of work to just add that type of information now, when we go to no sql databases, we have a very different view of what we can do with that type of environment. Tim Jones: So i've heard art so from Michelle Benson i've heard of structured and unstructured db How does this relate so structured is what we're talking about with relational databases. Tim Jones: And unstructured databases what we're talking about with no sql right so and so you'll get a bit more of an idea of what that means in a second.
Tim Jones: Right, would there be any any advantage to dual wielding relational and non relational databases absolutely so there's always great reasons to have particular types of databases so. Tim Jones: As I mentioned instagram probably wouldn't have a relational database for the way that they access their user information and looking at key information for. Tim Jones: The way users access the application itself for the front door right so there'll be some sort of database, but more likely and no sql database, however. Tim Jones: Within the organization, they would have things like financial records payroll HR and other systems, but very heavily rely upon structured data that we expect to see. Tim Jones: In our normal world of how we deal with systems where we're dealing with scale of thousands, rather than millions or 10s or hundreds of millions right so so those types of systems are perfectly.
Tim Jones: suited for that type of thing and so it's very much horses for courses and with any enterprise you'll see. Tim Jones: just about every flavor of different database so Whenever someone is deploying a new application. Tim Jones: or you're bringing some application in from an external vendor there'll be a requirement for a particular type of database or a particular version and so you'll end up with.
Tim Jones: Microsoft sequel you'll end up with five versions of that six versions of Oracle you'll probably have some db to in the back end. Tim Jones: New application riders will be doing things in Maria db or my sql and then you'll have the avant garde new developers going through and doing a whole bunch of stuff on Cassandra or dynamo db or because my duty. Tim Jones: And do we still need to keep backup database in our own data Center after the db moves to the cloud right, so this is all around your risk appetite so the the expectation is that you can backup your database to the cloud environment and the. Tim Jones: expectation from saving an object which would be a database backup to the.
Tim Jones: To the s3 environment, so the object store the durability of that is 11 nine so the expectation is that your data is still going to be there in you know about. Tim Jones: 10 million years or something like that, but you're not expected to lose any bits per euro from your database backup so it's incredibly durable now one of the. Tim Jones: biggest risks for anyone though within any organization is someone doing something wrong from within the organization itself whether it's an administrator accidentally deleting things they shouldn't do. Tim Jones: or someone doing malicious damage a lot of this can be controlled through identity and access management again but there's always risks of that and some companies like to make sure that they've got ongoing offsite cold side site type access to their storage. Tim Jones: But you can do a lot of those sort of capabilities and things through. Tim Jones: correctly managing and implementing your your cloud service capability.
Tim Jones: Okay, so we'll launch now into no sql. Tim Jones: Alright So here we have again dynamo db or as on Neptune from aws we've got cosmos db as your table storage units your time series insights. Tim Jones: From as your and JP has cloud data store and add big title now the main area that i'm going to look at is is the unstructured side of databases, so this is. Tim Jones: Using the stated no sql that's based off Cassandra type technology for how you deploy a no sql database and there are other types of unstructured or no sql databases such as graph databases time databases, etc, so this gives us allows us to have. Tim Jones: Information on physical locations or on time based data that might come from security systems or anything else right so there's some specialized use cases where you utilize a specific type of no sql or unstructured.
Tim Jones: or non relational database to take the non that's where we have something like and as on Neptune or the as your time series insights and specific types of no sql database. Tim Jones: So, just a quick one there's just a question of where did the no sql name come from says basically that it's it's it's not your standard sql right so it's it's just something something different to to standard sql. Tim Jones: So. Tim Jones: All of these no sql type systems are fully managed non relational and their key value. Tim Jones: database services right, so what we mean by key value is that we within standard relational databases, you might have heard the term of primary key. Tim Jones: And this essentially is the sort order of the database when we're talking about relational databases when we're looking at a key value.
Tim Jones: database what we're saying is that in a similar way it's like a primary key but it's also the only thing that we need to know about a user that absolutely needs to be there and because we're able to look up just this information, we can find them super quick and so. Tim Jones: bear with me i'll learn, while this is. Tim Jones: Going I will. Tim Jones: Cross here. Tim Jones: To show you what it looks like within dynamo db.
Tim Jones: And it just takes a bit of time to create a table, so how does. Tim Jones: My table, so this isn't one rip. Tim Jones: So this will go away and create a new dynamo db. Tim Jones: table for me so running stock reading things. Tim Jones: So. Tim Jones: What we have with this is the ability to really scale in a huge way
2022-03-24