Introduction to Singularity: Containers for Scientific and High-Performance Computing

okay it's a minute past 11 pacific time uh thanks for joining everyone uh let's go ahead and get started we should have a big turnout today uh today we'll be seeing a presentation by marty candas one of sdsc's computational and data science research specialists on an introduction to singularity containers for scientific and high performance computing but before we get started i want to make sure we uh point out the the exceed code of conduct uh everyone's usually on their best behavior but if you do observe any inappropriate behavior please uh submit a report to exceed.org code of conduct and also in the interests of fostering inclusion and showing respect for everyone we want to be sensitive to terminology in the following presentation or slides so please report anything that you're concerned with to terminology at exceed.org as well and with that i will go ahead and hand it off to marty all right jeff can you see my screen yes looks good okay great okay um all right hi everybody um thanks for uh coming uh let me see if i can move this stuff out of the zoom thanks for uh coming today to the uh the exceed webinar um introduction to singularity um we're gonna talk about what singularity is what a container platform is and how it's useful for um doing scientific and high performance computing and sort of how you can possibly integrate this technology into your own research and computational workflows in the future i've been giving this talk for a couple years now and this is the first time i sort of tried to start rehashing it um singularity um the software we'll talk about today has gone through some changes over the last few years moving from sort of major version releases singularity two to three and there's been a significant number of changes um from singularity two to singularity three including sort of a push by um the singular developers to commercialize the software added a bunch of new features that even i'm not completely familiar with yet um but so i thought it was appropriate this time to start um uh rethinking the presentation here the the the previous talks which you might be able to find still recordings for online um we're aimed at really getting to some of the nuts and bolts of how to use singularity um and show some examples of how to build containers and run them uh on hpc systems such as comet and now uh expansive sdsc um but all of the exceed systems have have singularities so um the the aim of this talk now that i've sort of made it's a little different instead of sort of trying to teach you exactly all the commands that you would use to go build a singularity container and run it i i sort of wanted to more have this talk be an inspiration for your own exploration of if you you know sort of it's more of a pitch to why you should think about using singularity if it fits with your use case and why um it's an important technology to sort of understand out that's out there that you could use if if you need it for your use case um in general another reason for the redesign here is it's it's really difficult to do sort of live demos and make everything work within an hour and so i thought sort of structuring it this way would be a little bit better and i'll talk about that um briefly a bit more so let's go ahead and get started for today just make sure i can double check my time okay so as jeff mentioned there's the exceed code contact i have these in slides but we can skip that so today what i really want you to um take away from the talk is um answers to these three questions for yourself um what is a container if you're not familiar with what it continues what is singularity and what singularity can do for you okay and the idea again with redesigning the talk for today was i have some ideas about you know how we can if you are interested in singularity more of the nuts and bolts details and more advanced features of singularity there's sort of you know two other talks i can think about really um creating which would be sort of uh the follow-on to this introduction talk would be sort of how to create a containerized workflow with singularity walk through all the steps that you would need to understand to really build containers yourself and run them you know with you know before you do that you really have to understand some of the basic concepts and definitions and that's really what today is about and like what what's possible essentially right and then there's more advanced features now in singular e3 such as encryption and uh you know verification sort of integrity checking with you know you can actually sign the the containers upload them to different registries and there's a whole bunch of other things you can do with singular right now that you couldn't really do with singularity in the version two series essentially and so there's a bunch of other topics here that you can dig into and we just really don't have time today so this talk again is really just about to inspire you to maybe want to become a singularity pro here um oh and so one you know in the so these talks are things i'm thinking about for the future such as our summer institute so you'll you might see an advertisement for the summer institute this is sort of a week-long hpc forum that we put on and so one of these more in-depth talks will probably be part of that curriculum so if you're interested in this material today um you might want to check out that more advanced stuff later on this this year okay so let's talk about um containers so when you if you start learning about containers and maybe watch some talks by people there's always one analogy that people give to sort of prompt people to understand the concept of what containers are useful in the software world and it comes from a very different place in the physical world and so this is sort of a um a leading question but basically um you know i would ask you know what can hold 10 000 ipads and travel from shanghai to hamburg for approximately five hundred dollars and the answer is a shipping container right so a shipping container is somewhat of a modern invention right it was really invented in its modern form in the 1950s so it's only about 70 or so years old right but it transformed the way that modern freight shipping is done around the world and created the rise of globalization around the world essentially because before shipping containers were invented in this sort of standardized box that can fit on a ship a plane a boat a a truck a train um you had a very different situation right you had what was called um uh bulk brake cargo essentially so before containerization um in in the shipping world you had essentially goods were loaded and unloaded individually mostly by hand you know from warehouses that the manufacturers brought their goods to the warehouses they sat in the warehouses at the dock then once the ship arrived they loaded the goods into the holds of the ships and this whole process was really inefficient and back in the day most of the time the ships would actually be in dock more than they would be sailing right so you spent a lot of time handling the goods themselves and this could create problems like insecurity where if you have a lot of goods sitting around for a while being handled by a lot of people um you know there's a higher potential for loss or theft of goods as well and this this whole inefficient insecure process um costed more right and so really you would only have shipping of goods from manufacturers locally essentially you could only really afford to ship luxury or specialty goods that fetched a high price could really be shipped from long distances right but after containerization um you know you had this modern infrastructure created essentially you had a standardized shipping container that you knew the dimensions of everyone in the in the world basically has the same um standard for what the size of a shipping container is what the permissible weight tolerances are this allows you to efficiently load and unload the containers and they're more secure basically you can send the container to the manufacturer they can load whatever they want into the container that needs to get to the end customer and the the intermediaries who are transporting it they might not even know what's in the container all they need to know is how to move a container you eliminate the process of this bespoke process of how do you you know move a bunch of barrels of wine versus a bunch of sacks of grain you know there's different loading unloading process for those goods but if you just have everything in a single standard everybody knows how to move that single thing right and this created a much more effective way of shipping goods around the world and you can pretty much ship anything from anywhere for cheaper than it used to be right now so the question is okay so what does this have to do with computing well when we talk about um you know what we do right i mean we're you know application developers scientists we want to run our code get our work done move data around and we need to do that on different hpc systems we need to do it or we need to do it on a laptop maybe you have cloud credits that you want to run in a virtual machine on aws you have all these computing resources around you and they're very different in certain ways but at the end of the day you want to run your code and against your data or generate your data with your code but you have all these platforms that it needs to run on right just like you have a bunch of different goods that need to get to different places you can package your software in a container and move it around easily between these different systems so the problem the analogy here is really about um is about software deployment right so this is looking at through time right you can imagine obviously just like shipping computing and software has changed over time with the hardware that it runs on right you know back in the day you could you know you had to wire your software up you know got a little bit better maybe you had some punch cards that you had to you know print out and feed into a machine um on a large mainframe that took a team of people to run your software and you know eventually you had pc and then now today you can just order your computing infrastructure from your command line on your laptop right out in the cloud right so the evolution of the hardware changes how you deploy the software right and that makes sense over time but just today right on exceed right we have many different systems that you might have time allocated on that you might want to run your software on and even though these systems are very similar you know from comet to stampede 2 to bridges 2 and now expansive sdsc right they have very similar infrastructure but each one is sort of still like those old shipping those old sailing ships and cargo ships where they're all slightly different yeah they're all meant to do the same thing they all have you know places to store things like software and libraries but they're all slightly different sizes and not all they don't run on all the exactly same software and libraries and so that creates a problem for you when you have to compile your software on each of these systems and run it right each time you bring your software into one of these systems usually what most people are doing is recompiling their package to meet that will run on that hardware with the software that's been deployed there for them to support your application right and so this is one example that i did today this is like a code that i wrote myself and this is just two i mean this is one one one way i'm compiling it on comets at sdsu one way i'm compiling it on stampede and so i'm just downloading the code right i have a make file that i need to run but first i need to check what sort of software modules are loaded what operating system is available you know what might what dependencies does common and the sdsc team here here at sdsu provide on comment versus you know in this case i'm looking at a compiler and a type of mpi and then at stampede 2 it's completely different where they have you know different default software modules available here they have like an intel compiler with intel mpi and you can see when i run the make command eventually once i have my sort of unique software environment for each each machine even the output from just compiling the software is different right in this case it's not that big of a deal but it's just to illustrate that just to compile the software you have a different process on each machine and then of course you can imagine once you have a different process of compiling it running the software is going to be slightly different as well now this is a fairly simple code but you know some packages that you might be working with that you maybe you didn't even write you know they can be these huge you know exascale project you know codes that you want to use yourself have you know hundreds thousands of dependencies versus like the two dependencies that i have for this code essentially right and that becomes really hard to satisfy when you move from machine to machine right because each time you try to recompile things it's going to be slightly different because there's different libraries that have been put there for you different compilers different mpi different math libraries and so it's it's it's kind of like the old bulk break cargo thing you know we give you on each you know machine we run kind of a unique environment you kind of have to figure out how to fit your goods into you know our platform right and so that's what containers are really for right so a software container um is really it's not a physical container um or there's not really a thing so i put this microsoft windows thing here just because you know software used to come in an actual container but in this case a container when we talk about in terms of software it's really more of an abstraction there's not really a physical thing you could point to that's the container right it's really a set of technologies that allow you that you bring together that allow you to run software reliable when you reliably when you move from one system to another okay now there are things that we think of as containers when you start using them the first one that you'll likely encounter when you start using containers is a container image so a container image which some people just you know you'll just generally hear called a container because it's one sort of facets or instantiation of a container it's really a collection of files that are just saved it's either a file or collection files that are saved on disk and stores everything you need to run your target application or code um on any system essentially right so it has like you know source code runtimes system tools libraries etc configuration files whatever you need essentially to run a certain application you uh is is associated with that file or set of files which is called a container image so the image here is just so the actual image shown on the screen here is actually just listing some of the containers that we have on common here so these are kind of the two most popular ones that we support essentially because tensorflow and pi torch are hard to compile in on the operating system that is usually run on hpc systems it's a very good they're very good candidates for using containerization and so here you can see these are just they each of these files is a container image right that i'm listing here right this is a different version of tensorflow and pi torch that run on gpus and sort of have some dates attached to when they were created but these are sort of container images right these are just files so these are actually singularity container images now when you take one of those images and you want to run the application that it's sort of targeting right you run a container process essentially so sometimes when people are talking about containers they're really talking about container processes right and these are just standard linux processes that are running on top of the host operating system and kernel on the hardware of the system that you're you're on the the difference is that inside the container it's really an isolated software environment that's defined by the contents of the container image right and so inside that container when sort of your application looks around inside the container all it sees really is the environment that you've defined within that container right you can allow it to see outside the container that's sort of you know your either some options to do that when you're running these container platforms like singularity but essentially it's really just another process that you're running so here i have an example where i've just taken our tensorflow example i've started it just like any other batch job and if you have access to comment you can go find that directory and take a look at the example essentially it's just another batch job that i've submitted to slurm and i can go to the node on comet to the gpu that it's running on and just see that i'm running if you're not familiar tensorflow is a python usually interacted with through a python interface so here's my python process that's running that's the containerized python process that was started from that patch job and it looks like any other process that's running on these gpus where here in this case i'm running on gpu number three it's really it'll look exactly like any other process that you start on these systems it's really the only thing that's different is what the software environment around it the container process sees and that's the the unique thing and this is kind of the analogy that i've used in the past where um you know if you've ever seen the movie the matrix they have you know they go into this virtual world and before they go into the virtual world um called the the matrix right so they have this thing called the construct where basically it's it's the program that they run before they go into the virtual world to bring it to get all the supplies that they need before they go into the matrix right so you can think about a container as this kind of loading program it's it's where you're going to load all the software you need to run on any hpc system on any supercomputer or your laptop or you know a cloud virtual machine on say aws and you could then bring that anywhere you need to go right now i've mentioned um virtual machines and so that might be what you're more a bit more familiar with if you've used them in the past and there's a fundamental difference between containers and virtual machines and why containers are more interesting in say the hpc space or basically any space but um containers right have really direct access to the underlying uh hardware um through your kernel on the system right so if you're not familiar with how you know sort of a linux operating system works right you have sort of a user space where your applications run right and um all of the sort of utilities that you interact with sort of run user space and that those those applications call to the linux kernel and that's the layer that really talks to the physical hardware right and so the containers are able to actually use the kernel on the host machine that you're running on versus a virtual machine they actu a virtual machine what it really does with this so-called hypervisor layer is it emulates the hardware um a hardware layer that your application's running on so you have these additional layers of translation that are occurring when your application's running in a virtual machine versus a container which um you know is would seem pretty obvious if you have more layers of sort of abstraction and translation happening from your application running versus you know between you and the hardware you're going to have a significant performance hit right virtual machines can have significant performance overhead if you're running application on them versus a container is almost native performance in most cases you won't even be able to tell the difference right so that's one of the the big advantages so performance is is definitely the the biggest advantage of of containerization right so but it the you know it's versus say a virtual machine but the the next thing is the freedom you can bring your own software environment wherever you go right you don't have to come to our team and ask hey can you compile the software for me can you put this software on the system in some cases you can't really containerize things and we'll talk briefly about that but in general if you want to put anything you want into a container that will work for your workflow you can do that right and so you can and it should and in general there are some caveats it will run on any system that supports that sort of container platform such as singularity right the other thing for scientific computing in general right like one of the big topics in science in general is reproducibility right if you're running some simulations or doing some data analysis and you run that analysis or run that simulation say on comment versus stampede 2 you really hope if you put the same inputs you get the same answer on both right but there's all these slightly different you know software environments that you might have compiled against maybe one math library has a slight bug on one system versus the other and you might get different results right so if you containerize your package you at least have a snapshot of the exact software applications and dependencies you use to run a piece of software or an analysis or a simulation right and so it will always and should always reproduce the same output for the same input because you have exactly the same applications and libraries you've compiled them against or built your software against um and the last two you know compatibility you know linux is you know widely available on many different systems these days um and um all of these container platforms that you've heard of such as docker now singularity i mean they're built on this similar set of standards that are sort of still forming but you know widely uh widely used in in both the research academic community but also in industry right and of course the the portability aspect right being able to build your application once and run it almost anywhere is kind of again the freedom to sort of bring your own environment and only have to do it once essentially right now there are some limitations uh to containers right so um architecture dependence does matter and the binary format of that architecture so you can't for example take a container that you built on an x86 machine to an arm machine right you're going to have to rebuild the container for the arm architecture right um portability is somewhat limited sometimes depending on uh you know it can depend on sort of the versioning of the underlying c library and your kernel uh on first in the host versus the container so for example if you have a scent os 6 machine you the kernel on a stentos 6 system is too old to run say newer containers that are built with say an umbuntu 2004 operating system in the container right so there is some kernel compatibility issues sometimes depending if you get too far away and also you can run into issues with hardware drivers is kind of where the real caveats come in you have to have some sort of compatibility between hardware drivers that are either on the host systems you're running on and the ones that you have in the container there's some solutions that are out there for getting around these a little bit easier these days but in general these are kind of some of the sticking points um and file system isolation this is i don't know if this is a limitation but basically this is kind of a another another point like the file system inside the container looks different than the one outside the container which is understandable but you can manipulate what the file system looks like on the inside so for example if you want to see your luster scratch space in a container on comet you can do that you just have to add like another option basically to bind that sort of file system to your container now i've mentioned docker if you're not familiar with docker this is probably the most if you have heard of a container platform before it's kind of most common one you've probably heard of um and this is really the first container platform that brought together all those different technologies that really made containerization popular the problem for us is it was really designed for sort of network-centric services you know it's really designed for industry like you know people running web servers and databases and while it's easy to install like well-developed documentation there's a huge community out there it's not really designed for hpc systems right hpc systems are shared resources right you can be running on the same node with other users running their scientific code right next to you right and docker security model doesn't really support this it really assumes you have sort of trust between all users running on the systems and you know in in in most hp systems um you know the the system administrators don't even trust the users right because um you know we don't allow you to have say root access or suitable access to install your own software that way right i mean so you're probably familiar with that if you ever tried to sudo app get something on to uh one of the the exceed systems right you can't do it right um but also things like um you know docker wasn't really designed to support you know batch job schedulers or tightly coupled applications that need to communicate between lots of processes such as you know mpi applications and so you know when when scientists were first asking you know system administrators to hey we want to run docker and do all this cool stuff on you know hpc systems and it just really wasn't tenable right and there just wasn't really a path forward and so that's what sort of inspired singularity right it was designed from the ground up at um berkeley lab to be the doctor essentially for hpc like do all the things that containers can do in a way that allows them to run on supercomputers essentially but you get all the advantages of portability um uh you know easy to share and distribute and reproduce you know the the containerized software the two things that are really different with singularity is it allows the sort of no trust security model so it fits in the hpc space by you know allowing the container to run as you you don't need root access to run the container it's just sort of untrusted users can run untrusted containers on most hpc systems if singularity is installed right so you can just build whatever you want your container upload it to comet and run right or expanse or any of the other exceed systems and it was designed to you know be compatible with hpc hardware like you know high-speed networking like infiniband or you know gpus and things like npi so these are some of the features of singularity um which i've sort of touched upon probably already but um one thing that's different about singularity versus say docker if you're familiar with docker is each container image file for singularity is a single file right so there's not this whole layering structure that you find in docker which can be a good thing or a bad thing depending upon the situation um but the single file single image file um is kind of nice it's easy to share with say a collaborator you know it's just you know it's easy to keep track of all of your it's easy to keep track of basically your your your software environment right you don't have all these different layers that might have dependencies that change if something gets patched and it's it's it's nice it's basically a static file that doesn't change once you build it it's going to be the same file it's not going to change it's it's read only actually once you once you build it and so it's it's sort of an immutable read-only file that you know if you want to change it you just build a new version of it again the the security model there's no sort of root owned processes there's no way for the user to change or escalate privileges to cause these security concerns that system administrators would be worried about so it supports this mult shared sort of multi-tenant resource environments that you find at hpc centers right and of course like as i mentioned you know it supports things like infiniband gpus and mpi so the most common use cases that we see at sdsc are for for using singularity containers sort of beyond i think hopefully what i've convinced you the benefits of building your own environment bring it wherever you want is what we use it for is usually to support applications that require newer system libraries that are available on on the host system right and what i mean by that is if you you know right now we've recently deployed expanse and we're running one of the newer the newest operating system cenos eight and but we haven't deployed necessarily every single package um you know that cenoas has in their repositories on the system but we get requests from time time for people like oh hey can you install this can you sell that um we can but um it's it's a process for upgrading um you can imagine you know a few hundred or a thousand nodes you know thousands of nodes um for the system administrators right there's a process that has to go on so it's the turnaround time for requests like that if you need something installed in the base os of any um operating system on an hpc system right that's kind of a lot of it's not necessarily a lot of work but it's a long process for that change to get pushed out to the system right so if you really need um you know software you can i mean containers are one way to get that right you can build your own and just upload it and run it if you need like abusive software right so but sometimes when those requests come in you know maybe we can't support those newer libraries because maybe the you know the kernel version uh that's that's running the operating system isn't compatible with what you need um and as i mentioned um earlier in the talk you know one of the two of the most common ones are sort of tensorflow and pi torch a lot of the machine learning applications right these things are evolving really fast and they usually assume you're using the latest and greatest operating system and in both those cases usually they assume you're running some sort of debian based ubu operating system which most hpc systems just aren't doing um and so there's a lot of incompatibilities and it's not necessarily easy to compile those packages natively on those operating systems right and so that's how that's what's what's our most common use case i think is what we use it for ourselves other cases might be like if you have an application that you pay for from a company they might only be give for your research they may only give you um a compiled binary right you might not have the source code you might not be able to build it um you know your license agreement does allow you to to have access to any of that that kind of stuff right and so as you know the again those companies might be assuming you have a certain set of requirements and operating systems that we can't meet on those systems well you can then use a container to package that binary with all of the things that the company says you need to have installed in your operating system you can put it in your container and run it from that container and then the the last one is uh probably i would say probably the second use case we see most common is is people find some you know have some existing docker containers that maybe were developed you know previously by the research group or you know a collaborator and you know you can't run the docker containers directly on say comment or expands but you can convert a document container to a singularity container with singularity right and then run it so that's those are kind of the three three big ones i think i think i would say now um sort of the last half of the talk here is really to sort of talk a little bit a little bit more about the nuts and bolts but not go into too many details um so hopefully sort of i've inspired you to sort of get curious at least to take a look at singularity for maybe one of your use cases but this is sort of how you use singularity kind of this last last bit of the talk with some some quick examples essentially but um this is kind of the workflow to think about when you're talking about using singularity and what you need to think about can you use singularity um one of the key things is is if you want to build your own custom container you need to have root access to the system essentially right you need to have a personal computer a virtual machine say could be on say for example exceeds jet stream you can get virtual machines there it could be on aws but the biggest stumbling block i see people who are interested in singular but don't really go their own way of say building custom containers for their own workflow is they don't have access to a linux machine where they have root right usually the majority of users have say a mac or a windows machine and that's not necessarily going to work you can work i'll show i'll touch on briefly how you can get started with it but ideally it's not really a long-term solution necessarily for you if you really want to get serious about building custom containers and using them in your workflows because essentially at the end of the day actually building your containers and you're sort of maintaining a mini linux operating system in the container so you kind of need to get familiar with with that sort of level of detail at some point um right so this is the the one stumbling block on on some for some users essentially where you you really need to build your containers elsewhere there are some new features of singularity which we have not tested yet for example sdsc and whether or not we would ever allow them you could in theory build containers on comet or expanse or any exceed system but right now i don't see that happening anytime in the near future maybe in the next year i think i'm going to push for us to evaluate that but um that's a topic for one of the more advanced uh talks essentially so but if you uh go down the route of you know wanting to build your own containers this is your first requirement you need to have somewhere to build them um and then all you have to do is you know once you have your your your custom software environment built in your container you just transfer that file to that singularity container to the system you want to run on and you run that container on the system that's it so the first part is the hard part now there is a way to do this on your mac or pc the the way that i recommend the way that i've been doing recently since i my linux laptop died and now i have a mac um is you can install virtualbox so this is a virtual machine software from oracle you can run pretty easily on your laptop or desktop so i would definitely recommend checking it out if you're interested in playing around with singularity or just linux in general you would you know install virtualbox create a virtual machine uh with your favorite linux flavor and then you can install singularity on that machine and use it um and the recommendation here that i have with the star is just if you're going to do that and actually you know run the containers on any of say any of the exceed systems you really the general rule of thumb is don't use a version of singularity that's higher than the one that you want to run on the hpc system essentially there are still like like other you know software singular is kind of evolving a little bit fast and the backwards compatibility from the the container image format sometimes breaks between versions if it's not um or i mean you can't you know run a newer version necessarily if you use a newer version of singular you can't necessarily run that newer container from that newer version on an older version of singularity maybe you can but it's not guaranteed um and so these are probably kind of running out of time here i might skip over this but um if there's three commands that you need to learn when you first get started with singular it's these essential commands it's sort of you know the build command you know to build your own container um shell sort of allows you to interact um with your container you can essentially log into your container and um you know run the software interactively like you would on an interactive node essentially or it's also useful if when you're building containers to actually modify the containers on the fly if you need to which isn't ideal but it's a topic for another day um and then the exec command is you this is what allows you to essentially execute arbitrary code within your containers right so anytime you want to run say your tensorflow script essentially you do a singularity exec you know python and then the name of the script that you want to run right it's it's it's it's how you it's just the the the the one command to run them all essentially right so any command that you would just run on the command line and essentially you can be fed into singularity exec and it'll execute the only difference is it's executing it within the container that you specify so like i said i think we're running a little longer than i wanted so i'm going to skip over these commands all these slides will be posted with the talk eventually so if you want to flip through them you can check them out again i mentioned that you know i've given this talk several times so you can find the old version of this talk where i probably went through more details of this kind of stuff in in that version of the talk if you if you want to take a look at those those should be available on the sdc website um so i'm going to skip over this for now and just again this is my first pass through this new version of the talk so still need to flesh it out a bit more um one thing i will mention is okay if you want to build singularity containers you need something called the definition file this is sort of the manifest of everything that you want loaded into your container right so essentially it's you can see here in this example i'm doing a bunch of act get install commands essentially if you want to build a container from scratch you have to write one of these definition files right and so this is where you put everything that you want installed in the container when you run that singularity build command this is how you specify it right and i have a repository here available on my github that tracks all of the containers that we support on say comet and now they're sort of in the process of updating most of the containers for expanse and essentially it's just a collection of different definition files for different applications that you can check out so they're good examples um maybe the application that you want to use is in there and you just need to slightly modify the container to add a few more dependencies that you might need um for example i often get the questions like if someone say using my tensorflow container and they're like oh can you install this other package in it because it's not already in there it's like well it's like you know i have to weigh the the cost of you know how much time it takes to rebuild the container is this package going to be useful for other people you know if you want to modify your own version of our containers this is you know the recommended way to do it you know learn how to build it yourself you can modify it put anything else you want in the in in those containers right so this is really base examples that you can start from essentially or you can if you have a request you can send me a request if you think i need to add a library to one of them so building containers interacting with containers running containers again so there's there's a few caveats that i need to touch on at the end here if you have an mpi based application uh things like this is one of those caveats that um where portability can become an issue right and the the fundamental thing that you need to keep in mind is that essentially you run a singularity container with an mpi based application the same you would natively on the system right you have to have you have to choose some flavor of compiler and mpi on the outside of the container and that has to be compatible with the version you install inside the container right and then there's all these things i mean and then the system if it's running with infiniband you have to make sure that the drivers inside the container um are compatible with the drivers outside the container uh on the host system so if you know say comet and expanse have very different um infiniband driver versions you know the container if you take one from one system to another it might not work out of the box you might have to modify it slightly to get the mpi working so this is one of those special cases that you need to you know take a little bit more care of if you're if you're going to build containers for mpi based applications um and it's a similar situation with gpu containers however singularity now has really built-in options for um for gpus both nvidia and i guess now i haven't tried it myself but amd gpus so they have special flags now allow you to sort of load the drivers from any system that you're on and use those from the underlying host system so you don't necessarily have to match the same versions inside the containers so if you go to my repo and actually get into some of the details of building containers you'll see i i do install like nvidia drivers and uh different versions of cuda in some of these containers um i do that usually just to match our systems but you could use those on any system if you do this sort of import method with these flags that allow you to bring the external driver modules essentially outside into the into the into the container when you run it and i think i'll end here with one minute over what i wanted to go but uh so this is kind of the summary um for my pitch for singularity for you if you're interested in learning more it really is a way to um install software that you need on any hp system without having to ask the system administrators or users sports staff like me i mean we we spend a lot of time um installing and maintaining software for for for you for for our users and you know sometimes it takes us a long time because we have a lot of requests and so if you but if you learn how to use singularity like you don't have to sit around and wait for that those requests to get filled um or in some cases um you know we won't be able to fill them in for certain reasons um but uh you know this the nice thing about having singularity if you do it yourself with singularity and you know it fits your model your computational workflow you don't have to go i mean you know you know asking for a piece of software just on expanse and then comets and then bridges 2 and stampede 2 right those are you know three sets of different people that have to do the same thing for you on these different systems if you figured out how to do it once in singularity and it was exactly what you wanted um you could use that same container on all of those systems right that's kind of the the the the key the key thing this this idea of portability and reproducibility on any system that you really want to run on right um yeah and so that's kind of my my pitch for singularity so um we have a few more minutes left i mean that i can either take some questions or um uh i don't know what is jeff so yeah there was a pretty good backup there was pretty good dialogue happening on the chat um maybe mary and miriam mahda were being very helpful in handling those uh maybe they have something they want to bring to your attention that you may be able to help clarify sure there was a question early on about whether there are instances in which a virtual machine um is required or or better than a container and things like that but it's a fairly extensive chat so marty or mary do you think there's anything that that i mean uh mahi anything that marty should should be aware of yeah i think there's one that uh right at the end that i haven't looked at yet and i think maybe marty can comment on that have you looked at fake root uh yeah yeah so um yeah that's a good question so this is uh i i sort of sidestepped this but briefly alluded to it that there is this way to um in in the newer versions of singularity i think it's particularly three six and above that we would have to move to um which were on at least the sdsc were at um 3.5 and i don't know i haven't checked if bridges uh psc and stampede i know they have singularity 36 on their systems and that should support the full fake roof um option for singularity which essentially for for you know without going into too many details it allows you um sort of a fake root access to your container on a system where you have no root access and so in theory you can install or build your own containers on our systems so if you don't have access to a linux system you want to say log into expanse and build your own container on expanse in theory we could do that with the newer versions of singularity however this also requires special permission in the configurations of singularity which is why i don't know if i haven't checked the psc and stampede or the intact folks support this yet so it's it's it's an open question whether or not i think we'll support it for users in general it really does require a specific special request at this point um and we have not evaluated ourselves yet so um i would yeah it's it's it's a possibility down the road is what i would say but it's not it's not supported at this time at sdsc yeah there's another question there um is there a repository in singularity uh where already built images could be run later uh to save time yes yes yes so this is one part that i uh sort of need to get back into um the uh i'll make some comments on this um yes so there are so this is one topic i didn't touch on so singularity does have does supports um what we would call container registries right so these are essentially places where you can um usually the registry itself usually stores the containers usually they're attached to some sort of build service where you can actually take your definition file this manifest of all the software you want upload it to this sort of cloud service that will build the container and save it in the registry for you and there are um if you go to scilabs is the company that's popular that is commercializing singularity which started as sort of a research open source project at berkela right this is the company that is uh sort of commercializing it and they do have um uh services such as the container library which is this container registry where you can find stored images so you can browse what containers are available there they also have a build service so you can upload a build file there a definition file and get your container built um it's sort of a freemium model so if you want to try it out you uh you know create a login you know you get i think 10 gigabytes of storage i don't know how long i mean there's some limitation on the build service itself too um and so that's the the singularity sort of specific uh registry um so it's like docker hub for singularity right um there's also uh i'll mention singularityhub which is the sort of original uh registry and build service for singularity um however while i was uh working on this talk this last week and updating everything and getting familiar with singularity three there was actually an email from the developer who made the admin who maintains singularity hub and she's actually uh not going to be able to maintain it anymore so it's sort of up in the air how long the singularity hub here will um stay up so it's it might go away this year it might not i think it's looking for a new home and a new uh caretaker so um so if you're if you're if you're if you're um so yeah so there so there are um there are singularity container registries and so you can find different collections of containers here so i have a bunch of if you go to my naked singularity project with the definition files um you know the if you look at the the readme there uh you'll actually see that i have a bunch of our base containers are built on singularityhub so you can actually download those base containers and play around with them and sort of build from there uh so i so i myself have to sort of think about a migration plan if singularity hub goes away but uh um so yeah there are there are registries and also the in the newer versions of singularity they also are now singular is sort of what is called oci compliant open container initiative and so from what i've read i have not tried myself you can actually use um say the amazon container image service the azure container image service um to uh store your your images there as well so it's i think it's called oras compliant is sort of the the open container initiatives sort of registry standard you can use basically any registry that is compliant with that standard you can use now but again these are sort of the more advanced topics that maybe in the in future talks beyond the this sort of pitch talk you can we can get into so marty there was one more about uh you started off saying that commercialization is coming will we end up in the center stream situation well um yeah i mean i i mean honestly yeah so i don't know um right so this is i mean i'm sort of i mean you know a minor a minor detail is is sort of the situation i found myself in this last week now you know i've learned that you know singularityhub might be going away which is kind of a free open service you know provided by stanford in collaboration with google right so google sort of gave them free nodes to build the containers on and store things in their cloud and you know i don't know if those resources are going away if since the the maintainer can't if if no one's going to help maintain the service it's sort of up in the air um but scilabs right it's a paid service so maybe we'll pay for storing our containers here now and if it's yeah it's it's an open question i think i mean it's how my impression is a lot of if you look at a lot of the features of the more recent singularities is they're they're they're supporting a lot more of sort of the enterprise um aspects that were never really part of singularity from the hpc aspect right and so there's there is a more hybridization of what singularity is becoming i would say um but that's good if if it supports singularity long term right if it meets um you know the scientific computing hpc needs for here into the future you know that that sort of focus on enterprise features and aspects is helpful for us because you get things like encrypted containers and uh container verification which is important in enterprise situations right there there you know some some compliance uh issues really mandate that sort of thing and that's helpful for you know if you know that's those sort of features are helpful for say if you have uh protected data that you need to analyze right you could have an encrypted container and run it on any exceed system um safely is you know i mean i'm not super familiar with all the the ins and outs of the these features but down the road i mean that's the types of features that are being added that could be helpful for the hpc and scientific community so but yeah it's it's you know choosing a container um platform is yeah it's an investment in your time and energy and whether or not uh i mean if it works for your workflow it's it's gonna be helpful i think long term um but it's yeah which one survives um there's i mean there's other hpc container platforms there's something called um shifter i think was one of the ones i think that's still being developed um and maybe there's one other one so there's other ones out there it's just singularity was the most popular one most widely when used essentially up until now so anyway i'm probably going on too long there um any other questions so there's one that said uh how do you compose a reproducible workflow that requires applications from multiple containers so um there's two answers there um so in the past i mean so this is sort of my own journey uh with singularity is when i first started maintaining all of the containers that we supported sdsc um for the most part uh i had the definition files were [Music] completely unique per use case essentially so they bootstrapped from a base say debian operating system or sent operating system and installed everything into that one container now that adds like build time and it's harder to maintain so for example like if i had a base um you know container with kind of the operating system and all the sort of utilities that i usually want in it for everything that i'm going to do say downstream with different applications i'd have to sort of keep everything um synced up manually where if i took that base container then added cuda to it i would have sort of a ubuntu cuda container and then in cuda um mpi container right and so they were all unique and but they were single build files like you only need that one file and that comes because comes a bit of a problem um to maintain however in i've reached recently in the last year so restructured our you'll see if you look at my my definition file examples i've restructured things to build off of other containers so i start with say a base operating system container with the libraries and things that i think i'll need for pretty much any other container that i'm ever going to build and then i start if i want to have that umbutu container with cuda then it's like okay i point that definition file back to a previous build of that other one of the base operating system without cuda right and so you started it's almost sort of recreating its own sort of layering structure and chaining between definition files to build up the container you want at the end or so you could build multiple applications from those different bases and the one thing that i saw that's really nice in the the latest versions of singularity which i need to which i'm definitely gonna look into myself is with all of this um um uh sort of uh encrypted verification stuff that you can use now you can actually um part of the definition files you can actually specify um the uh sort of the signature of the upstream containers that you want to build off of right so you could specify in a definition file like you can only build this definition file from this container if it has that signature right and so you can really hard code in a complete verifiable secure workflow um if you want to with the latest versions of singularity so that which is kind of cool if that that meets your use case anyway hopefully that helps answer the question yeah thanks we'll make that the last question um a little bit past the hour now thanks very much marty thanks everyone for joining us we will make the recording and slides available within the next day or two so uh have a great rest of your day everyone thank you thanks everyone have a great day thanks marty great job thanks everybody bye

2021-01-26 19:53

Other news