Using Docker Hub at Scale to Support Micro Focus' Delivery and Deployment Model
Welcome. To docker count 2020, live my. Name is Patrick dule I am a senior product manager at, macro focus, driving. Our IT operations management. Platform, and shared services. My. Session will cover how at macro focus, within the past three years, we. Were able to scale docker hub from 0 to 400 plus repositories. While. Enabling a new delivery and deployment model of our, containerized portfolio, for. Enterprise. Customers, thanks. For watching. This. Session will cover an advanced use case of docker hub and, how. We were able to leverage and scale this service, to publish, and distribute our, premium, software to our customers. We'll, start with a quick introduction on macro focus we are global, British an organization. Founded, in the 70s, with. Over 12,000. Employees. The. Company grew, through a set of M&A s over, time I personally. Came with a merger of macro focus a new-laid, Packard enterprise, software. We. Are one of the top pure software companies. With. 40,000. Plus enterprise, customers, worldwide. We. Have all the tools you need to build operate. Secure. And analyze, the enterprise, and, by. Design those tools bridge the gap between existing. And emerging technologies. Which. Means that you can innovate faster with. Less risk in. The race of digital, transformation. Talking. About digital transformation we. Actually started our own journey, about. Three years ago with, key initiatives, to. Modernize, and simplify. Our IT. Software portfolio. We. Decided to transition our IT operations management. Portfolio, by leveraging container technology. While. Bringing a new sense of agility into. R&D organization. Improving. Team velocity. And software. Quality, in the process. So. We went all in with container. Technologies, and embrace, the cloud native ecosystem. We. Embarked, into refactoring, or software and so. Indeed it is a journey, we. A joined. The CN CF foundation last, year to, get more visibility into. The ecosystem. So. Why does it look like what does it look like well. We. Move from 40-plus, legacy, products to five key containerized, Suites also. Known as applications, in, areas, such as service management cloud, management. Enterprise. Monitoring. Data. Center, and network automation. Those. Are our core skills, we. Also offer different editions, based, on the use cases or customers, would like to deploy. We. Ended up delivering a kubernetes based platform, that. I've been driving since the beginning, and it, provides, us with a solid infrastructure, abstraction. And a, set of key shared services. That. Were common to run or core portfolio. Services. Such as Identity, Management for. Authentication, and authorization. Collaboration. With, chat ups licensing. Log, monitoring. Orchestration. Dashboarding. And analytics. By. Bringing the portfolio together under a common platform.
It. Improved, the overall we use and sharing across the organization, it. Put a stop to duplicated. R&D, efforts and increase. The level of integrations, between those Suites. Now. When we studied this project, we, had few important, goals in mind especially. For our first production, release, back in the summer of 2017. We. Certainly wanted to ship our platform bits but, also get a strong, adoption, from aportfolio, applications. Another. Goal that was very important to me was how we could accelerate, well, really fast track, the, delivery, of our assets into our customers, environment. Something. That could take days or week with, our existing, legacy processes. And. Software. Distribution, channel and. Obviously. This is not to. Be expected this, is not the. Expected. User experience. That, you want to provide when. Dealing with cloud ready or SAS based enterprise software nowadays. We. Had been building, enterprise, software for thirty years and so, there is definitely a rich knowledge in-house but. There are always opportunities, to. Improve and in. This new context, we had the perfect mix of DevOps container. And cloud. Enablement. We. Had found ourselves ready to address some of our customer, pain points the right way so, how. Could I offer a simple, docker, pool experience, for, our customers, how. Could we start an in-house DevOps practice, to. Ensure that we're able to automate, the publishing of our container images in. A repeatable, and consistent, fashion. Every, time we, release our. Update. Or software. So. We went looking for a docker registry. One, that, could meet a very. Tight deadline deadline. At the time because we were about 30 days left before we week alive and. Knowing. Also our current skill, set which was also important, so we looked at few options, such. As artifactory. Amazonas. Er. Porters, from Susie, because Susie was, part, of micro focus at the time and. And. Finally docker hub now. We needed to ensure this was SAS friendly. But. We also knew that we could not set up shop in the cloud right away we. Did not have a strong DevOps culture, at the time or a group. That. Could quickly help us host and manage such. Service. Even. Our own IT organization. Was. Not able to meet our tight, deadlines at the, time. So, we looked at Amazon ECR, but. We we, found couple hurdles the first one was of overall. A lack, of knowledge of he situ. Within. Our teams we. Also found out that the ec2 security, model was maybe. A bit too complex, to, bring macro focus and, our. Customers, together in a very simple, and efficient, way, also. On the pricing, front we would have to pay not. Only for the amount of data stored, in our repositories. But. Also for the data transferred. And so, we're not able to gauge. The. Cost associated, with our expected, set up and downloads. We. Did, a first analysis. And. It indicated, that we would end up paying about 20, times the. Amount compared. To a docker. Hub set up for example. So. We went back to docker hub we. Already. Had few public, facing. Images. In the darker store at, the time and so, we knew. We. Could use docker. Through. The use of private repositories. If we wanted to. The. Are back model. Variable. To us was also simple, enough so that we can move forward quickly and, the. Pricing, also was quite, attractive, as docker. Was charging. Only on the number of repositories. Per. Organization, so we knew we could start small and grow. As we. See fit. We. Also had to get some approvals, from our usual stakeholders.
Something, That a large, organization. Such as macro focus has, to deal with so. I spent a great deal of time and effort explaining. The benefits of, our new solution. As. We're. About to roll out you know the new containerized. Portfolio. So, we met with legal because, obviously we were introducing, the. Use of a third party service in. A distribution channel so something to look at a bit more carefully, we. Work with security to, ensure we meet, their. Mandates, when, it comes, to. Image. Signing. Credentials. Management, for internal, and external stakeholders. As well, we. Also partner, with release, operations. People. Just, to ensure that we have the right governance, model that. We are able to, report. Back on. Who. Is using those repositories at. Any time and, we. Also enabled, our support, organization. Mainly. The. Impact felt was more on the operational, side than the sudden. Download the download aspect. So. We ended up establishing a, new delivery pipeline for, a premium software, and the, same pipeline is used today for. Initial. Downloads as well as ongoing updates and, patches. So. It starts with our R&D building in-house, a set. Of you, know software. Images, for. Daily and sprint activities. And we, use a combination of github, artifactory. Docker. And, Jenkins, as our, main tooling for this work. We. Use also a dedicated dev up pipelines, to automate and publish our container images on docker hub. When. We get ready to release so, we release, right now our cadence is every. Three months, so. You can expect a bit more activity on, docker, hub towards, end of cycle as we need to potentially, create, noodle, new, repositories. For. Our applications, or maybe additional. Shared, services. We. Also scan, and sign, those, artifacts, so. We work, with our chief security office, to improve. Our. Docker. Images, or. Software images. Based. On the scan result, we. Sign those images, using docker notary, so that our customers, are aware that those images are indeed coming from us and that they, have not been tampered prior to any deployments, in their data center. We. Pujoles assets into da crab in many, private repositories. So. None of those item premium. Assets can be found in the public facing side of docker hub, you. Always have to authenticate, and. Finally. Our customers, and partners can, access and download all the required images, by, accessing darker, hub using, their own dock your ID. We. Have automated and improved the streaming of those assets so based on their existing, software entitlements. And chosen. Capabilities. Our. Customers. Only download what matters to them and, that. Does reduce. Pretty. Substantially. The amount of images that needs to be pulled, some. Of our largest applications. Require over 100 plus images to be downloaded, so. We were not going to we're. Not going to let our customers. Visit. Docker hub and pick. And choose images, now that's that's not how we're going to proceed. The. Download of those assets, are. Is. Fully, scripted. And can. Be executed, either ahead of installation. Or during, the installation process. If. You look at our customer, base today 75%. Of our deployments, are, happening, on premise, with. No access to the. Internet so behind the firewall so. We, had to find a way to, provide our customers a way. To download those assets from an internet friendly host and allow, them to bring them back into their private data centers. So. Now that we're given the keys to dock your hub how. Where. We going to use it efficiently for our use case. We. Had to build a little model around, it. For. That we wanted to keep it simple. But. Certainly keeping it secure. Simple. Because we were able to we had to go and explain it and roll it out very, quickly as we're. Going to go live we. Wanted to minimize the account management overhead. Because. We started with no to mention inside. We. Wanted to delegate, the, overall. Administration, to. Other parties. Within, the organisation as, quickly as possible. Because, at the end the day we don't want to be the gatekeeper, of. Of. Docker from a software entitlement, standpoint, now. Even, though there were no automation. In. Place at the beginning of that journey, we. Definitely. Intended, to automate, from the get-go so we designed, that model, to, ensure. Automation. Would, be possible, and to. Also allow for maximum, portability, in.
Case, We. Need to switch technology. So, the answer looks like this we. Actually delivered, on a very strong, push-pull. Model, we. Allow each hour in the organization. To. Push within their own repositories. Because for them there is no overlap, of image, ownership. However. I mentioned. Earlier the platform, delivers. On, a set, of shared services and those. Services have to be distributed. And, accessible. Across the portfolio so. We have also our own dedicated, platform, R&D. That. Needs to be able to push. Those. Images as well. This. Solution allows from a customer, point of view to simply, download, what, they need using. Their own doctor ID, so. From a pool point, of view it's, a very simplified, model the. Entire complexity. Of building. And delivering those assets, are. Not. Our customers. Concerns. At. The end of the day our customers, want a simple, and reliable way, to download, what they are officially, entitled, to. Overall. This push-pull, model provides, us with. Three important, benefits, the first one is a nice separation. Of roles. And responsibilities. On, both. Pool and, push. / application, ownership. It. Also provides, a very nice. Application. Level isolation. Within. Macro focus and the model can be actually extended. As new. Offerings come along. So. We currently set up with set with. Seven. Applications. But. We're ready to. Onboard new applications, when we see fit. We, have a single point of management for, customers, lifecycle. So. That we can all board customers, at any time and we, can also, terminate. And revoke, access, to. Dock your hub. I will. Now describe, the, underlying, implementation. That. We put in place. Based. On, the. Security. Model offered, by docker hub so. It is based on organization. Repositories. Teams. And members. So. You build your model in the context of a single organization we, actually use two mirrored. Organizations. At microfocus, one. To serve our sandboxing, needs and one. For. Our production needs. So. First, we're. Going to create repositories. For each of our application. With. A special, prefix, this. Helps us understand, the overall ownership, of those repos. Now. We're going to do the same for the platform. Create. Additional repositories. For shared, services. So. Now we need to define, a set of teams to. Control those repositories now. You'll find a predefined, honours, team, this. Is a team that are the most. Privileges. On any. Repository you create. We're. Going to add few, administrators. To, this team. And. We also going to assign a custom, public, distribution, list, for. A release, operations, team the one that will unlink, that will handle, customer. Entitlements. Once, we're up and running. Next. We're going to create two teams per application. One. For pool and one for push activities. Again. We're. Going to be using the same prefix to give us a hint of the ownership and we. Will start adding some members as well. As you. Can see we're preventing the use of a common distribution list, instead. Of adding individual. Macro. Focus accounts. We. Use outlook to actually manage the overall PDL, distribution, membership, so. It's easier to handle the. Overall user management. We. Can add and remove micro, focus employees, out of the, outlook. Distribution, list instead, of doing, it twice. Maybe one an Outlook and one. After. The fact on docker hub, it. Also reduces, noise when we review docker up team membership, through the docker hub portal and. In. A way it's very helpful if we need to reset the docker hub password, for, whatever reasons, such as a security, breach or simply.
Force, A refresh of our passwords, on a regular basis so. One. Distribution list for each of our push/pull teams. Now. Let's do the same for the platform, push-pull, teams, and as, we seen members so, for this only two. Teams are required one, pool and one push. Now. Let's move over to the repo access, control. And we're, going to assign teams to each repository, with the right permissions. So. We're going to give readwrite, access to. All application. Push teams. And. We're going to give a read-only access, to. All application, pool team. Clean. And simple so far. We're. Going to repeat the same for the platform repositories. Readwrite for the push team and. Read-only. For the pool team. Now. Finally, let's not forget that our applications, do. Need access to the shared services. To run properly. So. What we do here is, we actually, add, a, read-only. Access, to, each of the application, pool team. Okay. So. Now we are done sitting or secured model, now. Let's have a look at what it means when. We bring customers. In, the, in the picture. The. First thing we do is we, tell our customers to, actually register to docker hub. All, right they need to get their unique ID and, if. They've been already, using some of the darker services, they can certainly reuse, them so. They have to come and see us with your doctor ID. Once. It provide us with it, we can do a quick look up in our internal, systems, to. Check their current software entitlements. So that we know which application, they. Have access to, one. Or many and, for. Us as translates, very nicely. Into. Adding the. Darker, ID into. The right application. Pool teams. Here. You go they're all set. And. Identically. If. For example they were on an evaluation period or, customers. Run runs, out of support, for example we. Can simply. Take. The talker ID out. Of the application, pool team and, suddenly. They don't have access to the, docker images, any longer. So, in this journey we were able to actually move the needle quite, a bit. With. This new delivery and deployment model, we. Move from a. Setup. That had, many, different products, different, sizes. Different. Packaging. Different. Installation, software and methodologies. And different. Always. Targets, as well and. We. Move to a container only model for maximum, portability. And, we. Provide. A more, consistent delivery. And deployment experience to our customers. We. Now use docker hub as our main distribution, service. Instead, of using our homegrown custom, websites. And. So, we use docker up for both initial, downloads as well as ongoing updates and patches, we. Are no longer sending patches, over emails, with, some additional, instructions or documentation. Our. Customers, don't have to visit additional. Support, websites and find, their way into. Downloading. Some. Set of patches, or files. What. Does our support, metric, metrics. Look like well there, are no more PDF, files. To read for sure, it's. Not all software-defined, and, what. I mean by that we. Are now actually carrying. The. Overall application, build. Of material, digitally, and, so. We're able to bring the signature, of this application, from, dev. To, QA to, production, into. Support, as well so, we know at any time, the. Composition, of a specific, release of an application. We. Were also able to also consolidate. Our licensing. Through. The use of a shared licensing, service. So. A, single. License, file can be redeemed, per suite and deployed instead, of actually downloading, and deploying. Many. License. Files one, per products. Or capabilities. So, let's look how far we landed with docker up after, using the service for the past three plus years. We. Have two, organizations, on docker hub one, for sandbox, and one. For production. We. Use sandbox, for early evaluation of, our beta programs. And we. Use a production organization. For the official. Delivery. Channel, of our. Paid premium, software. So. We studied from from, zero repositories. But now, we have around 450. Plus per. Organization. And this. Number continued to grow. As. We release every three months we continue, to decompose, our large monolith, into. Smaller consumable. Services. While. Introducing new capabilities. So, the, number. Of repositories. Tend. To increase as, we go as we. Go along we, also, keep an eye on the overall you, know image reuse through the portfolio and we're right now about front, 15%. Because. Again we're trying to play a lot around, with introducing.
A Lot. More share technology, and shared services. We. Have enabled over. 1600. Plus customers, partners, and professional, services, on, this new container I stack and we're just at the beginning of this journey. We. Keep the number of teams very small as I. Mentioned earlier which. Only trying. To create. Two, new teams every time we bring a new offerings to market so we try to keep sync still, very, much tidy and clean so. They can be, managed, properly. And. Efficiently. We. Have. Delegated. The overall customer and, link, a stomer management, to, three operations, team worldwide. One per go-to-market. Region, and it's. Been working, very well so far so we're very happy. With. The. Use of docker hub. Now. There are always, still, ways. To. Improve our or existing, implementation. For. Example, we would love to have a more granular permission. Model or. Release operations, Engineers do. Have admin, privileges. Within. Our daughter up space and that we do find that a bit too dangerous at times. We. Would love to bring a higher level of automation, for repo management. When. It comes to team membership, and team assignment. We. Don't have yet the. Right API is from docker hub and so. That does limit a bit. The. Amount of automation and. Visibility. We want to put in place. Now. We. Would like to be able to have indeed a better visibility into, the overall. Operations. Happening. With inaudible Co, system and so. For, now we also put a circle journaling, system, in-house so that we can track all incoming, requests, coming from customers, and partners. However. I would, love to be able to leverage a set of API so that I can actually interrogate. Everything. About my ecosystem, within docker hub. And. Try. To bring back a set, of metrics on on a regular basis for example. Well. There are some very good news we. Were quite excited to, see. That. Docker put their roadmap online, for everyone to see I think. It happened a few weeks back you. Can influence their product, backlog and actually, play an important, role here, in. The overall process you, can actually vote and add comments, and it's. Available online so everyone. I. Encourage. Everyone to, go. Look at this more carefully it's, under github comm / docker / roadmap, so I. Extracted. A few hints mints that, are very important to micro focus. The. User. Interface, and the docker hub search. Have. Been improved, and already released so this is great news.
There's, An audit system with api's in, as work. In progress. We. Some ways to get notified, when repositories. Are being managed, that's. Certainly something that we would leverage. Repository. Scanning I mentioned we are using our own scanning, tool but it doesn't hurt to see what would. The. Scanner coming, from docker hub would would report on our, docker. Images. We. Are looking at trying, to facilitate. The. Integration between, the, macro focus, customer. Accounts, and the. Docker world and so. Yeah sam'l or some, other means to be to. Provide. Some. Way of connecting. Back, to macro, focus would, be fantastic. Over, I think the most pressing, issue is to actually get access. To, those public API those public rest api's that's. Certainly something that we, will leverage extensively. Within. Micro, focus. Meanwhile. We could also make use of an activity dashboard, certainly. To surface, out some important, metrics for us overall. This is very positive we, are now able to track and review the plan and the overall execution of, those features online. So. Thanks to dr. up for that. Now. Finally, while we're waiting for those public areas to comes we have actually built a bit. Of. Automation. Scripting, in-house, and we would like to. Be, able to contribute back to the community. And. Make. That available open, source so, we. Could. Not publish, our code on time for Darkon but. If you are interested you, can visit our landing page at. Micro, focus github. Hyoh, at. A later time or you, can simply ping me and I'll, try to to. Publish this as soon, as I get approval. So. This, wraps up my talk thanks, again for watching I, will now be answering your questions, in the live Q&A talk. To you later bye bye.
2020-08-22 14:10
