Building resilient, mission-critical apps with Azure Database for PostgreSQL | Open Azure Day
(upbeat music) {\an8}<v ->Hello, I am Sridhar Ranganathan,</v> {\an8}a Program Manager for Azure Database for PostgreSQL {\an8}at Microsoft. I've been working with relational database technologies, including Oracle and Postgres for nearly 20 years in both technical and PM roles. I'm passionate about data protection, high availability and business continuity aspects of a data bases. I truly am really excited to talk to you about deploying and running your mission critical applications with Azure Database for Postgres.
Let me begin this talk by discussing about Postgres and why it is a popular database. I'll then go over how Azure builds on top of Postgres capabilities. Then I will discuss few Postgres deployment options for the applications on Azure. And finally end the talk with what makes Azure the best destination for Postgres. As many of you probably know, Postgres is the world's most advanced open source relational database. Why is it? The primary reason is it's open source and with a large developer community with the developers and DBAs who love Postgres.
Second, it's over 20 years in development with proven resiliency and stability with thousands of mission-critical applications running on Postgres. And third, it has rich feature sets, right? Such as a lot of data types that you can configure, rich indexing, full tech search, JSONB support, and also extensibility. All are by staying true to its principle of being standard complaint and highly programmable, Postgres has really solidified its position as the most loved database. Now, Postgres is more popular than ever, right? At the forefront, Postgres has shown that it is an open source database built for every type of a developer. DB-Engines ranked Postgres as the DBMS of the year for both 2017 and 2018.
Similarly the Stack Overflow 2019 developer survey revealed that Postgres is the most loved and wanted database. And Microsoft loves Postgres and is committed to Postgres. As part of it, Microsoft acquired Citus Data, the Postgres company, reaffirming its commitment and continued to release periodic updates for Citus back to the community. We are also very excited to have Postgres committers, Thomas, Jeff, Andres and David join our team. Having such core expertise in our team is helping us to get a better and deeper understanding of Postgres and also allowing us to contribute back to the community.
So with all those rich set of features and capabilities Postgres is really built for the cloud. And Azure builds on Postgres and makes it easy to deploy and run your Postgres database by adding capabilities such as security ecosystem integration, scaling operations, high availability, automated backups, and so on. So essentially, Azure Database for Postgres is a fully managed, intelligent and scalable Postgres service.
I show this short url here so if you want to know more details, please use that URL. So what you can build with Azure Postgres? So I'm going to show some examples of applications that you can deploy with the Azure Postgres. First one is using AKS, Azure Kubernetes service, right? It's basically a fully managed Kubernetes service that helps deployment and management of containerized applications.
So you can engage cloud native developers and deploy the AKS with the flexibility and also about the extensibility of Postgres, and build modern applications. You can enable real time operational analytics, where you don't have to have many databases separately for transactions and analytics. You can converge them in Postgres. You can build geospatial data like leveraging PostGIS extension or you can use time series. You can convert Postgres to be a time series database and you can make just-in-time decisions based on event driven applications.
So Azure Database for Postgres, we use the community version of Postgres and not a forked one. Which means that whatever applications you are creating on the database, s 97 you can pretty much port to any other location and continue to run. So how does Azure in the managed service help you building a much faster and efficient applications, right? Primarily the purpose of the managed service is to take care of the basic operations of management of database so that you can focus on your application. So Azure basically ensures your database is periodically updated, automatically updated and keeps up-to-date with all bug fixes and security fixes so that you can focus on your application.
And allows you to scale your resources independently, both compute and storage. It has built in complaince and security. It automatically leverages enterprise grade security that is already built in with Azure. You'll receive security alerts with Advanced Threat Protection and last but not least be rest assured that you have the best in industry uncapped defense and indemnification coverage from Azure. Let's start looking into some deployment options and some use cases and some applications how we can deploy with Azure Database for Postgres, right? Let's say you want to migrate to cloud or do you want to modernize your application, right? So we recently launched a new deployment model Azure.
It's called Flexible Server, right? So currently it is in preview. So we'll go over some of the concepts of a Flexible Server and how we can deploy our application with this deployment model. So Flexible Server effectively runs your enterprise workload, why? Because it provides you maximum control for your database.
Though it's managed service and we offer optimized parameter settings and depending on what the computer you choose, storage size you use, you can still go and tune a few things to suit your business and application. Such as you can configure your network configuration, what clients can access and how it can access. It can tune parameters and you can define what time that periodic updates should happen, right? Essentially you can choose the date and time of the week for example and that is when the updates happen.
So you can also build resilient applications. So the zone redundant high availability. So essentially you are deploying a standby database on a different zone, so to protect from zone level failures. And it's in synchronous mode to provide zero data loss.
Hence you can co-locate application and database in the same zone for much better performance, right? We also optimized a lot for simplified developer experience. For example, it provides a simple end to end deployment, like very easy to run a command or go through UI to deploy database. And it's a fully compatible community Postgres. It's not a forked version. And we have cost optimization such as a stop and start where you can, if you're in a development environment and if you don't want to keep it running over the weekend, you can turn off on Friday, for example, and you can start it on Monday morning, right? So that way you are not incurring any charges for the compute. You know when you're deploying a Flexible Server, you can choose which region you want to deploy and which availability zone you want to deploy in it.
So essentially region is a separate geographical regions and then each region have different availability zones and you can deploy your applications, VMs and databases to be co-located together. So you know you have optimal performance. Now we are going to spin up a Linux virtual machine and we installed Postgres database in it. So we currently support 11 and 12 and the storage is basically Azure Premium Storage for storing data and logs that is redundant multiple copies within the zone for resiliency and high availability. And the periodic backups are done to the zone-redundant storage. So that backup is available across all the zones within the region.
So you can also deploy your applications along with the database for low latency and high-performance. So if you want to protect your applications from zone level failures and need the ability to fast fail over, you can then choose the zone-redudant high ability deployment. So when you choose that option, a standard database has provision on a different zone. A synchronous model, physical replication is established and that provides a zero data loss in the event of a failover. All application commits and writes on the primary are acknowledged only after the standby received the data and then it's persistent.
So this setup basically gives you a good high availability and faster fail-over capabilities. So what happens in the event of a failure, right? So let's say like a node crash or a serious networking issue or a serious storage fault on the zone. And then your database is no longer available, right? So the Flexible Server detects that and it fails over to the standby. The standby server is just basically a replica it's activated and it becomes a redirect node and your applications will connect to that which is a new primary, and they can continue the operation there, right? Then the new standby is established on the previous zone that you were previously operating, right? In this model, it gives you a much faster, fail-over, much lower RTO and it gives you high availability in the event of a serious fault of the primary. So from networking perspective, Flexible Server supports two modes of access.
So one is using a public access. So you're just using public endpoint, right? It is a resolvable publicly using DNS address. And you can restrict what IPs can access to reducing firewall rules. And Flexible Server allows for private access using VNet integration. Servers in Azure Virtual Network can only be reached and can only through a private IP address. And you can create a separate sub-net for deploying your Flexible Server.
And you can define rules on what clients can access that. From the client connectivity perspective, we enforce TLS 1.2, and above. So let's build an application with Azure Database for Postgres.
So it's a sample application. It could be a transaction application here, which uses AKS Azure Kubernetes Service. As I mentioned, it is a managed Kubernetes cluster in Azure. You can modernize your application and development practices to a basically a microservice and container based approach.
And on the database side, which is on the middle of your screen, you can deploy a fully managed Postgres and configure it with the VNet and zone-redundant high availability. And you can offload database demands by managing session sates and asset caching with Azure Cache for Redis And on the left, you can see as Azure functions where you get a server-less infrastructure that it can trigger other based on key events such as user activity using as Azure Functions. And on the left, you can see a notification hub that it can push a notification directly to your users on the preferred service or medium using the service. And Azure Databricks just in the bottom of the screen is an Apache Spark-based analytics platform. It's a big data tool where you can perform a deeper analysis of the data. And finally it can monitor your application performance for degradation or anomalies.
And auto-scaling your application to changing performance requirements. Now say like if your application demands more performance and you want to scale up to hundreds of nodes and handle terabytes or petabytes of data, then you can choose Azure Database Postgres Hyperscale (Citus) deployment. So as I mentioned, Microsoft acquired a Postgres company last year called Citus Data. That team created the Citus extension, which is an open source that transforms Postgres into a distributed database that it can scale horizontally.
So Citus now available in the cloud as a Hyperscale (Citus) built-in deployment option on Azure Database for Postgres. So Hyperscale (Citus) is a managed Postgres database service with data and queries distributed across nodes to provide unlimited horizontal scalability. You're basically sharding your Postgres database across multiple nodes to give your application more memory, compute, and storage. This is the GitHub page and visit this page to learn more about Citus extension.
So what can you do with Hyperscale Citus? First, you can scale out horizontally. You're no longer confined to one single node. You can scale out to hundreds of nodes. You can ingest data at a real time, sub-second response for billions of rows, with the blazing performance. It's a simplified architecture, right? So both transactions and analytics, you can store it on the same database. And you can stay current with Postgres innovations like it is based on open source extension.
Postgres itself is open source. So it can leverage all of the Postgres expertise to its latest innovation. So building an application with the Hyperscale (Citus). So previously we deployed a Postgres application using AKS and Flexible Server. So if you want to scale out, you basically replace the database component using Hyperscale (Citus) instead of Flexible Server.
With that you can basically scale out horizontally and fit more performance and fit more nodes. And another application example is the real-time analytics. So in this we are basically going to ingest data into Citus and then how you can connect with other Azure services to add more value from it. So Azure Event Hubs, in the middle of the screen, is a big data streaming platform and even such ingestion service. You can stream millions of events per second from devices and sensors to it.
And then you can build downstream pipeline to process, manipulate, and ingest your data. Ingest those millions of raw transaction events into Hyperscale (Citus), events per second and it allows you to query an alert on granular events. It can do a periodic incremental roll-ups directly in your database for real-time reporting and also for dashboarding for downstream applications. And you can take advantage of Azure Databricks to clean, transform, and analyze, streaming data and combine that with the structured data from operational database or data warehouse. It can provide insights to users and operators in the on current device status that you can see in the bottom of the screen.
And you can push timely notifications to your users on their preferred service or medium. A few components of Azure that they can put together to have real-time analytics with Hyperscale (Citus). So moving on to real world examples here, right? Microsoft Windows. So Microsoft Windows, diagnostic metrics are displayed on real time analytics dashboard called "Release Quality View". And that is populated from Citus. So essentially they deployed across 2 Citus clusters with the 2,800 cores within it and 19 terabytes of memory, right? And petabytes of data.
So essentially it is supporting about 6 million queries every day with 2 petabytes of data. And most queries are responded To in a little less than a second. So this is critical for Microsoft Windows, for mission critical shiproom decisions. And another example is ASB, which is a large bank in New Zealand. So they were having 40 minutes to onboard customers and with Citus they were able to cut it down to 2 minutes which is 20 times faster. Now customers' environments are getting complex.
They have tens of hundreds of thousands of applications such as containers, VMs, databases, right? And they're a diverse infrastructure such as data centers, posters, branch offices and so on. And they also deployed multicloud, right? And with that they also have applications that cannot move to public cloud due to regulatory and compliance reasons. So in that situation, if you want to run Postgres in your infrastructure because of any of those restrictions there, right? We have a new deployment options called Azure Arc enabled PostgreSQL Hyperscale, which is part of Azure Arc enabled data services. So what is Azure Arc? It's one of the three pillars of Azure Hybrid along with Azure Stack and Azure IOT. So Azure Arc for data services is offered through Azure Arc where the infrastructure is provided by the customer.
And also customer takes care of the operations while the software is provided by Microsoft. So Azure Arc enabled Postgres Hyperscale, right? So it's pretty much like Azure makes it possible to run Azure Data Services on premises, at the edge, in public clouds, such as Azure, AWS or Google Cloud. So the software provided by Azure is always current. It's automatically updated and you can scale your infrastructure using the same automation, of course like you're providing the infrastructure, but the software makes it easy for scaling and probably is a unified management, a single view for on-prem and cloud using familiar tools.
It's Azure policy is implemented and also cloud-based billing, right? It's like you are going to enjoy all the benefits of the cloud, but in your data center or your preference. So essentially it can run anywhere using Azure Arc. Man the only requirement is it requires Kubernetes any Kubernetes, And we have a short url there.
You can read more about Azure Arc Data Services. Let's switch to what's making Azure the best destination for running Postgres applications. Azure Postgres offers a variety of deployment options to choose from starting with burstable compute, to scaling to hundreds of nodes, to meet your requirements. Right? So we talked about Flexible Server where it gives you control flexibility and high availability, Hyperscale (Citus). It provides a distributed database with the unlimited scalability and we have PostgreSQL Hyperscale - Azure Arc that runs Hyperscale on your preferred infrastructure. And we also have PostgreSQL Single Server which is a fully managed Postgres database service.
So with that, there is deployment option. Let me go over some of the building blocks, making Azure the best destination for Postgres. To start with Postgres is the best open source database. It's a large community support, right? And the proven reliability and stability with rich features, right? And then add Azure's management capabilities. It's a fully managed, integrated, intelligent performance, highly scalable, and highly available.
And then you provide maximum control. So that all of the Postgres database capabilities, and then you put that all the management capabilities on top of it. And you can add all the integration with Azure ecosystem, right? We provide integration points with the Azure Active Directory, Visual Studio, Azure Data Factory and Power BI and so on. So that you can build your application into and on Azure. So all the Azure services has the identity, security, management, and compliance. It's all available in Microsoft Azure, right? And with Azure Arc, you are even extending that same capability to a non-Azure infrastructure too.
And so I covered how you deploy mission critical applications on various Azure Postgres deployment modes. To learn more about the latest happenings in Citus, Azure Database, and even for Postgres, you can subscribe to this newsletter that I put a short URL here. I also have action resources that you can take a look into, including that Azure Service page and Flexible Server details. There is a feedback forum and email.
So we want any questions or concerns or any requirements you may have. Please feel free to post your questions to either to the forum or to the email. So with that I conclude the session and hope you found this session useful. We are looking forward to hearing from you and thank you for your time.
And bye (upbeat music)
2021-02-22 09:26
