Microsoft Fabric: Vision & Roadmap
And now put your hands together. And welcome to T-Mobile arena. And welcome to T-Mobile arena. Corporate Vice President of Azure Data Arun Ulag. All right.
Hey, folks. Good morning. Good morning. Good morning. I'm just so excited to be here.
I'm Arun Ulag, I run the Azure data team here at Microsoft. I'm absolutely loving it here. I'm really looking forward to spending the next few days with 6500 of our closest customers, partners, and community members right here. FabCon, thank you so much for joining us.
We recognize that this is a big investment of your time, and you're really, really busy people. And we really are grateful that you chose to spend this time with us. Okay. One of the things that we're really, really happy about is that we've done our part. We have brought about 400 of the developers, architects, engineers, product managers and designers that built Fabric right here to FabCon so that they can talk to you, they can share with you what they're doing, what they're building, why they're building it, and how it can make a difference for you. We also have literally hundreds of customers here, partners, community members, so they can talk to you about their Fabric journey, what they are implementing, the success that they're seeing, as well as, you know, some of the choices that they have made in building out, architectures.
When we think about, you know, why we are all here today. We recognize that, you know, AI is rapidly changing the world, and it's no surprise to anyone here. However, we are all data people.
So, we also recognize that AI is only as good as the data that it gets to work on, because it is data that is the fuel that powers AI. The best AI models, unless it has good data to work with. You put garbage in.
Most likely you're going to get garbage out. So, it's become incredibly important for customers to get their data ready for AI. And this is exactly what we've been working on here at Microsoft And this is exactly what we've been working on here at Microsoft over the last decade or so. over the last decade or so. We have literally invented and built dozens and dozens of products and technologies that are used by hundreds of thousands of customers around the world.
Products like Power BI, products like Azure SQL, Cosmos DB, Data Factory, Cognitive Services, Azure Machine Learning and Azure Open AI. Now, one of the things that we're excited about is that we've put all of this innovation together. But it's not just that Microsoft has all of these end-to-end capabilities, But it's not just that Microsoft has all of these end-to-end capabilities, but all of these products are industry leading in their categories. If you look at the Gartner Magic Quadrant, there are four Gartner Magic Quadrants that cover the data and AI space in every one of the Gartner Magic Quadrants.
The Microsoft products have a leadership position. This is the Magic Quadrant for Data Integration with Microsoft is a leader. Cloud DBMS Microsoft is a leader. Artificial Intelligence for Cloud, Microsoft is a leader, and Business Intelligence where we are by far the leader.
and Business Intelligence where we are by far the leader. Now the reason. Thank you. So, if you look at the four Gartner So, if you look at the four Gartner Magic Quadrant, that comprises the entire data and AI space.
It's exciting to see that there's only one vendor with a leadership position in all four Gartner Magic Quadrant, and that's Microsoft. And the reason and the reason that matters to you is that working with Microsoft, you not only get all of the capabilities you need to go on your end-to-end data and AI stack, but you know that these products are best in class. However, we know that we can do better. When I talk to my customers, we often hear that building data and AI projects is still too complex, too fragmented, and takes too long and cost too much. So, the message that we consistently get here from customers is please unify.
So, the message that we consistently get here from customers is please unify. I want to be the Chief Information Officer. I don't want to be the Chief Integration Officer. Help me take AI and turn it into my competitive advantage. And that's exactly what we're doing with Microsoft Fabric. With Microsoft Fabric, we built an end -to-end data platform that helps With Microsoft Fabric, we built an end -to-end data platform that helps you go from raw data to AI or BI value in the hands of your business users.
Fabric has a set of core workloads, and these are the boxes that you see on top. And these core workloads are purpose built for specific personas like data engineers, data scientists, data warehousing professionals, business intelligence professionals. And each of them, you know, our industry leading and making sure that those developers can get to value very, very quickly. But Fabric is not just a bundle of products. We've taken the time, we've taken the effort to actually bring them together into a single unified platform.
Right. So, if you work with Fabric, you'll notice that all of these products have a unified experience. They have a unified architecture and they have a unified business model. Now Fabric launched about 15 months ago.
Now Fabric launched about 15 months ago. That's when it became generally available. And since it became GA, the momentum has been simply staggering.
Satya announced in our last earnings call that Fabric today has over 19,000 paying customers. Thank you. Fabric is also the fastest growing, analytics platform that Microsoft has ever launched. Today, among the 19,000 customers Today, among the 19,000 customers that it includes over 70% of the fortune 500.
When we look at how customers are using Fabric, they're really getting value from the fact that Fabric is an end -to-end data platform. So, we're seeing over 50% or half our customers are using Fabric end-to-end. Typically, what we see is customers using three plus workloads. And the most common workloads are using Data Factory to bring data into Fabric and using either Spark or SQL to transform the data, and using Power BI for semantic models and visualization on all of the data. Living in OneLake.
Now there's literally dozens and dozens and dozens of customer case studies that are available today where customers are talking about their Fabric journey, why they choose Fabric, the business results that they're seeing. I wanted to highlight just three examples. The second customer that I wanted to highlight is EY. EY is one of the world's largest organizations with over 400,000 employees, and they work with hundreds of thousands of customers, helping these customers with complex tasks like assurance, tax and audit.
They chose Microsoft Fabric enterprise -wide because it dramatically accelerates their ability to work with data, and it shortens their data supply chain, helping them get to business value faster. And the third customer I wanted to highlight is Epic. Epic is by far the industry leader, in medical software. And they help, you know, thousands of hospitals and medical care providers around the world to provide compelling care to their patients. When Epic was looking for the data platform for Cogito, which is their analytics platform that they make available to their customers, they chose Microsoft Fabric because they love the fact that OneLake is a multi-cloud SaaS data lake. They love the unified capacity model and the Data Warehousing capabilities and Fabric based on T-SQL that all their developers knew and loved was truly world class.
that all their developers knew and loved was truly world class. Now, these are just a few examples. And if you were to look at aka.ms
slash Fabric Featured Customers, you'll find dozens and dozens and dozens of customer case studies where they talk about their journeys. However, as excited as we are about the 19,000 customers, we see the opportunity to be so much larger to help every one of our customers. And one example where we have democratized data at scale is really Power BI.
How many Power BI customers do we have here? Make some noise. So, I've been working on Power BI a long time and today So, I've been working on Power BI a long time and today you know by far Power BI, we have over 375,000 customers, 30 million business users that use Power BI every single month, and 7 million developers. And these customers include over 95% of the fortune 500. Now, if you look at this chart that you see on this slide, this is the actual usage chart of Power BI since we launched. this is the actual usage chart of Power BI since we launched. And you can see that over the last ten years, Power BI continues to grow exponentially.
And ever since we launched Fabric and it became generally available, you can see that, you know, the growth as Power BI has only accelerated. And the reason I'm talking about this is for every one of us, 7 And the reason I'm talking about this is for every one of us, 7 million Power BI developers, for every one of our 375,000 Power BI customers. Fabric is just one click away. Fabric is just one click away. Now we have taken the trouble to make sure that it's very easy for every Power BI developer, for every Power BI customer, to very quickly get a Fabric trial. All they have to do is click on their profile picture and start trial.
You don't need a credit card, you don't need a subscription. Azure subscription, and we give you $17,000 of capacity to be able to build something real over two months. Now, this is the most generous trial program in the industry. And the reason we're making it available is because we want every developer to try Fabric to give it a shot to see what Fabric can do for them as they build their end-to-end data and AI solutions. Thank you so much. And as excited as I am about our technology, we know it's not just about the technology.
It is really about the people. It's about all the personas that work with data the data analyst, the solution architects, the information workers, the database administrators, the data scientists, the data engineers, and so on. And what's wonderful about Fabric is it really makes it really easy for all of these developers for the very first time, to be able to work together and to be able to collaborate with each other seamlessly, because all the artifacts live in the same place in Fabric. And because we know that data and AI is a team sport, now, you're very welcome. And as we bring these capabilities together, these developers and community can collaborate.
capabilities together, these developers and community can collaborate. We also know it's incredibly important for us to build the community. Now I have, personally me, myself, my leadership team, my entire team has been working very closely with the Power BI and SQL communities for more than a decade. and SQL communities for more than a decade. And we're really, really excited to see the massive community momentum.
We have over 2.4 million members in the Fabric community worldwide. Yes. And we have a lot of folks right here. Over 300 user groups and pretty much every kind of like 62 countries. And I'd highly encourage you to join one of these user groups, And I'd highly encourage you to join one of these user groups, because we have more than a quarter of a million members just in the Fabric and Power BI user groups.
So, if you go to aka.ms. slash Fabric Community, you'll be able to quickly connect with people who are passionate about Fabric, who are eager to share with you their learnings, and also learn from you. Now, this is a community that actually helps each other. community that actually helps each other. One of the things we were excited to share is every 99 seconds, a question from a Fabric community member gets answered by another Fabric community member.
So, it's a really a community that works together. It's also a community that learns from each other. Okay. I'm excited to share that within the last 12 months. We have 30,000 developers who attained the DP- 600 or DP-700 Fabric certifications. Thank you.
Congratulate yourself. You know, these have been the fastest growing certifications that Microsoft has ever launched. And what's also exciting is a lot of the training materials available for these certifications are built by the community, for the community. What I'm excited to share today is for those of you who want to take the DP-700 certifications, guess what? For every FabCon community member, every FabCon attendee, We are making the DP-700 completely free today. So, please take a look at this QR code.
And take advantage of this opportunity. We're also very, very grateful for the MVPs as well as the Super Users who have really, really leaned in. Now over the last six months, we're excited to add 30 new MVP's and 56 new Super Users. So, just wanted to celebrate all of the additions to the Fabric community. Thank you so much for leaning in, please congratulate all of these folks, who have leaned in to learn Fabric and communicate with everybody else and help people understand what the product can do.
We've also been working very, very closely with our Systems Integration partners, Global Systems Integrators, Regional Systems Integrators, Vertical Systems Integrators across the world. And it's also really, really exciting to see the momentum because we know when customers want to implement Fabric, they often seek out a partner that has all of the capabilities and can do it quickly and efficiently. We have 150 Fabric Featured Partners, but across the entire partner ecosystem We have 150 Fabric Featured Partners, but across the entire partner ecosystem that are 45,000 consultants that are trained and are actively implementing Fabric today and across the entire partner community. and are actively implementing Fabric today and across the entire partner community. They have delivered over 16,000 customer wins just in the last12 months. So, the momentum is really, really massive.
So, the momentum is really, really massive. I also wanted to thank our Fabric Featured Partners, and there's 150 partners on this slide. Many of these partners are here at FabCon. Thank you so much. You know.
You can find them in the Partner Pavilion. But what it means to be a Fabric Featured Partner is partners that have deeply invested in certification. So, they have a lot of DP-600 and DP-700 certified consultants. They have many successful Fabric implementations, and they have deep relationships with the product team.
So, they know our roadmap. They give us feedback on the capabilities we have and really make sure that they can deliver the capabilities with confidence. So, I'd love to make sure that we play a quick video so you can hear from some of these partners.
What excites me most about Fabric is having a unified platform, everything from the ETL to the data storage, the data modeling, the data analytics, and the preparedness and AI aspects of it all within the one unified platform is the speed to value. So, you can go from having nothing set up to having the platform turned on and ready to start implementing. Within ten minutes, you can have your first proof of concept done easily within a week.
Throughout other side of your production set up. Having things like Copilot enabling your business users to get to the insight they need faster. It's a game changer. Effectively. It’s the simplicity. You can, start, start with a small solution and then let that simply grow.
If you want to start with classical analytics, then you can add Real-Time Intelligence afterwards. I think the thing that excites me most about Microsoft Fabric is that it puts data at the center. Everybody wants AI, and for AI to work, you have to have data. That's what Fabric can do for us. It brings it all together in one platform for the customers to really accelerate the AI journey.
For me, pretty much everything excites me about Fabric. It's a great product with a lot of innovation from a developer perspective. I can say it's really a game changer, and I'm looking forward for all the new, exciting features that help our customers to get the businessvalue out of the product. But this changing 15 years of how we're building data warehousing and, it's unifying everything under one hood than the one platform selling something that I believe in, a solution that I can be confident about. And it's, making it faster, faster.
From the first time we meet the customer from the intro to when we go to the production, it's a solution that is changing everything that until now. So that was a quick overview of Fabric and how we're doing now to the most exciting part of the keynote. I'm going to invite Amir Netz.
I'm going to invite Amir Netz. Our technical fellow and CTO. And he's going to show you all the innovation we're learning this week.
Welcome, Amir. Thank you so much, Arun. How are you guys doing? This is what you've been waiting for. We are going to take a look at the product, the innovation, everything that is coming and cool.
Oh, we're going to have so much fun. So first we'll start with just framing the vision of the product. We have three pillars in the vision. First of all, we are providing an AI-powered data platform. First of all, we are providing an AI-powered data platform. This means that we bring all the data services together in a single unified platform.
Second, we're unifying your entire data estate, and this is where OneLake comes into the picture. And lastly, we are directly connecting with our business users who are sitting in the productivity apps. Who want to get some insights. These are the three pillars. This is the structure of the presentation.
And we are going to start with the first pillar. We're going to take a look at what does it mean to have a platform that is end-to-end. Everything comes together. And this is the platform you see in this picture before.
It has everything that you need for data. It has the Data Factory for data integration, data extraction, data movement. It has analytics with Data Warehousing and Data Engineering, Data Science. It has Real-Time intelligence for everything that is coming from sensors, for telemetry, from devices. And it has Power BI.
You know, I don't have to say what it is. And then you have the youngest member of the family, the Databases. Okay. We'll spend some time on that one as well.
Super, super excited about the Databases all sitting on a common platform. of AI common platform for governance and OneLake And we're going to, you know, the hallmark of everything that we're doing is integration, unification. And we've been providing for you guys this environment where everything comes together. You have a place where you have workspaces. where you have the projects. You have the task flows that gives you rails for running the the structure and the architecture of everything that you do.
And then you have, of course, the the Copilot in every experience to really accelerate your work. And this is great because if you come from Power BI, this all makes sense. It's kind of massively empowering low code almost all over the place and you can build so much. It's such a democratizing platform.
But if you come from the I.T. side of the house, you're a data professional, data engineer, application developer, well, you told us if you want more, more capabilities oriented to the professional developer, a developer professional development platform. So, I'm going to share with you a bunch of announcements, a bunch of new capabilities that we are adding and into the platform. And we start with service principle.
Yeah. You guys. Yeah. You always wanted service principle. Generally available today is service principle for GitHub and GitHub APIs generally available are the service principal working with the development pipeline API. So, you want to automate the deployment you want to make the CI/CD service principal is now available generally available. That's one. Another thing we make generally available.
And now the big request was that we deploy with Terraform. That's another one that is just generally available. And then we go through the things that you haven't seen yet, most of you, okay. So we are now incredibly excited that in public preview we have variable libraries.
Now what are libraries? You can, you can guess what they are right. It's the ability to define variables that can change from settings, variables that can change from one environment to another environment, to another environment. So, you can have, you know, one set of settings for dev, another set of settings for test, another set for production. And we'll see more of that. The other thing you told us is that professionals say we don't want to have the same code showing again and again. Again, we want to be able to have code reusability. So, we are incredibly excited to introduce the User Data Functions.
And the User Data Functions allow you to allow you to create functions in code in your preferred language it could be C#, could be Python, and then call them from all over Fabric and outside the Fabric. It's a SaaS version of Azure Functions. It's pretty incredible. We'll see more of that. And lastly, maybe my favorite one of this bunch is the Command Line Interface. So, you told us, it's one of those weird demos that is like green card. There's a black screen. It's really, really hard to see what's going on there.
But we wanted to make sure that those of you who are professionals that say, you know it's GUI and mouse and click and drag. It's just for woosies And, you know, real developers really want to just use the keyboard, that we give you a command line. And what's cool about it is and we, we kind of looked at the command lines available out there, and we realized that the best, the closest one that we can imagine is the command line of an operating system where, you know, operating system, you know, you use a command line of Windows use the command line of Linux, you know how to work with them.
So, we can actually look at Fabric through the lens of an operating system. So, everything you do is commands that you already know. So, if you want to look at the list of workspaces, if you have a Windows, you can do DIR. If you come from Linux you could do LS and then you going to create a new workspace. It's just a MKDIR.
and you MKDIR by workspace dot workspace you have when you watch this and you can CD into the workspace and do another DIR or LS and then create a lakehouse by making it lakehouse dot lakehouse So, you know the Command Line Interface before even opening in the documentation. Because you use Linux, you use Windows, you know it already is super cool. You can schedule jobs, you can monitor, you can put this command line, as scripts, you can put them as part of GitHub actions. It's incredible.
So really, really excited by the Command Line Interface. What do you guys think? Now let's move on to the workloads. Now let's move on to the workloads. And we have all this cool work. We're going to go one by one.
We love to start with Analytics. The Analytics as we said is the world of Data Warehousing, Data Engineering, Data Science. Especially when it comes to the Data Warehousing, this is where we make massive, massive progress. Like we started with the fresh newest data warehouse in the market a year ago. You know, it was a brand new MPP architecture.
It's operational storage and compute, all auto-optimizing, almost no knobs to play with. It's the most modern data warehouse in the market. And, you know, as a new data warehouse, sometimes it has the initial kinks. But we made massive progress over the last year. And to show us what's new with the Data Warehouse, I want to invite my friend Charles, Charles Webb to the stage. Hey, Charles.
What's going on Amir? It's been quite a year with Data Warehousing. Right? So, a lot of progress. Yeah, we've been really, really busy working on the Data Warehouse and it really starts with all of you behind the scenes. We've been working on shipping over 100 of your Ideas, and that means we've mined Reddit, mined Fabric Ideas, we mined even X or Twitter just to make sure that the things that are most important to you, we've shipped. So, a lot of you have been using the data warehouses. We have 15,000 customers who have been using the data warehouse.
And one of the things that they love the most is the experiences that we have in the Data Warehouse. Absolutely. We've been really working hard to make the Data Warehouse experiences the most intelligent warehouse out there. Do you want to share something? Let's take a look. So here we are in the Data Warehouse experience. And you're going to notice unified experiences.
Everything from ETL with no code and pro code queries with again no code and pro code modeling because BI and AI is not an afterthought. And of course, your favorite tools like VS Code and SSMS We've also got Copilot built in and it's infused everywhere within the editor. So, let's take a look. Let's write a query and we'll leave a comment here. And you can see as I start typing code you're going to see intelligent completions just at work.
And this is great. Even if you are an expert developer. It just saves so much time. Yeah. This is one of my favorite features as a SQL developer. But it gets better. We also have these live templates.
And so as I type code what you're going to see is even if I'm coming from a different data warehouse, it's easy to write SQL here. And you can see how it easily helps me. Pre-filled all the syntax needed to deliver SQL queries and check out this performance. Blazing fast. It's really, really fast. Now. Copilot. One of my favorite features of the Data Warehouse is also when I want to analyze data and ask questions and figure out what kind of insights I can gather, Copilot allows me to brainstorm very easily and not only does it generate SQL queries for me, but the SQL queries are things that I can now run.
And the best part about it is that it walks me through all of the way that it's thinking, so that the business logic, I can make sure that's something that I can trust. Now it gets better Amir, it gets better. I can also get all the power of the Power BI clicky-clicky, draggy-droppy and just a button click. So, check out this chart and visualization I can create in seconds.
So, this is instantly out of the result of the query. You can go build the reports, share them. Super easy. Super easy. That's awesome. Now a lot of developers also like Notebooks. Okay. Yes.
And we of course support Notebooks. So, I can easily develop and write SQL here as well. And you can see I can also chart my data too. But because all the data in the data warehouse is powered and sits on top of OneLake, that also means I could switch compute engines.
So if I need to do data engineering, if I need Spark or if I need Python, if I need R, it's right there at my fingertips. You know what the thing I love about everything that is happening here, it's all built on the same language. I joined Microsoft 28 years ago. I joined the SQL server organization when it was still SQL server. And, it's everything was built around T-SQL, and we have millions and millions of people who know T-SQL already.
And just the transition to move to this fresh new data warehouse is so easy because of the T-SQL knowledge that they have. Absolutely. I love that quote from EQ Bank. Just because when you think about the era of AI, a lot of organizations are trying to figure out how do they accelerate and innovate, how do they modernize? And with SQL and a data warehouse, you're already there. You already have all the skills you need to develop and do more with data. So a lot of you have the Synapse Gen2 Data Warehouse in Azure, and you're looking kind of a little bit with envy about what's happening in Fabric with the Data Warehouse and say, how do I transition? And we kind of have been working on that. We have a new innovation and other announcements.
Absolutely. I'm super excited to announce that for Synapse customers, starting starting with Synapse customers, we're announcing a seamless migration experience. And this is really going to allow you to migrate. Thank you, thank you, thank you I'm super excited about this because for our Synapse customers, our past data warehouse customers, we're moving you to the SaaS capabilities. And we're just getting started. Let's take a look. Okay! Let's take a look.
So, starting today, you're going to have a new migrate button in your workspace. And you're going to see a number of different items there offline options. But you can also see a card for Synapse Analytics.
And what you're going to be able to do is migrate your code first and your data. And it's pretty simple. All you have to do is connect to your data, select the objects or the schemas that you care about, and then you have to validate a few things. Maybe you want to see the different objects. You want to preview the data, but at the end of the day you're going to validate your connections. And you're going to click migrate metadata.
And that's it. Behind the scenes, we're going to apply best practices and already move your code and see Amir. That's pretty quick. And there's a bunch of AI that actually make sure that the code transitions code correctly. 100%! And it gets better. Suppose there's a few things that I need to fix. I'm not on my own.
You can see here this preview suggested fix is going to give me a diff view. And what I'm going to be able to do is now see my original code and the new suggested code from Copilot. So now it makes it easy for me to work through any issues that might arise. And it gets even more magical! Once I accept this. What you're going to see is the Copilot is going to work.
I'm going to run this query, and we're going to validate that that query was the right thing to run. We're going to work through all those dependencies automatically. Fix those up. And you can see how magical that is.
We went from eight issues down to seven. And I can keep working through this experience in a really intelligent way. So, all the views, all the stored procedures are passing.
What about the data? Good question. So once you've migrated all of your code, the next step is of course is to copy the data. And this is again where we have an intelligent experience folks. So, you can do it yourself or with an assistant.
I love the assistant because it makes it easy. You have automatically mapped types for you, so it makes it easy to say, okay, I'm moving from Synapse to Fabric, but it gets better. You can do a full copy or incremental copy, and there's intelligence built in to make this a breeze.
All you have to do is click on create. And we're going to now start a copy job behind the scenes to make your data movement a breeze. That's awesome! Now, all the data gets moved.
And then we have all these applications that we're connecting to the old data warehouse. What happened to that? So traditionally a cutover is quite difficult. But for us, what we've done is we've built in both guidance and again, intelligence.
So, if you choose Reroute with an assistant, it's kind of a magical experience, Amir. We're going to click on this button. And we're going to re-route all those Synapse Database .NET connections to Fabric. So, there's no change needed in any of the existing applications to connect to the new data warehouse, because it connects to the old data warehouse, again, getting re-routed to the new data warehouse automatically. Absolutely. This is quite incredible.
Yeah, to be very honest, when Charles told me he's going to do a migration demo. That's kind of boring. It's all going, but it's so exciting, you know? Yeah. And it's not just cool. It's not just looking great. It's not just kind of magical, but it's also very, very practical actually, working in the world. I really love this quote from Kantar, because they tried this during the private preview, and one of the things they thought about was, hey, a migration typically takes years and months and they set a really ambitious goal.
Can we migrate in a quarter? They tried our tool and they're able to migrate their scripts, 700 of them, thousands of lines of code in less than an hour. And that's pretty powerful stuff. But I just love this quote, because migrations can be easier. They can be much more simple.
And we're really excited to give it all to you folks. That's awesome. Thank you so much, Charles. Thank you. So we talked about data warehousing. Let's continue to Data Engineering. This is the world of Spark and Notebooks and Python.
This is the world where it's been exploding. There's so much going on. And to show all the cool stuff that we're doing, I'm going to invite my friend Justyna to the stage.
Hey, Justyna Amir. Hi, everyone! So, it's been it's been an incredible year for everything we're doing in Data Engineering. Yes, it's absolutely been an amazing year. You know we've just seen phenomenal growth with our Spark compute. It has grown 50 x since this time last year. And we've, of course, been working on our performance, with our native engine.
You can expect performance to be 400% faster than open source Spark. And we don't just focus on performance, right. We also focus on price-performance.
And so, you can see with our offering, it is 2.5 times more cost-effective than the other alternatives in the market. So, there is no better system. There is no more cost, more cost -effective data engineering system There is no more cost -effective data engineering system out there than the one we have in Fabric. This is quite incredible. And don't just trust me. You know, we have people who say they will testify to that.
Yes, absolutely. So, we've been working closely with LSE, the London Stock Exchange Group. And they've been using Fabric and Spark to bring in all of their disparate data sources. Right?
Everything from on-prem, across multiple different clouds. They've been starting working with Fabric and Spark to bring that data, transform it, and share it out. And if you want to learn more about what the London Stock Exchange Group are doing, make sure you join Phil's session tomorrow morning at 8 AM.
And the key tool for the trade here is the Notebook. And what's really cool about the Notebooks is that They're really bring together the best of Fabric. Yes, exactly. You know, we've been kind of focusing on making the Notebook the central hub, connecting to different items, languages, data sources, and maybe to see some of the latest enhancements. Let's jump into the demo. So, we're here, I'm in my Notebook and I want to work with the same data warehouse that Charles is working with.
But, I'm personally more of a Python user myself. Right. So, we're there in our Python Notebook, but you're going to see this new T-SQL magic command I can add. And what that lets me do is it just lets me insert a T-SQL code cell. in the middle of my Python Notebook. So you can see I'm just writing a normal warehouse query. So, you're going to see something really cool here Amir, you're going to see this “Bind Data Frame command.
And what this lets me do. is it lets me store that result set from the warehouse in a Python DataFrame. So now my Python Tree can access the results of users directly.
Exactly! So now I have the full power of Python directly here with this result from the warehouse. So, I want to do some customer segmentation modeling. So, for that I'm going to go and create myself a couple of parameters such as recency and frequency. And the great news is one of my colleagues has already written the code for the customer segments.
And the even better news is they stored it in a User data function, right. User data function! That's going to be, I announced this one, right? You did! I think you did. Just announced this one! So, here's my beautiful User data function. You can see it takes in a number of different inputs. It's doing a bunch of this segmentation logic. It provides an output.
And these are just awesome for just defining this reusable code. I can just use everywhere. And what I really love about this, as well, is that I can just test them out directly in-line in this experience. So, if I was to just input some recency and frequency data over here, I could just run it and test it out. So, this is just SaaS-ified functions. So, anybody can create those.
And I can use them outside of Fabric as well. Yes, exactly! They come with an endpoint that I can call from anywhere, including Notebooks as well. So, you can call a REST endpoint from outside the Fabric. But in Notebooks it gets even better.
Yes of course. So, we wanted to make this even better than calling just a REST endpoint. We've integrated it with our user, with our Notebook utils over here. So, you can see I can just scan through and use IntelliSense to choose the function I want to use. And I can now create myself a function object that I can just leverage.
So then in a single line of code, I can basically call this function with all of the different parameters I've just defined. And in a couple of seconds, you know, there it is! That's how easy it is to integrate. Yes, it's pretty cool, right? Thank you! Now what about what about all these parameters that you had for those. Yes. And I think you might have, you know, announced something that was pretty useful for that. Of course, that is the variable library. And this is probably one of my own personal favorite announcements from this keynote.
We have the variable library. I'm basically able to just go ahead and define all of the parameters I need to use. So, you can see I've already created one over here. And the great thing here is I've got my default value set, but I've also got my alternative TEST and PROD set. So, what does that mean now when I go and deploy this across DEV, TEST, PROD, these values are all just going to update automatically. So, I'm going to define one more parameter over here. For recency.
You can see just how easy that experience is. I'm going to go and say this is an Integer. I'm going to add a default value over here. And I'm just going to override the TEST and PROD sets. So again, once these deploy all of these will just automatically update.
So, no more hardcoded values in the notebook. So, no more hardcoded values in the notebook. You can actually put it all in the library. Yes!
And to use them is a breeze the Notebook. Exactly! So it is going to be very similar to just what you just saw with the User data functions. I'm going to go and create myself a variables object. And then I'm going to go and update these parameters from being hardcoded to dynamic. And take a look at this again I get just beautiful IntelliSense.
I can actually browse through my variables library. Just select which parameter I want to add. It is that easy to use. Okay! So now we're going to run this. And again, it's just going to automatically… Thank you. Thank you. Alright.
So, I've done all of this great processing. And I want to get this data back into my warehouse now. So, I'm going to add one step which is just going to convert this customer segments I've calculated into a JSON object.
And now take a look, what I'm going to do over here, inside my Notebook cell. I'm going to go ahead and actually define data warehousing transactions. So, I'm going to go and create this new segment column. I'm going to add the data I just went and processed with Python directly back into my warehouse and a whole bunch of other steps. And it's really marrying that asset transactions in the Enterprise Data Warehouse with the full flexibility and programmability of Python.
So, we're going to see a lot, a lot of data warehouse being maintained through Motebooks, getting all the value of the combination of the power of the T-SQL or the power of Python, with the ability to monitor them very, very nicely with the monitoring tool that we have for Notebooks. It's going to be incredible. Now, all that is, we haven't seen much of the Copilot, but we will continue to work on Copilot in the Notebook. Yes, absolutely. So, if you're looking into this, you're a little bit worried, like, hey, I don't really work with Notebooks.
Well, Copilot is here to help you out, along with a whole bunch of new updates coming to it in public preview today. So, this includes the in-cell Copilot experience for ease of use, a bunch of quick actions such as debugging, optimizing, explaining your code, and best of all, no more installation step. Yes. Yeah,
I know some of you might like that. I learned how to code in Python just using the Copilot. It's so cool. Now there's one more announcement we have, right? Because, we have something new, that you guys may be familiar with from databases.
And, you know, you, you know, about views in databases, you know, your ability to define a query that is stored in the database, and then you can run it and call it in your queries as if it was a table and a logical query. And then you have, in the database, the ability sometimes to materialize the view. So, the view is not computed when you call it, but computed ahead of time.
And we're going to do the same thing with Spark in the lake. Absolutely! We're bringing materialized views to Spark, to the lakehouse a whole bunch of goodness, a lot of exciting features over here. So why don't we just jump straight in and take a look at what this experience is going to look like. Okay, so we're starting out inside our lakehouse. And you can see I've already defined my Bronze layers over here.
And I want to go and build out my Silver and Gold layers using these new materialized views. So, I'm going to jump into a Notebook. The first step I'm going to do is I'm going to go and create my Silver schema. Alright. And as a next step, we're actually going to start building out these materialized views in the Notebook. And there's one really exciting feature that's coming to these materialized views, which is the ability to define data quality constraints.
So, I can go over here and say, hey, if my sales_quantity column is null, go and drop those rows, right. I can start defining these different types of rules. So, so data quality is just built into it. The Spark in Fabric. Exactly, exactly.
And I can define things like nulls or failures. So, I'm going to go and define the rest of this materialized view over here. I'm going to basically be doing a bunch of joins in my Bronze layer. And let's fast forward now, you can see I've built out a pretty complex Notebook, It's gone and defined.
a lot of views that are built on top of other views, that are built on top of tables. And you can see, basically, I'm going to go and run this ahead of time, But it's really important to run them in the right order. All these used to materialize them in the right order because it depends one on the other.
Yes, exactly! And so this is where our beautiful management portal for materialized views comes in. Now you can see this beautiful diagram. I didn't have to go and configure it or do anything. This is just defined declaratively.
It is provided out of the box. I can see how all of these different views are defined. You can see it updating in real time. Did you see that box just went green? That means that step has succeeded. It has finished running. I can jump in.
I can see all the details, just how long it took to run when did it complete, all of the details I need. So, we're going to keep running this. We're going to keep monitoring to it, and we can see that the next step is actually turned red, which means a failure has occurred. So again, I can just jump into this. I can see exactly what went wrong.
The error message is right there in line. And in this case I can actually see that one of our quality constraints got violated and I configured it to fail if that was the case. Now, we're going to fast forward a little bit here as well. I've gone ahead and fixed my issue and now we can see everything is succeeding. All the steps look green. All of the steps have completed. So as the last step, let's go ahead and actually jump into our lakehouse You can see our Silver and Gold layers here.
And the best thing Is, from a user perspective these just look like delta tables. It's more than just “look like delta table”. These “are” delta tables which means now everything works there. You can run SQL queries on top of those. You can do the Power BI DirectLake.
on top of that, everything works. Everything just works end-to-end So materialized views are coming to Fabric, to Spark. And, one, one more announcement. Yes, one more announcement We're going to do… So, I'm really, really excited to announce that Autoscale Billing for Spark is coming. It is out in Public Preview as of today. So, all those Notebooks you just saw me running, they can all run with this new billing model.
What does that mean? You are billed for what you use, with autoscale up, auto-scale down, which is great for this kind of Spark, spiky workloads that you may have. So. So, the most affordable, the most cost-effective, Spark platform on the market, just became even more cost-effective there, I guess. Right! Thank you, everyone, and thank you, Amir. Have a great conference! Now we're going to continue. We're going to move to the, really, one of the most surprise stars that we have in Fabric, which is the Real-Time Intelligence.
We did not expect to see the reaction that we are getting from you guys and the adoption we're getting from you guys for Real-Time Intelligence. Real-Time intelligence, all about working with real-time data. Data coming from sensors, from telemetry, from devices. We have all the tools that allow you to find all those streams of data from wherever they are, like a single catalog with a Real-Time Hub, massive technology to allow you to process the data as it comes on-the-fly with the eventstream. Eventhouse is just an absolutely magical place to store real-time data, to analyze semi-structured data, time series.
It's massive, massive scale and the Data Activator, which allows you to define conditions when things are happening to say, hey, when this kind of thing happens, I want to take action automatically on top of that. And to talk more and explain what's new with our Real-Time Intelligence, I want to invite Tessa on stage. Hey Tessa, good to see you. So Real-Time Intelligence. This is it. We have a massive, massive platform behind it.
Right? We do! Yeah. This technology is built on top of planet-scale infrastructure. So, we're talking about managing ten exabytes of events and logs per month, doing 5.1 billion real-time queries per day, and doing this all at over five nines reliability.
It's amazing. So basically, every cloud service with Microsoft, every service you have in Azure, every service we have in M365, all the telemetry, exabytes of data every day coming in. They're all being hosted on this platform. Yeah, exactly. And it's not just inside of Microsoft. You guys, in Fabric, we've been seeing a massive adoption of the Real-Time Intelligence.
Exactly. Real-Time Intelligence in Microsoft Fabric is really just taking off, as you said, Even though we just GA'd in November, we have already 8,900 active customers. And we have 360% year-over-year growth.
The response has just been amazing. And when we see that, you know, the growth with typically who's using it? It's typically companies that have mission-critical infrastructure, Mission-critical operations. It could be utility companies.
It could be, you know, transportation hubs. Really anything. We see customers across every single industry. One great example is Melbourne Airport.
Many of us probably took an airline to get here today. And we all know how important real-time and efficient operations are for those types of businesses. And so, Melbourne Airport has had some great success with Real-Time Intelligence, really helping optimizing their operations and just having an overall better experience for their customers. So do you want to show us a little bit of what's what's going on there, how you optimize operations with Real-Time Intelligence? Yeah, let's see Real-Time Intelligence in action. In this case, it's going to be at, a major sustainability company.
So, I'm going to play the role of a data engineer. And I'm going to start here in the Real-Time Hub. And what's so amazing about the Real-Time Hub is I can see all of the different data streams, coming in from across my organization, all in one place. I can easily manage, discover, and take action right from this view.
And if I don't see the source I'm looking for, we make it extremely easy for you to find and connect to it. So here you can see, we have a wide range of sources across Microsoft and across cloud. We have Apache Kafka, Amazon Kinesis, Google Pub/Sub, and more.
And really, we just keep this list growing. And we've been hearing all of you. You wanted even more sources for streaming data and we've been adding more. We have been, yes! So, thanks to this audience, we've been adding more and more connectors. We're excited to announce five new connectors this week, including public data feeds and MQTT.
And with that data streaming in, I can use eventstreams to really process -and route it in real-time. And coming soon, to route it using a language I'm familiar with, on-the-fly, and ultimately get it to any destination that I need. In this case, I'm landing it in Eventhouse, which is that perfect location for the high-volume, high-granularity data that helps me query petabytes of data in seconds. And what's even more powerful, I can actually use it to connect to data across my whole system in OneLake, bringing together my information technology, operation Technology, and engineering technology - or IT, OT, and ET - data all in one place, helping me find correlated insights across all my data estate, both in fast and affordable queries.
Yeah, this is quite incredible, right? Now you have all the like… First of all, if you haven't tried the Eventhouse, you have to try the Eventhouse, even if you don't have sensors or devices. Because you can actually use the technology of the Eventhouse to work on top of every piece of data you have in OneLake. And can get massive query acceleration on top of that. Number one.
Number two, what you have here is, the volume of data it can handle. We had, a couple of months ago, McLaren on stage here. And these guys were analyzing 10 trillion points of data from their hypercars.
You know, McLaren F1 stuff. You know, 10 trillion points of data from the car. In seconds, slicing and dicing with this technology. It's just remarkable. Now, all these data, all these sensors coming to the Eventhouse, what we needed a way to really associate it with the Real-time object.
Whether they're the cars or the trucks or the planes or the gateways, everything. And we really need. Or the window right, in your company. Right? Yep. So, we need a way to really associate and model, kind of create a semantic model of the real-world objects inside of Fabric to create the relationship, with streams of data and the the data that we have in Fabric and the real-world objects. So, this is where we are introducing, and we're going to show a sneak peek of the Digital Twin Builder. Yes, exactly. We're very excited to show you all a sneak peek of the new Digital Twin Builder, because all that information is really helpful, but it's really lacking that context to the information about the actual physical items and how the system works together.
And so, with the new Digital Twin Builder, we can do just that, and let's see it in action. So here you can see that we have the physical operations of my wind turbine and wind farm modeled as an ontology. And here I can actually see all of the different entities and the relationships as they work together. So, I can see, for example, the wind turbine has a gearbox, it has a rotor, it has the turbine blades. And all of these pieces have different properties.
And the relationships that connects them. And so, as the data is actually emitted from my physical devices, it's come in and mapped into this, And you can associate the streams it's come in and mapped into this. You can associate the streams with each one of the components, with each one of them and multiple.
You can bring it all together into one view. So, I can see the wind turbines connected to the broader wind power plant and how all of these pieces are connected together. And all of this data ultimately lands, of course, in OneLake.
And so, it allows me to take all of this personalized, contextualized information and bring it into things like a Real-Time Operations dashboard. So back at the turbine, at the wind operations, we have our technician here seeing the data coming in, in real-time. They can see anomalies as they come through.
And with the help of the Digital Twin Builder, now they can actually go down and see the individual fault codes and understand exactly what's happening in the system. So the real documentation of the wind turbine is in the product. It's directly here. And then also with the power of Activator we can immediately set alerts. And so, the whole team is notified on issues before they get even worse.
And so, with Real-Time Intelligence now we have, you know, the real-time insights. We have AI-powered insights. And with the new sneak peek of Digital Twin Builder, it really helps organizations improve their operations. So, the Digital Twins joined the family of Real-time Intelligence. Thank you so much, Tessa. Really excited!
Okay, so let's move on to the next pillar in our vision, which is the open and AI-ready data lake. This is the world of OneLake. And if you are here, you know what OneLake is.
It's one place for all your data for the entire organization and infinitely scalable planet -scale deployed. All the data from all the workloads. Every piece of data that has been generated is stored in OneLake in an open format. Data can be shared without having to export, import, move. It will handle your lineage, It will handle the PII detection.
It'll handle everything. It's a SaaS data lake. Pretty incredible. We've seen amazing, amazing, amazing momentum around OneLake. All of you have been adopting it in, in an incredible way.
All of you have been adopting it in an incredible way. So just giving you a little bit of stats here. We've seen 3 million shortcuts already being created. Every 15 weeks, you guys doubled the volume of data in OneLake. It's just unbelievable. Now, you'll ask, how does the data get into OneLake? Well, we have a bunch of techniques.
The first one is through the Data Factory, and the Data Factory is the stack that allows you to connect to any data source, create pipelines, do the data transformation. And to show us what's new with our Data Integration and Data Factory technologies, I’m going to have Shireen join me on stage. Hey, Shireen! Hi, everyone! Good morning. So, we have tens of millions of users using our Data Integration.
That is correct. And the growth has been remarkable. Extremely remarkable.
And I want to take a moment to reflect on the usage so far. So, our platform processes 705 trillion rows of data per month, orchestrates 18 billion orchestration runs per month as well, and moves 430 petabytes of data. And these numbers are only accelerating. And this scale is a testament to the impact that we're all driving together. And one of the things that I've been hearing is like, not only that they use it a lot, but they also want to see more professional enhancement.
I hear from you guys saying, hey, you don't want to have any hardcoded I hear from you guys saying, hey, you don't want to have any hardcoded values in our, in our Notebooks in our pipelines, in our dataflows. We don't want to be able to store secrets there. We want to keep the secret somewhere else. We don't want hard connections.
And we've been working on really making everything dynamic and integrating with all the goodness of the secret management. Yep, you hit the nail right on the head, Amir. So especially when you're building scalable solutions, you know, having those hard -coded values are extremely cumbersome and difficult to scale. So today I'm excited to introduce parameterization across Fabric that allows you to bring dynamic data ingestion. Exactly. Really exciting and seamless adaptability across environments. So, let's take a look to see how this works in Fabric.
So, the first feature I want to highlight specifically is connecting to the data behind for an Azure Key Vault. So, let's run the video really quickly. There you go. So, we're going to pull secret credentials directly from Azure Key Vault.
And I already have a connection to my Azure Key Vault here. And I have an Azure Blob Storage connection that requires an access key. So instead of hardcoding that value specifically, I'll go to my Connection settings and see that Key icon right there. All I have to do. Yeah. Super easy! All I have to do is reference the Key Vault, set my Secret value and that's it.
Now we can bring in secure values directly into Fabric. Really exciting! Yes, yes. So now let's go to my pipeline. We have more things to show you. So, this is my medallion architecture right over here.
And I'm bringing my Product Orders through the medallion ar
2025-04-22 14:25
