CDOIQ Symposium 2024 | Day 2 Kickoff with Karthik Ravindran | Microsoft

theCUBE.net

Clips
News
More from CDOIQ Symposium 2024

Karthik Ravindran

General Manager, Purview Data Governance

Microsoft

Unified metadata models pave the way for advanced data governance

Unified metadata models are emerging as a cornerstone in the changing landscape of data management, highlighting the critical need for integrated frameworks that ensure data accessibility, security and value creation.As businesses increasingly leverage artificial intelligence and interconnected data systems, the focus is shifting toward avoiding infrastructure pitfalls and achieving seamless data integration. This shift underscores the importance of strong governance frameworks and standardized data formats to optimize the value derived from organizational data across diverse environments, according to Karthik Ravindran (pictured), general manager of enterprise data at Microsoft Corp.

play_circle_outline Modernizing Infrastructure for Data Ecosystems: Importance of Governance in Organizations

play_circle_outline Unlocking Data Estate Potential: Microsoft Purview's Semantic Views Benefits Unstructured Data Governance Approach

play_circle_outline Role of graphs in metadata management and governance

play_circle_outline Simplifying data governance for business users through semantic views and unified catalogs

play_circle_outline Emerging Trends: Transition to Governance Catalog, Personalized Data Management, and Opportunities in Supply Chain Solutions

Info
Transcript

Day 2 Kickoff with Karthik Ravindran | Microsoft

Karthik Ravindran

General Manager, Purview Data Governance Microsoft

TheCUBE covers CDOIQ, where Sanjeev Mohan, Dave Vellante, and Paul Gillin discuss data governance. Karthik Ravindran from Microsoft leads product and engineering for Purview, focusing on data security and compliance. He talks about data integration challenges and the shift towards analytical data management. Purview offers a catalog of catalogs and emphasizes unified metadata models. Microsoft aims to cater to various data management needs, prioritizing customer choice. Recent industry developments include Tabular's acquisition by Databricks and Microsoft's i... Read more

explore Keep Exploring

What is the scope of responsibility of the speaker at Microsoft? add

What is Microsoft Purview and what is its purpose? add

What is the focus of Purview in terms of metadata management and governance? add

What is the benefit of businesses partaking in the ecosystem in terms of managing their data estate and setting policies that transcend systems? add

What gap is Purview aiming to fill in the market and who are the critical set of personas that it aims to serve? add

bolt Powered by CUBE AI

Day 2 Kickoff with Karthik Ravindran | Microsoft

search

>> All right, well, welcome to day two of CDOIQ, theCUBE's continuous coverage. We're here in Cambridge, Massachusetts overlooking the Charles River. The boats are out, all the students are practicing, and we are here with Sanjeev Mohan, Dave Vellante. Paul Gillin is also in the house. This is the Premier Chief Data Officer Conference. Of course, we're bringing in chief data and AI officers now, and we've got a public sector summit within the CDOIQ conference going on right down the hallway here at the Hyatt. We're here with Karthik Ravindran, who's the general manager for Purview Data Governance at Microsoft. We're super excited to have you, Karthik. Thanks for coming on theCUBE. How's it going?

>> Absolutely. Thank you so Much for having me. It's a pleasure to be here today.

>> Yeah. Is this your first time at this event, or have You've been here?

>> No, it's actually my second time. I was here the year before last, and glad to be back.

>> So last time I was here was 2019. It's just amazing to see the growth of this event. It's just skyrocketed. And I think it underscores the importance of data, of course, data and AI, and governance is the hot topic, which is awesome for this crowd. Give us a sense as to your scope of responsibility, if you would, at Microsoft.

>> Absolutely. Yeah. So my name is Karthik. Been at Microsoft now for about 24 years. This is going to be year 25. In my current role, I lead product and engineering for Microsoft Purview data governance. Purview is a much more broader solution that encompasses data security, data compliance, and data governance, and I lead the data governance pillar. Prior to my current role, I built and ran Microsoft's internal data office, which was a wonderful journey of transformation for us internally in terms of how we evolved four plus decades of organically evolved data estates to a connected foundation that helped better apply data for Microsoft's internal purposes. And that journey taught us a lot in terms of data transformation journeys, including governance, not just being about technology, but also being about people, culture, change management, organizational evolution. And what we're really doing now in Purview data governance is taking a lot of those learnings that we had ourselves in terms of evolving our practices, and making it easier for our customers to avoid some of those missteps, as well as to be able to get to doing what they should be doing first and foremost, which is creating value from their data versus building a whole ton of infrastructure and apps to go do that. And in our journey, it took us about four to five years to get to a good place with our data governance initiatives, but the reason for that is because we spent almost two years is building technology and infrastructure and the next three practicing it. So we are hoping that with investments we're making in Purview, our customers can get faster the value from data, governing it responsibly, and doing great things with their data versus having to build tech to do it.

>> So in the conversations that Sanjeev, myself, others at theCUBE research... For Sanjeev at Sanjmo is an independent, but it's part of theCUBE collective, and so we work very closely. When we talk to customers, they say that they want to move to open table formats. And when we look at the try to read the tea leaves in the marketplace, we've put forth the premise that the point of control appears to be moving from the DBMS to the catalog, but not necessarily the point of value because the catalogs are being open sourced in so many places, not everywhere obviously. What's Microsoft's point of view on that? Is the premise valid? Are there parts of the premise that are valid? Would you totally disagree with that? Help us understand your perspective.

>> Yeah, I think you well laid out the various moving parts that our customers are having to contend with today in terms of the decision making. And the way we like to tell the story is to take a step back and focus on what is actually driving these shifts, right? And in my perspective, that is this notion of connected data, which is starting to become increasingly important for our customers, where we included in Microsoft evolved through several decades where we focused on data silos that supported individual business functions and units And several of our customers' data investments have also organically evolved in a similar manner. But now as we get into the world of AI, and increasingly so, applications of AI, it's becoming very visible that the importance of connecting data from across the enterprise is going to be of paramount importance to truly get the value out of the data assets that you've grown over the years and that you're looking to grow forward. And when you start connecting data, you're going to have to wrestle with a lot of complexities in terms of the source systems all being in various architectures, being in various data formats, and being able to bring all of that data and connect it together and create a unified linear, not just for catalog and discovery, but also for data management is going to be something that is of much critical consideration. I'll give you a real example, customer data at Microsoft. We have 100 plus systems inside Microsoft that emit customer data, just basic account data, right? Who's the customer? Who's the account? Who's the contact? What's the address? What's the telephone number? Basic master data, 100+ systems. Now, you add onto that operational data, you add onto that engagement data, you're talking about another a hundred plus systems. No enterprise... And by the way, this is a very common thing that we see in several of our customers, and none of us have the luxury of saying, "Oh, we're going to hit a pause button, go to one master system that rules them all.

>> No way.

>> We get started." Each of these systems, different formats, different technologies, right? Integrating, bringing all of that data together into a common linear is not only going to increase the complexity of the compute you need to do that, but it's also going to require you to look at a way to present that data in a format that's common versus reflecting all of the differences and the underpinning systems, right? So we see shift towards what we call the analytical paradigm of data management, which is start, and then shift left. And what we mean by that is start right in the sense that take your data from wherever comes. You don't have the luxury to put the pause button and say, "I'm going to go remaster my source systems." No, bring in the data from across your systems. And by the way, this is going to be freaking big data, which means the computer you got to run to manage this thing and connect it and to keep it together is not going to be the same as the computer and infrastructure that you deploy on your edges.

>> Correct.

>> So it's increasingly going to become important to start thinking about modernizing your infrastructure to handle the data ecosystem, and then however the storage in turn get optimized for serving the kind of compute scale that you have to serve. That's why we see open table formats. The sanitization is being incredibly useful to help customers

>> If I may, when you say bring in, you mean make it accessible? You don't mean necessarily move it-

>> Ingesting, you're not making a copy of-

>> Yes. And there are two architecture patterns, right? Because this is a very important question. I'm glad you asked this. There's one architecture pattern, which is virtualization, which is keeping the data where it is, pointing to it through symbolic links, and

>> Yes.

>> But there's another genuine architecture pattern, which is a one copy architecture pattern, which might also need to fit into the picture depending on the state of the originating systems. And the reason I say this is because... And we learned this the hardware at Microsoft. We went first at it, we tried to do the 100% virtualization thing and pointing compute to whatever data sits, and we just tried to create a common veneer, which is through an open format. But then the veneer basically just points back to your internal systems. You know what the risk in that is? The risk is some of your Originating operational systems may not be ready to handle the scale of compute. And those systems are also serving other primary use cases, which could be brought down to their knees. One of the systems that we try to go to last scale analytical computer against, it literally choked up because it wasn't set up to manage that kind scale, right? So depending on how deeply you are in some of the legacy sort of like stores, you might have to consider a one copy a architecture where the right thing to do is to move the data into an analytical layer, which is optimized, both not just from a Compute perspective, but also from a storage for that compute, right? And then make the right choices between when to virtualize and make a new copy.

>> And you support both.

>> We support both. We have to support both, right?

>> Yes.

>> And to make it real for you, if you look at some of the recent announces in Microsoft Fabric, there's two concepts you've probably heard of, shortcutting and mirroring. Fabric shortcuts are virtualization, right? That's about keeping the data where it is, pointing to the data where it sits, versus fabric database mirroring is primarily designed to enable some of the data caching and mirroring that needs to happen for sources that are not set up to run and manage the scale of compute that is needed for today's workloads.

>> Right.

>> So I think it's an and. It's not necessarily an or.

>> So I have a question about what you said earlier about 100 different sources where customer information is stored, but then you have another 100 unstructured contracts, agreements, licenses. So is unstructured data management and governance purview of purview?

>> Fantastic question. In fact, I would say increasingly in the age of AI, unstructured data is going to come to the front and center, right? Any kind of governance solution-

>> So I want to know how. How do you do it? 100%. I'll speak to it.

>> So any governance solution that caters only to one or the other, as in structured or unstructured, is going to be incomplete.

>> Yeah, it's music to my ears.

>> It's going to require further investments together.

>> I love that.

>> So with Microsoft Purview, what we've done now in Microsoft is we've taken what used to be two, I would say orthogonal sets of investments inside the company. There've been a whole bunch of investments around our security and compliance solutions, which were very anchored on Microsoft 365 and the unstructured information estate. So you're probably familiar with products like Microsoft information protection, data loss prevention, data lifecycle management. All of those were very anchored on the unstructured estate. And then you've likely heard of what used to be Azure Purview, which was more the data governance platform for the structured data . And we took a step back about 12 months back and looked at the commonality that is needed to truly sort of capital G govern an information estate, right? So capital G governance encompasses security, compliance, risk management, and data management. It's not just traditional data governance. So when we looked at this opportunity, we said like, holy crap, look at this. You've got CISOs information security officers, compliance officers, data officers, all of whom are today largely operating in silos, investing in their own infrastructures, all of whom define and manage policies that are important to each other but don't really come together in the most seamless way possible because of the systemic constraints and limitations. How cool would it be if we could bring together all of these investments into a single solution offering that spans these structured and the unstructured information estate, takes all the goodness that we've innovated on the unstructured estate and bring it to the structured data, and take the innovation on the structured data estate and bring it to the unstructured data estate and combine it together into one whole solution. That's Microsoft Purview, right?

>> So I've heard Microsoft is starting a new lab, and they're making a big investment in Barcelona for graph use cases. So is the idea for unstructured data to extract entities into a graph, and then make it govern it through Purview? How-

>> Yeah. So let me talk a little bit about that. Great, great question. So the graph has actually existed within Microsoft for decades now.

>> I'm sure, yes.

>> If you think about Microsoft 365, what do you think it operates on underneath? It's the Microsoft graph, right?

>> Yeah.

>> And the Microsoft graph itself, the users of the applications never get to see it because experiences are so seamlessly integrated into the workloads. But the data structure that's fundamentally supporting all of that richness, all of that seamless interchange and exchange of information between the M365 apps, all of that is underneath powered by the Microsoft graph, right? And we have an internal infrastructure called the substrate, which hosts the graph, and there's a whole bunch of go underneath that, right? Now, similarly, if you look at the Azure data estate and all of the Azure data cloud and estates over there, you've got a graph in the Azure world that manages all of that for the Azure operations. And now really what we're doing in Purview is, at least from the metadata management perspective, from the governance perspective, we are bringing those graphs together, right? Where I think about one single graph, one single graph, which is not just a graph of your technical metadata, but which is also going to be a graph of your technical plus your semantic and business metadata, which captures the entirety of what we now expand to and speak to as the digital estate, which is not just confined to data and data systems structured and structured, but also applications, infrastructure networks, bringing together a single digital view of that entire sort of enterprise estate, which then provides the underlying metadata foundation on top of which the workloads, whether it is governance or applied analytics, can activate, right? And that's the journey that we are on.

>> So that unified metadata model is actually unique in the industry and very powerful because now you can bring co-pilots on top of that and you can have the co-pilots take action confidently if it's governed. Okay, so we understand Purview is sort of a catalog of catalogs, if you will. So when you think about Snowflakes effort to open source Polaris keeping horizon for the role-based access control and all the capital G, I guess you refer to it, the business data, and then the unity being open sourced, et cetera, that doesn't affect you. You're fine with all that. Let the customers do whatever they want to do, and then you'll provide a catalog, uber catalog, if you will. Can you clarify that, explain it, validate it?

>> Great question. Great point. So yeah, we see it as being totally complementary. And in fact, I would say the day and age of catalogs goes back to as early as the initial days of SQL Server. I'm sure that you're very well familiar.

>> dictionary.

>> Of course.

>> Or

>> You're very familiar with the master database in SQL Server.

>> Yes.

>> What do you think that is? That's the local SQL Server catalog, right?

>> Right.

>> So we look at... There's not a single data cloud that cannot or should not have a point catalog that is optimized for that store. Databricks have Unity, Snowflake have Polaris, Google have Googleplex or Dataplex, Amazon have Glue, right? And the reality is you do need those local catalogs. In fact, Microsoft Fabric will have its own local data catalog that's optimized for its data estate. And these catalogs are meant to surface the artifacts within those data states. They're meant to surface, I would say, data management capabilities for setting up and permissions and basic policies at the estate level. But now, step forward and think about it through the lens of functions such as the data office, the privacy office, the security office. It's a very complicated task. And I can speak to this because when I ran the data office for Microsoft, my team constantly had to do this, which is going down to each of these physical systems and making sure that policies are configured and set up consistently, figuring out how do you, for our business users, take all of the complexity of the technical metadata in each of these systems and abstracted into a business presentable catalog. Super complicated challenges which often require to build point connectors, point tools for these systems versus taking a step back and saying, "Hey, why do I have to deal with the physical sort of gnarliness of all of these systems? Is that a better opportunity where I can create a semantic view that is focused on business concepts, that is focused on business constructs, which can then be mapped at scale to underlying physical stores current and future?" And then the business then gets to partake in the ecosystem, not just in terms of consuming the data, but also contributing to managing the data estate, especially since they've got the deep domain knowledge and context and understanding, being able to set policies that can transcend systems, agnostic of where the data sits. Going back to the customer data example, customer data sits in hundreds of systems in most organizations. If each customer or organization had to go and set policies and govern each of those systems, the sheer volume of the human task and the openness of the kind of errors and malconfiguration of policies that could surface from the human error quotient that can continue to increase because of the complexity of the task is huge. So what we want to do instead is to create a venue where you can look at your data estate through a business lens, and where you can configure and manage your policies on the audible business concepts and have it trickle down into your underlying data estates without needing to know the physical underpinnings of each system. And then from a catalog perspective, being able to harvest these data estates and present a unified view of the catalog of catalogs where enterprise wide consumers and users can search by business terms, can type in natural language queries that are focused on business concepts, business needs, and discover data agnostic of where it sits is also a great opportunity for what we can do for the consumption side of the data.

>> So because you have the unified metadata, the operational, the technical metadata, you can now take responsibility for harmonizing that data, the semantic layer, and then you can take action on top of that.

>> On top of the semantic.

>> And how much of this is production?

>> Yes.

>> For example, you've got customer data, unified semantic view of customer data. Now, some of that customer data may be in Snowflake as a external, Polaris, Iceberg table or in their proprietary, some may be in Salesforce, and some may be in SQL Server. So now you've created the semantic view and you're going to do policy enforcement across these. So how much of this works today?

>> Right. So you asked two very interesting questions. One is the unification challenge and the broader sort of governance and management.

>> Governance, correct.

>> So the unification challenge, I would say that are some really killer solution providers in the market today, like folks for instance, like Prophecy, CluedIn, Semarchy, Reltio. These companies have been doing incredible work tackling this fundamental problem of master data sitting in multiple different systems, not just master for that matter, any data.

>> Any data.

>> And that's why it's applying probabilistic models, machine learning techniques with human reinforced AI to tackle the complex problem of stitching together these different disparate systems into what's called golden records of truth. So that investment, I would say has been happening for the last, I would say several, I would say years, right? But that's one facet of data management,

>> Correct.

>> That doesn't cover all of the further richness that is needed around data quality management. While you can argue that data quality isn't parts master data management, it's also a superset of just mass data management. Then there's data cataloging, then there's data access management, there's data observability. There's a whole bunch of other capabilities that need to surround that.

>> Correct.

>> Bringing all of that together, including integrating with the best-in-class providers for the first problem that you described, which is unification, is what we're doing in Purview, right? So in Purview, for instance, we've got very deep integrations with Reltio, with a Prophecy, with Semarchy, as well as with CluedIn, and then we are augmenting that with a whole set of surround management capabilities that can bring the whole spectrum of governance tools and solutions, right?

>> Got it.

>> And in doing this, where are we at right now? So we've launched our private preview in December 2022. We launched our public preview in March, 2024, the Fabric Conference. As of now, we've got 1400 active tenants who are looking at the-

>> How much?

>> 1400 active customers who are looking at the product, and several of them using it super actively and looking to go live on it. Our general availability, I think we announced that earlier this week, Rohan announced that, is coming up on September 1st. We're excited for that milestone. And so yes, we've got a lot of momentum, there's a lot more to be done, but we are super excited.

>> So back to the original premise. If the source of value is shifting, source of control, sorry, shifting from database management system to the governance catalog, but the value is leapfrogging that to the catalog of catalogs, and then presumably the ability to combine toolchains, and then take action, is that a valid premise? Do you see that as their new value point? And not every company's going to agree. Oracle's not going to agree with this. Value's going to be the DBMS for Oracle. And Snowflake would love it to stay into the DBMS, but the trends, given open table formats, seem to be pushing in that direction. Do you agree with that premise?

>> Yeah, I think... Look, data by its very nature is federated, federated at different layers, federated at the systems layer, federated at the ownership layer, and federated at the practice layer, right? The data systems historically are very well served, what you would call your personas that are associated with building and operating your data systems, primarily your data engineers, your data analysts, your data scientists, and ID operators. But in this day and age, there's a critical set of functions that complement those technical functions. This is the data office, the data steward, the data product owner. And these functions are not necessarily data engineers, analysts and scientists. Bringing together the cycle of data management for an organization needs you to stitch and create experiences for both these edges, right? And while the first set of personas are going to be focused on the individual estates that they have to build and operate for their specific needs, the latter set of personas need to cut across and look across these systems, and they're also personas who operate at the business conceptual level, not at the technical underpinning level, right? And they also happen to be the domain experts, the subject experts who actually understand the business that the data is serving. And they're today underserved by solutions and tools that they can use to responsibly govern a complex multi-cloud, hybrid often, estate. So that is a huge gap in this market today for a solution that can help that critical set of personas who are vital to the data supply chain in any enterprise to not just be productive, but to also be effective in performing their tasks for the organization across the spectrum of physical estates, right? And that's the gap that we're looking to go after with Purview, not just for data governance, but also for security and for compliance, right? So we see us being very complimentary, right? It's not an and or an or. It is truly an and.

>> And that's a prerequisite for systems of agency. It truly is unified metadata, that broader view of capital G governance, et cetera. We are out of time, but I have to ask you, are customers asking you about Iceberg, the acquisition of Tabular by Databricks? Do you have a point of view on that? What are you telling them?

>> Yeah, there's a lot of interest right now in OpenTable formats for two reasons. One is to be able to create a consistent format veneer across these multi-cloud estates, and the other is to benefit from that veneer, not just as a way to discover data, but also as a way to operationalize data management at scale versus having to deal with each of the different individual systems and formats. Now, there are several, I would say postures that folks take in terms of a strong preference for one format over the other. So I think those, I would say religious debates and wars around the formats will continue to happen. At the end of the day, I think there's a lot of similarity between these formats while they have the nuances and the differences. I think that are some brilliant moves that have recently happened around initiatives, like for instance, the acquisition of Tabular by Databricks, and even prior to that, I think Databricks that the uniform sort of strategy, and then vice versa. And Microsoft, we are behind the XTable initiative in Apache Spark. And the way we see it that Microsoft is like, look, at the end of the day you got to give the customer the choice. Give the customer the choice, right? So we are not indexed on any of these religious postures and points of view on which format is better than the other. We are more focused on saying there are viably, objectively good formats out there. Customers are going to make their choices and picks based on various considerations. Agnostic of the pick that they make, we need to be able to support them, right? And then eventually, will one of these formats prevail? Well, let's wait and see. Time will tell.

>> Yeah, that's a good strategy. You don't have to choose the religion for them. All right, Karthik, thanks so much.

>> Absolutely.

>> We appreciate you helping us understand these important issues.

>> Awesome. That's fantastic.

>> And also, congratulations. You're going to finish 25 years.

>> 25 years, wow.

>> Yes....

>> at Microsoft.

>> 25, this number, yeah, looking forward to that.

>> Oh, wow.

>> Flies, doesn't it?

>> Awesome. Fantastic.

>> You start when you were 10 years old.

>> Sanjeev Mohan. Paul Gillin is also in the house. This is Dave Vellante, theCUBE's coverage of CDOIQ from Cambridge, Massachusetts 2024. We'll continue right after this.