Cloudera Evolve 2024 | David Dichmann, Cloudera

Clips
News
More from Cloudera Evolve 2024

David Dichmann

Senior Director of Product Management

Cloudera

Unveiling the future of data management: Cloudera’s partnership with Snowflake

Spurred by demands in generative artificial intelligence, supercomputing and advanced analytics, data management is evolving rapidly. In tune with that evolution, Cloudera Inc. is partnering with Snowflake Inc., with several data flexibility integrations and enhancements slated to reach enterprise users.Cloudera’s David Dichmann talks with theCUBE about its partnership with Snowflake

play_circle_outline Partnership between Cloudera and Snowflake for data sharing and generative AI.

play_circle_outline Benefits of partnership include increased business value and reduced complexity.

play_circle_outline Security and governance considerations in managing data.

play_circle_outline Lakehouse Optimizer for optimizing Iceberg tables and reducing costs.

Info
Transcript

David Dichmann, Cloudera

David Dichmann

Senior Director of Product Management Cloudera

Cloudera announced a partnership with Snowflake in New York City to make data more accessible. The Iceberg REST Catalog allows easy communication of data for various purposes like AI training. This aims to drive open concepts for better business decisions by leveraging data across platforms. Customers benefit from reduced complexity and costs, increased value from their data estate, and enhanced security and governance. The integration with Snowflake enhances centralizing access to data, reducing the risk of data breaches. The Lakehouse Optimizer optimizes st... Read more

explore Keep Exploring

What is the focus of using data, generative AI, and the Iceberg REST Catalog in partnership with Snowflake and Cloudera? add

What is the value of customer's data and how can freeing it up into a common table format benefit the organization's flexibility in using the data for various use cases? add

What are some benefits and challenges of adopting Snowflake in organizations? add

What are the benefits of the Lakehouse Optimizer in relation to Iceberg tables and cost reduction? add

bolt Powered by CUBE AI

David Dichmann, Cloudera

search

>> Hello everyone and welcome back to theCube's Live coverage of Cloudera EVOLVE24 here in New York City, Pier 59. I'm your host, Rebecca Knight, and I'm sitting alongside my co-host and analyst Bob Laliberte. Bob, it is lunchtime here at the conference and we are competing with a lot of noise and presentations.

>> Absolutely. But we're going to power through it.

>> We are indeed.

>> We've got a great speaker coming up next and our guest, so looking forward to it.

>> And we're going to talk product management.

>> Product management.

>> A topic near and dear to your heart, I know.

>> Absolutely.

>> I would like to welcome David Dichmann. He is the senior director of product Management at Cloudera. Thank you so much. We're all Bostonians here.

>> We are.

>> And we're here in enemy territory, so we're going to be okay. We're going to be okay. But I want to talk about an announcement that Cloudera made today, a partnership with Snowflake.

>> Yes.

>> Why don't you tell our viewers a little bit more about what this partnership entails?

>> Sure. So fundamentally what this is about using our data, especially with the advent of generative AI and this focus on how we can get more secondary use cases out of data that we use for managing our business data today and make that data available to anyone and everyone who needs it. So there's a lot of data in Snowflake's data state, and there's a lot of data in Cloudera, especially on-premises and in multiple clouds. So with the partnership with Snowflake focusing on this thing called the Iceberg REST Catalog, which is basically the easy button for different data engines and data compute environments to be able to talk to any data anywhere so that we can actually share this data in a more uniform way. And the key ingredient is this Apache Iceberg table format. And so I try to tend not to talk about these technical things too much, but Iceberg is a phenomenon in the marketplace today as the de facto standard for large data sets in cloud. It's basically released data from the confines of proprietary formats. That data is now more freely accessible provided you know how to use it and how you can find the start of the data, get the permissions to the data, all those sorts of things. So the partnership with Snowflake and Cloudera starts with this notion that we're going to use this common layer to communicate in each other so that folks that are doing their business dashboards and analytics in Snowflake can work on data shared by Cloudera. And then that same data that's being used for our business can also now be used to do things like train for generative and other forms of AI.

>> Yeah, I think that's great. That really helps to drive that open and flexible concept that you've been talking about today, we've heard about today. We know customers want to have that. They don't want to be locked in to any one vendor. So that came across really clear in the keynotes that you're trying to enable organizations, whether they've got slightly competitive products or not and be able to, it's all about better business decisions leveraging all the data that you have. I'm wondering if you could talk a little bit about when we think about this integration, some of the specific benefits or what you've seen from the customer's reactions about this integration and how they plan to leverage it.

>> Absolutely. And so I think you picked on the most important dimension of this, which is about the customer's data. We see this data as a very valuable business asset. It has tremendous value to the organization. It's something they own and they possess, and being able to free it up into a common table format means that they are more flexible, as you were saying, to use that data for all these additional use cases. So some of the customers that we were already working with in combining Cloudera's capability with Snowflake's capability prior to the partnership announcement, were using it to be able to do things like remove a significant portion of the complexity of moving data around by moving from a traditional data warehouse approach to an open data lake house. The open data lake house allows us to put all of the different types of data together in one place, structured, unstructured, curated raw, and then be able to use it for all of the different use cases as needed. Well curated data for the stuff where example, we're creating a report where we're paying tax to a government, you can't be more or less within a margin of error. You've got to be spot on. But other things like through analysis of social media, do customers like our brand? You'll make the same business decision at 80% plus or minus 5% because 75, 80, 85, yeah, they're happy with our brand. We're doing well. So different types of data fit for purpose for different types of things are now all together usable by Snowflake for those business analytics reports and those data warehousing use cases, but also now useful for all of the broader traditional lake use cases like statistically motivated predictive analytics. So what we're finding is we're able to create more focused single copies of data that can be used in more additional use cases, providing additional business value from that precious asset, which is that data estate that they now have in a single cohesive environment.

>> So it's producing more value, it is reducing complexity, but is it also reducing the total cost of ownership too?

>> Absolutely. So when we talk about total cost of ownership, we don't just look at the data estate and how can we take cost away from the IT budget, and that's certainly there, when you have fewer ETL processes, you need to manage, fewer point-to-point connections and reduce a lot of that spaghetti. First, you reduce a lot of the maintenance burden, you reduce a lot of the development burden for new use cases, and of course you reduce a lot of the hardware and software needed to run all of that. When you have the open data lake house, you can consolidate a lot of that. That takes a lot of cost and pressure out of IT, reduces the number of people that need to spend their time on that work, freeing them up for more creative work. But that's the second part of this. The total cost of ownership is also about value return to the business. When you get to spend less time on the plumbing and more time on things that actually impact business, like exploring and experimenting with AI and finding places for it to drive business value or creating new kinds of reports on more data than you thought was available to you to augment those reports, like dashboards that now include weather streams to tell you how to change your decisions based on cold days and hot days. All of this new information that's now aware to you, make available to you, can increase the value of the data that also affects the overall impression of TCO. So adding more business value with business use cases while also removing costs from IT.

>> Yeah, absolutely. That makes a lot of sense. One of the things I wanted to ask you about when we've looked at and done conducted research and associated with our research partner, ETR, they've come back and said, security and governance, top of mind for people trying to manage their data. We need to make sure they can't have any data breaches, et cetera, make sure it stays within their organization. I'm wondering if you could touch upon how this integration with Snowflake helps address some of those concerns while still allowing organizations to scale as well.

>> Absolutely. And so if we take a look at the way a lot of organizations have adopted something like Snowflake, the great thing about Snowflake is its flexibility to allow anyone in the business to be able to grab a Snowflake, get at the data they need, and make decisioning and not have any friction with waiting for provisions and waiting for services and approvals, just go get it and do it. And this is amazing because it makes the business flexible. The challenge is if by doing so, we also move more data out of secured environments into edge environments, we're now making that exposed. And what we've seen with some of our customers with front-end data warehouses, Snowflake, even Cloudera data warehouse and otherwise that have been built around a data warehouse philosophy, we now end up with 70, 80, 90 front doors to keep locked. It's like, no wonder some keys went missing. This is no one's fault. No one built a product that was hackable. We just implemented in a way that was a little less controlled. And this is where the open data, lake house philosophy really pays dividends. By putting all of that data into a single open data lake house, the unstructured data, the landed data, the curated data, all of this in one open lake house, we can then put that behind one firewall, one security, one governance, whether the data's on premises or in any cloud, one place to go and ask, "May I have permission to that data?" So now if you break in the front door of the front-line data warehouse, you're only in the foyer of the building. You haven't got past the security desk, you didn't get the badge on your shoulder, you didn't get to go up the elevator, you didn't get into the data, you just got into the lobby. And so that gives us a tremendous amount of comfort that with that firewall and our one copy of the data then made available to all the endpoints, whatever they may be, we can restore that control over security without limiting the freedom to use that data in every business unit everywhere they want.

>> So in highly distributed environments, you've dramatically reduced the attack surface and simplified the management and updating everything of that data.

>> Absolutely. And I like the way you said the surface area, because that's exactly what happens. When we get into what we like to call the chaotic clouds, where you're just popping up cloud instances of all kinds of things, you're creating silos. And if you're also putting the data inside those silos in proprietary formats with its own security and governance, you are creating a dramatically larger surface area of risk, especially to cyber attack. And so to defend that, you just bring all that into the virtual open data lake house. You're not moving the data necessarily. You're moving where the control is to the metadata layer where you have that one set of lock and key.

>> Yep, absolutely. That makes a lot of sense. Yeah, so definitely mitigating that risk and because the unfortunate part is data is everywhere now, it's all being and so much at the edge as well with a lot of the IOT devices and things that people need to make and derive business insights from has to be protected. So dramatically simplifying how you protect and how you secure that and how you protect the access to that as well. So I think that makes a lot of sense.

>> Absolutely.

>> One of the other things we want to talk about since we have you here is the Lakehouse Optimizer that you talked about going into preview. And I'm wondering if you can maybe share a little bit more about how that helps with the Iceberg tables, kind of impact it has on reducing costs and so forth.

>> Absolutely. So when we look at Iceberg as a table format and compared to some of the proprietary table formats where the engine and computer are tightly bound, there's a lot of automatic optimization that happens. But when we move that data out into traditional data stores like AWS's S3, ADLS File Systems, Ozone on-premises with Cloudera, the advantage is this is now files in a file system with massive parallel processing on top. So we can get at that data really quickly. That data can be any shape and size it wants, but there's a phenomena that happens with that that creates a bunch of little tiny files depending on how that data's being interacted with. And that sets us up for a less than optimal way of getting at that data. The more you fragment that data up, the harder it's to get at. But we're talking about file systems at all these different, so no one engine's going to own the responsibility of cleaning that up. So the Lakehouse Optimizer sets up a set of rules that you define and then does some intelligent analysis of the data estate that's managed by Cloudera and looks at those Iceberg tables and identifies whether it's a hot table or a cold table, and how large that table is. It uses telemetry to understand how that table's being used, when it's being accessed, how it's being written to, is it starting to fragment? And will automatically run at the right time with the right resources ways to basically optimize the storage behind the Iceberg. With things like the snapshotting, so one of the magic tricks with Iceberg is this thing called time travel. It's wonderful because you can go to Iceberg and run a query into the data as of a certain point in time. You can faithfully recreate data reports at any point in time without additional manual work like we used to by creating copies of tables and other things and managing it by hand. But every time these are created, we're creating additional inefficiencies in the overall storage. And so there's another technique that we can use that cleans all that up, essentially vacuums up the bits you don't need. So with the Lakehouse Optimizer automatically carrying for your data estate for you, the first thing it does is reduces the administrative burden, you have to keep this clean. We keep it clean for you. But the second thing is when you have these clean tables, you run less compute to get at the data to use it. So you save yourself on the consumption bill when running any compute, Cloudera or otherwise against this data estate. So you can save money, get better, faster results, meet your key performance indicators more effectively, and keep this all clean without having to grow your staff.

>> So I'm really interested in what you're talking about, the reduction in complexity, reducing the administrative burden as well, and as you said, really allowing people in it and frankly across the organization, spend less time on the plumbing itself and more time on the more creative aspects of our jobs and being able to spot patterns, identify potential problems, and as well as solutions. So I'm curious about how this changes both how you run your product organization and also customers you work with in terms of how people do their jobs differently and how people think about innovation and spotting potential product opportunities.

>> So that's a very interesting question. From a product perspective, we are looking, and this ties into some of the other announcements we've made with our true hybrid approach, we look for a lot of synergies across the platform to be able to deliver an environment that not just gives you the Iceberg table estate management, the partnership with Snowflake, with the REST API and other partnerships to follow around the REST API so that everybody can talk and understand the Iceberg estate. We also want to operate on that in a more uniform way, be able to do data preparation with the same tools, either an on-premises or in any cloud. So what this does is it also allows us to reduce the amount of skills we need to learn, the amount of different products and technologies we need to learn. We can do everything in Spark for data preparation. We can do everything in something like a Trino or a Hive or an Impala or Snowflake or whatever to do the analytics so we can use the skillsets we know and bring those skillsets to the table without having to learn new ones. So it allows us to be more efficient and streamlined, and we're looking at building and delivering our product in a more efficient and streamlined way, which also gives our customers something and ready to use in a more streamlined way. So that leads, I think, to the second part of your question, if I understood it correctly, is what is the impact to the business of having this? And this is allowing a lot of organizations, especially highly regulated organizations, where Cloudera tends to play to have the freedom to be more creative and do more experimentation in safe ways, such as working in on-premises environments and do their experimentation, do their tests, build the business applications that will run in cloud without having to change any of the business applications because they've deployed in cloud because it's the same base environment that they're running against. It's the same bytecode, the same services, it's the same application. So when I write to Cloudera, I can deploy in any cloud and it runs the same way. I can even move freely between clouds. I can even expand global coverage. When we look at all the different cloud service providers, international corporations have an interesting phenomena. By using Cloudera as kind of the universal services, I can run those over AWS in regions where AWS is a better support for me, run the same business application in Azure where that has better regional support for me, but I'm running the same business application with the same code on the same data sets the same way. I don't have to build in AWS and an Azure equivalent and then try and integrate them together. So this gives us a lot more flexibility to again, extend the data estate in our best business asset, the data we own, and make it more useful with less friction, whether it's between cloud and on-premises or between different types of services or different use cases, whether it's predictive AI, quantitative AI, or even just traditional business dashboards and reports.

>> More useful, less friction. I think that's what everyone wants to hear.

>> Absolutely.

>> David, thank you so much for coming on theCUBE, a really interesting conversation.

>> Thank you. This was great. Thank you so much.

>> I'm Rebecca Knight for Bob Laliberte. Stay tuned for more of theCUBE's live coverage of Cloudera EVOLVE24. You're watching theCUBE, the leader in enterprise tech news and analysis.