Databricks Data + AI Summit 2024 | CUBE Collective

theCUBE.net

Clips
News
More from Databricks Data + AI Summit 2024

How data transformation is shaping the future of AI: analyst insights from Howie Xu

The data transformation sweeping through the artificial intelligence industry marks a significant shift toward integrated, user-friendly platforms and innovative growth strategies.Spearheaded by companies such as Databricks Inc., this evolution is characterized by rapid expansion and strategic acquisitions designed to bolster capabilities. This transformative period is driven by the increasing need to manage diverse workloads, enhance programmatic functions and address complex issues surrounding data governance, security and compliance. As the industry adapts to these demands, it sets the stage for a new era of technological advancement and efficiency

play_circle_outline The Significance of Data Pipelines and Lineage for Business Success, Examining Databricks' Rapid Growth

play_circle_outline Simplifying AI and Data Science: The Democratization of Access, Compute Gravity, and Virtualization

play_circle_outline Unlocking Potential: The Significance of Agent Frameworks, Streamlining Complexity, and Navigating the Building Phase in Today's Industry

play_circle_outline Open source contributions and the role of Databricks in the open source community

play_circle_outline Excel spreadsheets as a hindrance to enterprise AI transformation

Info

CUBE Collective

search

>> Good afternoon CUBE community, and welcome back to Databricks Data and AI Summit. We're here in San Francisco. My name's Savannah Peterson for theCUBE. Very excited to be closing out our two days of Power Pact coverage with a fantastic foursome, Rob, Howie, and John, thank you so much for hanging out.

>> Love it.

>> I feel like the bell of the nerd ball right now. It's a wonderful feeling.

>> Don't sell yourself short on that. You're in the nerd fam. There's no doubt.

>> Thank you, Rob. It's nice to have you on the desk. I've missed you this week.

>> Well, I've been busy sequestered over in the analyst wing.

>> Have you quarantined over there with all the-

>> Not quarantined, sequestered. There's no quarantine going on.

>> So I'm going to open with you, actually, because I'm curious. We've had two totally different experiences. How's it been for you over there in the analyst land?

>> I think it's been really interesting seeing them deep dive into where they are today, where they're going in the future, some of which is under NDA and I can't get into. But I think, again, the success that they're having within the accounts to bring those who've started with them on the data science side over to the Databricks SQL side and expand inside those accounts has been really fascinating. And the growth rate that they have is just unbelievable.

>> 60% growth rates. That's at a 2.4 billion ARR, that's amazing.

>> That's super impressive, Howie.

>> Not too many companies can claim that.

>> Yeah, no wonder they are claiming it. I'd be shouting that from the rooftops because that's where I was. What an exciting time to be them. My goodness. Howie, this is our first time getting to sit on the desk together at the show and in particular. Nice to see you.

>> Great to be here.

>> Yeah. What have you learned? What's been some of your key takeaways?

>> Well, I just wandered around the floor a little bit. Data is the gold mine, as Jensen Huang said, right? The problem with data is it's gold, but it's pretty tough to mine it.

>> Right.

>> Right? It's always easy to have a PowerPoint to say, "Hey, here's all the reports," but when people really do, it's always a lot of work, governance, security, silos, and then who owns what, data lineage, compliance, dedupe, all sorts of problems. So I see so many companies solving so many problems, so it's a renaissance.

>> It's a renaissance. It's great. It's definitely a really unique time. John, we've had the pleasure of interviewing a lot of great guests on the show. Who was your favorite? Not that we have to pick a favorite child, but-

>> Well they all brought a little bit of different game to the table. Joel from Databricks actually gave the Databricks perspective. We had some customers. I really like Excel data because they're doing the data pipelines. I'm fascinated with the data pipelines. As how he pointed out, data's messy, but you start to see kind of in the neural network kind of vein, not directly, but data pipelines will become very important because separating compute from data will allow the ability to support diversity of workloads and various engines, whether it's compute engines, whatnot. So what that means is a lot of different things that was once diverse and messy can now be connected, and that's going to enable programmatic cool things. So I think that's a trend that I love, and that came out in that interview. But Databricks brought game this thing. They brought uniform general availability on the Delta Lake.

>> They brought game, baby.

>> Unity Catalog, open source, Mosaic AI adding more tools, Lakeflow on the pipelines, and then serverless taking away all the infrastructure so there's no knobs and clusters to manage it. So they're starting to kind of, I want to say grow up a little bit and get out of the command line-

>> There's a maturity though....

>> command line data developer nerd to okay, scale, simplicity, value at the business level. So I think they're going right down the right path, so there was a great set of interviews.

>> Last year, they bought a Mosaic ML. This year, they bought a Tabula, so they seem to be doing well on the M&A side. They're leveraging that-

>> That's a good point, Howie....

>> tool pretty well, even though they're a private company. Usually it's the public company companies doing that, but company.

>> With the valuation of over 45 billion on paper, they got a lot of, Rob, lot of powder to work with, so to speak, in terms of equity.

>> That's one way to put it.

>> I think Snowflake and Databricks are going to be competing. If Databricks makes product advancements, Snowflake goes on the M&A front. They make an M&A move over to Databricks, Snowflake kind of response. So we're going to see that back and forth action.

>> At the end of two weeks of data weeks, two weeks of it, you start to look at how these two are really going head to head, and I don't want to say tit for tat with the announcements, but when you start to look at, hey... There was a direct shot across the bow when he said-

>> For sure.

>> When Matei goes up there and pushes and makes the repo public for Unity right there and then on stage versus in 90 days for Polaris, that's a direct... That was

>> That was a direct frontal-

>> Right....

>> humiliation point.

>> And it was a move for the community. I think it was a real moment of uniting people, pun intended, but I thought that was a special moment. And I think anytime-

>> Oh, it was a nice shot that Snowflake, right through the arm, bullet through the arm, hey, here you go.

>> Polaris.

>> 90 days, how about now?

>> This is how we feel about 90 days.

>> 90 days.

>> We feel it should be instant.

>> I think it's more than that. Both of them, they are open... They wanted to have an open community. Then the question is, if it is a developer, where do I go? Right? Maybe there's another open community that's on top of both of them, right? So I actually wrote up on the LinkedIn yesterday. There are companies, open source project that they are more vendor neutral. Potentially, possibly, they have more advantage.

>> Yeah. And I think, again, when we talk to them again over in our little nerd fest in their sequestered area, part of it is how do they go after verticals they haven't necessarily been strong in? And I think they've done good at, for instance, at the credit union and FinTech level, but how did they get into the hardcore banking aspect of things. Where Snowflake has historically been really, really strong, I think they're bringing a lot of features to bear to be a more complete solution, and I think that is definitely a way that they've seen... And again, Ali talked about it. "Hey, I heard from a hundred CIOs, we need Lakeflow, and I need to be able to bring my data from SFDC and from Salesforce into here to join that up with this other to make my intelligence and my AI internally so I can answer these queries." That, to me, is, again, looking at what other personas and what are other customers and TAM, and I hates the word TAM, but going after which TAM do I go and pull in?

>> Well, no, they have a big TAM because they got data engineering and BI and SQLs for writers, right? So that's key. Now, the question is that, Rob, like Snowflake, they do a really easy way to get into the platform. They got a compute engine built in. It's easy to develop and to deploy apps, data apps on Snowflake. Now, the question is that Databricks has got much more robust flexibility, but the ease of use might be more like an iPhone. Okay, it's easy. It's all right there. Databricks might have a better architectural solution and more horsepower, but will they move down? But I think they coexist. I think all the analysts that we talked to, including yourself, mentioned that the collision of Snowflake and Databricks is much more of a market optics. The customers are voting with their data. If they stay on Snowflake, they're going to stay in Snowflake because it's easier and the computes there. If Databricks does its job, it can get Federation to suck the data out of Snowflake, change the economics, then that might be interest. So the question is, what's the switching cost between Snowflake and Databricks, and vice versa?

>> That's actually a very interesting point. Speaking of switching costs, we always care about switching costs. I do think the next few years, there will be some layers of technology to lower down or actually allow you to switch easier, right? Because a lot of times, we have been talking about data gravity for the last decade, right? But believe it or not, because of the AI, because of the Jensen Huang's GPU costing so much, actually I believe that there is a compute gravity going on right here. So before, it's like a compute moving to the data. I wouldn't be surprised, in the next five or 10 years, we want data moving to the compute because compute... Electricity is a scarce resource. The coating is the scarce resource. You want data to move there. Now, it's not going to be wholesale move, right? It's about can you do that incrementally? Can you virtualize the data layer? I do see a number of smaller startups, startup selling just message. I'm actually quite excited. That's why I see that there will be a lot of innovation over there.

>> In fact, we had a discussion about that over in the other place about Delta sharing. And when you bring in stuff into Unity Catalog and you're able to share without moving the data around, bringing the data to the AI in a location that has more power, lower cooling costs and things like more sustainability even from that perspective.

>> Good that we're speaking the same language, because in the last decade, there is only about data gravity. I do think the compute gravity is going to be more a thing, to your point.

>> I think you're right, Howie.

>> To your point, it's more than just the data. There's a lot of the data lineage, the optimization. So I'm quite looking forward to the new things.

>> And I think the lineage thing becomes really important because when you get hallucinations or you get issues within your data, and especially when you go to these compound systems or complex systems, whatever you want to call them, where you have multiple LLMs involved and other AI, traditional AI understanding where things went wrong, becomes really, really difficult. We actually had a session with Harlan, the CTO on Mosaic ML and really talked about the agent framework and how you break down in that. And he said, "Hey, we kind of went backwards and forgot about all of this circle of really iterating through, because we just said, oh, throw it all at one big model. And it's not going to be about that, and we kind of all got caught up in that. And now, we are doing things in more of a way where it's, hey, I have to understand all of these pieces, how they flow. And really, it comes down to where did that data come from, and how do I replace it?"

>> I'm a huge fan of agents. I think agents are going to be really important.

>> You've been talking about it for a while.

>> And to your point, if it takes too many steps to do it, in every single inflection point wave, that's been important. All the winners were simplify, make it easy, intuitive to understand, and reduce the steps it takes to do something, every single.

>> Simplicity, uniform layer.

>> Every single time. So your point there, it doesn't get done, I can't build my app faster than before.

>> And it's all about meeting people where they are. One of the things that was the theme of the conversation all week was not just appealing to the lead data scientists within an organization or a specific dev, but actually making it so that everyone can be a data scientist to a degree if they have access to the AI and to their data.

>> The key thing with the generative AI is the AI is being democratized, right? AI is being democratized, but the data access is not, right? So I think that's why so many vendors are innovating how to... I came from virtualization backgrounds. I have to use a virtualization, right?

>> Hypervisor.

>> Really virtualize the data layer in terms of the data access, the catalog. So I'm not convinced that-

>> We call it a hypervisor, a data hypervisor.

>> Yes. There's a company called Data Straddle, right? That's sort of another layer.

>> Joking aside, what you're getting at is that this harmonization we talked about on our rope day one yesterday with George Gilbert, you get the plumbing right, and then the data harmonizes through interactions. I think we're going to start to see something new we've never seen before, once people get their lakes set up, once the plumbing is in place, pipelines have AI to it. So I just think that we're in this grind mode right now. Things start to get-

>> It is gritty.

>> You got to wire this together, Iceberg with this, get the formats right. But once that's under the covers with a hard and top, then it has to get better. I just don't see... I like Mosaic. I think that's got a great action, but it's still in the... You got to grind it out, knock it down. So whoever nails the simplicity on the agents will take a lot, will take a lot.

>> Think of the province statement I mentioned, right? If you want the compute to follow data, there are so many problems you need to solve, so there will be some grinding period.

>> Oh yeah. Well, and it sounds like everyone we talked to, a lot of the guests we talked to are seeing a lot of different customers across verticals, and they spoke about that exactly. We're in that gritty building phase. When I ask everyone my standard savvy question, what do you hope to be able to say when we're here this time next year that you can't say today, it was all, "We're at scale with our customers doing X." It was less sandbox, more true implementation and rolling it out.

>> Normally, we get different answers. Normally, it's like, "Hey, I want my life to be easier.

>> They're all the same.

>> Now it's here, it's all about the features, product.

>> Yeah. It was so product heavy in the answers, I thought it was kind of wild.

>> This reminds me of AWS in the early days because you have builders, you have developers here, not corporate. I think this is not a sales event in the sense of-

>> No....

>> it's heavy swag, which is an indicator of developers.

>> I was just going to say that it's open source, it's definitely dev heavy. But I think exactly to your point, it's less salesy, more community.

>> I think in general, the Databricks ecosystem has more developers, and then the Snowflake ecosystem has more enterprise people.

>> Here's my take on Databricks right now, and this is... I'm going to watch this very closely. I'm not down on this, but I'm pointing it out. Databricks has been such a proponent of open source. Everything about Databricks is open source, but they've been very one-sided in their open source. They've been open source heavy on the Databricks side as the committers driving the project. There's nothing wrong with that. But to get true open source, you have to show a lot more non Databricks interactive. Otherwise, it's just a databricks product in the open.

>> What I would say is though, the one comment that I, and I said this earlier today in one of our wraps, that 66% of the contributions to Spark are now coming from non Databricks companies

>> Spark.

>> But Spark is not the core part of the Databricks product at this point.

>> I understand. That's a good point. That's a good point. It's a good point. It's an evolution for all of these different projects. I think-

>> It's a good signal that it could go the other way. My point is that Databricks has to be careful, and they have to emphasize and invest in this, just like they're investing in $2 billion in Tabular. Open source works only if it goes direction of 65%. Otherwise, it's just a Databricks company in the open. So I think that'll be the true test. And if that infiltrates into open source in a positive way, I think the data engineering will have a very Kubernetes like effect where that stuff happens fast.

>> And the one to watch is Unity Catalog because that really is the glue at that data layer.

>> Unity Catalog could be a data engineer's dream, not so much, just from the government side. Just like that could be just like I would've set the whole thing, and I predict that you're going to see complete redos. If you look at the math of how much it costs to actually go into an enterprise, take the big enterprise, take a pharma company, anyone, say, "I'm going to redo the entire data estate from scratch, reset the whole thing from the ground up," 100 million, maybe less. Rob, do the math.

>> But again,-

>> Don't ask Rob to do math.

>> Yeah, math might be too at the moment.

>> Let me ask OpenAI to do that.

>> Let me tell you. We're chatting.

>> Careful John.

>> It says math now. But I think, again, why I think... I've already had two different organizations say to me, at two different parties this week, "I'm really excited to get my hands on the open source Unity Catalog to go and see what I can actually do with it, and how much of it's actually open source, right? And I think that's the key, is what features that they have on top of Unity actually land in the open source versus what pieces are actually Databricks built on top of Unity?

>> Basically, is that truly, truly open? That's the question.

>> Yes .

>> Well remember-

>> I think the jury is still out.

>> Remember when Ali went up there yesterday on day one, he said basically the three things on the industry was demand for AI, check, we all know that, security and privacy, and the fragmented data estates. And again, that speaks to what we know, and everyone knows data is messy. But also organizationally from the data warehouse, old school data warehouse, it's siloed and it's like-

>> Silo. Silo, silo....

>> you can't... Data's got to be free to fly around to be integrated.

>> It's got to be free.

>> It's got to fly around horizontally scalable and in the application.

>> But that's not the enterprise has been set up in the last several decades.

>> Right. No, not at all. It's the opposite of free. It's... Yeah.

>> The script's flipping. So again, hypervisor, I'm just joking, but that concept of the control plane, interesting. It's got governance.

>> I hardly believe that a unit catalog will be that layer as a starting point of hypervisor. I think people will have some skepticism. They have to prove. There is a lot of work for them to do. So I truly believe that it's not just a collision between the Snowflake and the Databricks. There will be opportunity for third party to provide a more truly open-

>> Well, when you're at VMware, you wrote one of the early lines of code, first code for the network side.

>> Casual.

>> When hypervisor was not even a conversation that was happening, what was the motivation behind that? It was obviously server virtualization, but what was the motivation between the hypervisor? Why did it come about?

>> Well, 20 some years ago, believe it or not, I remember I went to VMware and I saw the stats. I couldn't believe that. Worldwide, the server utilization was below 5% across the board, because Windows... You give a lot of the CPUs, the Windows was not able to drive multiple CPU. Today, it's hard to imagine.

>> threaded application.

>> Here's a server. We're talking about expensive servers, utilization across the board, not just one company, across the board was less than 5%. So that's the motivation of hypervisor initially, to increase, basically to write an operating system right, basically.

>> And I think you have that same thing happening with data. It's like, how much of my data is actually being utilized? I'm just flooding it in.

>> Fingernailing it probably

>> Exactly.

>> In theory, there are the CPUs. But in reality, are you using it at same thing with data? It's the same thing.

>> Exactly. And how much of it's hot data? How much it's useful data. And we were even talking about how-

>> Hot data, there you go.

>> Hot data, and how you could actually, within answers and within Delta, you could actually go in and take something out from Mosaic using the framework, the AI framework. And if you deleted it out, you deleted it out of Delta, it deleted out of the actual answers inside of Mosaic. So it would not become and give you a hallucination. But again, you go how many copies of an Excel spreadsheet or somewhere? And then how do I know which one is the good one? And it's getting where I have multiple copies of customer support-

>> Literally....

>> mag bulletin and it says, "Hey, this one had the wrong answers in it" from

>> So there's opportunity to create a mechanism. And remember the hypervisor helped create the hyperscalers. If it wasn't for a hypervisor, it probably wouldn't have existed.

>> Well, it enabled the cloud, made the cloud happen.

>> Made the cloud happen, again, a new thing that popped out of the web services. So what will pop out of the data world?

>> So 20 years later,-

>> Hopefully, some wonderful solutions for humanity.

>> 20 years later, hypervisor is no longer the coolest word, but some other word, data straddle, data whatever layer will be needed, something to simplify things, to make things uniform. I think that that's absolutely needed.

>> We're going to take it back to theCUBE research team and we're going to bang out a data visor.

>> Data visor.

>> We're going to workshop that folks. We're going to... Yeah.

>> Mega hyper data visor. A mega hyper data-

>> I'm not saying that live on television, so you're going to have to figure that one out. Something I thought was cool, just shifting... Well, actually, I want to say one thing, Rob, that you brought up. You brought up the Excel spreadsheets. And I thought it was really interesting, when we were at AWS Financial Services Symposium last week, Malcolm deMayo of Nvidia was mentioning that the partnership with them in Deloitte popped up because they read a Wall Street Journal article where the quote was Excel is standing in the way, Excel is the thing standing in the way of enterprise AI transformation. And it's like it's that simple, right? Excel is where everyone's kept their data. It's the worst program. Nobody wants to use it. Sorry, spreadsheet lovers. I'm dyslexic. It's literally just a pile of numbers to me. And so it's really nice to see this whole movement towards data. I feel like the data nerds are getting their moment right now. It's such a hot time for data, to your point, Rob. And something I thought was really remarkable that we don't have this happen a lot on theCUBE, at least not that I've noticed, was we had two guests on theCUBE, we had Elliot and Andrew. Both of their companies are invested in by Databricks. Now this is Databricks who isn't public to your point earlier, so it's a very interesting dynamic there where we're seeing some really deliberate investments in the ecosystem and talking about putting your money where your community is quite literally and optimizing for solutions in that way. If you're not going to buy it, help them build it. And the partnerships there, you could just see the glowing love they have for Databricks, not just because they're sitting on the show talking to us about it, but because they're actually a true strategic partner, which I thought was honestly refreshing to see. I also thought it was great.

>> It's a fine line. It's very hard to walk on that line, partner versus a builder myself, or buy a company. It's actually pretty difficult.

>> Right.

>> Databricks as a company, they need to learn. And by the way, when you say that Excel being sort of the in way or whatnot, I would say the other way around, that Excel is actually the motherland of all the innovations. Because think about every single SaaS company, literally a SaaS company, every single SaaS company is a feature of Excel, right? Salesforce, Workday, ServiceNow, they all come from Excel.

>> Right, right, right, right.

>> So you extract the one functionality out you, it's $100 billion dollars SaaS company. So I love Excel. You can build a so many SaaS companies off of that.

>> You can do my Excel at any time.

>> Well, one of the companies that I built out was based on me using macros in Excel and getting sick of doing that and needing a more advanced way of doing it.

>> Every company is like that.

>> So Databricks is just Excel on steroids is what we were trying to say.

>> Well, you have bad software

>> Plus AI. Plus AI.

>> Plus AI. Yeah, plus AI.

>> Yeah, plus... Geez. Yeah, quite literally. One thing I have to point out a little bit on the disappointing side for me, this show, we do a lot of work with women in data science. I know you've been a supporter. Their goals are to see 30% of women represented in data science. We didn't have a single female guest. The only female on the stage this week was me.

>> I tried. I tried.

>> That's not good enough.

>> We almost had one,-

>> We can do better....

>> but she left early. Okay.

>> That's excuses. There's-

>> Savannah did a great job

>> I agree. I agree.

>> Thank you, and I just want-

>> I won't help you

>> It matters. Please don't interrupt me right now. What I'm trying to say is for the women watching, there are women in data science. This room has been full of really awesome, powerful women. We saw women in the keynote. I talked to a lot of really awesome women at Booth. There are a lot of powerful women, so don't watch this and just think it's a bunch of bros and Brooks Brothers blue jackets. It's a lot more than that. And I know that we can tell those stories. We can get those customer stories here on this stage, and I think it will be awesome. So that's my... What do I want to say? This time next year, I want to say that we had 50 50, or at least 30% women on this stage telling the most important companies in our industry. And on that note, I'm actually going to just say thank you because this has been a long segment, and I don't know that we can really pivot from that.

>> Thank you.

>> You're welcome, John.

>> I think it's great.

>> It's been great hosting with you guys.

>> I appreciate having here.

>> Howie, Rob, great work. Come on on the end angles.

>> Yes, thank you.

>> Thank you.

>> Great job.

>> Yes, really, really wonderful.

>> Great job for the team. Well done guys.

>> Working with all of you. I was just going to say that this wonderful group of well hatted hotties over there just crushing it. As always, we cherish you and you making us look good for the guests and for my own vanity when I rewatch every single segment because that's what I do. I also want to give a shout out to all of you who are watching. It's Pride month. Happy pride. We're proud allies up here. I've got my rainbows in the ears, friends. And without you tuning in, we don't get to sit here and learn from the smartest people on earth. We're at Databricks Data and AI Summit signing off for the last time here in San Francisco, California. My name's Savannah Peterson. You're watching theCUBE, the leading source for enterprise tech news.

CUBE Collective

search

>> Good afternoon CUBE community, and welcome back to Databricks Data and AI Summit. We're here in San Francisco. My name's Savannah Peterson for theCUBE. Very excited to be closing out our two days of Power Pact coverage with a fantastic foursome, Rob, Howie, and John, thank you so much for hanging out.

>> Love it.

>> I feel like the bell of the nerd ball right now. It's a wonderful feeling.

>> Don't sell yourself short on that. You're in the nerd fam. There's no doubt.

>> Thank you, Rob. It's nice to have you on the desk. I've missed you this week.

>> Well, I've been busy sequestered over in the analyst wing.

>> Have you quarantined over there with all the-

>> Not quarantined, sequestered. There's no quarantine going on.

>> So I'm going to open with you, actually, because I'm curious. We've had two totally different experiences. How's it been for you over there in the analyst land?

>> I think it's been really interesting seeing them deep dive into where they are today, where they're going in the future, some of which is under NDA and I can't get into. But I think, again, the success that they're having within the accounts to bring those who've started with them on the data science side over to the Databricks SQL side and expand inside those accounts has been really fascinating. And the growth rate that they have is just unbelievable.

>> 60% growth rates. That's at a 2.4 billion ARR, that's amazing.

>> That's super impressive, Howie.

>> Not too many companies can claim that.

>> Yeah, no wonder they are claiming it. I'd be shouting that from the rooftops because that's where I was. What an exciting time to be them. My goodness. Howie, this is our first time getting to sit on the desk together at the show and in particular. Nice to see you.

>> Great to be here.

>> Yeah. What have you learned? What's been some of your key takeaways?

>> Well, I just wandered around the floor a little bit. Data is the gold mine, as Jensen Huang said, right? The problem with data is it's gold, but it's pretty tough to mine it.

>> Right.

>> Right? It's always easy to have a PowerPoint to say, "Hey, here's all the reports," but when people really do, it's always a lot of work, governance, security, silos, and then who owns what, data lineage, compliance, dedupe, all sorts of problems. So I see so many companies solving so many problems, so it's a renaissance.

>> It's a renaissance. It's great. It's definitely a really unique time. John, we've had the pleasure of interviewing a lot of great guests on the show. Who was your favorite? Not that we have to pick a favorite child, but-

>> Well they all brought a little bit of different game to the table. Joel from Databricks actually gave the Databricks perspective. We had some customers. I really like Excel data because they're doing the data pipelines. I'm fascinated with the data pipelines. As how he pointed out, data's messy, but you start to see kind of in the neural network kind of vein, not directly, but data pipelines will become very important because separating compute from data will allow the ability to support diversity of workloads and various engines, whether it's compute engines, whatnot. So what that means is a lot of different things that was once diverse and messy can now be connected, and that's going to enable programmatic cool things. So I think that's a trend that I love, and that came out in that interview. But Databricks brought game this thing. They brought uniform general availability on the Delta Lake.

>> They brought game, baby.

>> Unity Catalog, open source, Mosaic AI adding more tools, Lakeflow on the pipelines, and then serverless taking away all the infrastructure so there's no knobs and clusters to manage it. So they're starting to kind of, I want to say grow up a little bit and get out of the command line-

>> There's a maturity though....

>> command line data developer nerd to okay, scale, simplicity, value at the business level. So I think they're going right down the right path, so there was a great set of interviews.

>> Last year, they bought a Mosaic ML. This year, they bought a Tabula, so they seem to be doing well on the M&A side. They're leveraging that-

>> That's a good point, Howie....

>> tool pretty well, even though they're a private company. Usually it's the public company companies doing that, but company.

>> With the valuation of over 45 billion on paper, they got a lot of, Rob, lot of powder to work with, so to speak, in terms of equity.

>> That's one way to put it.

>> I think Snowflake and Databricks are going to be competing. If Databricks makes product advancements, Snowflake goes on the M&A front. They make an M&A move over to Databricks, Snowflake kind of response. So we're going to see that back and forth action.

>> At the end of two weeks of data weeks, two weeks of it, you start to look at how these two are really going head to head, and I don't want to say tit for tat with the announcements, but when you start to look at, hey... There was a direct shot across the bow when he said-

>> For sure.

>> When Matei goes up there and pushes and makes the repo public for Unity right there and then on stage versus in 90 days for Polaris, that's a direct... That was

>> That was a direct frontal-

>> Right....

>> humiliation point.

>> And it was a move for the community. I think it was a real moment of uniting people, pun intended, but I thought that was a special moment. And I think anytime-

>> Oh, it was a nice shot that Snowflake, right through the arm, bullet through the arm, hey, here you go.

>> Polaris.

>> 90 days, how about now?

>> This is how we feel about 90 days.

>> 90 days.

>> We feel it should be instant.

>> I think it's more than that. Both of them, they are open... They wanted to have an open community. Then the question is, if it is a developer, where do I go? Right? Maybe there's another open community that's on top of both of them, right? So I actually wrote up on the LinkedIn yesterday. There are companies, open source project that they are more vendor neutral. Potentially, possibly, they have more advantage.

>> Yeah. And I think, again, when we talk to them again over in our little nerd fest in their sequestered area, part of it is how do they go after verticals they haven't necessarily been strong in? And I think they've done good at, for instance, at the credit union and FinTech level, but how did they get into the hardcore banking aspect of things. Where Snowflake has historically been really, really strong, I think they're bringing a lot of features to bear to be a more complete solution, and I think that is definitely a way that they've seen... And again, Ali talked about it. "Hey, I heard from a hundred CIOs, we need Lakeflow, and I need to be able to bring my data from SFDC and from Salesforce into here to join that up with this other to make my intelligence and my AI internally so I can answer these queries." That, to me, is, again, looking at what other personas and what are other customers and TAM, and I hates the word TAM, but going after which TAM do I go and pull in?

>> Well, no, they have a big TAM because they got data engineering and BI and SQLs for writers, right? So that's key. Now, the question is that, Rob, like Snowflake, they do a really easy way to get into the platform. They got a compute engine built in. It's easy to develop and to deploy apps, data apps on Snowflake. Now, the question is that Databricks has got much more robust flexibility, but the ease of use might be more like an iPhone. Okay, it's easy. It's all right there. Databricks might have a better architectural solution and more horsepower, but will they move down? But I think they coexist. I think all the analysts that we talked to, including yourself, mentioned that the collision of Snowflake and Databricks is much more of a market optics. The customers are voting with their data. If they stay on Snowflake, they're going to stay in Snowflake because it's easier and the computes there. If Databricks does its job, it can get Federation to suck the data out of Snowflake, change the economics, then that might be interest. So the question is, what's the switching cost between Snowflake and Databricks, and vice versa?

>> That's actually a very interesting point. Speaking of switching costs, we always care about switching costs. I do think the next few years, there will be some layers of technology to lower down or actually allow you to switch easier, right? Because a lot of times, we have been talking about data gravity for the last decade, right? But believe it or not, because of the AI, because of the Jensen Huang's GPU costing so much, actually I believe that there is a compute gravity going on right here. So before, it's like a compute moving to the data. I wouldn't be surprised, in the next five or 10 years, we want data moving to the compute because compute... Electricity is a scarce resource. The coating is the scarce resource. You want data to move there. Now, it's not going to be wholesale move, right? It's about can you do that incrementally? Can you virtualize the data layer? I do see a number of smaller startups, startup selling just message. I'm actually quite excited. That's why I see that there will be a lot of innovation over there.

>> In fact, we had a discussion about that over in the other place about Delta sharing. And when you bring in stuff into Unity Catalog and you're able to share without moving the data around, bringing the data to the AI in a location that has more power, lower cooling costs and things like more sustainability even from that perspective.

>> Good that we're speaking the same language, because in the last decade, there is only about data gravity. I do think the compute gravity is going to be more a thing, to your point.

>> I think you're right, Howie.

>> To your point, it's more than just the data. There's a lot of the data lineage, the optimization. So I'm quite looking forward to the new things.

>> And I think the lineage thing becomes really important because when you get hallucinations or you get issues within your data, and especially when you go to these compound systems or complex systems, whatever you want to call them, where you have multiple LLMs involved and other AI, traditional AI understanding where things went wrong, becomes really, really difficult. We actually had a session with Harlan, the CTO on Mosaic ML and really talked about the agent framework and how you break down in that. And he said, "Hey, we kind of went backwards and forgot about all of this circle of really iterating through, because we just said, oh, throw it all at one big model. And it's not going to be about that, and we kind of all got caught up in that. And now, we are doing things in more of a way where it's, hey, I have to understand all of these pieces, how they flow. And really, it comes down to where did that data come from, and how do I replace it?"

>> I'm a huge fan of agents. I think agents are going to be really important.

>> You've been talking about it for a while.

>> And to your point, if it takes too many steps to do it, in every single inflection point wave, that's been important. All the winners were simplify, make it easy, intuitive to understand, and reduce the steps it takes to do something, every single.

>> Simplicity, uniform layer.

>> Every single time. So your point there, it doesn't get done, I can't build my app faster than before.

>> And it's all about meeting people where they are. One of the things that was the theme of the conversation all week was not just appealing to the lead data scientists within an organization or a specific dev, but actually making it so that everyone can be a data scientist to a degree if they have access to the AI and to their data.

>> The key thing with the generative AI is the AI is being democratized, right? AI is being democratized, but the data access is not, right? So I think that's why so many vendors are innovating how to... I came from virtualization backgrounds. I have to use a virtualization, right?

>> Hypervisor.

>> Really virtualize the data layer in terms of the data access, the catalog. So I'm not convinced that-

>> We call it a hypervisor, a data hypervisor.

>> Yes. There's a company called Data Straddle, right? That's sort of another layer.

>> Joking aside, what you're getting at is that this harmonization we talked about on our rope day one yesterday with George Gilbert, you get the plumbing right, and then the data harmonizes through interactions. I think we're going to start to see something new we've never seen before, once people get their lakes set up, once the plumbing is in place, pipelines have AI to it. So I just think that we're in this grind mode right now. Things start to get-

>> It is gritty.

>> You got to wire this together, Iceberg with this, get the formats right. But once that's under the covers with a hard and top, then it has to get better. I just don't see... I like Mosaic. I think that's got a great action, but it's still in the... You got to grind it out, knock it down. So whoever nails the simplicity on the agents will take a lot, will take a lot.

>> Think of the province statement I mentioned, right? If you want the compute to follow data, there are so many problems you need to solve, so there will be some grinding period.

>> Oh yeah. Well, and it sounds like everyone we talked to, a lot of the guests we talked to are seeing a lot of different customers across verticals, and they spoke about that exactly. We're in that gritty building phase. When I ask everyone my standard savvy question, what do you hope to be able to say when we're here this time next year that you can't say today, it was all, "We're at scale with our customers doing X." It was less sandbox, more true implementation and rolling it out.

>> Normally, we get different answers. Normally, it's like, "Hey, I want my life to be easier.

>> They're all the same.

>> Now it's here, it's all about the features, product.

>> Yeah. It was so product heavy in the answers, I thought it was kind of wild.

>> This reminds me of AWS in the early days because you have builders, you have developers here, not corporate. I think this is not a sales event in the sense of-

>> No....

>> it's heavy swag, which is an indicator of developers.

>> I was just going to say that it's open source, it's definitely dev heavy. But I think exactly to your point, it's less salesy, more community.

>> I think in general, the Databricks ecosystem has more developers, and then the Snowflake ecosystem has more enterprise people.

>> Here's my take on Databricks right now, and this is... I'm going to watch this very closely. I'm not down on this, but I'm pointing it out. Databricks has been such a proponent of open source. Everything about Databricks is open source, but they've been very one-sided in their open source. They've been open source heavy on the Databricks side as the committers driving the project. There's nothing wrong with that. But to get true open source, you have to show a lot more non Databricks interactive. Otherwise, it's just a databricks product in the open.

>> What I would say is though, the one comment that I, and I said this earlier today in one of our wraps, that 66% of the contributions to Spark are now coming from non Databricks companies

>> Spark.

>> But Spark is not the core part of the Databricks product at this point.

>> I understand. That's a good point. That's a good point. It's a good point. It's an evolution for all of these different projects. I think-

>> It's a good signal that it could go the other way. My point is that Databricks has to be careful, and they have to emphasize and invest in this, just like they're investing in $2 billion in Tabular. Open source works only if it goes direction of 65%. Otherwise, it's just a Databricks company in the open. So I think that'll be the true test. And if that infiltrates into open source in a positive way, I think the data engineering will have a very Kubernetes like effect where that stuff happens fast.

>> And the one to watch is Unity Catalog because that really is the glue at that data layer.

>> Unity Catalog could be a data engineer's dream, not so much, just from the government side. Just like that could be just like I would've set the whole thing, and I predict that you're going to see complete redos. If you look at the math of how much it costs to actually go into an enterprise, take the big enterprise, take a pharma company, anyone, say, "I'm going to redo the entire data estate from scratch, reset the whole thing from the ground up," 100 million, maybe less. Rob, do the math.

>> But again,-

>> Don't ask Rob to do math.

>> Yeah, math might be too at the moment.

>> Let me ask OpenAI to do that.

>> Let me tell you. We're chatting.

>> Careful John.

>> It says math now. But I think, again, why I think... I've already had two different organizations say to me, at two different parties this week, "I'm really excited to get my hands on the open source Unity Catalog to go and see what I can actually do with it, and how much of it's actually open source, right? And I think that's the key, is what features that they have on top of Unity actually land in the open source versus what pieces are actually Databricks built on top of Unity?

>> Basically, is that truly, truly open? That's the question.

>> Yes .

>> Well remember-

>> I think the jury is still out.

>> Remember when Ali went up there yesterday on day one, he said basically the three things on the industry was demand for AI, check, we all know that, security and privacy, and the fragmented data estates. And again, that speaks to what we know, and everyone knows data is messy. But also organizationally from the data warehouse, old school data warehouse, it's siloed and it's like-

>> Silo. Silo, silo....

>> you can't... Data's got to be free to fly around to be integrated.

>> It's got to be free.

>> It's got to fly around horizontally scalable and in the application.

>> But that's not the enterprise has been set up in the last several decades.

>> Right. No, not at all. It's the opposite of free. It's... Yeah.

>> The script's flipping. So again, hypervisor, I'm just joking, but that concept of the control plane, interesting. It's got governance.

>> I hardly believe that a unit catalog will be that layer as a starting point of hypervisor. I think people will have some skepticism. They have to prove. There is a lot of work for them to do. So I truly believe that it's not just a collision between the Snowflake and the Databricks. There will be opportunity for third party to provide a more truly open-

>> Well, when you're at VMware, you wrote one of the early lines of code, first code for the network side.

>> Casual.

>> When hypervisor was not even a conversation that was happening, what was the motivation behind that? It was obviously server virtualization, but what was the motivation between the hypervisor? Why did it come about?

>> Well, 20 some years ago, believe it or not, I remember I went to VMware and I saw the stats. I couldn't believe that. Worldwide, the server utilization was below 5% across the board, because Windows... You give a lot of the CPUs, the Windows was not able to drive multiple CPU. Today, it's hard to imagine.

>> threaded application.

>> Here's a server. We're talking about expensive servers, utilization across the board, not just one company, across the board was less than 5%. So that's the motivation of hypervisor initially, to increase, basically to write an operating system right, basically.

>> And I think you have that same thing happening with data. It's like, how much of my data is actually being utilized? I'm just flooding it in.

>> Fingernailing it probably

>> Exactly.

>> In theory, there are the CPUs. But in reality, are you using it at same thing with data? It's the same thing.

>> Exactly. And how much of it's hot data? How much it's useful data. And we were even talking about how-

>> Hot data, there you go.

>> Hot data, and how you could actually, within answers and within Delta, you could actually go in and take something out from Mosaic using the framework, the AI framework. And if you deleted it out, you deleted it out of Delta, it deleted out of the actual answers inside of Mosaic. So it would not become and give you a hallucination. But again, you go how many copies of an Excel spreadsheet or somewhere? And then how do I know which one is the good one? And it's getting where I have multiple copies of customer support-

>> Literally....

>> mag bulletin and it says, "Hey, this one had the wrong answers in it" from

>> So there's opportunity to create a mechanism. And remember the hypervisor helped create the hyperscalers. If it wasn't for a hypervisor, it probably wouldn't have existed.

>> Well, it enabled the cloud, made the cloud happen.

>> Made the cloud happen, again, a new thing that popped out of the web services. So what will pop out of the data world?

>> So 20 years later,-

>> Hopefully, some wonderful solutions for humanity.

>> 20 years later, hypervisor is no longer the coolest word, but some other word, data straddle, data whatever layer will be needed, something to simplify things, to make things uniform. I think that that's absolutely needed.

>> We're going to take it back to theCUBE research team and we're going to bang out a data visor.

>> Data visor.

>> We're going to workshop that folks. We're going to... Yeah.

>> Mega hyper data visor. A mega hyper data-

>> I'm not saying that live on television, so you're going to have to figure that one out. Something I thought was cool, just shifting... Well, actually, I want to say one thing, Rob, that you brought up. You brought up the Excel spreadsheets. And I thought it was really interesting, when we were at AWS Financial Services Symposium last week, Malcolm deMayo of Nvidia was mentioning that the partnership with them in Deloitte popped up because they read a Wall Street Journal article where the quote was Excel is standing in the way, Excel is the thing standing in the way of enterprise AI transformation. And it's like it's that simple, right? Excel is where everyone's kept their data. It's the worst program. Nobody wants to use it. Sorry, spreadsheet lovers. I'm dyslexic. It's literally just a pile of numbers to me. And so it's really nice to see this whole movement towards data. I feel like the data nerds are getting their moment right now. It's such a hot time for data, to your point, Rob. And something I thought was really remarkable that we don't have this happen a lot on theCUBE, at least not that I've noticed, was we had two guests on theCUBE, we had Elliot and Andrew. Both of their companies are invested in by Databricks. Now this is Databricks who isn't public to your point earlier, so it's a very interesting dynamic there where we're seeing some really deliberate investments in the ecosystem and talking about putting your money where your community is quite literally and optimizing for solutions in that way. If you're not going to buy it, help them build it. And the partnerships there, you could just see the glowing love they have for Databricks, not just because they're sitting on the show talking to us about it, but because they're actually a true strategic partner, which I thought was honestly refreshing to see. I also thought it was great.

>> It's a fine line. It's very hard to walk on that line, partner versus a builder myself, or buy a company. It's actually pretty difficult.

>> Right.

>> Databricks as a company, they need to learn. And by the way, when you say that Excel being sort of the in way or whatnot, I would say the other way around, that Excel is actually the motherland of all the innovations. Because think about every single SaaS company, literally a SaaS company, every single SaaS company is a feature of Excel, right? Salesforce, Workday, ServiceNow, they all come from Excel.

>> Right, right, right, right.

>> So you extract the one functionality out you, it's $100 billion dollars SaaS company. So I love Excel. You can build a so many SaaS companies off of that.

>> You can do my Excel at any time.

>> Well, one of the companies that I built out was based on me using macros in Excel and getting sick of doing that and needing a more advanced way of doing it.

>> Every company is like that.

>> So Databricks is just Excel on steroids is what we were trying to say.

>> Well, you have bad software

>> Plus AI. Plus AI.

>> Plus AI. Yeah, plus AI.

>> Yeah, plus... Geez. Yeah, quite literally. One thing I have to point out a little bit on the disappointing side for me, this show, we do a lot of work with women in data science. I know you've been a supporter. Their goals are to see 30% of women represented in data science. We didn't have a single female guest. The only female on the stage this week was me.

>> I tried. I tried.

>> That's not good enough.

>> We almost had one,-

>> We can do better....

>> but she left early. Okay.

>> That's excuses. There's-

>> Savannah did a great job

>> I agree. I agree.

>> Thank you, and I just want-

>> I won't help you

>> It matters. Please don't interrupt me right now. What I'm trying to say is for the women watching, there are women in data science. This room has been full of really awesome, powerful women. We saw women in the keynote. I talked to a lot of really awesome women at Booth. There are a lot of powerful women, so don't watch this and just think it's a bunch of bros and Brooks Brothers blue jackets. It's a lot more than that. And I know that we can tell those stories. We can get those customer stories here on this stage, and I think it will be awesome. So that's my... What do I want to say? This time next year, I want to say that we had 50 50, or at least 30% women on this stage telling the most important companies in our industry. And on that note, I'm actually going to just say thank you because this has been a long segment, and I don't know that we can really pivot from that.

>> Thank you.

>> You're welcome, John.

>> I think it's great.

>> It's been great hosting with you guys.

>> I appreciate having here.

>> Howie, Rob, great work. Come on on the end angles.

>> Yes, thank you.

>> Thank you.

>> Great job.

>> Yes, really, really wonderful.

>> Great job for the team. Well done guys.

>> Working with all of you. I was just going to say that this wonderful group of well hatted hotties over there just crushing it. As always, we cherish you and you making us look good for the guests and for my own vanity when I rewatch every single segment because that's what I do. I also want to give a shout out to all of you who are watching. It's Pride month. Happy pride. We're proud allies up here. I've got my rainbows in the ears, friends. And without you tuning in, we don't get to sit here and learn from the smartest people on earth. We're at Databricks Data and AI Summit signing off for the last time here in San Francisco, California. My name's Savannah Peterson. You're watching theCUBE, the leading source for enterprise tech news.