Databricks Data + AI Summit 2024 | Raj Bains | Prophecy.io

theCUBE.net

Clips
News
More from Databricks Data + AI Summit 2024

Raj Bains

CEO

Prophecy.io

Data transformation gets a boost with Prophecy’s AI copilot for Databricks’ users

Data transformation is entering a new era with the advent of generative AI, significantly altering how users access and manipulate data for enhanced productivity. Prophecy Inc.’s latest innovation, the industry’s first copilot for Databricks, exemplifies this trend by streamlining the preparation of raw data for analytics and AI applications.Prophecy CEO, Raj Bains, talks with theCUBE about data transformation.“What’s happening with data transformation is people are like, ‘I’ve got to do more, I was doing this table stuff, I was doing reporting, I was doing dashboards and now I’ve got to do generative AI,’” said Raj Bains (pictured), founder and chief executive officer of Prophecy

play_circle_outline Interview with Raj about their product, Prophecy

play_circle_outline The Transformative Power: A Comprehensive Overview of the Data Transformation Ecosystem and the Essential Role of Productivity

play_circle_outline Exploring the Shift from Legacy Silos to Open Collaboration in Data Transformation for Survival

play_circle_outline Unlocking the Power: Exploring Speed, Automation, Scalability, and Agility in Data Transformation for Success

play_circle_outline Predictions on the future of the market and the growth of data platforms like Databricks and Prophecy

Info

Raj Bains | Prophecy.io

Raj Bains

CEO Prophecy.io

search

>> Good afternoon data nerds, and welcome back to Databricks' Data and AI Summit here in our fabulous hometown of San Francisco. My name's Savannah Peterson, here joined by my co-host and co-analyst John Furrier. John, we look like the beautiful blue sky outside today.

>> theCUBE has got two days live coverage. It's blue skies, got the blue jacket. Looking good.

>> I know.

>> It's all good here.

>> More sunshine as always. Really excited though to invite our next guest who's also bringing the dance moves with me up here. Raj, welcome to the show. How are you doing today?

>> Thank you so much. I'm doing great, having a great time at the conference. Love it.

>> What's your first impression walking around on the floor?

>> There's lots going on, lots of energy. Energy is moving all to AI and that's a big change. It's data, but it's data and AI, so it's like that's changing stuff and people are excited again. The data conference has got a little slower and now they're picking right back up. Exciting stuff. AI is changing stuff, it's shaking the market.

>> You guys got an interesting product. You've got customers that are big customers doing a lot of data. Explain what your company does. I want to get into a little bit, because you got unique use case there. Take a minute to explain what you guys do.

>> Sure. Prophecy is a data transformation copilot. We are all about making data users productive with the data platforms they already have. If you use Databricks, you are going to be coding. Then you have data analysts who are like, "I don't write Python, I don't write SQL." It's all about saying, okay, they are focused on providing maximum functionality, everything that you might need, everything under the sun. We are like, "But what about the users and the people?" GitHub Copilot made programmers so much more productive. What about the data people building data pipelines, they're like, "Where's my Copilot? Who's going to take care of me?" We're like, "Here's your Copilot." We make them productive. They can do their job in half the time."

>> When did Copilot come out? When you guys pushed that out, how long has that been out?

>> It's been out, so actually our whole product now is Copilot. We earlier had a low code visual drag and drop interface to build data pipelines, which was helping you write code for Databricks. Now what we did is generative AI came and attached to it. Now generative AI and visual pipelines, everything's helping you write code. But the AI features have come just now, last few months.

>> You guys were well-prepared for that movement. You moved quickly.

>> We got lucky. Just the right product at just the right place-

>> I don't believe in luck, you make your own luck.

>> You have to work hard, but luck shows up.

>> But you've also been preparing for this moment for a little while. You mentioned that you founded the prophecy in 2017. You bootstrapped for those first two years until 2019. Obviously, really good timing. That is a part of startup success. Well, business success, doesn't matter if you're a startup. Data transformation, the ecosystem right now is really having a moment, which is probably why that grin on your face is so good. What are you seeing? What is the data transformation ecosystem like right now?

>> Data transformation ecosystem is, most people have all their operational data, they use a Fivetran or something. You get that data into the lake house and if you seen the press release this morning, you don't need anything but Databricks now. Once you get there, that's where the real work starts. This is your data factory, this is where you're going to combine data. Let's say I'm doing some credit card stuff. I got to get my customers and their spend and my rewards and get them all together, join them, combine them, clean them, make a report, make the business dashboards. Also, put it all together for AI. All of that is like more and more is being asked of the data teams. What's happening with data transformation is people are like, "I've got to do more. I was doing this table stuff, I was doing reporting, I was doing dashboards and now I got to do generative AI. We are pushed beyond our limit. How do I do more?" Everybody's like, "Okay, I want productivity." Productivity is name of the game in data transformation and that's why I have the smile on my face.

>> Well, and across the organization though, one of the things that stands out to me is, if you're looking at upping the data literacy across the organization so that everybody knows what good data looks like, can manage good data and then use all these tools.

>> Definitely. Earlier it used to be that the line of business people were either out of the picture. Sometimes the business people who are data analysts who are experts in the data or data scientists would send requirements to the data platform team and say, "Oh, go code this for me." Then they code it up. They came back and it's like, "Oh, that's not exactly what I wanted. Can you change this?" Going months and months. Or Alteryx came in and got some users who are like, "Hey, here's your own system, just doesn't scale. Runs on your desktop. Windows desktop." That's an era. These guys have not been all productive. Now we are looking at it and saying, oh, you got this really good platform, like the Databricks or the data platforms and it's like I want to enable all the users. The data engineers? Yes. Data analysts? Yes. Data scientists? You can't move fast with data if people are depending on other people and using Word docs. Everybody who wants data has got to have access to it. You want to transform it for your business use case today, you have got to be able to do it. You're not waiting three months for data platform team. Those days are disappearing.

>> And so you're providing the tooling layer for that product?

>> Yes, we are providing the tooling layer that makes data engineers also and data analysts and line of business users, all of them productive with data. Like a data analyst using visual drag and drop and some English, generative AI, just say, "Build me a pipeline that does this," and we build it for them. And that runs on Databricks native code at scale, massive like a top data engineer. And now generative AI puts that in the hands of the data analysts. They are at parity with the data engineer.

>> So, you like what he was saying on stage there about, "Hey, just plug into our lake and then just manage the data. May the best engine win."

>> Oh yes, made the best engine win. But you see, for data processing, may the best engine win, but it's like what about the data user? All the engines are competing, adding more functionality. You had ingest, you have transform, you have quality, you have observability, you have generative AI and it's like, okay, who's got to understand all of this and get the job done?

>> Yeah, who owns that?

>> Who thinks about the poor user? As Databricks is anchored on and focused on data functionality, we are anchored and focused on the data user. We are saying that's the second anchor in the data market. That's the most important thing for the organization.

>> Okay, so I want to ask you, as gen AI comes in into focus, there's the old way and new ways. You're starting to see the new ways. You guys had the low-code, no-code now Copilot. Yes. What's the old way that's going to be disrupted by the new innovative solutions that are coming out now? Because as this new future emerges, it's a disruptive enabler, it's going to disrupt the other stuff. What will be disrupted? What do you see in the old legacy companies and how they deal with data transformation?

>> There are legacy companies who used to have a vertical stack where they did everything. Informatica would say, "Here is our product. You do visual drag and drop and build pipelines. And by the way, we'll store them in our proprietary format and then you're going to get you locked in. Then you have to run it on Informatica." The customer moves to the cloud and say, "Informatica doesn't run at scale. Databricks does. I want to run it on Databricks. I want to run it on Snowflake." It's like, so you can't lock people in. In the old way, you had the stack built by, okay, okay, engineer. Informatica, Data Stage, . These people, these are dinosaurs, these are dead. People are moving to the cloud and they're like, "We want to leave all of them behind." Those companies revenues going to dwindle like anything. And Alteryx, God, yes. They have a product for line of business, doesn't scale Windows desktop. Let's say you have a new use case. You have a new line of business. What are you going to buy? You move to the cloud. Generative AI, you're going to buy a desktop app? It's like no. If people are coming to us and say, "I want that Alteryx experience natively on Databricks." We're like, "Here's your tooling, you don't need. And we'll also import them for you, by the way." All those companies are dead, they kind of know it, but the revenue is going to dwindled. All of that is going away.

>> And if there's no unification, they become just a legacy silo that's just sitting around until its end of life.

>> That's right.

>> Yeah, that's legacy silo you got. Yeah.

>> I mean, well that's just it. I feel like there's this transformation wave that's happening here and I love your hot takes rise. We're going to have to bring you back on as an analyst and just laying down the sound bites right now. But I like it. What I think you're hitting on, and I'm curious to confirm this with you, we're really at a juncture where you're either on this fast train like we're talking about and willing to open up to be open source like we were talking about all day. Or at least be open to great collaboration and tool sharing or you're out.

>> Or you're out. This locking in silo business is done, but also as people are... We got some music in the background, I'll continue.

>> That's okay.

>> Oh ,yeah, we can always pause for the beat on theCUBE.

>> All right. But like I'm saying, as a startup, we are getting generative AI. Generative AI is building half your pipeline. Are you going to go back to that old way and build it all yourself, locked in a silo? You're not going to do that. Basically, the Copilot has got to work with the ecosystem. You have the Databricks ecosystem, it's going to work with that, plug into everything, make you more productive. Generative AI doing half the work. Now all the old tools, are they based on generative AI? No. They'll put a sticker somewhere saying, "Oh, we have this generative AI. Informatica will put clear." Does it do anything? Doesn't do shit. Those people don't have the best generative AI engineers.

>> Nope.

>> Nope. That's not your future. You betting your future. If I'm a customer buying, I'm betting my future. I'm not betting my future on one of those guys to have the best generative AI talent to build all these pipelines. Uh-uh.

>> What are some of the best use cases that you've seen for best practices? As people start to rethink about horizontally scalability, agile, more speed, more automation, agents are coming. What's that best practice for getting ready for it and actually implementing it. Besides throwing away the old legacy, but if I'm going to reset set the table, what does it look like?

>> I'll give an example. We had one large customer, one of the largest pharma customers with a name that has the name twice. That customer, they went to Gartner, they talked about it. They had data engineers and business data users. Business data users were not enabled sending them Word documents and saying, "Build that for me." They put the Copilot straight in the data user's hands, they're building their pipelines themselves. All the dependency, the back and forth with data platform team is gone. They're productive. They're getting data 10x faster that their throughput is 10x more. They're saying, "We are creating 10x the tables, 10x the assets and reports." Then the same team's productivity has shot through the roof, and some other companies like the largest health insurance company, Fortune 5, what they have is they moved 21,000 jobs from Ab Initio to Prophecy on Databricks. That's a massive, massive footprint.

>> That probably made you feel good.

>> That made us feel good. Hundreds of data engineers using us. They are coming in and they have a data platform team that sets the standards. They write the code, you can do that in Prophecy and say, "Here is the standards." And then they push it out to all the business users who are like, "Okay, we are going to do the day-to-day work." Fast, fast, fast data platform team setting the platforms, both of them getting productive. Of course, with generative AI, that's the future.

>> You see companies essentially do a full reset. Ali went up on stage today hinting that they had to redesign everything around serverless and they had a big technical debate. This is kind of like a do-over for data. You got to reset everything. That's what you're basically saying, you get the data engineering team in-

>> Yeah. It's a do-over. Yeah, completely. The data engineering team gives you the platform, sets the standards. The line of business team builds all the data pipelines themselves. I mean, they can build a few ones, central ones, but most of the day-to-day pipelines go to all the business users who understand their data, who understand what the need from that data. You're not playing telephone with these people. I think that's the future and the copilots are going to make everybody a solid, solid data engineer. I mean, this is not going to be like, oh, you have these few people in the back room who can do that. That world is going away fast.

>> It's going to up-level everybody.

>> Up-level everybody.

>> It's going to use the AI and the data literacy that we've all been hoping for, but across the organization, it's upskilling. It's when everyone's worried about jobs, we're actually just all going to get smarter together, which I think is awesome. I want to bring up a question. So, it's clear you have your finger on the pulse because all of your customers or a large group of them are in the Fortune 50. You're working with the biggest and the best and the brightest right now. You mentioned healthcare as an earlier example. I know we were talking about tech and you're working with some of the biggest tech companies in the world too. Is the behavior or the trends, are they similar across vertical? Are you seeing different things in their adoption of copilots? What's going on?

>> Yeah, actually the tech companies, initially we thought the banks, the pharma companies, the healthcare companies would buy us, but now the tech companies are using us. And what's happening is they have their own clouds and their own clouds have competing products. But actually some hyperscalers are using us and are our largest customer, like millions of dollars. And those customers are like, "I want to get productive." Think of a search team. Think of those people who are in core engineering, they've started to adopt us, but when they get us they hit the ground running so much faster. We had a customer come in and grow 5x within a year. In a bank, it would have taken us a year to get everything together. Some banks move faster. One of our investors in the last round was JP Morgan. There's bank who moves faster on technology, but then there is other people who are just taking months and months to get set up.

>> Your customers are big, big established whales like the big banks right now, because they have the big data states that are fragmented and need to be rebuilt from the ground up.

>> Yes, they are bringing all multiple legacy products all into us on top of the data platform. They'll be like, "Oh, get rid of the Ab Initio, get rid of the Data State. They had some products in different business units, they acquired some more companies. They have these 10 silos on-prem. They're like, "Let's all move it into one data lake." One single place, make everything standardized and they don't want to get locked in. That's very, very important to them. Us just helping them write open code, they love it.

>> Who do you think are the five hottest companies that everyone's going to be using in the next year as we see this separation?

>> You mean five other-

>> Like tools? I mean, obviously they're going to be using you.

>> Sure, sure.

>> Obviously they're going to be using Databricks.

>> No, I think it's actually going -

>> But who's going to survive, I think of the legacy systems?

>> Legacy system, who's going to survive? I think Informatica would be okay because they cost okay-

>> They have governance.

>> They have a governance, and they don't have the highest... When we go to Fortune 50 and really big use cases, they're not there. They're in slightly smaller use cases, not very expensive. They're like, okay, this kind of works, so that they're going to be okay. The other vendors, Alteryx is not surviving this, Ab Initio, not surviving this, Data Stage, not surviving this. And then modern data stack, not surviving this. Basically what's happening is the data platforms, here is the line. Under that is data processing. You had the Spark, you had SQL, you had Catalog, Databricks got Arcion, ingestion, they're eating up everything. Snowflake's going to look at that and say, "I'm going to eat up everything." If you are a point solution, modern data stack in the processing layer, you're dead. Sell to one of these, or you're dying.

>> .

>> The thing is once, if you get one data quality company in and Databricks let's say gets one and they integrate it, let's say-

>> Tabular just got bought for big money. You're seeing Lake Flow, we were talking about the TechCrunch article that talks about how they're not relying on some of the ecosystems.

>> Yeah, I just read the article today and Databricks is like, "We don't want the Fivetrans of the world, the DBTs, all these things." If you look at it, what does Fivetran do? It's just moving data from point to point. You know what, who's good at data processing? Databricks is, they're going to eat it up. So will Snowflake. What is DBT? It's like I help you write code. You know what Databricks is good at? Helping you write code. In that layer, you're going to die. We looked at it and said, "Uh, uh, uh, we are going up to the user, visual tools, productivity, UI, UX, the kind of stuff data platform people don't do. That's the other thing that's going to survive." And some catalogs might, like the Atlan and those guys, some of them will because they're so focused on the business user. Data platform, just the big ones on top, a couple of tools and it's consolidating.

>> What should Fivetran do if they see that that's going to be not an opportunity for them? Do they go away? They pivot? Because they have a good product?

>> No. Fivetran has built a product, I think the basic thing is they have to continue to be mostly mid-market. What they have is they have a bunch of connectors. When you have a connector, your top connector is going to make this much money. The next one, this much, this much. After five to 10 connectors, it's the long tail. They don't make that much money. What the data platforms are going to do is, let's pick the first 10 money makers, let's add that into our platform, give it for cheaper built into the platform. You don't need Fivetran for that. Now, they're left with a long tail. And that's the thing. The is on-premises, ETL companies were single consolidated products. Somehow the investors, A16z in this case and others, decided that point solutions in modern data stacks are the way to go. And the market is saying no. They were single platforms on-prem and they'll continue to be single platforms in the cloud. There is not going to be 50 different tools. In Informatica, you guys know that you should do everything. It's like data ingestion, data transformation, governance. They have the whole thing. So, we are going back to that world.

>> If the big platforms are going to fill the white space with their product, is there-

>> For data processing, yeah.

>> For data processing, is there any other white spaces where the ecosystem could thrive? Because Ali's on stage saying, "Ecosystem's super important to us."

>> Ecosystem, you actually have to add real value. It's like you can't tag on a small thing and say that's a company, I'll attach to it. Same is happening with OpenAI. You can't add a thin wrapper on it. OpenAI comes up with new stuff and they eat it up, so you have to add real, real differentiated value on top. Maybe you do something with generative AI that they're not going to build you. Maybe you build tools. It's got to be really differentiated. Look at MosaicML. Was it a small add-on? No, they had a serious product. You have to build serious deep technology. You don't have build these point solutions. Those are not going to fly.

>> Remember during the internet days, we used to say, "That's a feature, not a company."

>> .

>> That's kind of what you're talking about.

>> product's not a company. Yeah, it's something -

>> You got to have a differentiation that's going to be scalable and durable.

>> No. If you look at data quality, right? Informatica had data quality. Data Stage had data quality. What I'm saying is, all of these were features. We knew they were features, right? It's not like it's news to anybody. We move to the cloud and everybody said, "No, that's a company." I mean-

>> Okay, so Ali validate our research that said, "Okay, separate compute from the data, good things will happen."

>> Yes.

>> What do you see happening now that that's happening? Databricks is going to have their little area. There's going to be ecosystem area to play. What's going to happen, this new data platform, not the modern data platform, this next level, separate computer, what does it look like?

>> Yeah, I feel like the data platforms continue to get bigger and fewer. The cloud vendors cannot buy companies. Lina Khan, entire Bay Area is going to go vote Democratic. The Democrats are going to put Lina Khan. The Lina Khan is going to say, "You know what? You can't sell companies to Microsoft. You can't sell them to Amazon. You can't sell them to these big giants." Who else is going to buy the companies? Well, they're all available -

>> Maybe they'll vote Democrat and vote Lina Khan out so that we could have some liquidity.

>> It's very important for Silicon Valley to have liquidity exist. All these billion dollar valuation companies, who's going to buy them if there's no Microsoft, no Google, right? Basically, but till they can't acquire, who's going to acquire? Snowflake's going to acquire. They've acquired 6, 7, 8 companies last quarter. I think seven companies last eight quarters. So has Databricks. These are the people who keep getting fatter and fatter, wider and wider.

>> The rich get richer.

>> I don't know what's politically correct to say.

>> Just say the rich get richer.

>> More robust in their offering.

>> They're prospering.

>> The rich get richer, and we have to find new places perhaps with generative AI, you really have to have a paradigm shifting in company. You have to have an Uber, something like that, that never existed and has a deep technical mode. That's the way to survive these small little point features in-

>> I think Uber is a great example. I think they represent to me, they built from scratch with their bare hands, so to speak, a modern data platform that's more AI. They do a lot of things.

>> They do a lot of things.

>> They do a lot of things that's hard to do that enterprises need to do quickly. Do you see yourself kind of in that role where you're going to come in and help them get there?

>> Yes. I think the enterprise data stack, Databricks, Prophecy, this is all about getting that technology with much lesser cost, not with armies of engineers that the big tech companies had. Without that, getting this into the hands of enterprise and enterprises can catch up pretty quickly.

>> Raj, you're looking at analysts on our team now. We love you on theCUBE.

>> Literally. I know.

>> Let's get a plugin for your company. Tell us about Prophecy, what you guys are looking to do, why people buy you. Put a plug in for the company.

>> Sure. Prophecy is focused on making data users productive. We talked about that. You're a large enterprise, you want to move to the cloud, you want to get very productive. You want to spend half the time doing the old data stuff, free up the other half-time for doing generative AI, you get Prophecy. Prophecy, you come in, you get your business users productive, put data in every user's hand, get it ready for analytics, for reporting, for BI, for generative AI, ready, much, much quicker. That's what companies, it's very simple. And actually somebody said, I don't know who, must be one of the founding fathers. Good arguments are simple. We have a good argument. It's simple. You have users, we want to make them productive on the platform they already have. That's the argument. It's very simple, but that's what good value add is.

>> That's good value proposition.

>> Well, I'm sold. That sounds absolutely fantastic. All right, last question for you, Raj. Because when we have you back on the show many times now that we've seen how your brain works, I'm curious what you hope to be able to say on the show next time. We'll say the next Databricks Summit, so give you a year looking out that you can't yet say today.

>> In terms of the market or Prophecy. Prophecy?

>> Both.

>> Whatever you want.

>> Whatever I want.

>> At this point, tell us anything.

>> Do both, do both.

>> The weather, I mean, the state of healthcare.

>> Do both.

>> We'll take your opinion on anything.

>> Market and Prophecy. What's going to change? What's going to be different?

>> I think what I wish happens by next year is that everybody comes in convinced that, "Hey, I buy a platform, I buy a copilot and I'm good to go." That becomes the default stack, right? I buy a Databricks, I buy a Prophecy. That becomes that. And then the other thing I'm looking for is I want to see use cases of people who come in and say, "I use generative AI. My Copilot actually worked. These are the 15 users who got productive and they're doing 5x more." It's about taking it from this promise stage to delivering real value on the ground. That's what I think what's important. We've talked so much about AI, we've talked so much about the promise of AI. Now we need value delivered in the next year, and I think we're going to do it.

>> I love it. Let's deliver that value. My goodness, John, this has been quite a dialogue. I feel a lot of hot takes coming out of this. Raj, thank you so much for taking the time.

>> Thank you. It was such a pleasure talking to you.

>> Thank you, thank you.

>> And thank all of you for tuning in and grooving with us here on the Databricks Data and AI Summit, a show floor where it is absolutely a pumping. My name is Savannah Peterson. You're watching theCUBE here in San Francisco, the leading source for enterprise tech news.

Raj Bains | Prophecy.io

search

>> Good afternoon data nerds, and welcome back to Databricks' Data and AI Summit here in our fabulous hometown of San Francisco. My name's Savannah Peterson, here joined by my co-host and co-analyst John Furrier. John, we look like the beautiful blue sky outside today.

>> theCUBE has got two days live coverage. It's blue skies, got the blue jacket. Looking good.

>> I know.

>> It's all good here.

>> More sunshine as always. Really excited though to invite our next guest who's also bringing the dance moves with me up here. Raj, welcome to the show. How are you doing today?

>> Thank you so much. I'm doing great, having a great time at the conference. Love it.

>> What's your first impression walking around on the floor?

>> There's lots going on, lots of energy. Energy is moving all to AI and that's a big change. It's data, but it's data and AI, so it's like that's changing stuff and people are excited again. The data conference has got a little slower and now they're picking right back up. Exciting stuff. AI is changing stuff, it's shaking the market.

>> You guys got an interesting product. You've got customers that are big customers doing a lot of data. Explain what your company does. I want to get into a little bit, because you got unique use case there. Take a minute to explain what you guys do.

>> Sure. Prophecy is a data transformation copilot. We are all about making data users productive with the data platforms they already have. If you use Databricks, you are going to be coding. Then you have data analysts who are like, "I don't write Python, I don't write SQL." It's all about saying, okay, they are focused on providing maximum functionality, everything that you might need, everything under the sun. We are like, "But what about the users and the people?" GitHub Copilot made programmers so much more productive. What about the data people building data pipelines, they're like, "Where's my Copilot? Who's going to take care of me?" We're like, "Here's your Copilot." We make them productive. They can do their job in half the time."

>> When did Copilot come out? When you guys pushed that out, how long has that been out?

>> It's been out, so actually our whole product now is Copilot. We earlier had a low code visual drag and drop interface to build data pipelines, which was helping you write code for Databricks. Now what we did is generative AI came and attached to it. Now generative AI and visual pipelines, everything's helping you write code. But the AI features have come just now, last few months.

>> You guys were well-prepared for that movement. You moved quickly.

>> We got lucky. Just the right product at just the right place-

>> I don't believe in luck, you make your own luck.

>> You have to work hard, but luck shows up.

>> But you've also been preparing for this moment for a little while. You mentioned that you founded the prophecy in 2017. You bootstrapped for those first two years until 2019. Obviously, really good timing. That is a part of startup success. Well, business success, doesn't matter if you're a startup. Data transformation, the ecosystem right now is really having a moment, which is probably why that grin on your face is so good. What are you seeing? What is the data transformation ecosystem like right now?

>> Data transformation ecosystem is, most people have all their operational data, they use a Fivetran or something. You get that data into the lake house and if you seen the press release this morning, you don't need anything but Databricks now. Once you get there, that's where the real work starts. This is your data factory, this is where you're going to combine data. Let's say I'm doing some credit card stuff. I got to get my customers and their spend and my rewards and get them all together, join them, combine them, clean them, make a report, make the business dashboards. Also, put it all together for AI. All of that is like more and more is being asked of the data teams. What's happening with data transformation is people are like, "I've got to do more. I was doing this table stuff, I was doing reporting, I was doing dashboards and now I got to do generative AI. We are pushed beyond our limit. How do I do more?" Everybody's like, "Okay, I want productivity." Productivity is name of the game in data transformation and that's why I have the smile on my face.

>> Well, and across the organization though, one of the things that stands out to me is, if you're looking at upping the data literacy across the organization so that everybody knows what good data looks like, can manage good data and then use all these tools.

>> Definitely. Earlier it used to be that the line of business people were either out of the picture. Sometimes the business people who are data analysts who are experts in the data or data scientists would send requirements to the data platform team and say, "Oh, go code this for me." Then they code it up. They came back and it's like, "Oh, that's not exactly what I wanted. Can you change this?" Going months and months. Or Alteryx came in and got some users who are like, "Hey, here's your own system, just doesn't scale. Runs on your desktop. Windows desktop." That's an era. These guys have not been all productive. Now we are looking at it and saying, oh, you got this really good platform, like the Databricks or the data platforms and it's like I want to enable all the users. The data engineers? Yes. Data analysts? Yes. Data scientists? You can't move fast with data if people are depending on other people and using Word docs. Everybody who wants data has got to have access to it. You want to transform it for your business use case today, you have got to be able to do it. You're not waiting three months for data platform team. Those days are disappearing.

>> And so you're providing the tooling layer for that product?

>> Yes, we are providing the tooling layer that makes data engineers also and data analysts and line of business users, all of them productive with data. Like a data analyst using visual drag and drop and some English, generative AI, just say, "Build me a pipeline that does this," and we build it for them. And that runs on Databricks native code at scale, massive like a top data engineer. And now generative AI puts that in the hands of the data analysts. They are at parity with the data engineer.

>> So, you like what he was saying on stage there about, "Hey, just plug into our lake and then just manage the data. May the best engine win."

>> Oh yes, made the best engine win. But you see, for data processing, may the best engine win, but it's like what about the data user? All the engines are competing, adding more functionality. You had ingest, you have transform, you have quality, you have observability, you have generative AI and it's like, okay, who's got to understand all of this and get the job done?

>> Yeah, who owns that?

>> Who thinks about the poor user? As Databricks is anchored on and focused on data functionality, we are anchored and focused on the data user. We are saying that's the second anchor in the data market. That's the most important thing for the organization.

>> Okay, so I want to ask you, as gen AI comes in into focus, there's the old way and new ways. You're starting to see the new ways. You guys had the low-code, no-code now Copilot. Yes. What's the old way that's going to be disrupted by the new innovative solutions that are coming out now? Because as this new future emerges, it's a disruptive enabler, it's going to disrupt the other stuff. What will be disrupted? What do you see in the old legacy companies and how they deal with data transformation?

>> There are legacy companies who used to have a vertical stack where they did everything. Informatica would say, "Here is our product. You do visual drag and drop and build pipelines. And by the way, we'll store them in our proprietary format and then you're going to get you locked in. Then you have to run it on Informatica." The customer moves to the cloud and say, "Informatica doesn't run at scale. Databricks does. I want to run it on Databricks. I want to run it on Snowflake." It's like, so you can't lock people in. In the old way, you had the stack built by, okay, okay, engineer. Informatica, Data Stage, . These people, these are dinosaurs, these are dead. People are moving to the cloud and they're like, "We want to leave all of them behind." Those companies revenues going to dwindle like anything. And Alteryx, God, yes. They have a product for line of business, doesn't scale Windows desktop. Let's say you have a new use case. You have a new line of business. What are you going to buy? You move to the cloud. Generative AI, you're going to buy a desktop app? It's like no. If people are coming to us and say, "I want that Alteryx experience natively on Databricks." We're like, "Here's your tooling, you don't need. And we'll also import them for you, by the way." All those companies are dead, they kind of know it, but the revenue is going to dwindled. All of that is going away.

>> And if there's no unification, they become just a legacy silo that's just sitting around until its end of life.

>> That's right.

>> Yeah, that's legacy silo you got. Yeah.

>> I mean, well that's just it. I feel like there's this transformation wave that's happening here and I love your hot takes rise. We're going to have to bring you back on as an analyst and just laying down the sound bites right now. But I like it. What I think you're hitting on, and I'm curious to confirm this with you, we're really at a juncture where you're either on this fast train like we're talking about and willing to open up to be open source like we were talking about all day. Or at least be open to great collaboration and tool sharing or you're out.

>> Or you're out. This locking in silo business is done, but also as people are... We got some music in the background, I'll continue.

>> That's okay.

>> Oh ,yeah, we can always pause for the beat on theCUBE.

>> All right. But like I'm saying, as a startup, we are getting generative AI. Generative AI is building half your pipeline. Are you going to go back to that old way and build it all yourself, locked in a silo? You're not going to do that. Basically, the Copilot has got to work with the ecosystem. You have the Databricks ecosystem, it's going to work with that, plug into everything, make you more productive. Generative AI doing half the work. Now all the old tools, are they based on generative AI? No. They'll put a sticker somewhere saying, "Oh, we have this generative AI. Informatica will put clear." Does it do anything? Doesn't do shit. Those people don't have the best generative AI engineers.

>> Nope.

>> Nope. That's not your future. You betting your future. If I'm a customer buying, I'm betting my future. I'm not betting my future on one of those guys to have the best generative AI talent to build all these pipelines. Uh-uh.

>> What are some of the best use cases that you've seen for best practices? As people start to rethink about horizontally scalability, agile, more speed, more automation, agents are coming. What's that best practice for getting ready for it and actually implementing it. Besides throwing away the old legacy, but if I'm going to reset set the table, what does it look like?

>> I'll give an example. We had one large customer, one of the largest pharma customers with a name that has the name twice. That customer, they went to Gartner, they talked about it. They had data engineers and business data users. Business data users were not enabled sending them Word documents and saying, "Build that for me." They put the Copilot straight in the data user's hands, they're building their pipelines themselves. All the dependency, the back and forth with data platform team is gone. They're productive. They're getting data 10x faster that their throughput is 10x more. They're saying, "We are creating 10x the tables, 10x the assets and reports." Then the same team's productivity has shot through the roof, and some other companies like the largest health insurance company, Fortune 5, what they have is they moved 21,000 jobs from Ab Initio to Prophecy on Databricks. That's a massive, massive footprint.

>> That probably made you feel good.

>> That made us feel good. Hundreds of data engineers using us. They are coming in and they have a data platform team that sets the standards. They write the code, you can do that in Prophecy and say, "Here is the standards." And then they push it out to all the business users who are like, "Okay, we are going to do the day-to-day work." Fast, fast, fast data platform team setting the platforms, both of them getting productive. Of course, with generative AI, that's the future.

>> You see companies essentially do a full reset. Ali went up on stage today hinting that they had to redesign everything around serverless and they had a big technical debate. This is kind of like a do-over for data. You got to reset everything. That's what you're basically saying, you get the data engineering team in-

>> Yeah. It's a do-over. Yeah, completely. The data engineering team gives you the platform, sets the standards. The line of business team builds all the data pipelines themselves. I mean, they can build a few ones, central ones, but most of the day-to-day pipelines go to all the business users who understand their data, who understand what the need from that data. You're not playing telephone with these people. I think that's the future and the copilots are going to make everybody a solid, solid data engineer. I mean, this is not going to be like, oh, you have these few people in the back room who can do that. That world is going away fast.

>> It's going to up-level everybody.

>> Up-level everybody.

>> It's going to use the AI and the data literacy that we've all been hoping for, but across the organization, it's upskilling. It's when everyone's worried about jobs, we're actually just all going to get smarter together, which I think is awesome. I want to bring up a question. So, it's clear you have your finger on the pulse because all of your customers or a large group of them are in the Fortune 50. You're working with the biggest and the best and the brightest right now. You mentioned healthcare as an earlier example. I know we were talking about tech and you're working with some of the biggest tech companies in the world too. Is the behavior or the trends, are they similar across vertical? Are you seeing different things in their adoption of copilots? What's going on?

>> Yeah, actually the tech companies, initially we thought the banks, the pharma companies, the healthcare companies would buy us, but now the tech companies are using us. And what's happening is they have their own clouds and their own clouds have competing products. But actually some hyperscalers are using us and are our largest customer, like millions of dollars. And those customers are like, "I want to get productive." Think of a search team. Think of those people who are in core engineering, they've started to adopt us, but when they get us they hit the ground running so much faster. We had a customer come in and grow 5x within a year. In a bank, it would have taken us a year to get everything together. Some banks move faster. One of our investors in the last round was JP Morgan. There's bank who moves faster on technology, but then there is other people who are just taking months and months to get set up.

>> Your customers are big, big established whales like the big banks right now, because they have the big data states that are fragmented and need to be rebuilt from the ground up.

>> Yes, they are bringing all multiple legacy products all into us on top of the data platform. They'll be like, "Oh, get rid of the Ab Initio, get rid of the Data State. They had some products in different business units, they acquired some more companies. They have these 10 silos on-prem. They're like, "Let's all move it into one data lake." One single place, make everything standardized and they don't want to get locked in. That's very, very important to them. Us just helping them write open code, they love it.

>> Who do you think are the five hottest companies that everyone's going to be using in the next year as we see this separation?

>> You mean five other-

>> Like tools? I mean, obviously they're going to be using you.

>> Sure, sure.

>> Obviously they're going to be using Databricks.

>> No, I think it's actually going -

>> But who's going to survive, I think of the legacy systems?

>> Legacy system, who's going to survive? I think Informatica would be okay because they cost okay-

>> They have governance.

>> They have a governance, and they don't have the highest... When we go to Fortune 50 and really big use cases, they're not there. They're in slightly smaller use cases, not very expensive. They're like, okay, this kind of works, so that they're going to be okay. The other vendors, Alteryx is not surviving this, Ab Initio, not surviving this, Data Stage, not surviving this. And then modern data stack, not surviving this. Basically what's happening is the data platforms, here is the line. Under that is data processing. You had the Spark, you had SQL, you had Catalog, Databricks got Arcion, ingestion, they're eating up everything. Snowflake's going to look at that and say, "I'm going to eat up everything." If you are a point solution, modern data stack in the processing layer, you're dead. Sell to one of these, or you're dying.

>> .

>> The thing is once, if you get one data quality company in and Databricks let's say gets one and they integrate it, let's say-

>> Tabular just got bought for big money. You're seeing Lake Flow, we were talking about the TechCrunch article that talks about how they're not relying on some of the ecosystems.

>> Yeah, I just read the article today and Databricks is like, "We don't want the Fivetrans of the world, the DBTs, all these things." If you look at it, what does Fivetran do? It's just moving data from point to point. You know what, who's good at data processing? Databricks is, they're going to eat it up. So will Snowflake. What is DBT? It's like I help you write code. You know what Databricks is good at? Helping you write code. In that layer, you're going to die. We looked at it and said, "Uh, uh, uh, we are going up to the user, visual tools, productivity, UI, UX, the kind of stuff data platform people don't do. That's the other thing that's going to survive." And some catalogs might, like the Atlan and those guys, some of them will because they're so focused on the business user. Data platform, just the big ones on top, a couple of tools and it's consolidating.

>> What should Fivetran do if they see that that's going to be not an opportunity for them? Do they go away? They pivot? Because they have a good product?

>> No. Fivetran has built a product, I think the basic thing is they have to continue to be mostly mid-market. What they have is they have a bunch of connectors. When you have a connector, your top connector is going to make this much money. The next one, this much, this much. After five to 10 connectors, it's the long tail. They don't make that much money. What the data platforms are going to do is, let's pick the first 10 money makers, let's add that into our platform, give it for cheaper built into the platform. You don't need Fivetran for that. Now, they're left with a long tail. And that's the thing. The is on-premises, ETL companies were single consolidated products. Somehow the investors, A16z in this case and others, decided that point solutions in modern data stacks are the way to go. And the market is saying no. They were single platforms on-prem and they'll continue to be single platforms in the cloud. There is not going to be 50 different tools. In Informatica, you guys know that you should do everything. It's like data ingestion, data transformation, governance. They have the whole thing. So, we are going back to that world.

>> If the big platforms are going to fill the white space with their product, is there-

>> For data processing, yeah.

>> For data processing, is there any other white spaces where the ecosystem could thrive? Because Ali's on stage saying, "Ecosystem's super important to us."

>> Ecosystem, you actually have to add real value. It's like you can't tag on a small thing and say that's a company, I'll attach to it. Same is happening with OpenAI. You can't add a thin wrapper on it. OpenAI comes up with new stuff and they eat it up, so you have to add real, real differentiated value on top. Maybe you do something with generative AI that they're not going to build you. Maybe you build tools. It's got to be really differentiated. Look at MosaicML. Was it a small add-on? No, they had a serious product. You have to build serious deep technology. You don't have build these point solutions. Those are not going to fly.

>> Remember during the internet days, we used to say, "That's a feature, not a company."

>> .

>> That's kind of what you're talking about.

>> product's not a company. Yeah, it's something -

>> You got to have a differentiation that's going to be scalable and durable.

>> No. If you look at data quality, right? Informatica had data quality. Data Stage had data quality. What I'm saying is, all of these were features. We knew they were features, right? It's not like it's news to anybody. We move to the cloud and everybody said, "No, that's a company." I mean-

>> Okay, so Ali validate our research that said, "Okay, separate compute from the data, good things will happen."

>> Yes.

>> What do you see happening now that that's happening? Databricks is going to have their little area. There's going to be ecosystem area to play. What's going to happen, this new data platform, not the modern data platform, this next level, separate computer, what does it look like?

>> Yeah, I feel like the data platforms continue to get bigger and fewer. The cloud vendors cannot buy companies. Lina Khan, entire Bay Area is going to go vote Democratic. The Democrats are going to put Lina Khan. The Lina Khan is going to say, "You know what? You can't sell companies to Microsoft. You can't sell them to Amazon. You can't sell them to these big giants." Who else is going to buy the companies? Well, they're all available -

>> Maybe they'll vote Democrat and vote Lina Khan out so that we could have some liquidity.

>> It's very important for Silicon Valley to have liquidity exist. All these billion dollar valuation companies, who's going to buy them if there's no Microsoft, no Google, right? Basically, but till they can't acquire, who's going to acquire? Snowflake's going to acquire. They've acquired 6, 7, 8 companies last quarter. I think seven companies last eight quarters. So has Databricks. These are the people who keep getting fatter and fatter, wider and wider.

>> The rich get richer.

>> I don't know what's politically correct to say.

>> Just say the rich get richer.

>> More robust in their offering.

>> They're prospering.

>> The rich get richer, and we have to find new places perhaps with generative AI, you really have to have a paradigm shifting in company. You have to have an Uber, something like that, that never existed and has a deep technical mode. That's the way to survive these small little point features in-

>> I think Uber is a great example. I think they represent to me, they built from scratch with their bare hands, so to speak, a modern data platform that's more AI. They do a lot of things.

>> They do a lot of things.

>> They do a lot of things that's hard to do that enterprises need to do quickly. Do you see yourself kind of in that role where you're going to come in and help them get there?

>> Yes. I think the enterprise data stack, Databricks, Prophecy, this is all about getting that technology with much lesser cost, not with armies of engineers that the big tech companies had. Without that, getting this into the hands of enterprise and enterprises can catch up pretty quickly.

>> Raj, you're looking at analysts on our team now. We love you on theCUBE.

>> Literally. I know.

>> Let's get a plugin for your company. Tell us about Prophecy, what you guys are looking to do, why people buy you. Put a plug in for the company.

>> Sure. Prophecy is focused on making data users productive. We talked about that. You're a large enterprise, you want to move to the cloud, you want to get very productive. You want to spend half the time doing the old data stuff, free up the other half-time for doing generative AI, you get Prophecy. Prophecy, you come in, you get your business users productive, put data in every user's hand, get it ready for analytics, for reporting, for BI, for generative AI, ready, much, much quicker. That's what companies, it's very simple. And actually somebody said, I don't know who, must be one of the founding fathers. Good arguments are simple. We have a good argument. It's simple. You have users, we want to make them productive on the platform they already have. That's the argument. It's very simple, but that's what good value add is.

>> That's good value proposition.

>> Well, I'm sold. That sounds absolutely fantastic. All right, last question for you, Raj. Because when we have you back on the show many times now that we've seen how your brain works, I'm curious what you hope to be able to say on the show next time. We'll say the next Databricks Summit, so give you a year looking out that you can't yet say today.

>> In terms of the market or Prophecy. Prophecy?

>> Both.

>> Whatever you want.

>> Whatever I want.

>> At this point, tell us anything.

>> Do both, do both.

>> The weather, I mean, the state of healthcare.

>> Do both.

>> We'll take your opinion on anything.

>> Market and Prophecy. What's going to change? What's going to be different?

>> I think what I wish happens by next year is that everybody comes in convinced that, "Hey, I buy a platform, I buy a copilot and I'm good to go." That becomes the default stack, right? I buy a Databricks, I buy a Prophecy. That becomes that. And then the other thing I'm looking for is I want to see use cases of people who come in and say, "I use generative AI. My Copilot actually worked. These are the 15 users who got productive and they're doing 5x more." It's about taking it from this promise stage to delivering real value on the ground. That's what I think what's important. We've talked so much about AI, we've talked so much about the promise of AI. Now we need value delivered in the next year, and I think we're going to do it.

>> I love it. Let's deliver that value. My goodness, John, this has been quite a dialogue. I feel a lot of hot takes coming out of this. Raj, thank you so much for taking the time.

>> Thank you. It was such a pleasure talking to you.

>> Thank you, thank you.

>> And thank all of you for tuning in and grooving with us here on the Databricks Data and AI Summit, a show floor where it is absolutely a pumping. My name is Savannah Peterson. You're watching theCUBE here in San Francisco, the leading source for enterprise tech news.