Data + AI Summit 2025 | Craig Wiley, Databricks

Clips
News
More from Data + AI Summit 2025

Craig Wiley

Sr. Director of Product, Mosaic AI

Databricks

Inside Databricks’ AI revolution: 10 key takeaways from the Data + AI Summit

Artificial intelligence is becoming more intuitive, interactive and embedded, not just in systems, but in the way companies build, think and create. From live coding and autonomous agents to unified governance and automation, Databricks Inc. is moving AI closer to the application layer and aligning it with developer intent.TheCUBE Research’s John Furrier and George Gilbert break down the keynote analysis during Databricks’ Data + AI Summit.“You sort of tell the AI engineer, the software developer, an agent, what you want to do,” said George Gilbert, principal analyst for theCUBE Research. “In the past, all it did was generate code

play_circle_outline Innovative customer collaboration and ideation drive the development of new AI capabilities.

play_circle_outline The role of research teams in enhancing product performance at Databricks highlighted.

play_circle_outline The future of SaaS may evolve into more agent-based functionalities according to industry leaders.

Info
Transcript

Craig Wiley, Databricks

Craig Wiley

Sr. Director of Product, Mosaic AI Databricks

Craig Wylie, senior director of product, Mosaic AI, at Databricks Inc., joins theCUBE’s John Furrier at the Databricks Data + AI Summit 2025. The conversation highlights Mosaic AI’s latest research breakthroughs and how Databricks is advancing observability and precision in AI systems.

Wylie shares how Mosaic’s work on large language model judges is shaping the development of Agent Brick, with a focus on optimizing model accuracy for high-stakes use cases. He outlines the importance of network management, observability and performance in generative A... Read more

explore Keep Exploring

What are the steps involved in identifying and collaborating with customers to explore innovative possibilities? add

What contributions has Jonathan Frankle and his team made to the development of certain systems? add

What are the best approaches for companies to set up their small language models, and should they distill from larger models or create their own using proprietary data? add

bolt Powered by CUBE AI

Craig Wiley, Databricks

search

>> Welcome back, everyone, to theCUBE's live coverage here in San Francisco for Databricks' Data+AI Summit. Always a must attend event because there's always great product announcements, platform evolution, but also open source action and, of course, innovation around AI's here. Craig Wiley's here, Senior Director of Product at Mosaic AI, a part of Databricks, a big part of the apps and a big part of the Agent Brick is in his hands. He runs all the products. He's got the keys to the Kingdom. Of course, he's a CUBE alumni, been a 00:00:36] big part of our Supercloud 6 event, which is similar to the Above the Clouds paper that I think they copied from us over there at Berkeley. I'm only kidding. But a similar concept. Welcome back.

Craig Wiley

>> Thank you much.

>> I love the Supercloud, Above the Clouds, because we were riffing on that earlier around how the original paper, the original DNA of Databricks looks a lot like what we're seeing here today. If you think about the origination story of how we got here, zigs and zag, but at the end of the day, the North Star and the first principles, open source, make things simpler, Patrick and I were just talking earlier, he's the co-founder, that was the knowledge. You're now trying to put it together and when Mosaic came in, they brought some mojo in which you see rendering itself. But first, summarize for the audience, the part of the products that came from Mosaic and the parts that came from Databricks. How did those pieces come together for the big announcements that were announced today, the apps, and then now, obviously, the Agent Brick, which everybody loves.

Craig Wiley

>> Yeah. The research team at Mosaic has been absolutely central to building the custom judges and the high quality state-of-the-art LLM judges that are really needed to drive the accuracy to a level where companies can deploy these systems in higher and higher risk situations, if you will. And so that really, I think, has been the tailwind that's been able to drive a lot of the innovation we've seen over the past year.

>> And talk about the judging thing, because that's a term that talks about some of the human piece of the feedback. Explain that concept for the folks watching. I think this is a really key part.

Craig Wiley

>> Yeah, fundamentally, if we can identify a series of judges that can, not really evaluate whether or not your response is a good response for the prompt, but instead evaluate each step of the agent or of the chain and look for technical failures around why this thing might not be working the way it is. And then, ultimately, if we can train a judge to act as that human judgment would for this specific domain problem, then we can really start to optimize these things for high performance.

>> This is where I judge, management, litigate, understand. This is understanding the agent. This is where network management, observability, a lot of these theories come in. What is the science behind these products? Can you share how it's all working? What's the key science and mechanisms that keep track of all this?

Craig Wiley

>> Yeah. The real challenge of this is, when I find a problem with my agent and I say, "Hey, I no longer want you to do the following," or, "I want you to do more of X," how do we not have that just jammed in the prompt? But how do we optimize that across the vector store that might be used, the prompt that might be used, other tools or models that might be used? How do we ensure that, that feedback really hits all pieces of the agent? And I think that's a lot of what the Mosaic team has brought and that ability is really the differentiator in driving these holistic improvements to the agent.

>> When I see all the stuff going on in the past year, a couple of notable points jump out at me. The rise of MCP, that was a nice surprise. The DeepSeek happened, which is, you see some algorithms. You see what you guys have done with the brick, the Agent Brick, which is you're seeing this automation and simplification of agents, the unification with Unity Catalog Iceberg. Okay, this all sounds like an operating system to me. This is a word that's been kicked around on the keynote and on the hallway. Certainly, we covered it in our session. I love operating systems because they operate stuff, but there's a science and there's computer science behind this. If you had to describe the Gen AI operating system because generative is runtime, that's a concept.

Craig Wiley

>> That's right.

>> What does this mean? Can you share? It's an opaque question, but I'm trying to put my finger on the pulse because if this gets done right at the data layer, the infrastructure's getting faster. Okay, that's enablement. But you got distributed computing paradigm. We'll call it multi-cloud or whatever, it Supercloud, doesn't matter. It's just you're running over stuff. We've got network compute and database, the four pillars.

Craig Wiley

>> I think when we start looking at this compared to an operating system, really where it comes into is, do we have the foundational capabilities that would exist in an operating system to make it so that, if I hit print, it just goes and takes this document from one application, moves it through a number of firmwares and what have you, and then prints it? And that's an extraordinarily complex activity despite the fact that it's a button for me. And so I think you're right in your analogy, given that the goal is really, hey, how do we duplicate that by building out those primitives such that we can make it so that you hit optimize or learn my agent and we can just get that up and running and going?

>> So again, I love that print example because we preview. You're just tooling around. There's user experience around that. What has to happen to get essentially an operating system of data? Because I've been arguing for years on theCUBE that there's going to be a Google SRE-like role for data, because it's large-scale. Data engineering would be data coding. This is now happening. A lot of the startups that the venture team and the startup program over there are going to be doing at Databricks are the one founder. The three founders in the enterprise, some junior coder or junior IT person could change the makeup of JP Morgan Chase.

Craig Wiley

>> It is amazing.

>> You don't know where it's going to come from.

Craig Wiley

>> The leverage that this technology is able to drive is really, I think, unlike anything we've seen before. And as we get from a world of I built an agent to, I built an agent that's supervising a series of agents to, I built an agent that's spawning a series of agents, I think that's really the path we all know we're on. And it's really, again, just ensuring that those fundamental building blocks of governance, of evaluation are there so that we can make sure that as we get there, we're building real quality software, not just toys that I can use with my buddies.

>> So I have to ask you about your role, because being product management, agents will also do some PM work.

Craig Wiley

>> Indeed.

>> I'd like to hear your thoughts on that in the next question. But as a product priority, you have customer facing inbound telemetry, you're talking to customers, huge customer examples. We saw them on stage. But you also got to look internally to engineering, work with the team, and balance out items on the roadmap. Always a struggle. How has your priorities changed this year? What's your focus? And what do you optimize for?

Craig Wiley

>> Yeah. I think it seems like ever since the GPT moment hit that every six months or so, the hot new capability or what have you is brand new and we hadn't seen it before. And, certainly, agents came around maybe last summer or so and became really what folks were talking about as the next step here. And I think our goal from the beginning has been, hey, how do we help the large enterprises get these things operating at a level of quality? That means they can actually deploy them in serious situations, not just in, hey, let's search our collective documents quickly and easily. How do we actually get them to perform in these tasks of financial or reputational risk?

>> Engineering is involved too. You're talking about data engineering. You're talking about DevOps, DevSecOps. It's a cloud culture. So now you've got the convergence of platform engineering. I'm going to oversimplify this to illustrate my point. I'd love to get your reaction. Platform engineering colliding with the analytics world, decades of work. In the middle is this new cultural wave that Berkeley has spawned and that's the vibe, coding. It's a new generation. It's going to be the intersection of those two worlds. What's your landscape of that? As a product leader, you have to start thinking, well, I have people that know dashboards. They handle all the data. It's data science. But then there's also users now.

Craig Wiley

>> And not only are there users, but we find ourselves in a position where the use cases or even the functions that the folks we're talking to are doing are functions that I can look across the table and say, "Hey, we're all in agreement that three years from now this is going to be done completely differently," and they will uncomfortably nod. And we just have to be comfortable with the fact that we're getting to build this exciting, exciting future right now with these tools.

>> Okay, so your customer, let's just say I'm a customer, we think we are a customer, but I run siliconangle.com and I have my media brick video data lake, got a bunch of AI. I tell my team that in two years, siliconangle.com could be a prompt.

Craig Wiley

>> There you go.

>> What blog and what does media look like? But that's an unknown. Radical things create an unknown future.

Craig Wiley

>> Yeah.

>> So as a product leader, that's a hard task.

Craig Wiley

>> It is indeed.

>> How do you enable that to happen? Because it might spin in that direction, and I think it will. People just have to look up their business saying, and Jamie Dimon said, "Lean into it. Don't wait."

Craig Wiley

>> I think that's it. It's finding those customers who also share that belief of lean into it, and then really ideate with them on what the art of the possible is. What do they want to accomplish that seems like sci-fi to them? Bring in the Mosaic research team and see if we can't figure out how to do that. A lot of what you see with Bricks came about because we were trying to tune rag models for folks and what have you, and we realized, hey, we could automate this. And so once you've gained expertise, the ability to then automate it is right now something that is easier than it's ever been.

>> Well, as you know from Amazon, that was a growth strategy of AWS, was first generation identification of opportunity through the problems you were working on.

Craig Wiley

>> Yeah.

>> So what are those problems that you're working on right now with your team? Because, again, you got a lot going on. You got a lot coming at you. What are you dealing with on a daily basis?

Craig Wiley

>> I think on a daily basis, it really comes down to working with our customers to understand what's keeping them from being able to... We hear Satya say, "Hey, this is going to be the end of SaaS," and things like this. What's keeping them from being able to deploy those systems into customer facing situations or deploy those systems into critical workflows? Not just assistive workflows, but really critical workflows.

>> I think Satya's comment was taken out of context. In fact, Dave Vellante did a post on this with George Gilbert around him and Satya and Mark Benioff as Rock 'Em Sock 'Em Robots. But I think what he meant was, and I clarified this with Matt Garman, the CEO of AWS last week was, does SaaS get replaced by agents? Now, I was saying it's going to be 100X bigger TAM opportunity. That's my hyperbole and over the top view, which I think is accurate. But it's interesting. He said it's an evolution, and I agree with him because I think that's what Satya is saying. SaaS turns into agents.

Craig Wiley

>> That's right.

>> Because they have the data and the workflow. So it's not like SaaS gets decimated. Software as a service or service as software, it's just all an evolution.

Craig Wiley

>> Agreed.

>> Do you agree?

Craig Wiley

>> I think the one caveat to that I would say is that the barriers to development are going way down right now. And so what used to be a six-month lift to build might now only be a couple of weeks. And in a world where that's true, I see more and more companies maybe wanting to bring some of these functions in-house if they're able to achieve those velocities.

>> Craig, you know Google as well. You worked there. You've worked at some great places. Right now, native AI is really the key. Google recently talked about this being native AI for the . You guys have built a lot of stuff internally with AI in your products to make it better for customers. How important is it to have research involved in the product team? Not like applied R&D. There's an old school R&D where it's like, "Let's solve the climate change or whatever." And then there's applied R&D that's usually product or business driven agenda. Then there's just research that's tied to products. So we're seeing that Deep Mind with Google is helping them out. Is there a deep mind inside Databricks where you're tapping internal research to make the product at an acute level faster or better, smarter?

Craig Wiley

>> Certainly, Jonathan Frankle and the team that came over with Mosaic, and I think we've more than doubled the size of that team since they joined us, they are really at the heart. If you look at some of these systems that are getting built, there's the operational orchestration of the computer science, but then there's also some LLM and agentic magic in these systems, and a lot of that is coming from the expertise those folks have in exercising this technology.

>> If you look at DeepSeek, that innovation was based on a constraint. It wasn't even designed to build a model. They would build them from light sources, a CUDA clone. So they backed into it. This is where you don't know where innovation's going to come from, right? Do you agree?

Craig Wiley

>> That's right. I think the other thing to realize from DeepSeek is just that the ability for us to scale these models through just scaling compute is we're probably at a bit of a ceiling there. So really how you bring the data into these systems and how you tune the systems with the data is really increasingly the way in which you get these state-of-the-art results.

>> All right, so I have to ask you about some of the innovation around models because you've got computer vision. LLMs are great, that's language, but computer vision's hot right now. You got robotics. We're having a big event this month on robotics, three days of coverage. Because AI and robotics are beautiful together because you got open source, you got machine software integration, really been well done there. So we see that being a great wave. But they have computer vision. Computer vision is a model.

Craig Wiley

>> Yeah.

>> You got languages. How do companies get their small language models set up? Do they distill from big? Should they take their data like our data, like we have because we have our own data? Everyone's going to have their own data, but how do they build the models? Should they distill off the large ones or build their own? This was the debate I was having with Jonathan. With what you're working on, what's your advice on how people should think about dealing with their models?

Craig Wiley

>> I really think of the model as only one of a series of tools that the system has in order to do this. And it's, really, you can get a lot from fine-tuning a model or prompting the model particularly effectively, but you can also get a lot from the way in which you're managing your search. And one of the things we're seeing is that while the model quality is critical, it might be just as important as to the context you're bringing to that model. And we all thought, oh, for rag, just go do a hybrid search and you're good. And no. What we're finding is that, hey, I may want to do multiple searches. I may want to expand it and some of these other techniques to really drive the necessary context to get the results you want.

>> And one of the examples on stage was the fact that you can prune it. You can say, "Hey, don't have to do the whole thing, just subset it out and just past five years ago or whatever." Is that a user or is that an agent or both factored into that? So do you envision that being just autonomous or just voice come command, "Hey, only use three years"?

Craig Wiley

>> Yeah, I think it really is going to depend on the use case. So we're going to look to the human to help inform us what the right answer is. Now, certainly, if the human keeps telling us, "Hey, be more careful about dates or what have you," then that LLM judge that we're training should at some point be able to pick that up and do it on the human's behalf.

>> Well, final question for you is one of the things we've really been going in the weeds on and deep coverage on SiliconANGLE theCUBE and CUBE research is the enterprise adoption. Enterprise AI and a lot of their data is not being leveraged. You guys take advantage of that with Unity and the intelligence platform. Startups want to play too. We had Databricks Ventures on. Andrew was on earlier with Steve, who runs the startup program. A lot of the startups don't know where to rally in the stack. Kubernetes has done its job. People are rallying around the de facto orchestration layer, but inside the AI stack, there's not a lot of clarity on, what do I run? Because these large enterprises have a lot of Dell, HPE, NVIDIA, Juniper, Arista. There's a lot of legacy stuff that's already been deployed. So there's a lot of integration. If you had to give advice or direction, what should startups think about rallying around in the AI stack? Obviously, Databricks, you guys want them to be Databricks, but what else? Because they want to sell to them and you're getting the customers.

Craig Wiley

>> I'll tell you, I was meeting with a large bank recently and they said, "Here's our challenge. A huge amount of our data is inside legacy software that doesn't even have API access." And so we're sitting there.

>> Face palm.

Craig Wiley

>> And they're like, "Yeah." They're saying, "How are we going to go do this?" And so we all think of the connectors to, oh, do you have a ServiceNow connector and do you have a Workday connector or a Salesforce?

>> Doesn't everyone have a connector?

Craig Wiley

>> Right. But what is the future for these organizations that have these 20, 30-year-old mainframe systems running where they're dependent on these systems, but they don't even know how to exfiltrate the data at scale? And I think this is a ripe area. These companies are going to have to modernize this. And so figuring out how you can go deep with those organizations to help them modernize it, I think is going to be critical.

>> Craig, great to have you on theCUBE. It was great commentary. Final question again, final final question is, what are you optimizing for? What's your goals now coming out of this event? What's your plans?

Craig Wiley

>> Yeah, I want to get a lot of bricks built. Ultimately, what's the plan? The plan is to help organizations get these systems into critical high ROI use cases. And so we're going to spend every day with customers learning about how we can help them do that.

>> Simplify the onboarding. That's a great first step.

Craig Wiley

>> There you go.

>> All right.

Craig Wiley

>> Absolutely.

>> Thanks for coming on. Appreciate it.

Craig Wiley

>> Thank you.

>> All right. We got the experts in the house here. We've got a mixture of experts on theCUBE today, pun intended. Of course, live covers, more coverage after this short break. Stay with us all day from San Francisco. I'm John Furrier, host of theCUBE. Thanks for watching.