theCUBE + NYSE Wired: AI & Retail Trailblazers | Jonathan Frankle, Databricks

Clips
More from theCUBE + NYSE Wired: AI & Retail Trailblazers

Jonathan Frankle

Chief AI Scientist

Databricks

play_circle_outline AI as a form of computing, driving innovation

play_circle_outline Innovative AI Applications Impress Databricks: Tools for Dedicated Customers

Info
Transcript

Jonathan Frankle, Databricks

Jonathan Frankle

Chief AI Scientist Databricks

TheCUBE is covering Retail Week at NYS on the East Coast, focusing on technology and AI, particularly data and AI. Jonathan Frankle, chief AI scientist at Databricks, emphasizes the importance of customer success and the impact of AI on data utilization. The partnership between data and AI is crucial for success, with customers renewing deals and growing with Databricks to gain deeper insights and analytics. Use cases now go beyond traditional forecasting and analytics, focusing on unstructured data and AI assistance. The goal is to simplify AI and data proce... Read more

explore Keep Exploring

What are some of the things specifically that's driving the customer activity? add

What has been the greatest delight in working with customers? add

bolt Powered by CUBE AI

Jonathan Frankle, Databricks

search

>> Hello, everyone. Welcome back to theCUBE. We are here at the NYS. It's theCUBE's new studios on the East Coast. I'm John Furrier host. Dave Vellante is with me here for three days for Retail Week, also known as NRF. That's the show that talks about all the things going on in retail. Of course now it's become a technology AI show. It's data plus AI, that's the story. I mean, it's had tech before, IoT devices, some analytics, but this year we're seeing the surge of AI discussions and of course we're here breaking it all down for you. Jonathan Frankle is here, chief AI scientist with Databricks. John, thanks for coming in. We were talking before camera, MosaicML. You came over with Naveen. You guys are excited over there. Big lift. Congratulations on the big exit, by the way.

Jonathan Frankle

>> Hey, wait, there will be no exit until Databricks is a public company. We're just continuing the journey.

>> Okay, you're okay. Wait, I won't go there, but I why don't we just jump in? First of all that, that's great news. Huge fans of what you guys have done. Love open source. We're open source content, so our new set here. You're going to be lucky enough to get the bell ring here.

Jonathan Frankle

>> This is not a bad view.

>> So our new set, we're going to build out set next door.

Jonathan Frankle

>> Can I get an apartment here or anything? This is a good view.

>> We are the first podcast media company on the floor, the history of the stock exchange, the rest of these guys out there, cable first, independent operator. So thanks for coming on. Shout out to Ali and the team at Databricks, if you're watching. First of all, congratulations on the overall valuation and momentum. We're seeing numbers and what we're hearing numbers, although no official numbers, just significant revenue growth on Databricks. The data lake house is doing great. The stuff that you guys announced last year at your event is thriving. The open source communities is frothy and they're just hungry to get more horsepower out of the hyperscalers. They want those GPUs. So you have the confluence of this AI infrastructure that's going next level as fast as they can, and the app side, just whether it's no code, code assistance and then creativity driving massive collision of innovation. What is your perspective? Because you've seen the pre and you're in the during, you're going to see the afterglow of this. What is your perspective of market forces relative to AI and all the innovation and opportunities that are going to come out of it?

Jonathan Frankle

>> So I mean, the first thing I'll say is you said a bunch of numbers, but the one thing you didn't mention was our customers. And that's the most important part, keeping them happy. They're renewing deals, they're growing their footprint and Databricks, and that's a sign that we're doing a good job for them. If we do a good job for them, all the other numbers will be good, but it's customer success number one.

>> So customers, good scoreboard, the numbers reflect the customers. Talk about the customers. What are some of the use cases that you're seeing that's driving adoption? Is it cloud? Is it the AI piece? What are some of the things specifically that's driving the customer activity?

Jonathan Frankle

>> So the way I think about it is kind of in some sense, and this answers your previous question, AI is just a different kind of computing. I know people are talking about AI as like automation and agents and all these buzz words. At the end of the day, you go to AWS, you go to Azure or GCP and you get CPUs, and getting CPUs clearly isn't enough. That's why Databricks exists. We built a data platform on top of that. But you now go to OpenAI and Anthropic and AWS and Google and Azure and you get intelligence. And intelligence is again kind of a raw form of computing. And the question is what does your platform look like on top of that intelligence? How does intelligence as a processing unit affect your entire data estate? And from my perspective, it's pretty simple. Intelligence allows you to get more out of your data and it requires your data to do that. There's a reason we at MosaicML wanted to join up with Databricks and vice versa. Databricks has the world's best data platform. We had some awesome AI technology and neither makes sense without the other anymore. I mean, Databricks has been having the Data and AI Summit, that was the name of our event, for years before anybody was really talking about AI because what do you do with data? You understand it, you use it, you leverage it. So we're seeing, there are always the traditional use cases, forecasting, analytics, understanding, business intelligence. But now we're seeing all sorts of newer use cases pop up. Places where AI can help you turn your unstructured data like documents into structured information you can leverage for traditional analytics. Where you can build LLMs, you can answer questions about your data just by talking to Databricks. We have this product called Genie or AIBI where you can just ask a question and we can go and convert that into a SQL query, run a bunch of steps and come back and just give you an answer. And speaking personally as someone who knows a lot about AI and is still learning Spark, that's revolutionary for me. I just want to know how many of my GPUs have been idle lately? Which of my research teams are using the most GPUs? I can just ask Databricks and get the answer. So this is how you get from-

>> Versus the old way would be what?

Jonathan Frankle

>> I'd go and set up a notebook or a dashboard. I'd write a SQL query, I'd write some Spark. I'd use Databricks assistant to help me do that because I've got a lot to learn about how to use Spark and Databricks. And I'd get there and I'd run it a couple times and try it. Now I just ask in natural language. And me as the boring research manager at this point, I haven't written Python in a while. So the idea that I can just ask Databricks and get answers, it's huge for me. It's huge for my team, it's huge for our customers. And this is how you get from a hundred thousand users to a billion users. This is how you get from the CEO needing to go and have a data scientist build them a dashboard, wait a week and come back and then get the answer to the CEO just getting the answer. And that's the value of AI and data in one place.

>> I mean, the speed and the time savings, huge.

Jonathan Frankle

>> Oh, yeah.

>> Domain expertise. I don't need to do Spark anymore, I just get what I want.

Jonathan Frankle

>> Yeah. And if you want to get to that level, there's still a reason to go and do Spark when you really need things to be fast and good. And the same way AI can help you write code, that's not going to mean we don't need software engineers. We need incredible software engineers who can write the best code, but it means it's more accessible, that anyone can harness the power of AI and data for whatever they're trying to do. And by anyone, I mean these days, me, the research manager and not the research scientist. I mean, my mom. All of these people should have access to being able to understand the world through data and that's what we're pushing toward.

>> Great democratization message there. I have to ask you, okay, so if I want that Nirvana scenario at Databricks, what do I have to do? Do I have to make my data certainly available, available in a certain way? If say I'm a customer, I have all these data silos, does it mean throw into a data lake? Take me through the progression of the criteria, requirements I have to do to set this up because some people say, "I don't want to centralize but I want to have intelligence," that would imply centralization. So take me through the data layer requirements.

Jonathan Frankle

>> Yeah, and I'll give you, this is a place where I'll tell you I'm much more ignorant because I come from the AI side of the house, less the data side of the house. But even speaking of someone who had to do this transformation recently. We arrived from MosaicML, we had trillions of tokens of data for all of the LLMs we were building. We had a bespoke solution. We were running on our own instances. We were like, it was a mess. We got to Databricks, they gave us one solutions architect and a week later we were up and running. We were in Databricks. All of our data was there and it was just really about moving it over, getting into the Delta format so that you can get the best performance out of Databricks. But even beyond that, Databricks has connectors to a lot of other services where you might have your data today. You can start taking advantage of our services before you ever move an ounce of data to Databricks, your life will get better as you migrate to Databricks.

>> They cleaned up your mess, basically.

Jonathan Frankle

>> Oh, yeah. Well, they taught us how to clean up our own mess. Gave us a little bit of help, but I mean literally we. Went from jobs where it was one of my hardcore research scientists mucking around on a bunch of CPU instances for two weeks to get me my data set to train our first set of LLMs to someone being able to run a new experiment in 15 minutes by writing a quick Spark query in a notebook. It was like, I had heard of Databricks. I was like, "Oh, this seems like a cool company."

>> I'm sold.

Jonathan Frankle

>> The valuation was going up and I was like, "Oh, shit.". Sorry, I don't know if I'm allowed to say that.

>> .

Jonathan Frankle

>> Okay, good. Oh, shit. I see why Databricks is now a 60-something billion dollar company and growing fast. I see why our customers love it and are renewing and renewing and renewing. I would've paid money for that and I should have been smart enough to pay money for that months earlier and my life would've been a lot easier.

>> Yeah, I mean it's a pain relief. It's a steroid, it's a boost. Take us through the AI world now because one of the things that we've been talking about, and we're trying to get our sense around this because what comes next is the hype around agentic. You mentioned buzzwords, that's the biggest buzzword here this week. But as models come out, the conversation around, "Okay, I want to have model integration." I've been kind of a fan of the idea of, hey, if models are smart and intelligent and they're clean or they're well-formed, why can't you just have all these models just talk to each other? Why have one big model? As people start to have models that either are proprietary information or intellectual property, they're going to need to have a kind of connective tissue, LLM routing's hot right now. As an AI scientist, what's the state of the art with AI right now relative to foundation models? LLMs and now computer vision, certainly retail's got tons of computer vision.

Jonathan Frankle

>> Oh, my God.

>> So a ton coming out.

Jonathan Frankle

>> So let me give you, I'm going to give you a hard time first. You just mentioned a bunch of buzzwords. You mentioned agents, you mentioned routers, you mentioned LLMs. I don't care about the buzzwords. What I care about is what does it do? And I think the word agent, it's really capturing, it's not even capturing an actual thing, it's capturing an aspiration. That you can give a task to AI and kind of let AI not just give you back an answer, but actually take an action. That you're trusted to take that next step without direct supervision. That's what an agent capture is. It's still just an LLM. The idea of this multi-agent and many agents there, it could all just be the same model, you're just asking it to do different tasks. You're now the journalist, please try to write an article. You're now the editor, please provide edits to this article back to the journalist. You're the journalist again, update this. It's all the same model. I wouldn't even call that... We can call it an agent-

>> Explain the mechanism because I want to get down to the -

Jonathan Frankle

>> To the brass tacks?

>> Yeah. So LLM routing in particular was a concept where people saying, "I want to route to a model here." There's been papers written about it, but how does someone deploy? How should they think about it? I like the simplicity, I love that angle. But if I want to actually have it, am I using my own models or my other models? Take us through the mechanisms of... Because I like the simplicity. That makes it very easy to understand.

Jonathan Frankle

>> It's actually, I think routing is also an overcomplication because routing is an optimization. The thing I tell all my customers is don't even worry about routing, don't worry about agents. Don't worry about any of those words. Figure out how to use an LLM, an expensive LLM, an off-the-shelf LLM to just get to a basic solution to your problem. Get to the simplest possible way that you think, "Okay, we've made some progress on this problem." Use one of the amazing-

>> Give an example.

Jonathan Frankle

>> Let me think of one right off the bat. I've got a bunch of tasks with my team where I'm actually trying to generate synthetic data for a particular task. I'm trying to figure out... Actually, let me give you a better one. I've got some customers and they're trying to build what we call LLM judges. They get an output from an LLM and they want to know is this safe or not safe? Does this output make sense in the context of a document that I gave it? Is it factual? Now, there are a lot of ways to do this. One is just to ask GPT 4.o, "Is this factual?" One is to do a multi-step process where you say to GPT 4.0, "First, look at all the atomic facts that are in this statement. Now look at all the atomic facts that are in this document. Now, check each atomic fact here to make sure that it's present there and then do that verification. Then come back with an answer." You can do something more complicated. That might sound to you agentic, because you're asking the model to do multiple steps and take on multiple roles. Call that an agent if you want. I would just call that, we're using the LLM multiple times. We're spending more tokens to get a better response. Now, one thing you can do is you can create a bunch of judgments and you can actually fine-tune a model to try to be a judge. You can take a Llama model that's 8 billion parameters, fine-tune it to be your judge. And then in lieu of using this big expensive model with a lot of tokens, you have a fine-tuned model that just outputs zero or one. That's much cheaper and much faster. Now that model may also be a little bit worse. That model can give you uncertainty though about, "I'm really sure this is good. I'm really sure this is bad. Or I'm somewhere in the middle." When you see something in the middle, one optimization is, let's kick it up to GPT 4.0, which is a smarter model. It's going to cost more, but it'll solve the problem. This is how you end up evolving your way to a complex system. But you start off with asking, "Can I just solve the problem? Can GPT 4.0 do this, even if I give it lots of tokens, I do all this highly curated pipeline, can I just get the problem solved?" Then customers come to me and say, "Oh wait, I need to do this a hundred million times a year. That's going to be too expensive. Can I make this pipeline smaller? Can I make the prompt smaller? Can I fine tune a smaller model that takes care of the easy cases and then route to a bigger case?" Those are all optimizations though. The first part is just can technology solve the problem? And I think people get so caught up in the hype of agents and routers and this and that-

>> Yeah, totally. We've done that.

Jonathan Frankle

>> We lose sight of thing.

>> Well this is great, great thread. So let me ask you another follow up on that because this is awesome. This is exactly where I wanted to go. So all right, so we now have this new methodology, mechanisms and all kinds of cool things. If you go back to early days of computer science, you'd program on a language, then go to the complex systems. I did Pascal before I did Assembler.

Jonathan Frankle

>> Don't age yourself too much.

>> Okay, I'm old, yeah. You guys did the coolest shit, but as a developer... Oh, here we go, closing bell. Jonathan gets the closing bell, he gets the closing .

Jonathan Frankle

>> Oh, man. Oh, man.

>> So every day, they close the market and then do the gavel.

Jonathan Frankle

>> Oh, this is so cool. That's awesome.

>> Opening bell's fun too.

Jonathan Frankle

>> This is amazing.

>> We're going to launch our new studio out on February 20th. I think we're going the ring an opening bell, but anyway.

Jonathan Frankle

>> Oh my God.

>> So getting back to the cool AI conversation. So not to date myself, I just dated myself by just saying that, so that's bad. No, but I mean, there's a lot of people building, wanting to get in to get their hands dirty and start solving problems, not getting into the hype and saying, "Oh, there's magic in there." So if I want to go in and start coding or it's kind like coding, I'm playing around, I'm figuring it out to repeat it. So there might be a little bit of friction to do some of these ad hoc things around figuring out what to do for the judge's thing. How do I take that and make that a process and how does Databricks, how do you guys think about that? Because at the end of the day, the faster you can get this frictionless, repeatable for people, then who cares where the models come from, what you call it? It's just data. Take me through, I guess, what's the craft, the trade craft?

Jonathan Frankle

>> So I can give you the trade craft and the real answer is I wish I could make product announcements here without getting in trouble. But all I'll say is stay-

>> Something's coming from Databricks .

Jonathan Frankle

>> All I will say is stay tuned and some really cool stuff is coming that will take what I'm about to tell you in terms of the pipeline and just make this almost a wizard that you just walked through.

>> Yeah, that's the dream scenario.

Jonathan Frankle

>> Well, the dream scenario is very close to being a reality, so keep your eyes out. But what I'll say is here's how it begins. The first thing you need to do is figure out what are you trying to do? And by what I mean there, just even write it down in a couple of sentences. "I would like to take some input and produce some output," and what do those inputs mean and what is the process? Just write it down. That is already enough to build an LLM judge to give that to GPT 4.0 or Claude or what have you. And if you give it an example input and output, it can probably tell you, seems like it was a match, seems like it wasn't a match. The idea is you need some way to measure. The first part to all this, and I think the thing that everybody misses is if you don't know how to measure whether you've done a good job on your task, you won't know when you've solved the problem. It's like anything else in life. So write it down in natural language and then you need a few examples. Give me some input-output examples of what good behavior looks like for your task and what bad behavior looks like. You can ask an LLM to help you with this. You can describe your task and say, "Hey, can you give me a couple of example inputs," and you'll say two of these are good and three of these are terrible. A couple of these are repetitive. Take the two and then do this a few times and you'll generate five to 10 even to start with. Generate some example answers. Ask it to create some bad answers and some good answers. Curate for yourself, give it some natural language feedback and say, "Hey, this answer is not good for that-

>> So natural language is the new pseudocode?

Jonathan Frankle

>> Natural language is the new code.

>> Write the pseudocode, write what you want, maybe put some arrows, if then this, we want to do that, and do figure out what you want to do, why you want to do it and how to measure it.

Jonathan Frankle

>> I'd actually say it's even more than that. Pseudocode is code.

>> Yeah, exactly.

Jonathan Frankle

>> You don't need to write code anymore and I love that. That's the-

>> That's cool.

Jonathan Frankle

>> And so natural language is the interface for this, but you still have to follow regimented steps.

>> You got to know your data. What do you have available?

Jonathan Frankle

>> Well, first you build a way to measure and then build a dumb system that seems to solve this problem. This could involve just giving a prompt to Llama or to any close model that you've got on board. Don't try to go and fine tune and boil the ocean right away. Those are all optimizations. First is like stand up the most basic system you can, your minimum viable product, just like you would do in code. And you might want to connect it to a vector database and do RAG if it has to be document centric. Again, that's really straightforward using our vector database on Databricks and using your existing data. From there, get it up and running and just play with it. Ask your colleagues to play with it a little bit and just get a little bit of usage data, get a little bit of feedback on kind of, okay here, terrible there. You'll be impressed and you'll be disappointed. But that usage data becomes the next iteration.

>> And again, this is the magic I think of what's coming. I like how you kind of simplify it and throw the wet blanket on the hype. I appreciate that, because what's happening is that there's things now that are gettable, they're doable in business that were really difficult before. That no one maybe thought about five years ago, but like, "Eh, screw it. I'm not going to do it because too much of a hassle. It's going to cost me too much money or it's not gettable. I can't attain that goal."

Jonathan Frankle

>> And I can't tell you what's gettable today because with AI it's still very fuzzy and uncertain. That's kind of in some sense, the whole point of AI is it does fuzzy things. What I can tell you is, it's great to give people the chance to try and you'll be surprised at the things that work and disappointed at the things you thought should work that fail and we're still getting there as a technology. This is like the internet in 1995. We got a ways to go before it's really solid, but it's already valuable.

>> John, talk about your thoughts on best practices. Let's just say I have a data set, just hypothetical. We've done theCUBE 15 years and started storing the videos in 2012 in the cloud. I have all these conversations that are stored in S3, multimodal, video, audio, got all that metadata, I've got all the artists written on SiliconAngle. They're all there. I convert them into a vector database and now I have all this data. What do I do?

Jonathan Frankle

>> So the first thing I'm going to say to you, I'm going to scold you and say, "Why'd you bother to get all this data until you figured out what you wanted to accomplish?"

>> I'm a data hoarder.

Jonathan Frankle

>> And data hoarding is great, but actually I see this mistake a lot with our Databricks customers and we kind of coach them around this. They're like, "I can't start on AI until I get all my data perfectly organized." And the answer is actually, it's the other way around. Start trying to do the AI-

>> So don't store the data?

Jonathan Frankle

>> Well, store the data, but always store your data, but don't try to make your data perfect until you figure out what you're doing with it. Let your use case motivate the work that you do.

>> So I throw all the data in the back closet, it's all sitting there, it's all messy. It's a dirty room that no one sees-

Jonathan Frankle

>> And now figure out what you want to do. Let's say you want to replace yourself as host with Gemini. You want Gemini to come through and be able to host this podcast instead of you, based on all these transcripts. So your task is now I want to be able to give, here's a description of a guest, here's a topic I want to talk about or a list of topics I want to talk about. Generate a bunch of questions and the guest will answer them for you. Then you can hook that up to a model and have it do audio generation. Give it access to a vector database so the model can look up, well here's what I've talked about in the past, here's my style, here's how I talk. And then you'll figure out, "Oh, I need to have all these things in the vector database. I should probably have a list of guests, I should probably download their LinkedIn profiles and add that to a vector database." Let the use case be the motivation for the data and cool, in a couple hours we've just replaced you as host of this podcast with AI.

>> I love this. Sorry,

Jonathan Frankle

>> You're out a job.

>> So first of all, how much do I owe you for that consulting session?

Jonathan Frankle

>> Come become a Databricks customer and you'll see my rate.

>> Do you have a rate?

Jonathan Frankle

>> Apparently, yes. It's high enough that people don't ask for it very often.

>> Okay, so since you want to go back to academia, but you can't because you're in a great venture. Let's do a-

Jonathan Frankle

>> Oh, whoever said I wanted to go back to academia? Databricks may have gotten me out of academia. They broke me out of jail. I'm free.

>> All right. Let's give a master class. So I'm a young computer science major and like, "Oh my god, I just went to school three years ago. I'm a junior, about to be a senior and everything I learned is out the window because-

Jonathan Frankle

>> Oh, it's not out the window.

>> So Jonathan, tell me what do I do?

Jonathan Frankle

>> Go get your hands dirty. Go do a project, go build something with AI. In computer science especially, there is no substitute for doing the actual work. Go get your hands dirty. That's how you learn. You learn, you get the intuition for AI is good at this, bad at that. Here are great frameworks, here are terrible frameworks. Here's my belief about the world. Those are the people who I want to hire.

>> All right, so here's the question about the Jonathan pre-Databricks, MosaicML, hair's on fire. You guys are innovating. What's the coolest thing you've observed now that you're in the Databricks fold besides the example you gave earlier around them cleaning up that awesome story you mentioned? What observation like, "Oh, wow. Holy shit."

Jonathan Frankle

>> Honestly, this is going to sound very cliche. Our customers are incredible. And I mean that, like going and talking to our friends at John Deere about what they're doing for AI in agriculture or going and talking to our friends at FactSet about how they're making FactSet query language much more available to people who don't know FQL. It's just, learning what ADP is. I didn't know what ADP was. I now realize they're critical infrastructure for the whole economy and they're doing awesome stuff with AI. But it's honestly the greatest delight has just been spending time with our customers and learning what they do, how dedicated they are, and feeling motivated to put tools in their hands.

>> You don't have to say a name of a company, or you can if you want. What's the coolest thing you've seen?

Jonathan Frankle

>> I can't pick one. I'm sorry. I love all our customers equally, but they're all incredible.

>> Johnathan, great to have you on theCUBE.

Jonathan Frankle

>> Thank you so much for having me.

>> Awesome conversation. Databricks has got great customers. Obviously you don't get great customers by having a bad product. Obviously AI and they're taking simplifying it, just stay with the data, figure out what you want to do. Again, you don't have to be super complicated. Get down to the brass tacks. I'm John for theCUBE. Thanks for watching.