Scaling the Agentic Era with NVIDIA Vera Rubin NVL72 on CoreWeave Cloud | Corey Sanders & Shawn Lewis, CoreWeave

More from Scaling the Agentic Era with NVIDIA Vera Rubin NVL72 on CoreWeave Cloud

Corey Sanders

Senior Vice President Product

CoreWeave

Shawn Lewis

General Manager of SaaS

CoreWeave

Dave Vellante

Co-Founder & Co-CEO

SiliconANGLE Media, Inc.

Info

Corey Sanders & Shawn Lewis, CoreWeave

Corey Sanders

Senior Vice President Product CoreWeave

Shawn Lewis

General Manager of SaaS CoreWeave

Dave Vellante

Co-Founder & Co-CEO SiliconANGLE Media, Inc.

HOST

search

Dave Vellante

>> Hi everybody, we're back at theCUBE's NYSE Wired studio. This is our bi-coastal program. We're tracking the first Vera Rubin installations by CoreWeave. Really appreciate the information and the support. Corey Sanders is back again. He's the Senior Vice President of Products at CoreWeave, and Shawn Lewis is the GM of SaaS at CoreWeave. We're going to dig into that. But SaaS, when people hear SaaS, they think CRM and human capital management. You guys aren't in that business, are you? What's SaaS? It's more vertically oriented, specialized SaaS. Explain.

Shawn Lewis

>> It's not Salesforce like you would typically think of. So I manage our software tools layer. I was the founder of Weights & Biases. So Weights & Biases makes tools used by millions of AI researchers over the last 10 years. And we really have two major product lines. We have W&B Models that's used for AI researchers to develop all the models that you're familiar with now. And then we have W&B Weave, which is used for AI engineers to build AI applications using those models. At CoreWeave, we have a couple other SaaS tools today. So we have marimo notebooks as one other example, which is a popular data science

Dave Vellante

>> notebook.Yeah. So Weights & Biases is a company that CoreWeave bought, I think last spring maybe. It's a really strategic acquisition. We're going to get into some of the innovations that have come out of that. But Shawn, the industry is moving from models to now we're talking about— everybody's talking about autonomous agents. So how do you describe the breakthroughs in this agentic era? What should we be focused on?

Shawn Lewis

>> Totally. Yeah. So our tools, like I said, Weights & Biases tools are used to build the agents that you're familiar with today, right? The models that are used inside the agents, W&B Models, is the software that people typically use to run all of their experiments. And keep track of the experiments that they run, analyze all the experimental data, and decide what to do next. W&B Weave is our tool for building AI applications, which today a lot of AI applications are agents. So we've moved from people building simpler AI chatbots or like classification apps to full-on agents. And so Weave helps users do this AI loop that we like to talk about, where unlike traditional software development, it's a much more experimental workflow because at the core of it you have a model that's stochastic. It's non-deterministic. You don't know exactly if you give it an input what the output will be, right? So this loop is that you put something in production, you put your app in production, and then you monitor what users do with it, what the inputs are that users give, and what the outputs are. And you try to find either failure modes or scenarios that you'd like your model, your agent, to do better in. you classify all those into groups, and then you form them into what we call offline evaluations. So that's a big data set of scenarios or cases that we want our application to be good at. And then you can run your evaluation offline while you test your system. You can improve your prompts, you can improve your harness, you can improve the system as a whole against that evaluation and get a score for it. And when you get a better score, you can ship it back to production. And now you've completed that loop and you can look for new cases that you'd like to optimize.

Dave Vellante

>> So when you— in the early days of Weights & Biases, you had researchers interacting with the system So what did you learn from that interaction and how did that impact and inform the way that you're building software for agents?

Shawn Lewis

>> Yeah, so it's really, like I said, this experimental process where you don't know, we don't know what's going to happen next when we put our model in production. We're building this frontier thing that has all these new capabilities. We don't know how users are going to interact with it. So one major thing that researchers have to do is build these evals. Which is basically a human deciding, what is important to focus on and optimize for. That's the hard part of the job of doing AI research. The thing that we're really excited about now is that we're actually building AI into our tools. So for the first time, there is an agent inside of Weights & Biases that helps those AI users train AI models and build AI applications.

Dave Vellante

>> And presumably those agents can learn from the reasoning traces of humans when Humans have to get involved. Is that right?

Shawn Lewis

>> Yeah. So the agent that we're building, it can do all kinds of things. So it acts as an assistant. So you can ask it questions about, I've done all these experiments or I'm doing a new experiment. How does that experiment compare to a baseline? So in the past, our users would log, say, millions of metrics into this rich UI for exploring experimental data. And they'd have to pore through all these charts manually. Even today, that's a lot of the time that AI researchers spend. The agent can help you analyze that data. It can look at all of those metrics and try to find interesting outliers and make explanations for why certain metrics changed over time. But that's just like one of the small things that it can do. It can also, the really cool part of it is it can execute that entire loop. So it can go you can take an agent that's running in production and say, find the areas that my agent's not doing very well. The agent will look at those traces and cluster them together and tell you, your agent's not doing very well on this type of problem, it'll create an eval for you and run a new experiment.

Corey Sanders

>> And it's really exciting because we're actually using a lot of this inside CoreWeave to build our own products and make our own products better. So a good example is we have an inferencing platform with a lot of growing customers, a lot of excitement, and our goal is to make it consistently perform better and better and better based on the way people are using the platform. And so part of that is we're actually using this loop leveraging the capabilities that Weights & Biases delivers to constantly make the service and experiment better for those end customers. Peter mentioned speculative decoding as a good example, right? It allows us to run it through an evaluation, validate whether what we've built is better or worse, and move from there all in this automated fashion. And so we're as we like to say, eating our own dog food or drinking our own champagne.

Dave Vellante

>> I like the dog food

Corey Sanders

>> analogy.

Dave Vellante

>> There Because it's not always

Corey Sanders

>> pretty.Yeah, exactly. So I appreciate that. The process is not pretty.

Dave Vellante

>> A lot of hard work goes into it. Champagne comes out of it. Takes time though.

Corey Sanders

>> That's right.

Dave Vellante

>> Shawn, can you paint a picture of what the architecture looks like? You're doing orchestration, you're doing evaluations, you've got this execution, agentic execution layer. What's the stack look like?

Shawn Lewis

>> There's a new backend that can execute the agent harness that powers the agent when you interact with it in the UI an internal system today. It's something that we could make public and sell to our users so they could host their own agents. There's also a feature in Weights & Biases called W&B Launch that connects the Weights & Biases toolkit, which you can think of as a UI that users spend time analyzing data in, back to infrastructure. So it allows W&B users to launch and execute jobs on CoreWeave infrastructure from the UI. So now the agent can, instead of a human doing it, the agent can decide to launch experiments onto the infrastructure. And then, a really exciting component, and I think a really important surface for consuming all agents today is our mobile app. So we shipped a mobile app after joining CoreWeave, which I'm really excited to say was the most requested feature that we've ever had on Weights & Biases because experimenters, researchers tend to kick off research and then monitor what's happening with their experiments. These can be really expensive experiments that take many days to execute. So imagine you're leaving the office, it's 5:00 PM and you're going home. You've just kicked something off and you really want to know is that thing going to work or not? And so you watch these training curves converge over time as you're at home. So it's a little challenging on a mobile app, right, to actually edit code and maybe change hyperparameters and kick off a new experiment. The form factor is just not really there. But with an agent built into that app, now it's much easier for users. You can be riding the bus home and kick off a new experiment by just talking to the agents.

Dave Vellante

>> Yeah, really excited about that. Talk to it through whatever your favorite chat app is. So we use this phrase from the legend Andy Grove a lot on theCUBE, let chaos reign and then rein in the chaos. So when you think about agents on the one hand you want to constrain them, on the other hand you want to see what they can do. Have you found, are there any surprises in terms of what your agents are doing as you go toward this agentic era?

Shawn Lewis

>> It's just really exciting to see people for the first time when they have an agent that can actually do part of their job. So for AI researchers that didn't have this capability in Weights & Biases to go when we watch them use the agent for the first time and they've got it connected to Launch and they say they're in a project, maybe they've been working on for a long time where they've been manually doing experiments. All of a sudden they can say, what experiment should I do next? Get a good set of answers and say, okay, go ahead and run them. And that's a real aha moment for people. It happens in Weights & Biases. When you put, when we put AI into our product, that's what we see happening. Kind of happens in all products. It's really amazing to see. When you add this huge amount of intelligence to any existing system, how users level up in what they can do. They end up being more like managers and less the executor of their work.

Dave Vellante

>> We've been doing a lot of research in theCUBE Research further up the stack, the application, the data and app layer. And one of the dots that we've connected is the interaction of the user and the intelligent backend. Very important because, as I said earlier, the backend will learn from the human exceptions and things like that. So where do the humans fit in in this world of AI research?

Shawn Lewis

>> Yeah, always a good question. It's like, will AI completely automate the job of AI researchers? I personally don't think that AI is going to completely automate anything. I think it's going to allow humans to do a lot more than we could do in the past. And I think it's the job of humans to decide what's worth working on. So if you're familiar with Andrej Karpathy, he's this famous AI researcher, right? And a few months ago he released AutoResearch, which showed the world that Claude Code actually could autonomously improve older AI models. so the model he's improving was a few years old. It's called nanoGPT. It's based on the GPT-2 architecture. I don't think that today you could autonomously improve GPT-5. I think that surely researchers are using GPT-5 to ask questions and help them along the path to improving GPT-5, but I don't think they're automating it. So if you're completely out there on the frontier, AI may not help you actually automate everything. If you're doing something that is a little more known to humans, it'll help you automate stuff. I think the other thing to know about that is that AutoResearch is optimizing against a single objective, right? There's a predefined evaluation, and the goal of AutoResearch in Andrej Karpathy's case is to improve against that objective. The other big part of AI research is defining new objectives to optimize against. So defining new evals. Like I said, in this AI loop, we collect data from production to see what users do and we make new evals. So Andrej showed the ability to optimize against a single eval but not to improve existing ones. And that's the really hard job that humans have to keep doing.

Dave Vellante

>> Yeah. And again, thinking about your customers and your customers' customers, It would seem that, of course, there are concerns about jobs, but it would seem that a lot of the coordination activities, what I like to call paper cuts, stuff we hate, AI is going to help us. It's that judgment piece that we're still going to need humans. Now, will that eventually get AI-ified?

Shawn Lewis

>> Maybe.

Dave Vellante

>> Maybe in the sense that there's tacit knowledge, tribal knowledge, we call it, within the organization. But even in that case, if Lisa leaves the company and Lisa had all the knowledge in her brain, I actually want an agent to be able to remember what Lisa knew and share that and scale that. So, as a software person, do you think about that that much? I'm sure you think about a lot of things, but I do. How do you see that playing out?

Shawn Lewis

>> Yeah, I think we can take evals as an example. So there's an eval called SWE-bench, which was the most important eval for AI for the last few years. SWE-bench was produced with a really automated process where they mined a bunch of GitHub repos and found, I think, 1,000 different tasks from GitHub repos. They turned that into the eval. There are humans reviewing some of the aspects of SWE-bench, but not very rigorously. SWE-bench is basically maxed out by AI models today. You can get 90-some, 92% accuracy on it with GPT-5. It's maxed out. Now, a new eval just came out 2 days ago from Cognition. The makers of Devin. It's called FrontierCode, I think. And it's really, really cool. It actually took 40 hours of the smartest programmers for each task. And there's 150 tasks in there. So that's 6,000 hours. It's like a couple of years of human work. So we're actually going in the opposite direction, right? As the models get smarter, it's harder to define tasks for them that are hard. And it takes even more human effort to build those evals.

Corey Sanders

>> I do think that a little bit to your point, and Shawn mentioned this earlier, just about the different thing, different tasks, different outcomes. I think the defining of the outcome and the creativity behind that, I think, will always be a premium human skill. And as I look to some of the amazing work being targeted towards researchers and the proliferation or democratization of AI across industry, that creativity is going to become even more of a premium across every industry in the world is going to be doing some form of what we call research today. And as they do that, they're going to need help to get there because they're not going to have this historical knowledge, but they're going to have ideas. They're going to have new application models. They're going to have new solutions, whether it be a bank or a retail shopping center or a shuttle to the moon. And so I think the ability to create the tools that allow that creativity, which will again always be that powerhouse behind what I think makes us human, able to turn it into productive outcomes. And that's really, I think, where our services from top to bottom are really quite unique in delivering.

Dave Vellante

>> Yeah. And when you talk to the hottest job now is the forward deployed AI engineer. When you talk to these individuals, the most important thing is understanding the business requirement, actually defining that outcome.

Corey Sanders

>> Correct.

Dave Vellante

>> And obviously they're pricing for certain milestones. That's how they do it. But I'm sure it's like eating dog food. it's course correcting. Is this right? No. And that's where the human judgment comes in.

Corey Sanders

>> 100%.

Dave Vellante

>> So Shawn, could you describe your software portfolio, how your customers engage with that? Are these for-pay services? Are they embedded in there? How does it all work?

Shawn Lewis

>> Sure. Yeah. So Weights & Biases, we've been selling it for the last 8 years to customers. We've got thousands of paying customers across the board. And so, yeah, they typically pay per seat, so they pay per human seat. Of the product. As agents start to use more and more of the product, we're thinking about ways to actually monetize agentic use. Because if— I think that human seats will continue to grow, but you'll see more and more experiments actually executed by a single agent. It's hard to call an agent a single seat because it can do way more experiments than a human.

Dave Vellante

>> Well, and that gets us to outcomes again, and maybe outcome-based pricing. Is that something that you can see in the future. And the whole software industry is trying to figure that out now,

Shawn Lewis

>> obviously.Yeah, that's the dream, that's the goal, that's the hardest thing to do anywhere. It's really hard to measure productivity of a programmer. We know that from years of running software engineering, right? There's no good metric for that that we found. There's lines of code, but there's all these problems with that. Everything can be gamified. So it's a real challenge to do outcome-based pricing, but that's where we'd love to be if we could.

Dave Vellante

>> Well, as a software pro, no matter what pricing rubric you choose, it's not going to be perfect. It just never is, whether it's consumption-based or seat-based or outcome-based. It's always a combination. There's so much diversity out there in the market. All right. We got to rapid-fire each of you. Now we're entering the Vera Rubin era. We've been— it was announced over a year ago. We're super excited. What's exciting you? What are you looking forward to over the coming months and year?

Shawn Lewis

>> I'm just always excited to see how the frontier keeps changing. And when I started this, I would say 10 years ago, Weights & Biases, we could predict a little bit further out into the future what the capabilities would be. Now they're changing so fast. The way that they're changing, the way that we work is changing so fast. It's just incredibly exciting to see what 6 months from now will bring and how we can do more stuff. I'm a manager. I personally love building software. And there were a couple of years when I didn't get a chance because I was managing so many people. Now with AI programming capabilities, I actually do get a chance again because between meetings I can talk to an agent that's actually doing stuff.

Corey Sanders

>> And the code's a lot better now. Yeah, now that the agent's doing it versus you.

Dave Vellante

>> Certainly.All right, Corey,

Corey Sanders

>> you're—Yeah, sorry, I didn't mean to interrupt and insult your coding in the meantime.

Shawn Lewis

>> My code's pretty good. The AI is really good at coding

Corey Sanders

>> today.Yeah, your code's better than mine, that's for sure. I haven't coded for decades, but yeah, for me, it's similar to the frontier, but maybe on the other side of the spectrum. I am so excited as we continue to see this AI, these AI scenarios, these AI applications proliferate. I'm just so excited to see again how the collective human intelligence and human ingenuity creates really new ideas. I think back to the early days of cloud and I worked in some of the early cloud products and the initial ideas that we had about how the cloud would be used exploded into this massive set of new scenarios that we hadn't even thought of. And I think we are on that precipice. And I think Vera Rubin is going to be a big component in helping us get there, which is basically creating this new era of application development that we can't even imagine what it's going to look like till we're there.

Dave Vellante

>> Very exciting, guys. Thanks so much. Appreciate you coming in.

Shawn Lewis

>> Thank you.

Dave Vellante

>> And thank you for watching. We're gonna toss it back to Palo Alto with John Furrier. This is Dave Vellante signing off from the New York Stock Exchange CUBE Wired program. Thanks for watching.

Corey Sanders & Shawn Lewis, CoreWeave

search

Dave Vellante

>> Hi everybody, we're back at theCUBE's NYSE Wired studio. This is our bi-coastal program. We're tracking the first Vera Rubin installations by CoreWeave. Really appreciate the information and the support. Corey Sanders is back again. He's the Senior Vice President of Products at CoreWeave, and Shawn Lewis is the GM of SaaS at CoreWeave. We're going to dig into that. But SaaS, when people hear SaaS, they think CRM and human capital management. You guys aren't in that business, are you? What's SaaS? It's more vertically oriented, specialized SaaS. Explain.

Shawn Lewis

>> It's not Salesforce like you would typically think of. So I manage our software tools layer. I was the founder of Weights & Biases. So Weights & Biases makes tools used by millions of AI researchers over the last 10 years. And we really have two major product lines. We have W&B Models that's used for AI researchers to develop all the models that you're familiar with now. And then we have W&B Weave, which is used for AI engineers to build AI applications using those models. At CoreWeave, we have a couple other SaaS tools today. So we have marimo notebooks as one other example, which is a popular data science

Dave Vellante

>> notebook.Yeah. So Weights & Biases is a company that CoreWeave bought, I think last spring maybe. It's a really strategic acquisition. We're going to get into some of the innovations that have come out of that. But Shawn, the industry is moving from models to now we're talking about— everybody's talking about autonomous agents. So how do you describe the breakthroughs in this agentic era? What should we be focused on?

Shawn Lewis

>> Totally. Yeah. So our tools, like I said, Weights & Biases tools are used to build the agents that you're familiar with today, right? The models that are used inside the agents, W&B Models, is the software that people typically use to run all of their experiments. And keep track of the experiments that they run, analyze all the experimental data, and decide what to do next. W&B Weave is our tool for building AI applications, which today a lot of AI applications are agents. So we've moved from people building simpler AI chatbots or like classification apps to full-on agents. And so Weave helps users do this AI loop that we like to talk about, where unlike traditional software development, it's a much more experimental workflow because at the core of it you have a model that's stochastic. It's non-deterministic. You don't know exactly if you give it an input what the output will be, right? So this loop is that you put something in production, you put your app in production, and then you monitor what users do with it, what the inputs are that users give, and what the outputs are. And you try to find either failure modes or scenarios that you'd like your model, your agent, to do better in. you classify all those into groups, and then you form them into what we call offline evaluations. So that's a big data set of scenarios or cases that we want our application to be good at. And then you can run your evaluation offline while you test your system. You can improve your prompts, you can improve your harness, you can improve the system as a whole against that evaluation and get a score for it. And when you get a better score, you can ship it back to production. And now you've completed that loop and you can look for new cases that you'd like to optimize.

Dave Vellante

>> So when you— in the early days of Weights & Biases, you had researchers interacting with the system So what did you learn from that interaction and how did that impact and inform the way that you're building software for agents?

Shawn Lewis

>> Yeah, so it's really, like I said, this experimental process where you don't know, we don't know what's going to happen next when we put our model in production. We're building this frontier thing that has all these new capabilities. We don't know how users are going to interact with it. So one major thing that researchers have to do is build these evals. Which is basically a human deciding, what is important to focus on and optimize for. That's the hard part of the job of doing AI research. The thing that we're really excited about now is that we're actually building AI into our tools. So for the first time, there is an agent inside of Weights & Biases that helps those AI users train AI models and build AI applications.

Dave Vellante

>> And presumably those agents can learn from the reasoning traces of humans when Humans have to get involved. Is that right?

Shawn Lewis

>> Yeah. So the agent that we're building, it can do all kinds of things. So it acts as an assistant. So you can ask it questions about, I've done all these experiments or I'm doing a new experiment. How does that experiment compare to a baseline? So in the past, our users would log, say, millions of metrics into this rich UI for exploring experimental data. And they'd have to pore through all these charts manually. Even today, that's a lot of the time that AI researchers spend. The agent can help you analyze that data. It can look at all of those metrics and try to find interesting outliers and make explanations for why certain metrics changed over time. But that's just like one of the small things that it can do. It can also, the really cool part of it is it can execute that entire loop. So it can go you can take an agent that's running in production and say, find the areas that my agent's not doing very well. The agent will look at those traces and cluster them together and tell you, your agent's not doing very well on this type of problem, it'll create an eval for you and run a new experiment.

Corey Sanders

>> And it's really exciting because we're actually using a lot of this inside CoreWeave to build our own products and make our own products better. So a good example is we have an inferencing platform with a lot of growing customers, a lot of excitement, and our goal is to make it consistently perform better and better and better based on the way people are using the platform. And so part of that is we're actually using this loop leveraging the capabilities that Weights & Biases delivers to constantly make the service and experiment better for those end customers. Peter mentioned speculative decoding as a good example, right? It allows us to run it through an evaluation, validate whether what we've built is better or worse, and move from there all in this automated fashion. And so we're as we like to say, eating our own dog food or drinking our own champagne.

Dave Vellante

>> I like the dog food

Corey Sanders

>> analogy.

Dave Vellante

>> There Because it's not always

Corey Sanders

>> pretty.Yeah, exactly. So I appreciate that. The process is not pretty.

Dave Vellante

>> A lot of hard work goes into it. Champagne comes out of it. Takes time though.

Corey Sanders

>> That's right.

Dave Vellante

>> Shawn, can you paint a picture of what the architecture looks like? You're doing orchestration, you're doing evaluations, you've got this execution, agentic execution layer. What's the stack look like?

Shawn Lewis

>> There's a new backend that can execute the agent harness that powers the agent when you interact with it in the UI an internal system today. It's something that we could make public and sell to our users so they could host their own agents. There's also a feature in Weights & Biases called W&B Launch that connects the Weights & Biases toolkit, which you can think of as a UI that users spend time analyzing data in, back to infrastructure. So it allows W&B users to launch and execute jobs on CoreWeave infrastructure from the UI. So now the agent can, instead of a human doing it, the agent can decide to launch experiments onto the infrastructure. And then, a really exciting component, and I think a really important surface for consuming all agents today is our mobile app. So we shipped a mobile app after joining CoreWeave, which I'm really excited to say was the most requested feature that we've ever had on Weights & Biases because experimenters, researchers tend to kick off research and then monitor what's happening with their experiments. These can be really expensive experiments that take many days to execute. So imagine you're leaving the office, it's 5:00 PM and you're going home. You've just kicked something off and you really want to know is that thing going to work or not? And so you watch these training curves converge over time as you're at home. So it's a little challenging on a mobile app, right, to actually edit code and maybe change hyperparameters and kick off a new experiment. The form factor is just not really there. But with an agent built into that app, now it's much easier for users. You can be riding the bus home and kick off a new experiment by just talking to the agents.

Dave Vellante

>> Yeah, really excited about that. Talk to it through whatever your favorite chat app is. So we use this phrase from the legend Andy Grove a lot on theCUBE, let chaos reign and then rein in the chaos. So when you think about agents on the one hand you want to constrain them, on the other hand you want to see what they can do. Have you found, are there any surprises in terms of what your agents are doing as you go toward this agentic era?

Shawn Lewis

>> It's just really exciting to see people for the first time when they have an agent that can actually do part of their job. So for AI researchers that didn't have this capability in Weights & Biases to go when we watch them use the agent for the first time and they've got it connected to Launch and they say they're in a project, maybe they've been working on for a long time where they've been manually doing experiments. All of a sudden they can say, what experiment should I do next? Get a good set of answers and say, okay, go ahead and run them. And that's a real aha moment for people. It happens in Weights & Biases. When you put, when we put AI into our product, that's what we see happening. Kind of happens in all products. It's really amazing to see. When you add this huge amount of intelligence to any existing system, how users level up in what they can do. They end up being more like managers and less the executor of their work.

Dave Vellante

>> We've been doing a lot of research in theCUBE Research further up the stack, the application, the data and app layer. And one of the dots that we've connected is the interaction of the user and the intelligent backend. Very important because, as I said earlier, the backend will learn from the human exceptions and things like that. So where do the humans fit in in this world of AI research?

Shawn Lewis

>> Yeah, always a good question. It's like, will AI completely automate the job of AI researchers? I personally don't think that AI is going to completely automate anything. I think it's going to allow humans to do a lot more than we could do in the past. And I think it's the job of humans to decide what's worth working on. So if you're familiar with Andrej Karpathy, he's this famous AI researcher, right? And a few months ago he released AutoResearch, which showed the world that Claude Code actually could autonomously improve older AI models. so the model he's improving was a few years old. It's called nanoGPT. It's based on the GPT-2 architecture. I don't think that today you could autonomously improve GPT-5. I think that surely researchers are using GPT-5 to ask questions and help them along the path to improving GPT-5, but I don't think they're automating it. So if you're completely out there on the frontier, AI may not help you actually automate everything. If you're doing something that is a little more known to humans, it'll help you automate stuff. I think the other thing to know about that is that AutoResearch is optimizing against a single objective, right? There's a predefined evaluation, and the goal of AutoResearch in Andrej Karpathy's case is to improve against that objective. The other big part of AI research is defining new objectives to optimize against. So defining new evals. Like I said, in this AI loop, we collect data from production to see what users do and we make new evals. So Andrej showed the ability to optimize against a single eval but not to improve existing ones. And that's the really hard job that humans have to keep doing.

Dave Vellante

>> Yeah. And again, thinking about your customers and your customers' customers, It would seem that, of course, there are concerns about jobs, but it would seem that a lot of the coordination activities, what I like to call paper cuts, stuff we hate, AI is going to help us. It's that judgment piece that we're still going to need humans. Now, will that eventually get AI-ified?

Shawn Lewis

>> Maybe.

Dave Vellante

>> Maybe in the sense that there's tacit knowledge, tribal knowledge, we call it, within the organization. But even in that case, if Lisa leaves the company and Lisa had all the knowledge in her brain, I actually want an agent to be able to remember what Lisa knew and share that and scale that. So, as a software person, do you think about that that much? I'm sure you think about a lot of things, but I do. How do you see that playing out?

Shawn Lewis

>> Yeah, I think we can take evals as an example. So there's an eval called SWE-bench, which was the most important eval for AI for the last few years. SWE-bench was produced with a really automated process where they mined a bunch of GitHub repos and found, I think, 1,000 different tasks from GitHub repos. They turned that into the eval. There are humans reviewing some of the aspects of SWE-bench, but not very rigorously. SWE-bench is basically maxed out by AI models today. You can get 90-some, 92% accuracy on it with GPT-5. It's maxed out. Now, a new eval just came out 2 days ago from Cognition. The makers of Devin. It's called FrontierCode, I think. And it's really, really cool. It actually took 40 hours of the smartest programmers for each task. And there's 150 tasks in there. So that's 6,000 hours. It's like a couple of years of human work. So we're actually going in the opposite direction, right? As the models get smarter, it's harder to define tasks for them that are hard. And it takes even more human effort to build those evals.

Corey Sanders

>> I do think that a little bit to your point, and Shawn mentioned this earlier, just about the different thing, different tasks, different outcomes. I think the defining of the outcome and the creativity behind that, I think, will always be a premium human skill. And as I look to some of the amazing work being targeted towards researchers and the proliferation or democratization of AI across industry, that creativity is going to become even more of a premium across every industry in the world is going to be doing some form of what we call research today. And as they do that, they're going to need help to get there because they're not going to have this historical knowledge, but they're going to have ideas. They're going to have new application models. They're going to have new solutions, whether it be a bank or a retail shopping center or a shuttle to the moon. And so I think the ability to create the tools that allow that creativity, which will again always be that powerhouse behind what I think makes us human, able to turn it into productive outcomes. And that's really, I think, where our services from top to bottom are really quite unique in delivering.

Dave Vellante

>> Yeah. And when you talk to the hottest job now is the forward deployed AI engineer. When you talk to these individuals, the most important thing is understanding the business requirement, actually defining that outcome.

Corey Sanders

>> Correct.

Dave Vellante

>> And obviously they're pricing for certain milestones. That's how they do it. But I'm sure it's like eating dog food. it's course correcting. Is this right? No. And that's where the human judgment comes in.

Corey Sanders

>> 100%.

Dave Vellante

>> So Shawn, could you describe your software portfolio, how your customers engage with that? Are these for-pay services? Are they embedded in there? How does it all work?

Shawn Lewis

>> Sure. Yeah. So Weights & Biases, we've been selling it for the last 8 years to customers. We've got thousands of paying customers across the board. And so, yeah, they typically pay per seat, so they pay per human seat. Of the product. As agents start to use more and more of the product, we're thinking about ways to actually monetize agentic use. Because if— I think that human seats will continue to grow, but you'll see more and more experiments actually executed by a single agent. It's hard to call an agent a single seat because it can do way more experiments than a human.

Dave Vellante

>> Well, and that gets us to outcomes again, and maybe outcome-based pricing. Is that something that you can see in the future. And the whole software industry is trying to figure that out now,

Shawn Lewis

>> obviously.Yeah, that's the dream, that's the goal, that's the hardest thing to do anywhere. It's really hard to measure productivity of a programmer. We know that from years of running software engineering, right? There's no good metric for that that we found. There's lines of code, but there's all these problems with that. Everything can be gamified. So it's a real challenge to do outcome-based pricing, but that's where we'd love to be if we could.

Dave Vellante

>> Well, as a software pro, no matter what pricing rubric you choose, it's not going to be perfect. It just never is, whether it's consumption-based or seat-based or outcome-based. It's always a combination. There's so much diversity out there in the market. All right. We got to rapid-fire each of you. Now we're entering the Vera Rubin era. We've been— it was announced over a year ago. We're super excited. What's exciting you? What are you looking forward to over the coming months and year?

Shawn Lewis

>> I'm just always excited to see how the frontier keeps changing. And when I started this, I would say 10 years ago, Weights & Biases, we could predict a little bit further out into the future what the capabilities would be. Now they're changing so fast. The way that they're changing, the way that we work is changing so fast. It's just incredibly exciting to see what 6 months from now will bring and how we can do more stuff. I'm a manager. I personally love building software. And there were a couple of years when I didn't get a chance because I was managing so many people. Now with AI programming capabilities, I actually do get a chance again because between meetings I can talk to an agent that's actually doing stuff.

Corey Sanders

>> And the code's a lot better now. Yeah, now that the agent's doing it versus you.

Dave Vellante

>> Certainly.All right, Corey,

Corey Sanders

>> you're—Yeah, sorry, I didn't mean to interrupt and insult your coding in the meantime.

Shawn Lewis

>> My code's pretty good. The AI is really good at coding

Corey Sanders

>> today.Yeah, your code's better than mine, that's for sure. I haven't coded for decades, but yeah, for me, it's similar to the frontier, but maybe on the other side of the spectrum. I am so excited as we continue to see this AI, these AI scenarios, these AI applications proliferate. I'm just so excited to see again how the collective human intelligence and human ingenuity creates really new ideas. I think back to the early days of cloud and I worked in some of the early cloud products and the initial ideas that we had about how the cloud would be used exploded into this massive set of new scenarios that we hadn't even thought of. And I think we are on that precipice. And I think Vera Rubin is going to be a big component in helping us get there, which is basically creating this new era of application development that we can't even imagine what it's going to look like till we're there.

Dave Vellante

>> Very exciting, guys. Thanks so much. Appreciate you coming in.

Shawn Lewis

>> Thank you.

Dave Vellante

>> And thank you for watching. We're gonna toss it back to Palo Alto with John Furrier. This is Dave Vellante signing off from the New York Stock Exchange CUBE Wired program. Thanks for watching.