RAISE Summit '26 | Jyoti Bansal, harness.io

Clips
More from RAISE Summit '26

Jyoti Bansal

Co-Founder & CEO

Harness

play_circle_outline Harness: Automating Testing, Verification, Security and Deployment to Unlock AI-Generated Code

play_circle_outline From Ferrari to Freeway: Building High-Throughput Pipelines and Risk-Based Autonomy for Shipping AI-Generated Code

play_circle_outline Tokenomics and AI cost management: model selection, caching, and efficiency reduce token spend

play_circle_outline Observability for AI: Linking Prompts, Token Usage, and Code Commits to Business Outcomes and Total Cost of Ownership

play_circle_outline Agent development lifecycle: authoring, testing, security, cost optimization, and operational tooling

Info

Jyoti Bansal, harness.io

Jyoti Bansal

Co-Founder & CEO Harness

search

>> Palo Alto Studio Connection, Silicon Valley and Wall Street.

>> I'm John Furrier, the co -host of theCUBE here with Dave Vellante, my co -host.

>> Welcome to the RAISE Summit coverage, brought to you by Solidigm, with support from Tensordyne, d-Matrix, and Argentum.

>> Welcome back to theCUBE's live stream in Paris, France. I'm John Furrier, host of theCUBE's The RAISE Summit, Day two coverage, wall -to -wall coverage here. The big story is AI infrastructure build -up continues to enable more and more real -world use cases that are coming online. We're seeing coding, agentic, a lot more, and robotics has been a big topic too on day zero with MACHINA event here. Again, all of it's around software, AI, and of course, hardware. We've got returning back on theCUBE and a TechForward Award winner from Harness, Jyoti Bansal, CEO, founder of Harness, the Harness company I call them. Jyoti, great to see you.

>> Hi, good to be here.

>> It's Harness.io is technically the name, that's the URL, but Harness has been, when did you start the company and name it Harness?

>> 2017.

>> So 2017, before the word Harness became vernacular for managing agents at scale, so congratulations on a great name.

>> Well, thank you, yes.I didn't imagine it would become so mainstream, the name, but we were doing exactly what Harness is supposed to do. Today with agents, but even before agents, when you have all the code that's coming out in the world?

>> How do you piece it all together? It's really good.

>> First of all, you're a visionary, multi -time entrepreneur. You had the vision, you're doing Harness before Harness was popular, and now, again, vernacular, de facto standard mechanism for running agents at scale. This is just an extension of software development. And now a couple things have changed. Obviously coding assistants and coding engines are thundering away. The production of code is amazing. Now you've got automation of workflows. So again, we're seeing the software revolution kind of change.

>> What's your, how do you describe what's happening?

>> Because the game is still the same, but the game has changed.

>> Yeah, yeah.Well, think of the number one use case of AI that's working out very well is coding. Everyone, there's not a single coder in the world that's not using AI in a significant way. And that's driving the most consumption of AI. but coding is only the first 30 -40 % of the job to ship software. The end goal is that you want to ship software for your business, for your consumers. The rest 60 -70 % of the job is verification, testing, security, governance, compliance, DevOps, all the things that your people have to do. And that's what our thesis at Harness was that we need to build Harness to automate all of that process and bring AI agents to manage that part of their SDLC. And otherwise the bottleneck just really shifts downstream. So we're seeing in the data, the code volume is going up two and a half times, but what organizations are really shipping is only going up 10%. And because if you don't have the verification and the testing and deployment and security processes running at the AI speed, they're still running at the human speed, you cannot really do anything. So that's what we're seeing manifesting, that the true ROI of AI coding is not coming yet until you bring the downstream.

>> So it's kind of like a backlog into production.

>> Yes.

>> A lot of backup. What needs to change in your opinion? Because it has to burst. At some point it has to come through. What is the success path?

>> Yeah, the analogy I like to use is, let's say you have the fastest Ferrari, but there are no highways, all you have is small town, village streets. What are you going to do with the Ferrari?

>> So that's what really is missing. So what's happening right now, you have the code that's coming out, like 10 minutes you can produce 100,000 lines of code. But now to bring the 100,000 lines of code to production,

>> that's where you need a highway. You can't have small roads with manual steps and processes. You need to bring AI to simulate the human judgment on most of that. On reviewing the code, on testing the code, making sure the security vulnerabilities are graded, the quality is graded.

>> and if the risk is high, you bring a human in the loop in the right way. So at Harness we believe in the concept of risk-based autonomy.

>> You provide, if the risk is low, it's fully autonomous, everything goes out there. If the risk is high on something, then you kind of manage through the Harness of what do you do when the risk is high. Where do you bring the human, where do you bring another set of AI, where do you bring another set of agents, and you tie it all together with Harness.

>> And this scales or translates to agents, obviously. And so what's the best practice?

>> Because we're hearing governance, besides Harness being a very popular word, governance has probably been the most popular word. I've probably said the word governance in the past year more than the 17 years I've been doing theCUBE.

>> it was always in this regulated sector, master key management, all that stuff. But now it's mainstream governance. What's the best practice to get all this going?

>> Governance and Harness go very hand in hand together. If you think of what is the big part of a Harness for agents, one is where do the agents run and what governs them, what secrets they can access, what is their blast radius of things, what they're allowed to do. That's a big part of any kind. You have the agents, you have the model selection, you have how the agents interoperate with each other, the agent chaining, et cetera. But the governance is one of the biggest parts of any Harness for agents.

>> The biggest risk with agents, agents get smarter. And the developers who are creating these agents, they really don't know what an agent would do. An agent can reason through things that you didn't probably expect it to do. Or invoke tools. Invoke tools that you didn't want them to invoke. So you start by building a governance as a core layer of any Harness around agents. which is governance to me is about defining policies on what they're allowed to do, defining risk controls on what degree of risk a human has to come in. You define auditability about everything, that every action of the agent is audited and tracked. And so there is no like you know, that you have no idea something happened bad and no one has any idea what happened really, who did what, agents need to track everything. And then finally is the access control, like what they're allowed to do. As humans we have an identity, we have this is your permission set. and an agent needs to have even more so, this is your identity and this is your permission set and what you're allowed to do.

>> Wow, that sounds super complicated, so simple, how does that get simpler? What are you guys doing to make that simpler? Because that's, you just laid out essentially the rulebook, basically. What's the process? Walk me through a day in the life of doing that. Is it easy? Is it hard? It sounds harder than a...

>> It doesn't have to be that hard if you have the right Harness. To be honest, it's really about, you have to define, so one of the biggest things you have to do is, where are you running the agents? So there's a cloud agent and there's agents that will run in someone's infrastructure. So one of the biggest things that we simplify is that you've got to control where the agents run and what they can do. Of course you have the cloud agents that are from the model providers, but you cannot have an agent running in the cloud somewhere to be in the large banks infrastructure doing something. So the first control you get in Harness is agents that can run in your infrastructure in a safe way with the safety framework around it. So a tool chain like Harness can make it simpler for you to get there otherwise it becomes very complicated. People make mistakes, people get risk. The second is that you also need a tool chain for auditability around the agent authoring and all the things. You need the right tools like you're in the kind of early stages of the right tools Without the right tools, it's very hard. With the right tools, it gets easier.

>> Talk about the business now, on your end. What's going on with the customers? Obviously, there's a huge opportunity to leverage the automation. Coding's great if you can Harness it. Rein in the chaos. What's going on with the product and the business?

>> Well, definitely a lot of excitement, a lot of acceleration, a lot of opportunity that we are getting now. There are three parts that are driving our business. One is , as people are writing more code, and people are realizing what people call the AI Velocity Paradox. You have so much more code coming out, but you're not really shipping it as much. So how do you fix that? How do you automate the downstream processes on coding? All the DevOps and DevSecOps and FinOps and testing kind of things. So that's the first part that's driving a lot of demand for us. The second part that we are seeing is a lot of concern around models like Mythos, finding security vulnerabilities.

>> If you look at people talk about how do we be ready for a Mythos world or any model that's pretty good at finding these vulnerabilities that were there for a long time. And so if you look at a Mythos model, it will find you the vulnerability, it will find you how to exploit it, it will tell you how to fix it. But the challenge is rolling out the fix. And so at the end, it's not a cybersecurity problem, it's more of a DevOps problem. Because for many organizations, even if they know this is the fix to roll it out because of the dependencies and everything could take two weeks or five weeks but you got to roll it out in like an hour so that's where we are seeing a lot of momentum in the business on how you get ready for

>> for like you know new models like AI tools from a security perspective and then the third area that you're seeing a lot of momentum is around actually the AI cost the concept of tokenmaxxing and how

>> do you get the right ROI of AI and we think as part of any AI Harness, managing the cost and efficiency is extremely, extremely important as well.

>> Yeah, I was just mentioning before we came on camera, Ali Ghodsi from Databricks just published a blog post and LinkedIn post on evaluating their own benchmarks. Hundreds of thousands of lines of code. He didn't want to trust external benchmarks, so they ran their own. And what they found was the model costs weren't the issue, it was the Harness selection. Why is that important? because he teases out a central issue on tokenmaxxing. Efficiency and also, that's an ops issue.

>> That's basically, you picked the wrong tool. If you pick the wrong tool, the cost goes up, not the model.

>> Yeah, yeah, totally.Well, the models are more or less doing similar things, but you can, based on the task you are doing, you can pick a model that's two X cheaper, three X cheaper, four X cheaper. So, but based on what model you picked, you have to have the right controls around it.

>> You pick a certain model, you have to do a certain set of follow-on tasks. If you pick another model, you have to do a certain set of follow-on tasks. And that's where the Harness comes in and kind of giving you the checks and balances in the framework.

>> It's interesting, Joe, we've seen this movie before, and I think it's Cloud, right?

>> We all experience, oh, I left that service running on the Cloud, and where's the bill? So a similar dynamic, it's not just the service, It's how you operate the service.

>> And how you tie it back to some ROI. The difference in the cloud world and the AI world is that in the AI world, everything is down to tokens. But now it's like, your one single unit is tokens, but how do you track it to, okay, what did I get out of those tokens? And that's what the hard problem is right now.

>> Say if you're a developer writing code, I spent 100,000 tokens in doing something.

>> What did I get? Did that code ever ship. Was it a small bug, was it a big feature request, what did I get, how do you tie back to it? And that's something, that's what we are trying to simplify at Harness, how do you track it all? The second part is, if you're writing agents to do some business process, say customer support, and you have a customer support ticket come in, and a human can manage that ticket for say $20, but it's a quick phone call, and a human can support it. And now you're an AI agent doing it, and it cost you $100 in AI tokens. Is it worth it? Yeah. So how do you tie the AI tokens back to the unit cost of any business process? People haven't figured that part out. So when we're looking at what are the...

>> It's a total cost of ownership kind of equation.

>> Yes. What's the value? What is the value? And what is the value out of doing that business process without AI and what is the most efficient way of doing that?

>> The pragmatic answer is what you just said. If it costs me $20 for a human, and it costs more with tokens, bad call.

>> Yeah, totally.

>> You're saying there's no metrics on that.

>> There are no metrics on it.

>> So what we have done at Harness is, my background before Harness was my previous company, AppDynamics, in instrumentation observability space.

>> So we thought, how about we bring the observability techniques to understand the token costs, and tie it back to what's happening at the code level. So, say, if you're doing, you write an agent to do customer support ticket, and now you're consuming some tokens on it, can we track it and tie it back to the ticket itself, and, through an observability mechanism?

>> So that's what we would do there. But we also then, can you also optimize it? from whether it's model selection, whether it's caching, whether it's, optimizing for sometime, maybe not using a model, suggesting to the user, the developer that for this particular thing, you probably don't want to

>> use a model for this part of the task.

>> It's interesting, it's funny because when we were only, it was an IT project or digital on the internet. You can connect things, it's instrumentable. You can actually instrument things. However, we're crossing over and blurring in with other lanes, if you will. We're crossing into the lane of business. So we know the cost, finance knows the cost of a human, but that never really trickled into IT projects. So as someone who's done this at AppDynamics, how do you look at this? Because we're living in an era now with AI where there's really not a lot of comparables or pattern recognition that you can really apply. You can do metaphors like, hey DevSecOps, we did these things, but how do you look at KPIs? How do you look at the instrumentation when you're starting to cross into these qualitative metrics?

>> Is there an approach you're taking?

>> Yeah, I think that it's a very emerging area, of course.but if you look at the last few years, the AI cost was an experimental line item for most enterprises.

>> Now it's a big line item, companies are like, I have a $100 million AI bill, or a $10 million AI bill, and now it's when the bill starts getting that size, you cannot just ignore it, you cannot just put it in the experimental bucket, you have to understand where is the money going. So I think that the number one thing that we strongly believe in is that you have to tie the cost to the business outcome, and see what's the business outcome of the money that you spent on it. How does it compare to other ways of doing it, you can't just throw everything to an AI model and expect it to do something.

>> Is there a mechanism though? This is what I'm trying to get at, because it makes total sense, I totally agree, but can I put a mechanism around it? Because we can tie it to outcome, hey, ship code, but it could be bad code and no one uses it.

>> But we hit the metric, it worked.

>> So the mechanism that we have built at Harness is that we create a small agent that runs on your laptop, a developer laptop.

>> And what it does, it ties to when you're using Cursor or Claude Code or whatever AI coding agents you're using, it will intercept the call to it.

>> And now we can tie it to, this is the prompt you wrote to write code, and this is the code that came out that you committed to Git. And now we're tying it back, and now we say, based on the commit that happened, this is the ticket that you had that you were originally working on, and how big the ticket was, was it a small feature, was it a big feature, and then we tie it to downstream what happened, like being a DevOps company, we can tie it to downstream, okay, that code that you wrote, was it rejected later on in a code review? Was it, did it fail the test? Did it ship? So we can see, in our case, you've got lineage, in a way. The full lineage from the prompt, the lineage starts from the prompt, which is the intent now, the lineage used to start from coding, now it starts from the prompt that you write. And then we can tie it to what you were trying to solve, what is the business value out of that, and what did you ship? So then we can start tracking, okay, for this number of tokens, your shipping velocity increased by this much. For the tokens that you use here, nothing really changed because most of it was rejected in the downstream testing. Yeah, bad, bad content. So that's where the mechanisms come in. You have to instrument and tie it back to the entire business process, not just the, okay, I can do this many tokens. The worst thing people did was Jensen Huang from NVIDIA, he went into this speech and he said Token purchase for everybody. Yeah, or not, he said every engineer should be using $250 ,000 of tokens, otherwise maybe they're not doing a good job. What he did was, I think he said, now it's like he made it normal for people to think, oh, my engineer not using $250 ,000 of tokens, maybe we need to give them more. The reality is you don't need $250 ,000 of tokens.

>> He put it in people's head, which I don't think is a good thing. No, it's a bad thing. He didn't mean that. What he was saying was, if you're a good engineer, you should be using tokens. Yes.

>> But he made it sound like it's a mandate. That's engineering practice.

>> And you don't need that much. For most engineers, even engineers who are

>> very, very, very highly productive on using AI for everything, every day, that's a very high amount of tokens. That's probably are not doing it as efficiently. And so if you have the right tool chain and the right Harness around it, we have seen internally that we cut down our per developer AI token cost by half. 50 % reduction, we could see, even before we productize our capabilities, our internal consumption, we could cut down by half.

>> This whole comment by Jensen has stirred a lot of problems. One, token maxing was one. I was talking to a friend in Palo Alto who works at Google, and we were riffing around the old school days of when I was in college, you were judged by how small your code was. Remember those days, Jyoti was like, nice and tight code, it wasn't really fat, it wasn't a book. And people were shamed by writing too long code, inefficient, back then, memory loading, all that stuff. I think that's coming back to tokens.

>> You should be judged by how efficient you are with tokens, not how many tokens. Or in the end, how much you ship to your consumers.

>> So if, let's say, you use $100,000 worth of tokens, and the capabilities that you ship to your consumers were not really that useful, is it really worth it?

>> So you have to eventually tie it to the outcome for your business, for your team, that what you ship, and how efficiently were you able to do that in the process.

>> Well great to have you on theCUBE as always. Time flies when you're on. Love chatting with you, we should definitely do a follow up in Palo Alto or next time you're in New York. Give an update on what you're working on now. What's your focus? because obviously Harness is prime time on all fronts of software development and pushing to production, and agents are only going to accelerate what you guys do.

>> What's your focus now? What are you optimizing for?

>> Well, we are looking at a few areas. One is, as people are building more and more agents, almost every software engineering team is building agents, so how do we create an entire development life cycle for agents? For agent testing, agent security, agent cost optimization, for everything that happens around the agent software life cycle, that's a big focus area.

>> Second is for organizations to get the right value out of AI coding investment.

>> Most organizations are spending a lot of money on coding, but the return is not coming because they're bogged down in the downstream processes, so how fast we can help modernize them in the downstream processes, that's where our focus is. More and more AI native software.

>> Make it easier and more productive for the developers, match it to business value.

>> Yeah. And the other area is security. everyone is really concerned about what happens with code security with AI models, with the hackers using AI models, AI models finding vulnerabilities that didn't happen before. So how do we enable people to deal with it faster?

>> AI coding has killed the supply chain concept of software supply chain, because now the software supply chain is like a data lake at this point. It's crazy. Great to have you on, as always.

>> Appreciate you stopping by, Jyoti. Great to have you on. CEO, founder, multi-time TechForward Award winner from SiliconANGLE's awards. Again, repeat entrepreneur back at it. The coding process is getting easier, but complex production workloads will ultimately be the judge, and AI's going to do a lot of great things, and we're going to see a lot more of this. So you got to get your Harness, right? And of course, called Harness. This is theCUBE doing its part here, harnessing RAISE Summit inside the booth here. Thanks for watching.

Jyoti Bansal, harness.io

search

>> Palo Alto Studio Connection, Silicon Valley and Wall Street.

>> I'm John Furrier, the co -host of theCUBE here with Dave Vellante, my co -host.

>> Welcome to the RAISE Summit coverage, brought to you by Solidigm, with support from Tensordyne, d-Matrix, and Argentum.

>> Welcome back to theCUBE's live stream in Paris, France. I'm John Furrier, host of theCUBE's The RAISE Summit, Day two coverage, wall -to -wall coverage here. The big story is AI infrastructure build -up continues to enable more and more real -world use cases that are coming online. We're seeing coding, agentic, a lot more, and robotics has been a big topic too on day zero with MACHINA event here. Again, all of it's around software, AI, and of course, hardware. We've got returning back on theCUBE and a TechForward Award winner from Harness, Jyoti Bansal, CEO, founder of Harness, the Harness company I call them. Jyoti, great to see you.

>> Hi, good to be here.

>> It's Harness.io is technically the name, that's the URL, but Harness has been, when did you start the company and name it Harness?

>> 2017.

>> So 2017, before the word Harness became vernacular for managing agents at scale, so congratulations on a great name.

>> Well, thank you, yes.I didn't imagine it would become so mainstream, the name, but we were doing exactly what Harness is supposed to do. Today with agents, but even before agents, when you have all the code that's coming out in the world?

>> How do you piece it all together? It's really good.

>> First of all, you're a visionary, multi -time entrepreneur. You had the vision, you're doing Harness before Harness was popular, and now, again, vernacular, de facto standard mechanism for running agents at scale. This is just an extension of software development. And now a couple things have changed. Obviously coding assistants and coding engines are thundering away. The production of code is amazing. Now you've got automation of workflows. So again, we're seeing the software revolution kind of change.

>> What's your, how do you describe what's happening?

>> Because the game is still the same, but the game has changed.

>> Yeah, yeah.Well, think of the number one use case of AI that's working out very well is coding. Everyone, there's not a single coder in the world that's not using AI in a significant way. And that's driving the most consumption of AI. but coding is only the first 30 -40 % of the job to ship software. The end goal is that you want to ship software for your business, for your consumers. The rest 60 -70 % of the job is verification, testing, security, governance, compliance, DevOps, all the things that your people have to do. And that's what our thesis at Harness was that we need to build Harness to automate all of that process and bring AI agents to manage that part of their SDLC. And otherwise the bottleneck just really shifts downstream. So we're seeing in the data, the code volume is going up two and a half times, but what organizations are really shipping is only going up 10%. And because if you don't have the verification and the testing and deployment and security processes running at the AI speed, they're still running at the human speed, you cannot really do anything. So that's what we're seeing manifesting, that the true ROI of AI coding is not coming yet until you bring the downstream.

>> So it's kind of like a backlog into production.

>> Yes.

>> A lot of backup. What needs to change in your opinion? Because it has to burst. At some point it has to come through. What is the success path?

>> Yeah, the analogy I like to use is, let's say you have the fastest Ferrari, but there are no highways, all you have is small town, village streets. What are you going to do with the Ferrari?

>> So that's what really is missing. So what's happening right now, you have the code that's coming out, like 10 minutes you can produce 100,000 lines of code. But now to bring the 100,000 lines of code to production,

>> that's where you need a highway. You can't have small roads with manual steps and processes. You need to bring AI to simulate the human judgment on most of that. On reviewing the code, on testing the code, making sure the security vulnerabilities are graded, the quality is graded.

>> and if the risk is high, you bring a human in the loop in the right way. So at Harness we believe in the concept of risk-based autonomy.

>> You provide, if the risk is low, it's fully autonomous, everything goes out there. If the risk is high on something, then you kind of manage through the Harness of what do you do when the risk is high. Where do you bring the human, where do you bring another set of AI, where do you bring another set of agents, and you tie it all together with Harness.

>> And this scales or translates to agents, obviously. And so what's the best practice?

>> Because we're hearing governance, besides Harness being a very popular word, governance has probably been the most popular word. I've probably said the word governance in the past year more than the 17 years I've been doing theCUBE.

>> it was always in this regulated sector, master key management, all that stuff. But now it's mainstream governance. What's the best practice to get all this going?

>> Governance and Harness go very hand in hand together. If you think of what is the big part of a Harness for agents, one is where do the agents run and what governs them, what secrets they can access, what is their blast radius of things, what they're allowed to do. That's a big part of any kind. You have the agents, you have the model selection, you have how the agents interoperate with each other, the agent chaining, et cetera. But the governance is one of the biggest parts of any Harness for agents.

>> The biggest risk with agents, agents get smarter. And the developers who are creating these agents, they really don't know what an agent would do. An agent can reason through things that you didn't probably expect it to do. Or invoke tools. Invoke tools that you didn't want them to invoke. So you start by building a governance as a core layer of any Harness around agents. which is governance to me is about defining policies on what they're allowed to do, defining risk controls on what degree of risk a human has to come in. You define auditability about everything, that every action of the agent is audited and tracked. And so there is no like you know, that you have no idea something happened bad and no one has any idea what happened really, who did what, agents need to track everything. And then finally is the access control, like what they're allowed to do. As humans we have an identity, we have this is your permission set. and an agent needs to have even more so, this is your identity and this is your permission set and what you're allowed to do.

>> Wow, that sounds super complicated, so simple, how does that get simpler? What are you guys doing to make that simpler? Because that's, you just laid out essentially the rulebook, basically. What's the process? Walk me through a day in the life of doing that. Is it easy? Is it hard? It sounds harder than a...

>> It doesn't have to be that hard if you have the right Harness. To be honest, it's really about, you have to define, so one of the biggest things you have to do is, where are you running the agents? So there's a cloud agent and there's agents that will run in someone's infrastructure. So one of the biggest things that we simplify is that you've got to control where the agents run and what they can do. Of course you have the cloud agents that are from the model providers, but you cannot have an agent running in the cloud somewhere to be in the large banks infrastructure doing something. So the first control you get in Harness is agents that can run in your infrastructure in a safe way with the safety framework around it. So a tool chain like Harness can make it simpler for you to get there otherwise it becomes very complicated. People make mistakes, people get risk. The second is that you also need a tool chain for auditability around the agent authoring and all the things. You need the right tools like you're in the kind of early stages of the right tools Without the right tools, it's very hard. With the right tools, it gets easier.

>> Talk about the business now, on your end. What's going on with the customers? Obviously, there's a huge opportunity to leverage the automation. Coding's great if you can Harness it. Rein in the chaos. What's going on with the product and the business?

>> Well, definitely a lot of excitement, a lot of acceleration, a lot of opportunity that we are getting now. There are three parts that are driving our business. One is , as people are writing more code, and people are realizing what people call the AI Velocity Paradox. You have so much more code coming out, but you're not really shipping it as much. So how do you fix that? How do you automate the downstream processes on coding? All the DevOps and DevSecOps and FinOps and testing kind of things. So that's the first part that's driving a lot of demand for us. The second part that we are seeing is a lot of concern around models like Mythos, finding security vulnerabilities.

>> If you look at people talk about how do we be ready for a Mythos world or any model that's pretty good at finding these vulnerabilities that were there for a long time. And so if you look at a Mythos model, it will find you the vulnerability, it will find you how to exploit it, it will tell you how to fix it. But the challenge is rolling out the fix. And so at the end, it's not a cybersecurity problem, it's more of a DevOps problem. Because for many organizations, even if they know this is the fix to roll it out because of the dependencies and everything could take two weeks or five weeks but you got to roll it out in like an hour so that's where we are seeing a lot of momentum in the business on how you get ready for

>> for like you know new models like AI tools from a security perspective and then the third area that you're seeing a lot of momentum is around actually the AI cost the concept of tokenmaxxing and how

>> do you get the right ROI of AI and we think as part of any AI Harness, managing the cost and efficiency is extremely, extremely important as well.

>> Yeah, I was just mentioning before we came on camera, Ali Ghodsi from Databricks just published a blog post and LinkedIn post on evaluating their own benchmarks. Hundreds of thousands of lines of code. He didn't want to trust external benchmarks, so they ran their own. And what they found was the model costs weren't the issue, it was the Harness selection. Why is that important? because he teases out a central issue on tokenmaxxing. Efficiency and also, that's an ops issue.

>> That's basically, you picked the wrong tool. If you pick the wrong tool, the cost goes up, not the model.

>> Yeah, yeah, totally.Well, the models are more or less doing similar things, but you can, based on the task you are doing, you can pick a model that's two X cheaper, three X cheaper, four X cheaper. So, but based on what model you picked, you have to have the right controls around it.

>> You pick a certain model, you have to do a certain set of follow-on tasks. If you pick another model, you have to do a certain set of follow-on tasks. And that's where the Harness comes in and kind of giving you the checks and balances in the framework.

>> It's interesting, Joe, we've seen this movie before, and I think it's Cloud, right?

>> We all experience, oh, I left that service running on the Cloud, and where's the bill? So a similar dynamic, it's not just the service, It's how you operate the service.

>> And how you tie it back to some ROI. The difference in the cloud world and the AI world is that in the AI world, everything is down to tokens. But now it's like, your one single unit is tokens, but how do you track it to, okay, what did I get out of those tokens? And that's what the hard problem is right now.

>> Say if you're a developer writing code, I spent 100,000 tokens in doing something.

>> What did I get? Did that code ever ship. Was it a small bug, was it a big feature request, what did I get, how do you tie back to it? And that's something, that's what we are trying to simplify at Harness, how do you track it all? The second part is, if you're writing agents to do some business process, say customer support, and you have a customer support ticket come in, and a human can manage that ticket for say $20, but it's a quick phone call, and a human can support it. And now you're an AI agent doing it, and it cost you $100 in AI tokens. Is it worth it? Yeah. So how do you tie the AI tokens back to the unit cost of any business process? People haven't figured that part out. So when we're looking at what are the...

>> It's a total cost of ownership kind of equation.

>> Yes. What's the value? What is the value? And what is the value out of doing that business process without AI and what is the most efficient way of doing that?

>> The pragmatic answer is what you just said. If it costs me $20 for a human, and it costs more with tokens, bad call.

>> Yeah, totally.

>> You're saying there's no metrics on that.

>> There are no metrics on it.

>> So what we have done at Harness is, my background before Harness was my previous company, AppDynamics, in instrumentation observability space.

>> So we thought, how about we bring the observability techniques to understand the token costs, and tie it back to what's happening at the code level. So, say, if you're doing, you write an agent to do customer support ticket, and now you're consuming some tokens on it, can we track it and tie it back to the ticket itself, and, through an observability mechanism?

>> So that's what we would do there. But we also then, can you also optimize it? from whether it's model selection, whether it's caching, whether it's, optimizing for sometime, maybe not using a model, suggesting to the user, the developer that for this particular thing, you probably don't want to

>> use a model for this part of the task.

>> It's interesting, it's funny because when we were only, it was an IT project or digital on the internet. You can connect things, it's instrumentable. You can actually instrument things. However, we're crossing over and blurring in with other lanes, if you will. We're crossing into the lane of business. So we know the cost, finance knows the cost of a human, but that never really trickled into IT projects. So as someone who's done this at AppDynamics, how do you look at this? Because we're living in an era now with AI where there's really not a lot of comparables or pattern recognition that you can really apply. You can do metaphors like, hey DevSecOps, we did these things, but how do you look at KPIs? How do you look at the instrumentation when you're starting to cross into these qualitative metrics?

>> Is there an approach you're taking?

>> Yeah, I think that it's a very emerging area, of course.but if you look at the last few years, the AI cost was an experimental line item for most enterprises.

>> Now it's a big line item, companies are like, I have a $100 million AI bill, or a $10 million AI bill, and now it's when the bill starts getting that size, you cannot just ignore it, you cannot just put it in the experimental bucket, you have to understand where is the money going. So I think that the number one thing that we strongly believe in is that you have to tie the cost to the business outcome, and see what's the business outcome of the money that you spent on it. How does it compare to other ways of doing it, you can't just throw everything to an AI model and expect it to do something.

>> Is there a mechanism though? This is what I'm trying to get at, because it makes total sense, I totally agree, but can I put a mechanism around it? Because we can tie it to outcome, hey, ship code, but it could be bad code and no one uses it.

>> But we hit the metric, it worked.

>> So the mechanism that we have built at Harness is that we create a small agent that runs on your laptop, a developer laptop.

>> And what it does, it ties to when you're using Cursor or Claude Code or whatever AI coding agents you're using, it will intercept the call to it.

>> And now we can tie it to, this is the prompt you wrote to write code, and this is the code that came out that you committed to Git. And now we're tying it back, and now we say, based on the commit that happened, this is the ticket that you had that you were originally working on, and how big the ticket was, was it a small feature, was it a big feature, and then we tie it to downstream what happened, like being a DevOps company, we can tie it to downstream, okay, that code that you wrote, was it rejected later on in a code review? Was it, did it fail the test? Did it ship? So we can see, in our case, you've got lineage, in a way. The full lineage from the prompt, the lineage starts from the prompt, which is the intent now, the lineage used to start from coding, now it starts from the prompt that you write. And then we can tie it to what you were trying to solve, what is the business value out of that, and what did you ship? So then we can start tracking, okay, for this number of tokens, your shipping velocity increased by this much. For the tokens that you use here, nothing really changed because most of it was rejected in the downstream testing. Yeah, bad, bad content. So that's where the mechanisms come in. You have to instrument and tie it back to the entire business process, not just the, okay, I can do this many tokens. The worst thing people did was Jensen Huang from NVIDIA, he went into this speech and he said Token purchase for everybody. Yeah, or not, he said every engineer should be using $250 ,000 of tokens, otherwise maybe they're not doing a good job. What he did was, I think he said, now it's like he made it normal for people to think, oh, my engineer not using $250 ,000 of tokens, maybe we need to give them more. The reality is you don't need $250 ,000 of tokens.

>> He put it in people's head, which I don't think is a good thing. No, it's a bad thing. He didn't mean that. What he was saying was, if you're a good engineer, you should be using tokens. Yes.

>> But he made it sound like it's a mandate. That's engineering practice.

>> And you don't need that much. For most engineers, even engineers who are

>> very, very, very highly productive on using AI for everything, every day, that's a very high amount of tokens. That's probably are not doing it as efficiently. And so if you have the right tool chain and the right Harness around it, we have seen internally that we cut down our per developer AI token cost by half. 50 % reduction, we could see, even before we productize our capabilities, our internal consumption, we could cut down by half.

>> This whole comment by Jensen has stirred a lot of problems. One, token maxing was one. I was talking to a friend in Palo Alto who works at Google, and we were riffing around the old school days of when I was in college, you were judged by how small your code was. Remember those days, Jyoti was like, nice and tight code, it wasn't really fat, it wasn't a book. And people were shamed by writing too long code, inefficient, back then, memory loading, all that stuff. I think that's coming back to tokens.

>> You should be judged by how efficient you are with tokens, not how many tokens. Or in the end, how much you ship to your consumers.

>> So if, let's say, you use $100,000 worth of tokens, and the capabilities that you ship to your consumers were not really that useful, is it really worth it?

>> So you have to eventually tie it to the outcome for your business, for your team, that what you ship, and how efficiently were you able to do that in the process.

>> Well great to have you on theCUBE as always. Time flies when you're on. Love chatting with you, we should definitely do a follow up in Palo Alto or next time you're in New York. Give an update on what you're working on now. What's your focus? because obviously Harness is prime time on all fronts of software development and pushing to production, and agents are only going to accelerate what you guys do.

>> What's your focus now? What are you optimizing for?

>> Well, we are looking at a few areas. One is, as people are building more and more agents, almost every software engineering team is building agents, so how do we create an entire development life cycle for agents? For agent testing, agent security, agent cost optimization, for everything that happens around the agent software life cycle, that's a big focus area.

>> Second is for organizations to get the right value out of AI coding investment.

>> Most organizations are spending a lot of money on coding, but the return is not coming because they're bogged down in the downstream processes, so how fast we can help modernize them in the downstream processes, that's where our focus is. More and more AI native software.

>> Make it easier and more productive for the developers, match it to business value.

>> Yeah. And the other area is security. everyone is really concerned about what happens with code security with AI models, with the hackers using AI models, AI models finding vulnerabilities that didn't happen before. So how do we enable people to deal with it faster?

>> AI coding has killed the supply chain concept of software supply chain, because now the software supply chain is like a data lake at this point. It's crazy. Great to have you on, as always.

>> Appreciate you stopping by, Jyoti. Great to have you on. CEO, founder, multi-time TechForward Award winner from SiliconANGLE's awards. Again, repeat entrepreneur back at it. The coding process is getting easier, but complex production workloads will ultimately be the judge, and AI's going to do a lot of great things, and we're going to see a lot more of this. So you got to get your Harness, right? And of course, called Harness. This is theCUBE doing its part here, harnessing RAISE Summit inside the booth here. Thanks for watching.