SC25 | Callan Fox & Betsy Chernoff, WEKA

Clips
More from SC25

Callan Fox

Principal Product Manager

WEKA

Betsy Chernoff

Principal AI Product Marketing Manager

WEKA

play_circle_outline Importance of memory hierarchy in complex, multi-turn AI inference systems.

play_circle_outline Analysis of potential cost reductions using fewer GPUs through augmented memory capabilities.

play_circle_outline Revolutionizing AI: NeuralMesh Unveils Augmented Memory Grid and Partners with OCI to Enhance Inference Capabilities

play_circle_outline The importance of reducing overhead in inference systems for better performance.

play_circle_outline WEKA's focus on solving bottlenecks in storage systems for enhanced AI performance.

Info
Transcript

Callan Fox & Betsy Chernoff, WEKA

Callan Fox

Principal Product Manager WEKA

Betsy Chernoff

Principal AI Product Marketing Manager WEKA

In this SC25 interview, theCUBE’s Dave Vellante and Jackie McGuire sit down with WEKA’s Callan Fox, principal product manager, and Betsy Chernoff, AI product marketing manager, to discuss breaking through the "AI memory wall." The conversation centers on the newly announced availability of Augmented Memory Grid on NeuralMesh, a technology designed to solve the mounting economic and technical challenges of AI inference. Fox and Chernoff explain how the industry shift toward complex, multi-turn agentic AI creates massive memory demands that traditional DRAM tie... Read more

explore Keep Exploring

What are the current challenges and advancements in memory hierarchy for inference systems? add

What is the cost comparison between using a hundred GPUs and using 24 GPUs, and how does this relate to system throughput? add

What are the key announcements regarding the availability of augmented memory grid? add

What challenges were encountered regarding the performance and efficiency of the inference system in relation to the agents' communication and memory requirements? add

What are the primary use cases that WEKA addresses in relation to storage systems? add

bolt Powered by CUBE AI

Callan Fox & Betsy Chernoff, WEKA

search

Dave Vellante

>> Hi, everybody, and welcome back to St. Louis. We're here at SC25. This is day two of theCUBE's wall-to-wall live coverage of Supercomputing, the supercomputing event. This event, Jackie McGuire, my co-host. I'm Dave Vellante, started in 1988-

Jackie McGuire

>> Yes....

Dave Vellante

>> right? We've come a long way. John Furrier.

Jackie McGuire

>> I was born then, if it makes you feel better.

Dave Vellante

>> That was the year you were born, or?

Jackie McGuire

>> No, no, no.

Dave Vellante

>> Oh, okay.

Jackie McGuire

>> I was born in the '80s.

Dave Vellante

>> John Furrier is also here. He and I were, well, he was in college at the time.

Jackie McGuire

>> Yeah,

Jackie McGuire

>> Savannah was saying yesterday that it was the year she was born, so I wanted to make you feel better. I was already walking and talking.

Dave Vellante

>> I feel good. I sleep good. We're going to talk about breaking through the AI memory wall. We've talked a little bit about that this week, and we're here with a couple of folks from WEKA. Callan Fox is the principal product manager at WEKA, and Betsy Chernoff is an AI product marketing manager at WEKA. And you guys have announced the availability of augmented memory grid on NeuralMesh, so we are going to geek out about that. But basically, this is a memory extension technology that's designed to attack the economics issue around inference. So folks, welcome.

Betsy Chernoff

>> Thank you.

Dave Vellante

>> theCube really appreciates-

Callan Fox

>> Thanks for having us....

Dave Vellante

>> having you guys on. So Callan, let's start with you. I mean, what is augmented memory grid, why is it necessary?

Callan Fox

>> Yeah, so today what we find is that as inference gets more and more complicated, and people have more demands of it, they want it to be affordable, they want it to be efficient. For, as conversations become more multi-turn and get longer and longer contexts, we find that the memory hierarchy of the inference system is more and more important. But today, the memory tiers that support that memory hierarchy are lacking, they don't have enough capacity. So what we've done with augmented memory grid is, we've taken the durable advantages of WEKA, of WEKA's product called NeuralMesh, and we've plugged that into inference systems in a supported way. And what that allows us to do is take the memory tier of DRAM that exists today, as a common one, and augment that, extend it into our system. And allow much, much, much larger capacities in the memory tier, but at the same speed as DRAM.

Dave Vellante

>> So are you essentially creating a virtual pool of memory, using less expensive resource?

Callan Fox

>> So what it allows us to do is, we're based on NVMe, so we're much, much denser than what you can get in DRAM. But because of the way our system's designed, and all the optimizations we take in our core products, that we've battle tested in AI systems for many years and many HPC environments, many of our customers are here today. All of the learnings and the IP that we have out of that system, we've plugged that into the memory tier, to allow it to be quite a fair bit larger.

Jackie McGuire

>> You were mentioning economics, and Dave and I actually did a breaking analysis a couple of weeks ago talking about the economics of agentic AI. We were talking before we started, and right now, Dave and I share the opinion that I don't really think people are fully conceptualizing the cost associated with this.

Betsy Chernoff

>> Yeah.

Jackie McGuire

>> So how does this memory, this different strategy of memory, how does that impact the economics of AI, and agentic AI specifically?

Callan Fox

>> You good there?

Betsy Chernoff

>> Yeah. So there's a couple different ways, I think, that we truly impact the economics, but one of the ways that we've worked hard in order to be able to highlight this, because I think it's so easy to go through the benchmarks that we've done, it's so easy to understand some of this. But what we've really found is that, within the world of AI today, there's this idea of faster AI at lower costs. And those two are often opposed, they're not one and the same, and it makes it very difficult for customers to understand how we sit and how we help. But what we found is that there's a couple different benchmarks that are really impactful. We've come from a world, and we can go into more detail on this as well, but we've come from a world where originally, AI, people were just doing single shot prompts. So think, "What is the capital of Arizona?" Just one simple question, very much like a Google user coming in and asking a question, but where we've come to now is multi-turn. People want to know more and more. There's coding co-pilots, there's this world that is getting more and more and more, and we need it faster. But costs are continuing to also go up within this world, as well. And it's a struggle to understand all of this, but where we see this really impacting for us is time to first token, so reducing latency overall. And then also, within that, time to last token, or total token throughput. What we've seen within multi-turn, and concurrent, large context benchmarks that we've done, we've been able to get to 6x lower, or 6x faster, time to first token. But then in addition to that, we've seen total token output turn into 4.2x. So, increasing it overall. These are a bunch of numbers for everybody, right? What does this all mean? And what we really see at the end of the day is, if you think about an H100, it's around $30,000 for a GPU. If you have a hundred GPUs, that's $3 million. 4.2x reduces that cost to somewhere around $2.28 million. So $720,000, or 24 GPUs, instead of a hundred. And that's the context that we want to get to. So you can certainly think about this less AI, or less GPUs, but another way to think about this is more throughput. How much more can you get through this system? And that's really where augmented memory grid, and what we've done, comes into play.

Dave Vellante

>> And I heard Satya Nadella the other day, talk about that in the future, the brilliance of software is going to be... You know what I'm talking about, to understand when you need latency, when you need, he called it performance, but I think he meant, you take more time and do reasoning and the software's smart enough to figure out which way to direct that query, to more cost-sensitive or more latency-sensitive.

Betsy Chernoff

>> Right.

Dave Vellante

>> And so, you're supporting the economics of that sort of software architecture in the future, are you not?

Betsy Chernoff

>> Yes. Yes, absolutely.

Dave Vellante

>> Okay. So his point was, that is going to be the brilliance of where software is, which I thought was kind of profound. You have hard news, this week.

Betsy Chernoff

>> We do.

Dave Vellante

>> What is the news?

Betsy Chernoff

>> So there's two primary things that I think I'm particularly excited about, and Callan's particularly excited about, for augmented memory grid. The first one is the idea that it's now available on NeuralMesh, and this opportunity for us is incredibly exciting, because I think it helps democratize AI inference for customers that we have today. In addition to that, we're also announcing this commercial availability on OCI, Oracle Cloud Infrastructure.

Jackie McGuire

>> Awesome.

Betsy Chernoff

>> And so, equal parts, we're so excited to announce that it's available on NeuralMesh, but we're also thrilled to announce that it's also available through OCI.

Jackie McGuire

>> Awesome.

Dave Vellante

>> All right, so Oracle's making moves in AI. So coming back to the techniques that you guys have applied to get to where you are, to attack this inference economics problem. Where did you come from? What was that journey like? What were the learnings? How difficult was it? How long did it take? What can you share with the audience?

Callan Fox

>> Yeah, maybe I'll take that. We have had a long journey to get here. We started off initially by really wanting to understand the problem. Like many people, we looked down the barrel of various leaders that say, "Hey, many organizations are going to have many, many agents supporting real workers." And we started to think about, "Well, how are the economics of that going to scale?" And the thing that we noticed is that as you scale the number of concurrent users that you need for an enterprise, unless you have a significant memory, they start to conflict with each other. So the system, it starts to not be able to be efficient.

Dave Vellante

>> It comes out of balance.

Callan Fox

>> Right, because you're basically having, you're overrunning what your system can support. So in a world that we're heading towards, and even really a world that exists today for the bleeding edge customers, we have this problem where the amount of agents that they need to support is decoupled from the compute power that they have. And so, we need to sort that. But as far as how we got here, from a testing perspective, we went all the way back to basics. So we have been collecting billions and billions and billions of tokens of our own usage in AI, too, for coding use cases and all sorts of other agentic use cases. And that allows us to really fundamentally understand how agents actually consume the inference engine themselves, and how they consume context, and that allows us to do much more sophisticated benchmarking. So we came from a world, which Betsy mentioned, where we started with single prompt performance, and we found that the biggest problem was actually not our product, it was the inference engines themselves, and the overheads they provide. So while we could get an industry-leading time to first token number, our component of it was only like a fifth of that time. The rest of it was overhead of the inference system. And then as we learned more and more about how the agents actually work, and how they communicate with each other, and how one person, like a human being may actually have 10 or more agents working together, we could start to model the amount of memory that inference system needed to support that workload. And that's what we saw. The DRAM, that you can get in these systems, it's just not sufficient. So that's where we really started focusing on, "Okay, how do we solve this problem? Can we get the performance that you need to keep a GPU fed, so that it's getting optimal efficiency?" And that's what we've been focused on for the last six to nine months.

Jackie McGuire

>> It may be worth explaining, so what happens when you don't have enough memory? Because this may not be something people were fully familiar with. So if you're using an agentic system and all these agents are passing information back and forth, doing these kind of segmented complex tasks, what do they do when they don't have enough memory?

Callan Fox

>> So how it works is, most people when they're talking to a chat application or an agentic application, most of those today are largely language models, and they're largely transformer based. And transformer models have two distinct phases. There's a pre-fill phase, and a decode phase. The pre-fill phase is very compute-dominated. So the power, or the flops, that a GPU can give is very important to that phase. Whereas the decode phase is very memory bandwidth focused. So they're two fundamentally different ways you optimize for those things. And that's why Nvidia has done things, like with Dynamo, for example, where they've separated those two so they can optimize for those separately. So what happens is when you send a prompt in, regardless of its length, it has to go through this pre-fill phase to prepare those tokens for processing. There's an optimization technique that comes out of that called the KV cache, and you can keep that KV cache persistently across turns of the conversation, and it will prevent you from having to do that pre-fill phase again. To give you some numbers, so when we look at, we have a data set that's measured in the tens of billions of tokens. If I take the average cache hit rate across the board, we can get a cache hit rate of 95, 95.5%. That doesn't mean much to humans. But if you think about that a different way, that means on average, every single token in that entire data set, without that caching, is getting computed many, many, many times over. So if an inference system, for example, was only able to have the cache live for one minute on that data set, you would be recomputing those tokens 16 times on average. If you could hold that entire thing for five minutes, that would go down to four times. You can hold it for an hour, two hours, three hours, we can get all the way up to days even, you can get down to one. So you only have a pre-fill once, and you can decode as many times as you want off that token. And that's where we're getting to, because that's the ultimate efficiency you can get to.

Dave Vellante

>> So low KV cache hit rates are an economic killer. Is that right?

Callan Fox

>> Massive economic killer.

Dave Vellante

>> Okay. And so, what you are doing is, you're increasing those hit rates by elongating the persistence window, if I can call it that.

Callan Fox

>> That's right.

Betsy Chernoff

>> Correct. I think what's fascinating about the system in particular is, there's two components within the GPU of memory. And to Callan's point, one hits it, it's high bandwidth memory, it's incredibly fast in the system, but there's also just not a lot of it. And then, as soon as that KV cache gets full, it's got to go somewhere else, so it ends up hitting the system memory or the DRAM. And then, DRAM is also finite, it's not very large. So as soon as that goes away, that's really where we come into play with augmented memory grid. We are that persistent layer of memory, if you will. And while those two may be ephemeral, or not be persistent, we provide that persistence in a way that's incredibly meaningful. So those systems typically have about terabytes, and we can move all the way up to petabytes.

Callan Fox

>> And to put terabytes in perspective, that's millions of tokens. That sounds like a lot, but an eight-way, say a Hopper class even, DGX system. You can pre-fill that in minutes. So that one terabyte of DRAM, potentially, is only minutes of cache that you're constantly cycling through very fast.

Dave Vellante

>> Yeah. So I'm curious as to how you... Did you see this coming? I mean, of course you did, that's why you've been working on it for a while. But when you listen to Jensen, he'll tell you six months, six months before we saw reasoning. He's like, "Oh, these systems are going to start reasoning." You're like, "Oh, that's going to be interesting." Or, "They're going to have memory." And so, how did you predict this? What kind of telescopic window did you have? And I want to know what's next, problems that you're working on.

Callan Fox

>> Maybe I'll start with that. So the main thing is all the research we've done, and for people that are listening at home, they can go check out our blogs. We have a lot of this publicly available, we've open sourced all the ways that we do our benchmarking suite, so people can replicate what we've been doing. We'd be very open with the community. From a benchmarking perspective, it's been so important for us to understand how the actual agents use the cache, because... I actually forgot what I was saying. We'll scrub that.

Dave Vellante

>> Well, I was asking you about what's next, what kind of research visibility do you have? How did you predict that this was going to be a problem? When did this project start? I mean, it was a couple of years ago, right?

Callan Fox

>> Yeah, so this project started about a year ago or more, at this point. We saw this coming, we saw the economics. Basically, what we saw is that the amount of floating point operations you can get out of a GPU is increasing, increasing, increasing. But the demands of an inference system are changing, I shouldn't say the demands, the expectation of an inference system are changing. People want it to be efficient, and to some of the science users around here, that could mean getting more science out of the same box. To an inference provider, that means making more money out of the tokens they sell. But the flops are important, but how you use them is more important. So we saw the fact that the way things were scaling, unless the pre-fill phase could be significantly reduced, as the context lengths increase of models, it was just going to become a runaway train. You can never compute enough to make it efficient. And that's why we've been focusing on that.

Dave Vellante

>> So I mean, it was relatively straightforward to predict that inferencing was going to become a thing.

Jackie McGuire

>> Yeah.

Dave Vellante

>> We predicted that, even pre-Gen AI. But you also had to understand that the KV cache, in and of itself, wasn't going to solve the problem. Actually, it was going to create the problem.

Betsy Chernoff

>> Right.

Dave Vellante

>> So was it, when you saw that you said, "Oh, we happen to have a solution," or you started to work on the solution? And I'm very curious as to what's next.

Betsy Chernoff

>> I think this also probably primarily falls under what we do at WEKA. We solve for bottlenecks today, within the storage system. So that's always been something that has been a primary use case for us. And as inference, and if you think about the world, we're still somewhat in this phase. But think about 2024, people were looking at POCs trying to get them deployed for inference, and it just was near impossible for people to get them up and running in a meaningful way. And as we started to look into the system, and look into those bottlenecks, and see where we could help from a storage specific mindset, we really saw that gap within the KV cache, in that area. And so this fundamentally has been something we've been focused on, and focused on where we're going. I think I probably take it a little bit from a different perspective. We just got the opportunity to announce not only that augmented memory grid is included with NeuralMesh, but now that we've got OCI, and we're partnering with them via their marketplace. I think about what's next for us in that context. We know, today, that inference is still incredibly difficult. People are still struggling to get their POCs into deployments. And so for me, partnering with OCI is such a wonderful opportunity for us to be able to help democratize inference, and make it more meaningful across the board for these customers. So that opportunity, in and of itself for us, really means leveraging their bare metal shapes. And then, being able to leverage their support of not only RDMA, but also GPU direct storage. And because of that, we can provide inference at scale for these customers that is so much more meaningful than what they have today.

Dave Vellante

>> I'm glad you brought that up, Betsy. There's two takeaways from what you just said. First of all, your superpower is reducing, removing bottlenecks. That's what you guys do.

Betsy Chernoff

>> Right.

Jackie McGuire

>> Yeah.

Dave Vellante

>> Okay. And then the second, I'm glad you came back to OCI, because when you think about Oracle, you're talking about mission-critical, the biggest banks, manufacturers, insurance companies. The most mission-critical applications and workloads, bringing inference to those customers, is really the opportunity.

Jackie McGuire

>> You bet.

Betsy Chernoff

>> Right, a hundred percent. And then if you think about, we're talking about faster AI at lower costs. I mean, these are all very basic ideas or understandings, but really for these banking industries, for healthcare, life science, for all of these industries that are struggling today to do this in a meaningful way, this is their opportunity to be able to take advantage of faster AI for lower costs at inference scale. And I think we're going to see a lot more of that when we talk about agentic, agentic swarms for 2026. Those are the things I'm most excited to see, what comes out of that space.

Dave Vellante

>> And the interesting about Oracle is, and it's ironic, they've got the best multi-cloud strategy in the business. They have a really strong, if not the strongest, on-prem and hybrid strategy, because it's the same, same. You can get real application clusters now in every single hyperscaler, plus OCI, and you can get it on-prem. So it's ironic, because when cloud first came out, I don't want to say Larry poo-pooed it. He was just like, "It's operating systems and databases and storage, and servers, and compute. No big deal." But now they've got that strategy in place, so it's a fantastic distribution channel for you guys.

Betsy Chernoff

>> Well, and you talk about the future, and how we were able to predict it. I see that with OCI as well, right? I think they had a clear line of sight within AI, and how they wanted to meaningfully bring it to their customers, and they've been able to successfully do it. And now, we get to be a part of that.

Dave Vellante

>> Yeah, Oracle's going for it. All right, we got to go. Betsy, Callan, thanks so much.

Betsy Chernoff

>> Thank you so much.

>> Working with -

Jackie McGuire

>> Thank you, guys.

Dave Vellante

>> It was great having you.

Callan Fox

>> You too.

Dave Vellante

>> Good conversation. All right. I'm Dave Vellante, for Jackie McGuire and John Furrier. You're watching theCUBE's live coverage of SC25 from St. Louis. We'll be right back, right after this short break. Stay with us.