SC25 | Shimon Ben-David, WEKA & Dion Harris, NVIDIA

Clips
More from SC25

Shimon Ben-David

CTO

WEKA

Dion Harris

Sr. Director, HPC, Cloud, and AI Infrastructure GTM

NVIDIA

play_circle_outline Revolutionizing AI: The Emergence of Factories for Large-Scale Intelligence Production Powered by Data and Energy

play_circle_outline Importance of optimized data pipelines for efficient GPU utilization and performance.

play_circle_outline Discussing KV cache acceleration and benefits for AI inference environments.

play_circle_outline GPU direct storage minimizes latency by allowing data access without CPU involvement.

play_circle_outline Exploring AI Factory Applications: From Chatbots to Genomic Research and Smart Cities in Biosciences and Robotics

play_circle_outline Continuous feedback loops in AI systems optimizing training and inference processes.

Info
Transcript

Shimon Ben-David, WEKA & Dion Harris, NVIDIA

Shimon Ben-David

CTO WEKA

Dion Harris

Sr. Director, HPC, Cloud, and AI Infrastructure GTM NVIDIA

In this interview from SC25, Shimon Ben-David, chief technology officer of WEKA, and Dion Harris, senior director of HPC cloud and AI infrastructure at NVIDIA, join theCUBE’s John Furrier and Dave Vellante to unpack the architecture of modern AI factories. The discussion defines these factories as systems that convert energy and raw infrastructure into intelligence, emphasizing the critical role of data pipelines in driving ROI. Harris and Ben-David detail the "extreme co-design" partnership between NVIDIA and WEKA, highlighting how storage has evolved from a... Read more

explore Keep Exploring

What is the concept and function of a modern-day AI factory? add

What factors should be considered to optimize the performance of data infrastructure in AI environments? add

What are the key components and benefits of KV cache acceleration and its role in improving AI inference environments? add

What is the purpose of GPU direct storage and how does it function in bypassing the PCI CPU complex? add

What are some notable use cases and broad market opportunities resulting from advancements in AI efficiency? add

What is the significance of the ecosystem's value in relation to existing infrastructure and AI workloads? add

bolt Powered by CUBE AI

Shimon Ben-David, WEKA & Dion Harris, NVIDIA

search

John Furrier

>> Welcome back everyone to theCUBE's live coverage of Supercomputing in 2025. I'm John Furrier, host of theCUBE with Dave Vellante, bringing out all the action and AI infrastructure and all the enablement. The tech is being built out ushering in an accelerated computing wave and value in this session, building AI factories at scale to drive ROI. We've got a great lineup here Shimon Ben-David, CTO of WEKA and Dion Harris, Senior Director of HPC Cloud and AI Infrastructure at NVIDIA. That's like the super computer market, Dion. That's how you class it. , just saying to the senior director of SuperComputing NVIDIA.

Dave Vellante

>> That's right. That's right.

John Furrier

>> Great to see you guys. The AI factory conversation really is exploding since two years ago when you guys were really putting it out there that this is what large-scale systems look like at GTC. Since then, just accelerated set of announcements, architectural designs, the role of networking, the role of storage, the density, all coming together again this year, liquid cooling, energy utilization, getting those all top of mind. So the AI factory is a thing, and so congratulations. So what is an AI factory? How would you guys define it in the sense of the current state of AI factories, still big build-outs, a lot of edge, more storage, how would you describe it?

Dion Harris

>> So I would describe the modern-day AI factory is really trying to tackle a problem at data center scale, and so when you think of a factory itself, you're taking power raw materials and out of it you get an end product. With an AI factory, you're taking energy and of course the core infrastructure and out of that, you produce intelligence and intelligence in the form of LLMs, intelligence in the form of AI, surrogates, intelligence in the form of all these different recommender systems. And so this capability of producing intelligence is truly unlocking every industry, every scientific domain. And so by the broad reach an AI factory in terms of scale, you mentioned it before, it might've been tens of thousands of GPUs or endpoints. Now we're talking hundreds of thousands of millions. And then of course, you're understanding that it all needs to behave as a single system. So that includes the compute, the networking, and of course the storage.

Dave Vellante

>> I'd like to add data to that recipe.

Dion Harris

>> And that's why put storage, exactly, yeah.

Dave Vellante

>> Energy and data in and intelligence out.

Dion Harris

>> You're absolutely right, yes.

Dave Vellante

>> But what role does data play?

Shimon Ben-David

>> So eventually, all of these GPUs, all of these tokens are generated by simulating a large amount of data into intelligence, into tokens. It's all about feeding the GPUs with data so they can be optimized in terms of taking decisions and generating tokens. And actually that's what we do with NVIDIA. We're creating these joint collaborative designs where a customer should just consume this factory and work with it and generate tokens without thinking about the underlying factory underneath.

John Furrier

>> Talk about the relationship with WEKA and NVIDIA because storage is already hot now, but it's going to get hotter we believe. We've been looking at networking first as that operating system connector brings the back walls together, that's well documented on theCUBE and SiliconANGLE, of course. Storage is right behind it. It's sitting closer and closer. More data's coming in. This is going to be a key design point and engineering focus for NVIDIA. They do co-design, extreme co-design, as they say. I love the word extreme. What does that even mean? But we'll get into that later. Talk about the relationship and why that's important now and how that storage piece or data layer in the factories matter.

Shimon Ben-David

>> So eventually it's all about the customer having an optimal decision. They're making an investment on GPUs, on infrastructure to generate their tokens, to generate AI outputs. Eventually, if the environment is not optimized, if you have multiple data pipelines, if data is moving around in an unoptimal way, these GPUs are sitting and waiting for that data. Also, if we're looking at AI environments, there's multiple different stages of AI, analytics environments, training environments, inferencing environments, and each of those have their own data requirements, whether they're ingesting massive amounts of data. And then how do you feed the data directly to the GPUs in an optimal way to create more models, more data, and then how do you change that data? And also, when you are building these inference environments, they're composed out of multiple components, vector databases, graph databases, multiple models. How do you make sure that they're running and loaded and unloaded in an optimal fashion? So the data infrastructure is an underlying accelerator for all of the AI environment. And what we're seeing recently actually with AI innovation with inferencing is there's a way for storage, or I would say for memory storage to actually accelerate inferencing even further with a, I'm sure you heard the term KV cache acceleration.

John Furrier

>> Yeah. So KV cache, all this NVLink stuff makes it all happen. Dion, what's the storage version of NVLink and MV cache? Is there something, because that has changed the game. You see how the density matters, certainly NVLink and KV cache killer components. What's the storage version? Because when the models come in, you're going to have a lot of, I won't want to say policy, I'll put that in air quotes, but this really AI, it's got to go to work because you want to maximize the memory and the volume of data coming in that you can throw into storage. That's not just a old school storage methodology. It's got to be modernized. What's your view of storage fabric?

Dave Vellante

>> In the persistence of KV cache so you can get a high cache hit rate? That's right.

Dion Harris

>> And when you look at the models, whether it's a agentic workloads, whether it's deep research, there's a huge requirement on data and being able to serve that data to adequately and quickly service the request. And so we've basically taken a very thoughtful approach of looking at not just the compute, obviously the networking and how you connect the GPU together, but also how do you expose data that's in the storage. And so we introduce something called Dynamo. And Dynamo is really about delivering AI inference at scale, but across the entire tiered memory hierarchy. In other words, you have data that sits in the CPU. We have our MB-Link C to C, which allows data transfer from CPU to GPU. You also have, of course the memory that's sitting in the GPU itself. You have system memory, but you also have off-memory or off-system storage. And so the great thing about our Dynamo platform, it allows you to have the intelligence to understand where that KV cache lies, and you can instantly restore that as opposed to recompute it. And so when you're interacting with your LLM model or you're interacting with your overall AI use case, it allows you to have that persistence of not just your existing interaction, but historical interactions to make sure your experience is very rich in intelligence. And instead of having to reissue that recompute that every time.

John Furrier

>> So I have to reflect-

Shimon Ben-David

>> Maybe I'll give you an example. Oh, sorry. Maybe I'll give an example of how tangible it is. What's a validated outcome? So we've been doing some work at Coreweave running such an environment, a WEKA environment Dynamo in a real KV cache use case. So imagine a code developer writing code ingesting massive amounts of data into the model. So to create the next code generation. And obviously the more code you are generating, the more data it requires, the more memory it requires. And then if that developer is now stepping out for a few minutes, that's very tangible. That's something that happens constantly. When that developer steps back in, do they continue working immediately or do they need to take a few minutes or seconds to continue? That's a big difference. So a tangible acceleration of code environments is something chatbots LLMs that are just running faster. It's the token generation, but it's the human outcome that comes afterwards. You're just able to work faster without waiting by spreading that KV cache in an efficient manner.

John Furrier

>> So last year at GTC, Jensen was on stage he said in his exciting bravado that he has, "KV cache," he said this, "KV cache is the operating system for AI factories." I'm like, "Oh yeah, that he said it." Now I should be corrected, it's Dynamo-

Dion Harris

>> Yes, that's right....

John Furrier

>> That should be a subset because what you're saying, and you guys are pointing is that KV cache is like, of course he's geeking out on why that's great but it's Dynamo. That's more the operating system just just taking in, not just the networking piece, but also all data layers.

Dion Harris

>> Yeah. And so through Dynamo, we've exposed a new protocol. It's called NIXL, which is NVIDIA Inference Transfer Library. And that allows us to expose this hierarchy to our storage partners like WEKA. And they're able to then immediately have this integration across that full orchestration, like you said, Dynamo being the operating system of AI factory. So that's how we expose that and they're taking full advantage of it.

Dave Vellante

>> And you're extending that memory essentially. .

Shimon Ben-David

>> Exactly. So when we had that idea of how do we implement KV cache efficiently, we had that notion that if you look at GPU memory, I'll geek out for a second.

Dave Vellante

>> Yeah. Go ahead.

Shimon Ben-David

>> GPU memory is nanoseconds, WEKA storage is microseconds, usually storage is milliseconds and seconds. So how do we bridge that gap? And then we understood that the improvement function is not nanoseconds to microseconds, but it's actually microseconds to recomputation of the KV cache. So then working with NVIDIA, working with Dynamo and NIXL frameworks, we actually were able to intelligently utilize that to data across different GPUs. I do want to also say that Dynamo, you call it the operating system of AI, which is brilliant because it actually takes the AI developers from, I'm doing a POC on my laptop, on my 234 GPU servers to how do I do AI at scale? And there's a big difference on how do you POC and test an environment, then how do you operate it at scale? There's different constraints, the LLM routing, the KV cache movement, it's a completely different topic.

John Furrier

>> I'll make sure Jensen makes a note of that next time.

Dave Vellante

>> If you can correct Jensen, okay, yeah.

John Furrier

>> If you watch those two keen Gene at GDC. Make sure you say Dynamo.

Dave Vellante

>> But the point is, if you can get a high KV cache hit rate, you don't have to go through the whole cycle again.

Dion Harris

>> Exactly.

Dave Vellante

>> And if it goes from 95% to 50%, that's expensive. So if you can extend that persistent window, if I can call it that, that your cost declines.

Dion Harris

>> And I mean it speaks to when we think about performance, I mean obviously we're known as a hardware company, but NVIDIA's always looking at ways to leverage software to introduce efficiencies and speed up because then you get exponential X-factors and so that's the situation you're seeing where we introduce technologies, and of course the hardware is the start of the platform, but a lot of the innovation, a lot of the efficiency comes from the software integration.

Dave Vellante

>> Well, we've heard this thing called CUDA all the lives that we're seeing there. I think that's software. That's right.

John Furrier

>> It runs on everything. How does all that speeds and feeds that you guys just went through, which is awesome. How does that translate into some of the challenges in the market? Because also you got some efficiencies you guys engineered co-design with NVIDIA around KV cache, which is the core jewels right now that dynamo, how does that translate into value? Is it GPU utilization? Is it more software intelligence in AI native applications? Is it just efficiency? Where's the rubber meet the road on this?

Shimon Ben-David

>> Maybe I'll start with a few topics. So we're talking about KV cache acceleration, and again, if you can or that's one of the topics, if we accelerate inferencing 4, 10, 20, 40 times faster, imagine that the amount of tokens that you can generate and the amount of outcomes. But it's not only about KV cache acceleration. Inference environments are very complicated. As I mentioned, they have many moving parts and their ability to work, to collaborate and create joint designs where storage is not a problem, AI is not a problem. You just get an outcome is very powerful for our customers as a result. Environments that are saying that they're doing 30 times more science, PhD finishing their designs faster. I think you had a really good example also.

Dion Harris

>> So when you described, like you said, the integration that happens, of course, like you mentioned KV cache is key, but there's also RAG, receive augmented generation. In fact, last year when we were here, we were announcing WARP, which described our integration to help facilitate RAG workflows where you're taking proprietary data and you are using that to boost the intelligence, reduce hallucinations. And so that's a core part of the data pipeline that is actually key for AI. But even extend beyond AI, in fact, we're at Supercomputing. In 2019. We introduced something called GPU direct storage, which was basically a way to bypass the CPU complex and have the data served directly through a PCI switch to the GPU to make that transfer process seamless. And what was incredible about that, and it really crystallized the importance of this data pipeline, there was a researcher that was basically demoing a NASA Mars lander. And in fact, it was basically serving up 150 terabytes of simulation data. And I remember it struck me because the researcher during the walkthroughs literally was in tears because she had saw her Mars lander simulation visualize and realize that she was never able to do before. And so it was really through this tight integration in this case with our GPU direct storage, integration of course with WEKA-IO, but really enabling, like I said, researchers to unlock these capabilities that just didn't exist and therefore advancing science.

Dave Vellante

>> And what exactly does GPU direct storage to?

Dion Harris

>> Yeah, so basically when you look at how NVIDIA built out our architecture first, we said, "How do you leverage RDMA?" So basically how do you bypass the PCI CPU complex to allow GPU to talk directly to each other and that deliver more performance? So that was GPU direct RDMA. But then we also recognized Azure serving data from the storage device. When you typically do it routes up through the PCI switch, goes through the CPU separate system memory, then it comes back through the PCI switch-

Dave Vellante

>> And you're eliminating all that overhead.

Dion Harris

>> So GPU direct storage eliminates that allowing direct access to data from the GPU to the PCI switch. So that delivers much better overall return time for those data serving use cases. And so having this integration, and we exposed something called CuFile, which is an API that allows WEKA to then take advantage of that and serve that data directly. So that's another example where it's really about not just the hardware, but how do you expose it through the software.

Shimon Ben-David

>> And by the way, I'll throw some numbers in. So just to understand the magnitude of the importance of GPU direct storage, imagine a amount point that works at 10, 20, 40 gigabytes per second, that's now accelerated to more than 300 gigabytes per second and more than Two million IOPS. That's how important GPU direct storage is.

John Furrier

>> What use cases are most exciting for you guys in this breakthrough? Because I think the efficiency is going to enable, is it broad market opportunity or is there use cases that jump out for the benefits of that performance?

Shimon Ben-David

>> It's a very interesting question. We've seen an adoption of a few very interesting use cases. Everybody's exploring AI from chatbots to RAG environments, semantic searches, and just raw inference accelerations. We've seen use cases such as video search and summarizations. We're actually working with our partners at NVIDIA on that as well. We're imagine smart cities, smart factories, security environments. We've seen a lot of biosciences, AlphaFold, OpenFold, running genomics research, running medication research. We're now seeing more and more physical intelligence where robotics environments are being generated, simulations of robots and actually operating system for robots. All the-

John Furrier

>> So you feel that the use case of physical AI, that you guys are in good line with WEKA and NVIDIA because smart cities, that's basically like an AI factory. I mean, it'll be a mini or a micro factory.

Shimon Ben-David

>> Smart cities would eventually incorporate everything. It start with video streaming. It'll continue with the NLP and NLU with speech recognition.

John Furrier

>> I mean, we've been saying it's going to be a hyper-converged edge factory with maybe a DGX box or a mini factory bundled into wireless. Why wouldn't we do that? Then have a big factory and then micro factory.

Dion Harris

>> So some of the work we're doing with WEKA, so we've talked about the KV cache management, we've also talked about GP direct storage. There's also something we developed called the AI data platform, which is essentially putting AI with the storage device itself, allowing you to interact with your data. So there's a whole new class of storage devices and platforms and solutions that are now powering AI, but also leveraging AI to power data access. So it really is this continuous virtuous cycle, if you will, of data feeding AI. But now ADA AI also accelerating and feeding and presenting data.

Dave Vellante

>> And when I think about what you guys are doing, so NVIDIA obviously does some amazing work, systems company, hardware, software, the whole system, AI factories, but you still leave a lot of white space for companies like WEKA to add value. That's the beauty of it. And you're doing things with KV cache, CPU direct, AI data platform, that's new innovations. These extend the value of the GPU. So my question is whether it's, let's say it's a training infrastructure, does it become tomorrow's inference or other use case infrastructure? Is that how we should be thinking about this because of the value that the ecosystem is adding on top of the existing infrastructure? Is that the right way to think about it?

Dion Harris

>> Again, think as you look at the workloads across AI, like you mentioned, there's data analytics and processing, there's pre-training, there's post-training and then there's this whole feedback loop that happens. And so we're starting to see those workloads and workflows converge. And having centralized and consistent persistent data sources becomes very key. And it provides a lot of opportunity, like you said, for the ecosystem to add value on top of the computing platforms that we build. And so when we approach the solutions, obviously we're thinking broad-scale data center scale. We were also recognizing that it requires partnership across the ecosystem to deliver that end-to-end integration.

Shimon Ben-David

>> And the reason-

Dave Vellante

>> For my question....

Shimon Ben-David

>> Funnily enough... Sorry.

Dave Vellante

>> No, please go ahead.

Shimon Ben-David

>> I'll give my view as well. Well, we are seeing that consolidated to some extent because eventually the most efficient a customer can go is with a single infrastructure that can do most of the things. If we're looking at training and inferencing. In the past there were completely different designs, different GPUs, different connectivity in the networking as well. Today when we're seeing Neoclouds adopting, Sovereign cloud adopting GPUs and networking environment, we're seeing the designs are being more consolidated. And actually something that we're doing with some of these customers is enabling to hot swap between inferencing and training on the same environments. So just-in-time inferencing, just-in-time training.

Dave Vellante

>> So this is so important because I think people are all very much concerned about the useful life of I want a GB300, so nobody wants a GB200, we joke. But of course that's not the case because the workloads are so diverse. If the cloud's taught us anything at that particular moment, I want to get something on demand and consume it and then go on to something else. The useful life of that infrastructure is I think, much longer than most people realize.

Dion Harris

>> Absolutely. Absolutely. And I mean just to drive that point home, I think when you look at the life of a GPU or even just an accelerated platform, there's a couple of transitions that are happening. The first is CPUs are moving to GPUs. So there's a whole list of classic simulation use cases and applications that are being ran on GPUs. And so what you're starting to see is as those applications to workload migrate to GPUs, they're looking for any and everywhere to land those applications. So even if it's not on the latest and greatest platform that's driving a lot of the AI workloads H100s, A100s are absolutely driving a lot of the core IFV simulations that are happening. They're driving a lot of the core genomic simulations. So the idea that you're getting utility across all of your data analytics platforms, your simulation platforms, your AI platforms, of course, we're here at Supercomputing, we're seeing Quantum. So the utility of the platform allows utility throughout all the various works stages and workloads that are being ran in the data center. So that's the first thing. But then the second thing is when you look at the generation over generation improvements, so when we went from Hopper to Blackwell, we've recently demonstrated a 10x performance per watt on an actual measured workload. So this isn't a PowerPoint slide. This isn't based on specs. And the reason we're able to do that is through this tight integration, this extreme co-design of both the compute, the networking, and of course the storage, making sure this all works together. And so by creating those generation over generation leaps, it's adding tremendous value, but it creates an opportunity to then go redeploy that hardware for any other workload that's being migrated to GPUs. And so there's a lot of opportunity that we see happening there.

Shimon Ben-David

>> Yeah. Even in the AI pipeline, we're always thinking of in terms of inferencing and training, but there's so many more details around it. There's embedding, there's vector indexing creation, that's GPU acceleration. There's so many details that you can utilize different GPUs, even older GPUs for your current needs. Also, we were talking a lot about KV cache. What if you could make older generation GPU get closer to a newer generation GPU? Obviously the newer generation with KV cache is accelerated as well, but you can get more by utilizing different infrastructure things than you would've without.

Dave Vellante

>> And the monetization opportunities for not only AI factory operators, but enterprises that are now just starting to figure out AI, they're virtually endless. They're going to require a lot of diverse infrastructure. And the fact that there's consolidation of the architecture, I think is a very positive for the industry. And I think Michael Burry is going to get burned.

John Furrier

>> Guys, we are running out of time, but thanks for coming on. And what I learned from this session is I got a shout-out from NVIDIA's LinkedIn handle. They quoted me by, I said, "Networking is the future of AI." And it's a nice quote but here what I learned here is networking and storage are one thing, just like HPC and AI is the same thing, it's like they're hand in hand because it's so close. Networking is fundamental to moving the data around, it's not just old school switching, send a packet from point A to point B.

Shimon Ben-David

>> Generation networking storage.

John Furrier

>> So I think that's a really cool walk away. Now I want to get some content from you guys while I'm here. I wrote on LinkedIn, the things I'm looking for. Is sovereignty real and what does multi-party ecosystem look like? And is AI factory a living network?

Shimon Ben-David

>> I'll start with sovereignty.

John Furrier

>> Yeah.

Dave Vellante

>> Yeah.

Shimon Ben-David

>> Sovereignty is a real thing. It's something that evolved organically. We're seeing a lot of requirements for GPUs worldwide, obviously. And there's multiple ways to consume it, whether it's the hyperscalers or the neoclouds. But recently we've been involved and we're seeing worldwide a lot of environments that they understand the power of AI and they understand that it's a tactical or strategic advantage for the nation that they're in. So actually, a lot of the sovereignty is demanded by the ability to not take the data out of that nation. So we're seeing multiple sovereign cloud. Sovereignty is a reality, it's already exists in multiple locations.

Dave Vellante

>> That's not going away.

Dion Harris

>> Well, and I just to add to that briefly, I mean beyond the data sovereignty issues, there's also just the ability to use AI and leverage AI to drive economic growth and prosperity. And so most countries recognize that this is an opportunity and so inherent in their AI sovereignty strategy is how do I leverage it to really make sure that they're not left behind. And so we're seeing a lot of investment just like people invested in telecom, just like people invested in the electricity infrastructure, this is the same opportunity to build out.

John Furrier

>> So it is a living network because intelligence is like electricity and you're going to have edge hyperconverging, almost like a grid configuration, connecting, talking to each other.

Dave Vellante

>> It's critical services for our country.

Dion Harris

>> Exactly.

Dave Vellante

>> Better have that in-house.

John Furrier

>> Contributing to my wish list of what the fish out of the show this week. Thanks for sharing that. Oh, it's really important. I mean, this is your questions. Thanks for coming on. I really appreciate it.

Dave Vellante

>> Absolutely. Cheers for having a conversation, guys, thank you.

John Furrier

>> I'm John Furrier with Dave Vellante here at Supercomputing. So much change is happening that's going to enable whole new use cases, national infrastructure, sovereign cloud, sovereign AI, and a new network and data architectures are emerging, accelerating really, really fast. We're doing our best to keep up. Thanks for watching.