SC25 | David Noy, Dell Technologies

Clips
More from SC25

David Noy

VP of Product Management

Dell Technologies

play_circle_outline Advancements in AI Data Processing: Dell Technologies’ PowerScale and NVIDIA’s KV Cache Innovations Since 2012

play_circle_outline Collaborating for Innovation: Dell's Storage and Server Teams Optimize AI Solutions with KV Cache Integration for Enhanced Performance

play_circle_outline Dell's AI factory as a high-performance, energy-efficient solution for AI infrastructure.

Info
Transcript

David Noy, Dell Technologies

David Noy

VP of Product Management Dell Technologies

In this interview from SC25, David Noy, vice president of project management at Dell Technologies, joins theCUBE’s John Furrier and Jackie McGuire to discuss the critical role of storage performance within the Dell AI Factory for NVIDIA. Noy details how "extreme co-design" between storage and server teams is optimizing data delivery to GPUs, specifically highlighting advancements in PowerScale and ObjectScale. The conversation focuses on the necessity of speed and efficiency, with Noy revealing that PowerScale performance has doubled in the last 18 months, wi... Read more

explore Keep Exploring

What advancements has Dell made in AI technology, particularly in relation to storage solutions and performance improvements over the last 18 months? add

What are the strategies for improving performance in server systems? add

What challenges do companies face when building exabyte scale storage environments in relation to energy and space efficiency? add

bolt Powered by CUBE AI

David Noy, Dell Technologies

search

John Furrier

>> Welcome back everyone to theCUBE's Live coverage here at Supercomputing 2025. I'm John Furrier, host of theCUBE with Jackie McGuire, Industry Analyst with theCUBE Research. This next session is Supercharging KV cache with NVIDIA, Dell, PowerEdge, PowerScale, and ObjectScale. David Noy is here, Vice President of Project Management with Dell Technologies. David, great to see you. Thanks for coming on.

David Noy

>> Thank you for having me.

John Furrier

>> First shout out for PowerScale, your baby since 2012 on the product side. Congratulations in the middle of all the action now.

David Noy

>> Absolutely, yep. Innovating fast.

John Furrier

>> So props to you and the team. Stories obviously we've seen the evolution, we've covered on theCUBE since we've been 16 years covering it, but now more than ever with NVIDIA, KV cache, Dynamo, it's becoming an operating system element of the AI factory.

David Noy

>> That's right.

John Furrier

>> And everybody wants FirstToken out, then Jensen put out those scaling laws. Now we're into thinking after reasoning, so it's like high performance all the time. They want more, more, more speed. Where do you guys sit in this? Explain how you guys are working in that construct.

David Noy

>> Well, as you know, Dell has AI factories, so it's a Dell AI factory for NVIDIA. Very popular solution. It's a holistic solution that includes servers, networking and storage. I focus on the storage part, which is how do we get that data to our customers' GPUs as fast as humanly possible. And to, you're right, it's all about speed. Look, in the last 18 months, brought up by baby PowerScale, we've improved the performance of PowerScale 2x in the last 18 months, and we've got line of sight to improving that performance by doubling it again over the next 18 months.

John Furrier

>> I mean, we're hearing 2x, 10x is on the NVIDIA side. You're starting to see things like the KV cache, Dynamo, the switches inside the scale up in these large systems and more data is coming in. There's only so much memory you can put stuff in.

David Noy

>> Yeah, that's right.

John Furrier

>> And the models are big, but there's also a small model. So you have all these nuances around, I call it the system software challenge. How do you manage it? What's the software? How does that work? Is it close to the chips?

David Noy

>> Well, there's a lot of things that we can do to speed up performance, and it's a combination of factors. We look at how do we make our systems run better on the servers that they run on because storage is just software running on servers with media. So we look at holistically as a system, to your point as a solution, how do we go faster? And then there's things that we can do in the software itself. You brought up KV cache, great example. So maybe we'll start with the servers and talk a little bit about what we're doing there. Tight collaboration with the PowerEdge team. As it turns out, Dell has a great server portfolio, so we know that our customers want to be on the latest and greatest servers. They want to have the latest and greatest components, network cards, memory, CPU. So we are in collaboration with our PowerEdge server team to build extraordinarily optimized solutions to drive data as fast as possible.

John Furrier

>> You collapse that wall between server and storage.

David Noy

>> Absolutely.

John Furrier

>> Can you just, while you're on that topic, share your role in, Jensen calls it extreme co-design. Because these systems, these co-design systems isn't just like, "Here's the reference spec. Deliver it." It's like really integrated.

David Noy

>> It's really integrated and we have a tight collaboration between the storage team and the server team. We have launched our ObjectScale, we mentioned the object storage product, and we also announced a coming of our PowerScale product running on the PowerEdge R7725xd. That's a high performance server for driving extremely fast per node performance into the GPU. And by the way, it's not just about getting performance, about getting efficiency because the more that we can drive performance out of the server, the less number of storage nodes you need. So if I can double the performance of each node, then I can cut the node count in half and that gives you energy savings as well.

John Furrier

>> And the other thing that's come up this week, a lot of, it's been a preamble going into the show, but GPU utilization has been a big topic. So this fits right in the narrative of PowerScale and the work you're doing because I won't say offload because that's been an older term, but they're kind of offloading memory. It is an offload. That's the modern version of that.

David Noy

>> Yeah, absolutely. So I mentioned that the other way that we can accelerate the performance is the software. So you talked about KV cache. When you're working in inferencing and you're keeping track of the memory of a conversation, what's happening, let's say at a conversational chatbot, there's all this context that's been built up over time. You don't want to have to rebuild that context every time. There is some amount of that that can be kept in memory. You don't want to reuse the GPU cycles just to recalculate context. If I can extend that memory onto the storage with direct access. So we've announced integration with VLLM and LMCache using the Nixle transport protocol. This is a protocol that allows the GPU to speak directly to the storage to save all the previous context of your conversation so you can have longer memory and not have to constantly do recalculation, that accelerates your time to FirstToken by 19x.

Jackie McGuire

>> For people who are not familiar, KV cache, I'm going to try to create an analogy because we have a huge global audience and not everybody may know what KV cache is. It's like the court transcript in a trial, right? It's like all of the different information, and then at the trial there's an outcome at the end of the trial. And so you want to be able to use that case law for everything else you're doing without having to re-litigate every single time, right?

David Noy

>> Absolutely.

Jackie McGuire

>> Because human memory doesn't really work that way, right? Human memory is very fallible, but we're trying to create this kind of base of context. I guess you wouldn't say it's immutable, but that is consistent because otherwise, because of the variability of probabilistic outcomes, you could end up with different outcomes of the same case if you don't have a really good KV cache, right?

David Noy

>> Right. It's the difference between having an oral history and actually writing it down.

Jackie McGuire

>> Yeah, yeah.

David Noy

>> The moment when you write it down, you have the books, but actually what KV cache does is it gives you a quick way to go look up in the book for the information that was figured out five or 10 years ago.

Jackie McGuire

>> Yeah.

John Furrier

>> We're getting questions from the crowd. So I got a text to ask you a question. Why is it important to have purpose-built block file and object versus all in one?

David Noy

>> Great question.

John Furrier

>> Thank you for sending the questions in. I'm getting DMs.

David Noy

>> Look, there's a lot of storage vendors who say, "Let's go and build a storage solution to serve AI that can do block file and object all from one platform." And actually object is becoming increasingly popular in the world of AI and I'll tell you why. A lot of people who start off in AI, they start in the cloud and so they're used to basically S3 object as sort of the paradigm for storage for AI. And then when they go to on-prem, they want that same paradigm. The thing is architecture matters, and you'll hear this from different storage vendors, architecture matters. A file system is a file system. It designed a certain way. It's a hierarchical folder-based approach to basically storing data. Just because a file system exports an S3 protocol doesn't make it object storage. Object storage systems are designed a certain way too. They're optimized for certain workloads. Just because they export NFS doesn't make them a file system.

John Furrier

>> Yeah.

David Noy

>> We take a best-of-breed approach. We say, "Hey, you know what? We're going to give you the best file system with optimal file system performance and features and best object storage with optimal performance features."

John Furrier

>> Dave, a lot of people are talking about the VMware storage integration with AI. It's kind of a reboot of that. You remember those days. So it's like, I guess my question is that, what's best of the old tricks and what are some of the new modern things around storage and data? Because they're data, we're talking about data platforms here. I mean, it's compute and data and the relationship between networking and storage is blurring like HPC and supercomputing. It's all one thing.

David Noy

>> Yup.

John Furrier

>> So we're starting to see this new modern architecture. What's some of the things that are best from the old school and new school?

David Noy

>> The new modern things is we're improving those protocols to make them even faster. So for example, at our file system, NFS was the way that you worked with files. We're introducing parallel NFS. Parallel NFS allows you to take a request, split it up and send it to multiple storage servers at the same time. That's just a fast way to accelerate getting your responses back. Something you don't typically see from a scale-out file system. With object storage, you ask like, "Why do you have best-of-breed file, best-of-breed object?" Now when NFS, we have a concept of something called NFS RDMA. This is direct access from the GPU to the storage over NFS protocol. That's now going to be available for S3 as well. So for people who are using object storage for their AI training can talk directly to the storage as well.

John Furrier

>> It's very nuanced, but explain why that's important because that is a key innovator, because you're piping directly to the GPU. Why is it important?

David Noy

>> There's two big benefits. Benefit number one is that if you're talking from the GPU to your storage over network, if you're using traditional TCP/IP as the interconnect, there's a lot of back and forth and there's a lot of network handshakes that happen and that just introduces latency and performance overhead. The other one is the calculation of all of what's required to do that TCP/IP handshake puts load on the client. So by going direct and not using TCP/IP, I can actually take some of the burden off the client. In some cases over 80 to 90% reduction in the burden on the client. That's resources you can use for something else.

Jackie McGuire

>> So I was talking to one of the vendors on the show floor about this and I said it's like the difference between Amazon and a regular retail store because by sending the data directly, you basically just put it in another box and send it. You don't open the box, you don't put it on a shelf, you don't have to repackage it like with TCP/IP. It's kind of the Amazon where you can just send the box as it is, right? Send the product, you just store the data, you don't actually open it and it's much more efficient. And you could keep significant-

David Noy

>> You described it a lot better than I do.

Jackie McGuire

>> Well, so I always say that the thing that made me good as an analyst is I love rap and hip hop. And one of the things that makes rap and hip hop one of the best art forms in music is analogies and metaphors. And I think this is such an abstract for most people in technology, and I think the more we can do to create physical parallels of in Networks in security for me too. So anytime I'm trying to explain, "Hey, this piece of identity is really important," I'm like, "All right, let's talk about this as if it was a building." And I think the more we can do to tie these abstract technology concepts to real world things people can understand, the more we can demystify this and bring... Because we're going to need millions of additional people in this industry and they're not all going to come from compute PhDs and HPC backgrounds. We need to bring people from every industry in. So I spend, and this is silly, but I do, I spend a ton of time trying to think about the court transcript thing. I've literally been thinking about for the last three hours since I talked to somebody else about KV cache and that like, "How do we explain this to people who are in some other area, but really want to understand what's going on here?"

John Furrier

>> I was talking to the Uber driver coming in today, and obviously it's packed here in St. Louis to that point, and he said, "What's going on here?" And I'm like, "It's the AI thing. All those data centers you hear in the news." So in his mind it's like ChatGPT, large scale data centers. So I guess, David, the question for you would be, how does storage at scale change? Because now it's all great to have all those KV caches all working with these unified racks. When you're at that large scale and you guys are doing exabyte deal, I know you can't talk about the public ones, but I know about a few of them where Dell's AI factories, I think Michael Dell was public on his investor meeting. He talked about some of the deployments are pretty massive. What's changes at scale, because that's rarefied air in terms of deployments, you're starting to see the level of scale. How does that affect your baby in the future of the data platform?

Jackie McGuire

>> I guess to add to that, are those deployments typically also hybrid?

David Noy

>> Good question. So a lot of the at scale right now is still on premise, but I would imagine that over time that there'll be more hybridization. Look, when we talk about, so you brought up a bunch of good points. One of them is people are definitely needing to learn new skill sets. When you think about we're buying a Dell AI factory for NVIDIA, right? I'm going to deploy it. I went and I bought it because everybody's using their budget right now to go buy these things because it's like the hot thing. And then suddenly they're finding out they need to skill up. They need to learn like, "Oh my gosh, now what kind of software do I put down? How do I use it?" So we also have built out the AI data platform for NVIDIA, which is a set of pre-opinionated tooling that we use for analytics, query, understanding what data you have, being able to search it, be able to transform it and prepare it for those AI training and inferencing use cases. At scale, understanding what you have, being able to give holistic view into what you've got, how old is it? Is some of it still relevant or not? Bring that down to a curated set of data for your training and inferencing workloads, that's critical. So we want to be able to make that as easy as possible and turnkey. That's some of the solutions that we deliver as well.

Jackie McGuire

>> Dell has this, you have a first mover advantage, obviously you've been on the forefront of everything. You've got a hardware advantage, and we were talking about this before we started. There were years that it seemed like hardware was a back burner thing. It was a commodity. And now we're at this point where it's the most exciting thing now. And then with this factory, one of the things that really struck me at DTW when we were looking at the AI factory the first time is the efficiency. Because when you put different components together that work together, but work designed together, they work, but they're not efficient. And then when you look at the AI factory and when all these components are designed to work together, the energy efficiency you get, the throughput efficiency, the data efficiency. So do you feel like that first mover advantage gave you a head start on this co-design with NVIDIA?

David Noy

>> Yeah, it's super, you asked a great question and let me combine that with the earlier one about exabyte scale. I met with one of the neocloud hyperscalers and they told me, "Look, 95% of our power is going to GPU servers and I got 5% leftover for storage." Now imagine you're in that situation, you're like, and I'm building exabyte scale storage environments.

Jackie McGuire

>> Sorry, you don't need memory. You don't need...

David Noy

>> They work about your data. So when we designed the next generation of PowerEdge, as I said, we have tight collaboration. We're absolutely thinking about how much performance can we drive per watt? How much can we drive per rack unit? And so we are hyper focused on collaboration with our teams to make sure that we are building the most energy efficient and space efficient solutions for AI infrastructure. And for example, if I can give you double performance per watt or double the performance per rack as one of my competitors, I took a 5% and turned it into a 10%. That's a big deal.

Jackie McGuire

>> Right. Yeah.

John Furrier

>> So you're just scratching the surface. I want to get your thoughts since you run product management, look out on the roadmap, there's a lot of dry powder potentially available to get more performance.

David Noy

>> Absolutely.

John Furrier

>> What's the headroom look like? Could you share your vision on where the headroom is, where you see the focus on the engineering, the software, and the co-design?

David Noy

>> So in working with platform teams, the PowerEdge teams, in working with our software engineering teams, as I said, we doubled our top end streaming performance per node. So if you look at per node, how much maximum read performance can you do? We've doubled that in the last 18 months. We have line of sight to doubling that again in the next 18 months. So right now, in an ObjectScale node as an example, we can do per node 40 gigabyte per second throughput per node. So in an exabyte, you're talking about thousands of nodes. That is a big a lot of... Now with the advancements in the platform technology and with the new network cards that coming out, and we're always going to stay on the bleeding edge of network cards because we don't want to be ourselves in a situation where, because we put a network card, and then all of a sudden we get software boost, we're now throttled by the network card.

Jackie McGuire

>> Yet super cars stopped at traffic lights.

David Noy

>> Right. So we always want to put the latest in, but by doing that, we have plenty of dry powder to where we could continue to ride this linear improvements in performance over the next 18 to 36 months.

John Furrier

>> I want to give you the last word on this. Say thanks for sharing all that good data. A lot of people look at Dell, Dell's got a lot of products, not a lot of people may know all the advancements that you have going on. Obviously we've been covering the AI factory a whole nother paradigm shift of scale. I mean basically it's a super computer, it's got a lot of other computer, it's like services storage all bundled into one thing. That's my metaphor. But what would be your message to folks watching about Dell's storage products, how that's going to roll out and the importance of it and some of the cool things you're doing that's relevant?

Jackie McGuire

>> Yeah. And also if people want to learn more, is there a good place for them to go for that?

David Noy

>> So we have lots of product pages on Dell.com. I would suggest going and reading those. I've actually put out a lot of blogs recently. It would be great to go and read some of those blogs because I talk about exactly the topics that we've talked about today. And my message would be like, look, Dell builds awesome AI factories. Our Dell AI factory for NVIDIA is bar none like world-class, but an AI factory is only as good as the data that feeds it. So we are absolutely determined and we are absolutely delivering on world-class, high performance, scalable exabyte scale data storage. And we're going to continue to innovate in that space.

John Furrier

>> Well, we're here at SuperComputing, which started in 1988. The name is now SuperComputing. It's actually happening. It's high performance computing meets AI, all one thing. It's accelerated computing. You guys are doing a great job. And of course we're huge fans of the AI factory, one of our most popular series. David, thank you so much for coming out and speaking with Jackie and I.

David Noy

>> Thank you. Thanks for having me.

John Furrier

>> All right. I'm John Furrier with Jackie McGuire here at theCUBE Live coverage SuperComputing 2025. The systems are rolling out and they got to perform great. We're doing our part to keep up. Thanks for watching.