Name: HPE Unleash AI Momentum: AI Infrastructure for Inference Speed
Uploaded: 2025-11-05T14:00:00.000Z
Duration: 24 min 27 s

Clips

Luke Norris

Co-Founder & CEO

Kamiwaza

play_circle_outline Overcoming CPU–GPU Data-Path Bottlenecks and Session-Affinity KV-Cache Constraints for Scalable Inference

play_circle_outline Importance of inference performance for enterprise AI adoption and ROI

play_circle_outline Centralized Key-Value Cache and RDMA-to-GPU Data Path for Concurrent Multi-GPU Context Sharing

play_circle_outline Demonstrated performance gains: ~15–20× overall, ~20× time-to-first-token

play_circle_outline Reduced need to add GPUs, lowering power, rack, and infrastructure costs

Info
Transcript

HPE Unleash AI Momentum: AI Infrastructure for Inference Speed

Luke Norris

Co-Founder & CEO Kamiwaza

In this interview from theCUBE's HPE Unleash AI Momentum series, Robin Braun, vice president of AI business development hybrid cloud at HPE, joins Luke Norris, co-founder and chief executive officer of Kamiwaza AI, to talk with theCUBE's Rob Strechay about how rethinking AI inference architecture can multiply GPU performance without adding hardware. Braun argues that the real bottleneck in scaling enterprise AI isn't compute — it's how data reaches the GPU. Norris breaks down why traditional inference architectures lock user sessions to individual GPUs throug... Read more

explore Keep Exploring

What challenges arise when architecting AI inference in enterprise environments—particularly with GPU-based setups that require routing a user's requests to the same GPU to maintain the KV cache? add

How should enterprises scale AI inference workloads beyond simply adding more GPUs or servers, and how does focusing on data access and networking (as Kamiwaza does) address the challenges this creates? add

How does centralizing the KV cache and using object-to-RDMA/direct-to-GPU memory access change the architecture, and what performance and GPU-utilization benefits does that provide? add

What speedup did you observe in time-to-first-token and overall GPU performance after changing the data path? add

How can organizations architect their infrastructure for AI inferencing to boost performance and sustainability (reducing energy, memory, and scaling costs) without large investments in servers or power, and what role do technologies like the Alletra X10K and the Kamiwaza platform play? add

bolt Powered by CUBE AI

HPE Unleash AI Momentum: AI Infrastructure for Inference Speed

Rob Strechay

>> Hello and welcome to HPE Unleash AI series where we're going to be discussing the momentum scene out there in the field and what's really going on with customers here on theCube. I'm Rob Strechay, and today we're diving deep into AI infrastructure for inference speed and why architecture really matters. Exploring beyond the noise around GPUs today, we're talking about what actually unlocks their performance and their value. To help me break this down, let me bring on some friends of theCube, Robin Braun, who's the VP of AI business development hybrid cloud with HPE. Luke Norris, who's the co-founder and CEO of Kamiwaza. Welcome on board, both of you.

Luke Norris

>> Thanks for having us.

>> Thanks.

Rob Strechay

>> So Robin, let's kind of start with you. It's been a little bit since we've had one of these and what's changed in enterprise AI that makes inference performance such a big deal right now?

>> Well, I think people are ... It's no longer a question of if AI, it's a question of how and when. And what I love about it is that people are trying to find the signal in the noise. They're trying to use their data to improve their efficiency, to improve their approach to their business. And that's where inference comes in. The models that are out there, the frontier models, the people who are doing those, that's a very specific and special part of AI. However, just trying to use that to get at the underlying understanding of your data is so important. And that's where I see so many customers are now really locking in and focusing on how are they solving some of their more mundane, messy data type of issues and that's really where inference comes in.

Rob Strechay

>> Yeah, totally agree. And I think again, inference is the long tail where people are going to get value out of AI for sure. But Luke, where do traditional architectures break down when you start scaling AI, especially with agents and large context?

Luke Norris

>> So I first love the whole concept of traditional architectures. I mean, AI is so new and so rapidly evolving every paradigm seems to sort of have to re-change and reinvent itself. And in the enterprise, the whole concept of architecting for inference is probably only two years old and it's got some pretty significant issues. To maximize the most expensive part of the infrastructure, which is the GPUs, you have to architect the environment so that when a user makes a request, the data and that request and the answers get loaded up into that GPU. And then that way when the user makes another request, it needs to be redirected back towards that same GPU that already has the cache, the KV cache. That already has the knowledge of what the user's interacting with. That's extremely complex and that's extremely limiting because you've now locked that user session to that GPU. And if you have many GPUs and you have many users, if you mess that up one time, the next response of that user for that same workload takes a very long time and it's very inefficient across the board. So new architectures, new paradigms are needed.

Rob Strechay

>> Yeah. But if you solve that, Luke, that data path problem, what does that unlock for customers?

Luke Norris

>> Yeah. So the data path problem gets exponentially harder, not just from an architecture standpoint. If you also think about the fact that data typically is loaded by a CPU into standard memory and then from the standard memory into GPU memory and then into GPU processing. So that's a long path. Plus you have the architectural path of having to send the same request back and forth through that path. Now, if you could completely unlock all of that, you start to get these amazing performance gains that we've been able to prove out. And we're talking in the range of 15 to 20X performance and I literally mean X as in multiple levels of performance because now we can take the data and route it directly to the GPU memory. And as soon as you do that, you can also then have that centralized data going to any GPU. So you almost break multiple architectural bottlenecks and not just speed this thing up, but multiply it by a magnitude.

Rob Strechay

>> Yeah. I mean, that I think is amazing when you talk times, 15 times plus. I mean, that's just crazy. But Robin, how does this change how enterprises should be thinking about scaling AI?

>> Well, I think one of the things enterprises have really thought about scaling AI by compute. That every time you want to add a new workload, you need to add another GPU or another set of servers. And so there's been a tremendous focus on the server side, which of course is vitally important. And then it started to transition a little bit into the network, but we haven't paid as much attention to how we're connecting. And as Luke was just describing, how we're accessing the data during that step-by-step process that actually happens during inference. We've treated that inference environment kind of like a black box that we couldn't tease apart. And what I love about what we've been doing with Kamiwaza is that we just kind of took that black box and blew it up and really broke those constraints that were hampering how the business could scale. Because where at least I was hearing from customers, and I'm sure you may as well, be that they're not just trying to scale out on compute, but now they're having to scale their power usage. They're having to scale what that is by rack, what that is in the data center. That can be a really tricky nut to crack for them. And by starting to really examine how you're accessing your data, not just how fast you're computing it, it starts to really open some incredible new performance and ideas.

Rob Strechay

>> Yeah. No, I can imagine that, especially with the limitations that these organizations, like you said, we hear about it all the time. So Luke, walk us through what's different about this architecture.

Luke Norris

>> I think the key differentiator on this is the caching capability becomes centralized. So now you can have that one big KV cache and you can have all of the GPUs accessing it nearly simultaneously. More importantly, the next request that a session, a user, a query would have around the data doesn't have to be directed back to that same GPU, it can be directed to the most available GPU. So now you start to have not only that speed up that I talked about because the data path is going direct to the GPU memory via object to RDMA, direct to GPU, but you now also unlock all of the GPUs. So you're no longer waiting or queuing a GPU for the next session to come back to it. You can go to any GPU. So you get 10 to 20X more performance and you get more GPUs in a cluster that you can then use across. It just is this compounding architectural change that is going to really change the paradigm of inference in the enterprise.

Rob Strechay

>> Yeah, I can understand that. I mean, these new technologies are amazing how they're helping take the burden off of some of what's going on. But Robin, where does HPE infrastructure play a role here?

>> And that is, I think something we're incredibly proud and excited for is that as you start to unlock this, it's going back to that core data and it's looking at it and being able to have that integration with our Alletra X1000, which is already set up as essentially kind of intelligent unstructured data, being able to bring that intelligence now with the integration with Kamiwaza to just a whole new level in support of AI. In support of truly using the data to unlock the information within it and being able to get at that knowledge in such a much faster way. We already were able to demonstrate the performance of the platform with incredibly fast time to recover from backups and things of that nature. And I know you've had Ed on to have those conversations, but this is an absolute leapfrog when you start to look at the capabilities we're unlocking for AI, for inference. For what our customers are trying to do in leveraging their data in real time and being able to make it more sustainable, more economical, and more achievable in the type of time that they're looking to deliver.

Rob Strechay

>> Yeah. Like you said, when I was talking with Ed from the storage division and talking about the X10000 and the RDMA and the fact that they were the first ones to actually get the certification with Nvidia for object storage over RDMA, which is just a great achievement. So Luke, let's kind of bring it back to some of the reality. Let's talk numbers. What did you actually see?

Luke Norris

>> Yeah. So when we started this, we were hoping for an increase in speed. And Robin and I and the teams, we got together and we had some lofty predictions and goals. But to be honest, once we really started to test this out, the numbers literally almost shook us. It was just amazing. So right out of the gate, we saw the time to first token drop dramatically. And by dramatically, we ended up right around 20X time to first token. So that meant if it took 20 seconds for a model to process the very first token, to start the very first word, we were now doing it in one second. And then that translated into just an astonishing level of more performance per GPU rounding out to about 19.4X. So effectively, on GPU is getting about 19.4 more GPU performance. That's like literally almost having 20 GPUs per GPU that you previously had just by changing the data path and adding the Alletra X10K.

Rob Strechay

>> Yeah. I mean, that's amazing results to put it mildly, but let's kind of take it up a level a little bit. And Robin, why do these numbers matter to a business leader?

>> Well, I think just like we were talking about that we've really changed that black box of inferencing, it's now being able to truly explore how do you architect your business for inferencing and make that investment wisely. And by the numbers we're showing, when you think about sustainability, energy cost, memory cost. All of the different costs that are going into scaling the environments that people are creating to be able to find these answers, the real magic this can deliver is that you can dramatically increase a performance without having to dramatically invest in more servers, and without having to dramatically invest in a larger power bill. And that to me is something that's really exciting when you start to think about that having your own structured data on something like an Alletra X10K by using an application or platform such as Kamiwaza together that we can actually fundamentally change how you're getting at those answers. That to me, we kept looking at the answers, we kept repeating the test because as Luke was saying, they were so powerfully different than even what we had originally anticipated.

Rob Strechay

>> Yeah. In such a positive way too, which is great. So Luke, where are you seeing the biggest impact today?

Luke Norris

>> So some of the biggest impacts I would say are in collaborative use cases where you have a tremendous amount of context loaded and you have multiple people hitting documents, hitting processes, hitting procedures, and they're just adding to that context, which would normally compound to the issue. This continues actually to speed it up. We believe there's even some use cases that we're going to find that might even have better performance than this. Also, we're seeing dramatic value in interactive use cases, chatbots, customer service arena, and definitely audio voice level services.

Rob Strechay

>> So Robin, let's kind of break this down for the organizations watching. How does this map to HPE customer use cases that are out there?

>> I think exactly where Luke was going is that do you have use cases, and yes, by the way, is going to be the answer, that are using a lot of data. Where you want to repetitively access data, you want to have multiple people accessing and getting at the understanding of that data. So I would say just as he was saying, whether you're looking at something where it's more interactive, where you need those answers incredibly quickly, whether that's a voice or chat agent, being able to engage with the public, or with a customer set. To being able to look at large and complex queries that multiple people would be doing such as records or document searches, things of that nature where they're going to ask potentially complex searches, multiple searches, or multiple layers of questions against the same type of documents. Like you would think about from either even a local government to records to legal to really the sky seems to be the limit.

Rob Strechay

>> Yeah. I mean, it brings to mind when we talked to Vail and we were talking about what they were doing with voice and things of that nature and all of the different languages that needed to be dealt with, especially during the seasons that they have there. And also to me, it brings to mind things like financial services and life sciences that are really regulatory heavy, big data and portfolio analysis and gene sequencing and things of that nature where this really could help, but definitely. So let's kind of-

>> I also think just really quickly to tie into that is obviously we've talked with the town of Vail and those type of use cases and it's incredibly impactful. But tagging on to when you think about that regulated, whether it's financial, whether it's health life sciences and even going into energy where you start to think about the large information that gets managed from oil and gas, things of that nature. That being able to speed that up 19.4 times versus having to buy 19.4 more GPUs is kind of crazy.

Rob Strechay

>> Yeah. Let's talk about that a little bit because I think to me, I talk to a lot of organizations and they're like tokens. I'm like, I don't get tokens and context windows and things like that. One of the points that often is hard to translate is really the actual business value. And I think we started to touch on that, but things like time to first token, tokens and GPU utilization, which can feel very abstract to a CIO or especially to a CFO for that matter when he's trying to figure out ROI. Let's start with you, Luke, and kind of break it down into what does it mean dollars and budget wise for these, and why does it matter to these organizations and to those leaders?

Luke Norris

>> Well, let's just break those use cases we've talked down one more level. If you have an interactive AI service like customer service or chatbot, et cetera, and you have a user that has to wait, let's say 20 seconds for that first word to start or that response from their voice back to them and you can now reduce that down to one second. Now it's actually going to create a much better service, a much better user interaction story, but also you can now do 20 times as much within that same period. So you could handle 20 customer interactions with the same workload. You could handle 20 chats going on with the exact same workload, the exact same GPU. And then like I said, you start to layer on the fact that that's just how fast you're able to do it. Then you also got the architectural advantages of this, you got the ease of the architecture. You don't have to overbuy now to make sure that you actually can dedicate that chat to one GPU. This is a fundamental shift in the way enterprises are going to look at it and that's just the structured use case. We keep dancing around this, but the real value in my mind when I talk to enterprises is when you're massing together and mixing data sets, just massive data sets that are all different, all of that gets put into context and it requires very expensive GPUs to have all that memory, to have all that context and all that availability. Now you're just going to have it in one central storage and you can have a tremendous amount more of that context and you load just the exact amount and the exact tokens you need right into those GPUs. Now you can open up and unlock much better workloads, much higher fidelity of the AI workloads, much easier programming to make those work. I mean, my head swims that this is just such a paradigm changer. The markets are just going to love this.

Rob Strechay

>> Yeah. I mean, I think that that is dead on with that. And I think again, with that productivity, Robin, how should leaders think about lower cost per token, better GPU productivity? And the idea that this architecture can effectively turn one GPU cluster into the economic equivalent of 10 to 15 clusters of concurrent workloads. What does that really mean?

>> Yay. Is, I think, how they should react. No, but seriously, as Luke's saying, we've been very much on the cutting edge of bringing together the technology to drive the customer benefit and to be able to really start to look at and simplify the inference architecture in a fundamental way that we've been tied into the way that it has worked. Which has been incredibly powerful and certainly does work, but are there ways we can do it faster, better, and more economically just by improving how you store your unstructured data? And what we found is the answer is yes, being able to work with partners such as Kamiwaza. So my yay is that what they can do is to start to understand how to right size their investment in AI to get at the intelligence of their data. They don't have to create a big data lake, they don't have to massage all their messy data. They just need to put it on Alletra X10K, and we can do all the heavy lifting for them.

Rob Strechay

>> Yeah. I think that is an outstanding way to think about it because again, this is so much that can be done within the X10K. It's amazing technology. I know I've talked and gone deep on that, but Robin, let's kind of bring this back to where does this fit into the Unleash AI program as a whole?

>> I think it really is a fantastic example of why we're doing Unleash AI and it's to bring together these amazing innovation partners and innovation ecosystem with HPE infrastructure to be able to drive benefit and value as a total solution for our end customers. And it's just a fantastic example of what working together can bring as a benefit to our customers and how we can also, by bringing Unleash AI together with the infrastructure, our partners, our partnerships, that we're truly able to create something unique and exciting in the industry that changes how people can actually approach their environment.

Rob Strechay

>> Yeah. And Luke, given that you're on the partner side of this partnership, what does this partnership enable you to do that you couldn't do alone?

Luke Norris

>> So I mean, we're an ISV, we're a software shop and we are trying to deliver that outcome, that end user value to our mutual customers. But the hardware and the architecture and the limitations of the data center typically prohibit our customers from moving forward. And the fact that the HP AI unleash partnership really takes all of that away from a complexity standpoint, from an acceleration standpoint and from a packaging standpoint, allows us to continue to focus on what we want to focus with our customers. So I haven't seen any other OEM, other than HPE, really attack this and really solve this. And then the fact that they doubled down on engineered solutions with us where we can get deep inside the code base of a storage platform, but actually get that to accelerate the output of a GPU and therefore the outcome of an AI solution. I think it's unheard of in the industry and I'm just proud to be a part of it.

Rob Strechay

>> Yeah. I mean, that is true partnership to put it mildly. So final question, Robin, what should organizations do next?

>> Oh, I think they should get a briefing. Really actually in all honesty, it's even just see a demo, be curious, find out more. The numbers seem so heroic that they seem unbelievable, but we have demos with actual data we can go through, we can have that architectural conversation. I think it's just be curious, in all honesty, in being able to open the consideration that there are new ways of doing new things with AI and that we're at the forefront and we're here to help.

Rob Strechay

>> I think that's a great thing because seeing is believing to put it mildly. So hey, Robin and Luke, thanks for coming on board today. I think this has been great and I think eye-opening. And some great numbers, but even better, how it ties to the true value in ROI that everybody wants to get out of AI. So thanks for coming on board.

>> Thanks for having us.

Rob Strechay

>> And thank you for watching this episode of HPE Unleashed AI. We've been talking about the momentum we've been seeing this year because it's going to be a lot about getting the ROI out of AI. Stay tuned for more on theCube, the leader in analysis and news. Stay tuned.