RAISE Summit '26 | Sid Sheth & Sudeep Bhoja, d-Matrix

Clips
More from RAISE Summit '26

Sid Sheth

CEO

d-Matrix

Sudeep Bhoja

Founder & CTO

d-Matrix

play_circle_outline NVIDIA–Parasail Partnership: Heterogeneous Disaggregated Inference Clouds for Low‑Latency Token Generation

play_circle_outline Memory bandwidth bottleneck (the "memory wall") limiting latency and interactivity

play_circle_outline Rack-scale deployment and Supermicro integration for seamless installation

play_circle_outline Turbo Boost AI Infrastructure: Pairing d-Matrix Decode with GPU Retrofits to Achieve Blackwell-Like Throughput

play_circle_outline Scaling and Commercializing 3D-Stacked DRAM and Logic to Enable Fast-Token AI Deployments

Info

Sid Sheth & Sudeep Bhoja, d-Matrix

Sid Sheth

CEO d-Matrix

Sudeep Bhoja

Founder & CTO d-Matrix

search

>> Palo Alto studio connecting in Silicon Valley and Wall Street. I'm John Furrier, your host for theCUBE here with Dave Vellante, my co -host.

>> Welcome to the RAISE Summit coverage, brought to you by Solidigm, with support from Tensordyne, d-Matrix, and Argentum.

>> Welcome back to theCUBE's live stream here in Paris, France. I'm John Furrier, host of theCUBE. We're here for the RAISE Summit Day 2, as we're winding down two days of wall -to -wall coverage. The top story has been AI infrastructure, and the big part is hardware has been a big part of it. The acceleration of AI is coming down from the chips. Big large scale AI factories that's enabling all the growth at the agentic level. Of course, the AI -native applications and the company here making it happen. The co -founders of d-Matrix are here in theCUBE. We're going to explain it all. And of course, we've got some news that hit while we were here in Paris, so we'll get down to it. Sid, the CEO, Sudeep, CTO, co -founders of d-Matrix. It's great to see you guys. Thanks for coming on theCUBE. Appreciate it. Good to see you again.

>> It's great to be back, yeah.

>> All right, so you've got to love this market, Sid. you guys work so hard to get the hardware out. You know how hard it is to do that. the demand curve is so great right now.

>> How are you feeling?

>> Well, it's a wonderful time to be in the industry, right? the demand curve just keeps going more parabolic. People have tried to size this opportunity multiple times, and every time they size it up, the numbers go up. CapEx keeps increasing, and a lot of that is just tied to deploying more inference. Token consumption is just going off the charts. So a lot of this is really tied to finding different applications that can unleash new use cases, and all of that leads to more token consumption. So what a phenomenal time for us as a company to be building products at a time like this.

>> And Corsair just went into production at volume. So congratulations, just last month or so. So you're up and running.

>> Very much. It's flowing good. Yeah, very much. And that's a huge milestone for the team. Right, because first product, as they say, for a chip company is always the hardest. Right, your first one, it's not just about the silicon, right? it's silicon and then, when we announced production, it was not silicon production. It was the silicon, the boards, the racks that go along with it, the servers. We had to strike partnerships with Supermicro and Broadcom and Arista and all the software, lift that goes along with it and getting models ready to run on the hardware. So yeah, it was ecosystem lift pretty much.

>> It's so hard to build that product. It's like jumping out of an airplane, like will the parachute open? You guys are doing great. Sudeep, you've got a lot of pressure on you. You've got to keep building the speed and velocity of this market. If you look at all the recent announcements, the life cycle is so fast. If you go back a generation, you had some time.

>> Exactly, yeah. So it's a disaggregated inference. Low latency is the name of the game today. Agents are running for a long time. Users don't want to wait. So there's a demand on trying to get the latency down, and that means disaggregated inference. There's GPUs and fast accelerators like ourselves. So we have Corsair that we're just bringing into production, we have our next generation already in the works, like you said, you can't skip a beat here, we're trying to stack memory and compute together in a 3D manner, we believe that's the future, it's really, really exciting times.

>> Yeah, you guys cracked the code on all the data points around performance, so that's a huge deal, especially on inference. Inference is one of those apps, or not apps, capabilities that you can see the speed. We all work with the chat prompts, and okay, reasoning, thinking, that's got to happen faster. And you guys just recently had news that broke here in Paris, deployment of the d-Matrix accelerators for AI inference with Parasail and NVIDIA involved. It's rare that they put their name on a press release, so they're totally behind it. Explain the news, what does it mean? Obviously, inference is a service, people want inference in AI token economy. Talk about this news and why it's relevant.

>> It is important and a very, very critical piece of news from our point of view because it is the first public announcement of a heterogeneous compute solution targeted specifically for fast token generation in partnership with NVIDIA. And it's really a go -to -market partnership where you have inference clouds. And by the way, this is the first. There's going to be many more that we do over the next few months. But this is the template. This is the blueprint. We have a lot of inference clouds coming to us and saying, exactly what Sudeep just talked about, is more low latency, deploying that in a GPUs only infrastructure just doesn't get us there.

>> So how do you take XPUs like the d-Matrix Corsair solution, which is really targeted for low latency and fast decode, pair it up with an NVIDIA GPUs so you get the throughput advantage and you also get the latency benefit and you put the two together and it's a win -win solution. So really, anytime you run into a situation like this, NVIDIA is essentially saying that we're going to work with d-Matrix to enable customers.

>> Yeah, and they have inference too, so it's a real testament to the product leadership, so congratulations. You mentioned there's going to be a template. Explain how this gets deployed. Just walk me through the configuration. Rack scale is hot right now. People want rack scale. So you see all the deployments are rack scale systems. this is a perfect scenario for it. How's it work? Does it sit somewhere? Where do you guys sit in the rack and what's the relationship? Just take us through the topology.

>> Yeah, so there's an NVIDIA rack. The specific demonstrations we showed, for example, there's a B300 server with the NVIDIA GPUs. And we have a Supermicro X14 server that's with our cards, Corsair cards. and the Corsair cards run parts of the model. Other parts of the model are run by the GPUs. It could be in the same rack, it could be in different racks. So there could be a complete NVIDIA rack, and there could be a companion d-Matrix rack, and by complementing each other, we are able to speed up the user experience. There's a fundamental trade -off of throughput latency, and we can push that curve out so that users don't have to wait as long, we can still get good economics, and that's what we're really offering in this partnership.

>> Talk about the economics and some of the performance. Scope the gains on performance, 10x, 20x, and what's some of the drivers on the cost savings?

>> what we're really unleashing here is a new class of fast tokens, right? And fast tokens today equals premium tokens, right?

>> You have Anthropic Claude Code today. They have something called Fast Mode, and essentially you can turn that on. you have to pay more when you do that, but in that mode you essentially have much higher levels of interactivity with the application. So the user has a much better user experience if they really care about a real -time experience with the application. And the application developers are charging more for those fast tokens. So it's essentially a new class of tokens, a new economy that is getting unleashed because of this. So today I think you see almost like a 10x differential between fast token pricing and throughput token pricing, but that'll most likely collapse over time. I think, I don't see who would not care about having a real -time experience if they could afford it. Today maybe the delta's too big and so you have to tier this thing, but over time I think everyone will say, no, I want access to fast tokens, and the pricing will collapse as the volume economics kick in, but at a time like that, the TCO of the solution is going to start mattering, because now you see a collapse in the pricing, and then it comes down to whose solution enables the best TCO, so that you can enable better margin structure for.

>> So there's a real need right now, obviously, for high end, low latency.

>> Correct, absolutely.

>> And some people who don't, they don't pay for it.

>> And it's very early days, right? it's very early days for, this literally started like less than nine months ago, right? When you had the agentic OpenAI moment, end of last year, and the agentic harnesses started getting developed, right? And so now people are like okay now there's a true agentic use case with coding.

>> Yeah.

>> Right, and so latency is beginning to matter, machines talking to machines, not just humans talking to machines. Capabilities with Claude Code, just step function improvement. Put the two together and it is just an explosion, right? And so yeah, this is very early days for this. We have a long, long way to go.

>> We are barely getting started. And so what are some of the performance gains that you're seeing?

>> Yeah, so there are some tiers where you cannot hit that performance with GPUs. So we can push the latency down and the interactivity by 2 to 4x. So that's one regime we can reach. You couldn't reach that with GPUs, but with our accelerators paired with the GPUs, you can turbo boost the GPUs to get more throughput, can go faster. That's one use case. Another use case is you want the same level of interactivity. You still want your 300 tokens per second, but can you increase the throughput? And you can get it two to four X higher throughput. So the economics look much better with the turbo boost options. And you can choose. You can go fast, or at the same speed you can be much more efficient.

>> Talk about the turbo boost impact, Sid, because one of the big discussions here at RAISE is the value to tokens. Obviously, we're in a token economy. I wrote a post on that before we came to the show. So, people are starting to match business values. They're going to be paying for the premium tokens that you mentioned, until you hit volume. That makes economic sense to me. But now businesses are starting to say, hey, I can put a value on productivity. I can put a value on an outcome. I can put a value on agentic tasks. Is that where you guys see the rationale on the token side? Are you starting to see people say, okay, I will pay for these use cases. Is that clear in the economics?

>> Is that coming out a lot with customers?

>> Oh yeah, yeah. Again, as I said, it's still early days for this stuff. It's nine months into this, and people are still trying to figure out what this new token economy looks like, right?

>> The use case today is primarily coding, agentic coding, right, and that is where I think all the trial and error, all the experiments are happening, right? But one thing is pretty clear, right, that there's pricing numbers as high as 10x higher than throughput tokens, right? So premium fast tokens being priced at call it $20 for a million tokens versus $2 for a million tokens, right? And so that's almost 10x delta, right? And people are willing to pay for that. People are willing to pay and they're getting completely consumed, right? So given the option if if you're an engineer, software engineer at an enterprise and if that's all you have to do just hit the button and take advantage of those fast tokens without really having to worry about whose budget it is coming out of, everyone's going after those fast tokens, right? Still people trying to figure out, oh my God, folks are token maxing on fast tokens and we got to trim that down.

>> No one wants slow inference.

>> no one's raising their hand for that. Sudeep, talk about the, I think we talked about this last time you were on theCUBE, but the technical people care about it too because the memory wall has been the villain in some of these deployments. This also factors, hence why the heterogeneous angle here on this disaggregated production environments are important because it's rack scale, but it's not just all NVIDIA. you guys are partnering with NVIDIA, that's a huge testament, put that aside for a second, we talked about that, but there's a technical issue here.

>> Exactly, it's all about memory bandwidth, right? It's all about having enough memory capacity, and then getting the bandwidth out of the memory. And the problem has been that AI is currently bottlenecked because of memory bandwidth. because these things live on different substrates. But if you can combine this into a single substrate, either in three dimensions or on a single piece of chip, then you can make memory bandwidth much faster because the distances that we're moving are much closer. And that allows us to get much better interactivity. And so the key to solving the latency problem is memory bandwidth. And what we offer at d-Matrix is to stack DRAM and logic together in a single chip, and doing that we get many times the performance of HBM memory. And so that's the way we break through the memory wall, and because we don't move many millimeters on silicon, we can do that at very low energy. And so for the same power that you have, and we are always power constrained, you can deliver much more bandwidth and much more interactivity.

>> Okay, so just to put my skeptic hat on for a sec, whoa, whoa, we've got our racks in place. There's motivation on the technical and business side. How much rearranging is needed? Take me through some of the changes that would have to be made, if any.

>> To the deployment specifically? First of all, the d-Matrix go-to-market, right?

>> The way we take our products to market, we have deliberately made some choices here where we can integrate our solution into a customer's environment fairly seamlessly. We don't insist on our customers having to buy entire racks from us. The deployment model is you tell us what you deploy and we'll find a way to fit into your deployment. So a Supermicro X14 server is a starting point and that fits into pretty much any 19 inch rack. And 19 inch racks are everywhere in data centers. So a customer can come and say, look I'm using a technique called speculative decode. Now this is a technique where you run a small draft model on the d-Matrix solution to accelerate fast token generation. It could be as simple a thing as a server that goes into somebody's rack, sitting next to other servers that could be NVIDIA servers. If you want to run other forms of disaggregation and you need a rack, you could take a standard 19 -inch rack and fit multiple servers inside. We have a solution called JetStream that can connect all those servers up. So it's very seamless. There is no major disruption needed in the way a rack has to be rolled in or a server has to be fitted into a rack, none of that.

>> We've worked through all those options for our customers.

>> And you mentioned the NVIDIA part of the press release that I saw on the news was interesting to me because again, they have Groq, so they have inference, so this is again the heterogeneous piece. What's the relationship to the GPUs? The GPUs do the pre-fill, you guys do the decode. you're coexisting, you're not replacing GPUs. So they pre -fill on the GPUs, and then you guys do the decode, is that right?

>> That's correct, that's correct. So the pre -fill, the GPUs are really good at it, they have a lot of compute flops packed into it, because they're really great training machines, and training machines do a really good job on pre -fill. So the GPUs do the pre -fill. We do parts of the decode. Sometimes it could be a full decode, Sometimes it could be just what is called a speculative decode, where we propose a bunch of tokens for the GPUs to verify. And that's the case where we turbo boost the GPUs. And sometimes it's another technique called expert disaggregation, where we only disaggregate the experts and we leave the attention, we leave the pre -fill on the GPUs. So there's many, many various types of disaggregation. We're still early days, but there are many parts of the workloads that can benefit from more memory bandwidth, and that's what we disaggregate.

>> So the GPUs love you because then they're more efficient.

>> Exactly.

>> You guys pair up nicely with them.

>> Yeah, and another use case is, for example, there are a lot of GPUs fleets out there with old Amperes and Hoppers, right?

>> We can go into those fleets and turbo boost those fleets and get them the performance of a Blackwell. So you don't have to upgrade your fleet to a Blackwell GPUs. You can just say, you know what, I'll just retrofit my existing fleet with a d-Matrix solution, and almost get another 2x improvement in performance without having to upgrade my GPUs fleet.

>> who's going to say no to turbo boosting? Absolutely.

>> Absolutely. It's going to be a household word. Turbo boost and more tokens. That's right.

>> Token maxing, token maxing has gotten a bad rap because people don't look at it, they've been using a lot of tokens, but token efficiency is a big deal right now. You guys fit into that because you're basically making the better tokens boosting.

>> Efficiency has always been part of everything that we do.

>> So even when we are talking about highly fast tokens, we are doing it in a very efficient manner. And when you talk about efficiency, it's like energy efficiency. For example, we have the densest amount of SRAM in a single rack today in the market. It's all air cooled, so no liquid cooling. So cost effectiveness is part of the solution that we have built. because we have the densest amount of SRAM, the hardware footprint needed to run these fast tokens is probably the smallest we have.

>> So, cost efficiency, energy efficiency, big part of what we do.

>> Congratulations, guys, great innovation, well needed.Everyone, inference is hot, it's going to be everywhere, it's going to be part of all the applications, so congratulations, I guess what's next? You mentioned just a little teaser earlier, what's next after this? It's hard to... You're going 3D.

>> 3D, yes, 3D.So we want to... 3D d-Matrix.

>> 3D d-Matrix, yes. Explain, what's that, what's it going to do?

>> So essentially the logic industry and the DRAM industry evolved in different ways, right? Because the same substrate is not good for both. So the memory guys specialize, the logic guys specialize. And they all separated and went far away and created this memory wall. So we're trying to bring all of them together and put them on a single piece of silicon, We hybrid bond it so it looks like one piece of silicon. So it can be made in two separate fabs, or two different companies. We'll put it together and build an architecture around that and break through the memory wall. So that's the first step is to, and we have demonstrated that technology of putting logic and memory together on the same substrate. Then we want to go high rise, multi -tier. We want to put four DRAM stacks on top of compute and get the capacity up, the bandwidths up, so you get essentially many more fast tokens in a smaller footprint.

>> The holy grail of inference is 3D memory.

>> Great, and of course doing more deals, getting...

>> Absolutely, that's the focus now. So now that we're in production, commercialization begins now for us, right?

>> seven years in, and yeah, that's going to be the focus is deploying the hardware, standing up, identifying use cases that people never thought of before, right? Unleashing these fast tokens and improving productivity.

>> It's great to see we've had previous conversations before you guys were shipping it in theCUBE and NYSE Wired, so congratulations. Everyone's a little bit relieved, but the game's just getting started.

>> Just getting started.

>> Thanks for coming on and sharing and getting us up to speed with some news too here at Paris. Thanks so much. Thank you. All right, I'm John Furrier. We're doing our part to turbo boost coverage on day two. A lot of action, again, memory is the hottest area. If you look at all of the trends, memory and GPUs together, that's what d-Matrix is doing. Again, another example of the innovation that you see entrepreneurs doing when the market has constraints, they work around them. The AI infrastructure's going to continue to be built out. Whole new ball game here on the infrastructure side, continuing to go forward and faster, and that's going to accelerate more apps, more agents, and better security. So of course, doing our part here in theCUBE. Thanks for watching. Thank you.

Sid Sheth & Sudeep Bhoja, d-Matrix

search

>> Palo Alto studio connecting in Silicon Valley and Wall Street. I'm John Furrier, your host for theCUBE here with Dave Vellante, my co -host.

>> Welcome to the RAISE Summit coverage, brought to you by Solidigm, with support from Tensordyne, d-Matrix, and Argentum.

>> Welcome back to theCUBE's live stream here in Paris, France. I'm John Furrier, host of theCUBE. We're here for the RAISE Summit Day 2, as we're winding down two days of wall -to -wall coverage. The top story has been AI infrastructure, and the big part is hardware has been a big part of it. The acceleration of AI is coming down from the chips. Big large scale AI factories that's enabling all the growth at the agentic level. Of course, the AI -native applications and the company here making it happen. The co -founders of d-Matrix are here in theCUBE. We're going to explain it all. And of course, we've got some news that hit while we were here in Paris, so we'll get down to it. Sid, the CEO, Sudeep, CTO, co -founders of d-Matrix. It's great to see you guys. Thanks for coming on theCUBE. Appreciate it. Good to see you again.

>> It's great to be back, yeah.

>> All right, so you've got to love this market, Sid. you guys work so hard to get the hardware out. You know how hard it is to do that. the demand curve is so great right now.

>> How are you feeling?

>> Well, it's a wonderful time to be in the industry, right? the demand curve just keeps going more parabolic. People have tried to size this opportunity multiple times, and every time they size it up, the numbers go up. CapEx keeps increasing, and a lot of that is just tied to deploying more inference. Token consumption is just going off the charts. So a lot of this is really tied to finding different applications that can unleash new use cases, and all of that leads to more token consumption. So what a phenomenal time for us as a company to be building products at a time like this.

>> And Corsair just went into production at volume. So congratulations, just last month or so. So you're up and running.

>> Very much. It's flowing good. Yeah, very much. And that's a huge milestone for the team. Right, because first product, as they say, for a chip company is always the hardest. Right, your first one, it's not just about the silicon, right? it's silicon and then, when we announced production, it was not silicon production. It was the silicon, the boards, the racks that go along with it, the servers. We had to strike partnerships with Supermicro and Broadcom and Arista and all the software, lift that goes along with it and getting models ready to run on the hardware. So yeah, it was ecosystem lift pretty much.

>> It's so hard to build that product. It's like jumping out of an airplane, like will the parachute open? You guys are doing great. Sudeep, you've got a lot of pressure on you. You've got to keep building the speed and velocity of this market. If you look at all the recent announcements, the life cycle is so fast. If you go back a generation, you had some time.

>> Exactly, yeah. So it's a disaggregated inference. Low latency is the name of the game today. Agents are running for a long time. Users don't want to wait. So there's a demand on trying to get the latency down, and that means disaggregated inference. There's GPUs and fast accelerators like ourselves. So we have Corsair that we're just bringing into production, we have our next generation already in the works, like you said, you can't skip a beat here, we're trying to stack memory and compute together in a 3D manner, we believe that's the future, it's really, really exciting times.

>> Yeah, you guys cracked the code on all the data points around performance, so that's a huge deal, especially on inference. Inference is one of those apps, or not apps, capabilities that you can see the speed. We all work with the chat prompts, and okay, reasoning, thinking, that's got to happen faster. And you guys just recently had news that broke here in Paris, deployment of the d-Matrix accelerators for AI inference with Parasail and NVIDIA involved. It's rare that they put their name on a press release, so they're totally behind it. Explain the news, what does it mean? Obviously, inference is a service, people want inference in AI token economy. Talk about this news and why it's relevant.

>> It is important and a very, very critical piece of news from our point of view because it is the first public announcement of a heterogeneous compute solution targeted specifically for fast token generation in partnership with NVIDIA. And it's really a go -to -market partnership where you have inference clouds. And by the way, this is the first. There's going to be many more that we do over the next few months. But this is the template. This is the blueprint. We have a lot of inference clouds coming to us and saying, exactly what Sudeep just talked about, is more low latency, deploying that in a GPUs only infrastructure just doesn't get us there.

>> So how do you take XPUs like the d-Matrix Corsair solution, which is really targeted for low latency and fast decode, pair it up with an NVIDIA GPUs so you get the throughput advantage and you also get the latency benefit and you put the two together and it's a win -win solution. So really, anytime you run into a situation like this, NVIDIA is essentially saying that we're going to work with d-Matrix to enable customers.

>> Yeah, and they have inference too, so it's a real testament to the product leadership, so congratulations. You mentioned there's going to be a template. Explain how this gets deployed. Just walk me through the configuration. Rack scale is hot right now. People want rack scale. So you see all the deployments are rack scale systems. this is a perfect scenario for it. How's it work? Does it sit somewhere? Where do you guys sit in the rack and what's the relationship? Just take us through the topology.

>> Yeah, so there's an NVIDIA rack. The specific demonstrations we showed, for example, there's a B300 server with the NVIDIA GPUs. And we have a Supermicro X14 server that's with our cards, Corsair cards. and the Corsair cards run parts of the model. Other parts of the model are run by the GPUs. It could be in the same rack, it could be in different racks. So there could be a complete NVIDIA rack, and there could be a companion d-Matrix rack, and by complementing each other, we are able to speed up the user experience. There's a fundamental trade -off of throughput latency, and we can push that curve out so that users don't have to wait as long, we can still get good economics, and that's what we're really offering in this partnership.

>> Talk about the economics and some of the performance. Scope the gains on performance, 10x, 20x, and what's some of the drivers on the cost savings?

>> what we're really unleashing here is a new class of fast tokens, right? And fast tokens today equals premium tokens, right?

>> You have Anthropic Claude Code today. They have something called Fast Mode, and essentially you can turn that on. you have to pay more when you do that, but in that mode you essentially have much higher levels of interactivity with the application. So the user has a much better user experience if they really care about a real -time experience with the application. And the application developers are charging more for those fast tokens. So it's essentially a new class of tokens, a new economy that is getting unleashed because of this. So today I think you see almost like a 10x differential between fast token pricing and throughput token pricing, but that'll most likely collapse over time. I think, I don't see who would not care about having a real -time experience if they could afford it. Today maybe the delta's too big and so you have to tier this thing, but over time I think everyone will say, no, I want access to fast tokens, and the pricing will collapse as the volume economics kick in, but at a time like that, the TCO of the solution is going to start mattering, because now you see a collapse in the pricing, and then it comes down to whose solution enables the best TCO, so that you can enable better margin structure for.

>> So there's a real need right now, obviously, for high end, low latency.

>> Correct, absolutely.

>> And some people who don't, they don't pay for it.

>> And it's very early days, right? it's very early days for, this literally started like less than nine months ago, right? When you had the agentic OpenAI moment, end of last year, and the agentic harnesses started getting developed, right? And so now people are like okay now there's a true agentic use case with coding.

>> Yeah.

>> Right, and so latency is beginning to matter, machines talking to machines, not just humans talking to machines. Capabilities with Claude Code, just step function improvement. Put the two together and it is just an explosion, right? And so yeah, this is very early days for this. We have a long, long way to go.

>> We are barely getting started. And so what are some of the performance gains that you're seeing?

>> Yeah, so there are some tiers where you cannot hit that performance with GPUs. So we can push the latency down and the interactivity by 2 to 4x. So that's one regime we can reach. You couldn't reach that with GPUs, but with our accelerators paired with the GPUs, you can turbo boost the GPUs to get more throughput, can go faster. That's one use case. Another use case is you want the same level of interactivity. You still want your 300 tokens per second, but can you increase the throughput? And you can get it two to four X higher throughput. So the economics look much better with the turbo boost options. And you can choose. You can go fast, or at the same speed you can be much more efficient.

>> Talk about the turbo boost impact, Sid, because one of the big discussions here at RAISE is the value to tokens. Obviously, we're in a token economy. I wrote a post on that before we came to the show. So, people are starting to match business values. They're going to be paying for the premium tokens that you mentioned, until you hit volume. That makes economic sense to me. But now businesses are starting to say, hey, I can put a value on productivity. I can put a value on an outcome. I can put a value on agentic tasks. Is that where you guys see the rationale on the token side? Are you starting to see people say, okay, I will pay for these use cases. Is that clear in the economics?

>> Is that coming out a lot with customers?

>> Oh yeah, yeah. Again, as I said, it's still early days for this stuff. It's nine months into this, and people are still trying to figure out what this new token economy looks like, right?

>> The use case today is primarily coding, agentic coding, right, and that is where I think all the trial and error, all the experiments are happening, right? But one thing is pretty clear, right, that there's pricing numbers as high as 10x higher than throughput tokens, right? So premium fast tokens being priced at call it $20 for a million tokens versus $2 for a million tokens, right? And so that's almost 10x delta, right? And people are willing to pay for that. People are willing to pay and they're getting completely consumed, right? So given the option if if you're an engineer, software engineer at an enterprise and if that's all you have to do just hit the button and take advantage of those fast tokens without really having to worry about whose budget it is coming out of, everyone's going after those fast tokens, right? Still people trying to figure out, oh my God, folks are token maxing on fast tokens and we got to trim that down.

>> No one wants slow inference.

>> no one's raising their hand for that. Sudeep, talk about the, I think we talked about this last time you were on theCUBE, but the technical people care about it too because the memory wall has been the villain in some of these deployments. This also factors, hence why the heterogeneous angle here on this disaggregated production environments are important because it's rack scale, but it's not just all NVIDIA. you guys are partnering with NVIDIA, that's a huge testament, put that aside for a second, we talked about that, but there's a technical issue here.

>> Exactly, it's all about memory bandwidth, right? It's all about having enough memory capacity, and then getting the bandwidth out of the memory. And the problem has been that AI is currently bottlenecked because of memory bandwidth. because these things live on different substrates. But if you can combine this into a single substrate, either in three dimensions or on a single piece of chip, then you can make memory bandwidth much faster because the distances that we're moving are much closer. And that allows us to get much better interactivity. And so the key to solving the latency problem is memory bandwidth. And what we offer at d-Matrix is to stack DRAM and logic together in a single chip, and doing that we get many times the performance of HBM memory. And so that's the way we break through the memory wall, and because we don't move many millimeters on silicon, we can do that at very low energy. And so for the same power that you have, and we are always power constrained, you can deliver much more bandwidth and much more interactivity.

>> Okay, so just to put my skeptic hat on for a sec, whoa, whoa, we've got our racks in place. There's motivation on the technical and business side. How much rearranging is needed? Take me through some of the changes that would have to be made, if any.

>> To the deployment specifically? First of all, the d-Matrix go-to-market, right?

>> The way we take our products to market, we have deliberately made some choices here where we can integrate our solution into a customer's environment fairly seamlessly. We don't insist on our customers having to buy entire racks from us. The deployment model is you tell us what you deploy and we'll find a way to fit into your deployment. So a Supermicro X14 server is a starting point and that fits into pretty much any 19 inch rack. And 19 inch racks are everywhere in data centers. So a customer can come and say, look I'm using a technique called speculative decode. Now this is a technique where you run a small draft model on the d-Matrix solution to accelerate fast token generation. It could be as simple a thing as a server that goes into somebody's rack, sitting next to other servers that could be NVIDIA servers. If you want to run other forms of disaggregation and you need a rack, you could take a standard 19 -inch rack and fit multiple servers inside. We have a solution called JetStream that can connect all those servers up. So it's very seamless. There is no major disruption needed in the way a rack has to be rolled in or a server has to be fitted into a rack, none of that.

>> We've worked through all those options for our customers.

>> And you mentioned the NVIDIA part of the press release that I saw on the news was interesting to me because again, they have Groq, so they have inference, so this is again the heterogeneous piece. What's the relationship to the GPUs? The GPUs do the pre-fill, you guys do the decode. you're coexisting, you're not replacing GPUs. So they pre -fill on the GPUs, and then you guys do the decode, is that right?

>> That's correct, that's correct. So the pre -fill, the GPUs are really good at it, they have a lot of compute flops packed into it, because they're really great training machines, and training machines do a really good job on pre -fill. So the GPUs do the pre -fill. We do parts of the decode. Sometimes it could be a full decode, Sometimes it could be just what is called a speculative decode, where we propose a bunch of tokens for the GPUs to verify. And that's the case where we turbo boost the GPUs. And sometimes it's another technique called expert disaggregation, where we only disaggregate the experts and we leave the attention, we leave the pre -fill on the GPUs. So there's many, many various types of disaggregation. We're still early days, but there are many parts of the workloads that can benefit from more memory bandwidth, and that's what we disaggregate.

>> So the GPUs love you because then they're more efficient.

>> Exactly.

>> You guys pair up nicely with them.

>> Yeah, and another use case is, for example, there are a lot of GPUs fleets out there with old Amperes and Hoppers, right?

>> We can go into those fleets and turbo boost those fleets and get them the performance of a Blackwell. So you don't have to upgrade your fleet to a Blackwell GPUs. You can just say, you know what, I'll just retrofit my existing fleet with a d-Matrix solution, and almost get another 2x improvement in performance without having to upgrade my GPUs fleet.

>> who's going to say no to turbo boosting? Absolutely.

>> Absolutely. It's going to be a household word. Turbo boost and more tokens. That's right.

>> Token maxing, token maxing has gotten a bad rap because people don't look at it, they've been using a lot of tokens, but token efficiency is a big deal right now. You guys fit into that because you're basically making the better tokens boosting.

>> Efficiency has always been part of everything that we do.

>> So even when we are talking about highly fast tokens, we are doing it in a very efficient manner. And when you talk about efficiency, it's like energy efficiency. For example, we have the densest amount of SRAM in a single rack today in the market. It's all air cooled, so no liquid cooling. So cost effectiveness is part of the solution that we have built. because we have the densest amount of SRAM, the hardware footprint needed to run these fast tokens is probably the smallest we have.

>> So, cost efficiency, energy efficiency, big part of what we do.

>> Congratulations, guys, great innovation, well needed.Everyone, inference is hot, it's going to be everywhere, it's going to be part of all the applications, so congratulations, I guess what's next? You mentioned just a little teaser earlier, what's next after this? It's hard to... You're going 3D.

>> 3D, yes, 3D.So we want to... 3D d-Matrix.

>> 3D d-Matrix, yes. Explain, what's that, what's it going to do?

>> So essentially the logic industry and the DRAM industry evolved in different ways, right? Because the same substrate is not good for both. So the memory guys specialize, the logic guys specialize. And they all separated and went far away and created this memory wall. So we're trying to bring all of them together and put them on a single piece of silicon, We hybrid bond it so it looks like one piece of silicon. So it can be made in two separate fabs, or two different companies. We'll put it together and build an architecture around that and break through the memory wall. So that's the first step is to, and we have demonstrated that technology of putting logic and memory together on the same substrate. Then we want to go high rise, multi -tier. We want to put four DRAM stacks on top of compute and get the capacity up, the bandwidths up, so you get essentially many more fast tokens in a smaller footprint.

>> The holy grail of inference is 3D memory.

>> Great, and of course doing more deals, getting...

>> Absolutely, that's the focus now. So now that we're in production, commercialization begins now for us, right?

>> seven years in, and yeah, that's going to be the focus is deploying the hardware, standing up, identifying use cases that people never thought of before, right? Unleashing these fast tokens and improving productivity.

>> It's great to see we've had previous conversations before you guys were shipping it in theCUBE and NYSE Wired, so congratulations. Everyone's a little bit relieved, but the game's just getting started.

>> Just getting started.

>> Thanks for coming on and sharing and getting us up to speed with some news too here at Paris. Thanks so much. Thank you. All right, I'm John Furrier. We're doing our part to turbo boost coverage on day two. A lot of action, again, memory is the hottest area. If you look at all of the trends, memory and GPUs together, that's what d-Matrix is doing. Again, another example of the innovation that you see entrepreneurs doing when the market has constraints, they work around them. The AI infrastructure's going to continue to be built out. Whole new ball game here on the infrastructure side, continuing to go forward and faster, and that's going to accelerate more apps, more agents, and better security. So of course, doing our part here in theCUBE. Thanks for watching. Thank you.