Advancing AI 2026 | Soni Jiandani, AMD & Omar Baldonado, Meta

Clips
More from Advancing AI 2026

Soni Jiandani

SVP & GM, Networking & Solutions Group

AMD

Omar Baldonado

Senior Director Data Center & AI Networking

Meta

play_circle_outline Networking Foundations for Large-Scale AI: Helios Rack-Scale Boosts Throughput and Tokens-per-Dollar

play_circle_outline Programmable AI NICs and DPUs for Ethernet-based fabrics

play_circle_outline Co-innovation between AMD and Meta on networking hardware and software

play_circle_outline Ethernet Surpasses InfiniBand for AI Scale-Out with Resilient MRC and UEC Transport Protocols

play_circle_outline Rapid bandwidth cadence: 400G→800G→1.6T to move at AI speed

play_circle_outline OCP-Driven Co-Design for Scale-Across: Overcoming Distance, Physics, and Resource Heterogeneity

play_circle_outline KV cache, storage disaggregation, and DPU offloading for agentic AI

Info

Soni Jiandani, AMD & Omar Baldonado, Meta

Soni Jiandani

SVP & GM, Networking & Solutions Group AMD

Omar Baldonado

Senior Director Data Center & AI Networking Meta

search

(INTRO)

Dave Vellante

>> Hey everybody, welcome back to Moscone West. My name is Dave Vellante. You're watching theCUBE's live coverage of AMD's Advancing AI. I hear 14 ,000 people registered for this event. Amazing keynote this morning by Lisa Su. I am super excited. We're going to go deep into networking with a long time CUBE friend, Soni Jiandani. She's here, she's Senior Vice President And GM of the Networking Technology And Solutions Group (AMD) at AMD. I'm joined by Omar Baldonado, Senior Director at Data Center And AI Networking at Meta. You've heard of them. Folks, great to see you. Thanks so much for spending some time with us.

Omar Baldonado

>> Great to be here, thank you.

Dave Vellante

>> Thanks for having us.

Dave Vellante

>> Great keynote by Lisa. she talked about how AI demand is absolutely exploding across, she laid out the workloads. AMD, amazing. Chip company turning into a full -scale systems company with rack -scale systems like Helios. We're going to get into that, really at the center of all this. She talked about 35 quadrillion tokens consumed every month, And that's an increase of 158x in just two years, but the real thing I took away was the curve is steepening. So the demand is going to be there for a little bit. Don't freak out too much about the AI bubble. It will be okay for a little while here, but I want to start by just asking you to talk more about the importance of networking in the whole context of AI systems, generally And Helios specifically. Maybe Soni, you could start off.

Soni Jiandani

>> So I think some of the best statements that I heard from Lisa And our customers today were, the unit of deployment used to be a rack, it's now the data center. So you have to think about all things coming together, which is building system level approaches to a data center, not just to feed the beast, the GPU, but also the CPU is now a first class citizen with agentic AI, And the role of the network is only increasing, because if you have to deliver these systems at scale with performance, with high availability, And this type of token consumption of 158X in two years, the network is foundational to deliver a balanced approach to processing those types of tokens And delivering the results that the customers want with the evolution towards these large frontier models And agentic AI, the network couldn't be more foundational. And that's how we have been able to innovate as part of Helios, where we are now claiming a 15 % higher throughput when compared to our competitors that have been delivering rack scale designs with Helios, as well as 30 % more tokens that we can generate per dollar, which ultimately is the metric for customers.

Dave Vellante

>> I'm going to push you on those benchmarks. We'll come back to that.

Soni Jiandani

>> Sure.

Dave Vellante

>> Because those were bold statements. But I wanted to add, Omar, one of the other key attributes And AMD's playing this card hard And the marketing standpoint is openness. I was there at the first OCP conference. You guys were very instrumental. I don't know if you were with Meta at the time, but very instrumental in setting that foundation for open infrastructure. even today the innovations like warm water cooling. that is changing the game. But I wonder if you could talk about, from your perspective, the importance of things like rack scale, the data center, is that how you think about it?

Omar Baldonado

>> Yeah, no, absolutely. Absolutely, we do think, And we do think of it as a data center scale problem And even beyond. I think Santosh And Lisa talked about that this morning. But let me go into the openness first.

Omar Baldonado

>> Yeah, please.

Omar Baldonado

>> With the OCP, we were founders of OCP, co-founders of a number of other companies, And in fact we helped start the OCP Networking way back when, And it was because we saw that networking needs the openness, because it's about plugging things together, And so many different technologies across the stack have to interoperate And work together, so our approach was to do it openly. And I think that's actually carried into the partnership with AMD on Helios. Be as open as we can on all the different parts of the network. So let's talk about some of the challenges in networking.

Dave Vellante

>> We were talking off camera, it used to be when you had spinning disks, spinning rust I call it, that was the bottleneck And you took some of the pressure off the network, that's changed with flash storage And new storage architectures, memory architectures, KV cache, that bottleneck is gone, so it pushes all the pressure onto the network. So let's discuss those bottlenecks, And then let's bring it to Helios, some of the innovations that you've had to deal with to address those challenges.

Soni Jiandani

>> So I think the beautiful part about our architecture is that it's an open architecture, which gives us an opportunity not only to partner with customers And the people that are leading the AI frontier models directly, And this has been a relationship over the many years where we have innovated to lead to the point we are at on Helios, by having an open platform you have the ability to see where the puck is landing by working with these large frontier AI model companies. Ultimately you care about driving the outcomes And having an open platform allows you to innovate alongside your partners, where you're not going at it alone. So the pace of innovation in an open platform like Helios becomes much faster. It's the force multiplier where you have multiple people coming And solving it together. You're looking at it at a data center scale. I think if you look at the network across Helios, whether it's across front -end networks where DPUs have a very, very important role to play, or across scale -out where the AI NICs become important, or even in the scale-up environment where you are bringing together dozens if not hundreds of GPUs in a large scale-up domain where you want to deliver memory as a shared resource, as a pool of resources, where your GPUs are acting as one big GPU, without the network participating And you innovating in that network, none of that is possible.

Dave Vellante

>> Okay, I'm really going to push you guys on this.

Soni Jiandani

>> I will go into that.

Omar Baldonado

>> I really want to dig deep on this And understand,

Omar Baldonado

>> but Omar, you had a comment.

Omar Baldonado

>> No, I was saying, if we talk about the network being under pressure And having to deliver, there are so many different ways to cast that problem. The networking in the rack, you can bring different technologies And co-design that together. Networking within a building, we can have a different discussion And different technologies. Networking across buildings, between data centers, that's yet another level of discussion. And so networking is not a single faceted problem, there's so many different domains that we have to work on together. And within Helios between the at scale, the NICs, even at the software layers, we're working together with AMD.

Soni Jiandani

>> And if you want I can give you a couple of examples. And at scale, we partnered very closely with Omar And Meta to make sure that the Ultra Accelerator Link (UALink) that becomes the fabric to bring together all the GPUs along with the large domain of memory which we enable, we have 50 % more memory than our competitor. How do we bring high availability attributes, because now the failure domain is dozens of GPUs, acting as one with a lot of memory. And these jobs are running for weeks, if not months. So you want to be in a position where it's a highly available fabric. And you want to recover dynamically from link layer failures, packet drops, packet corruption. So we partnered very closely with a company like Meta to ensure that at a software layer, not only with the hardware that we are delivering on Helios, but from a software layer, we are able to automate the recovery so the jobs are not interrupted, whether it's distributed inferencing workloads or large models for training.

Dave Vellante

>> Okay, so I want to ask how you guys work together, because when you hear the term extreme co-design from your competitor, we all know who you're talking about. When I first heard that term, I thought it was okay, the ecosystem, John Furrier And I talk about this all the time, yeah, it was an ecosystem play, it's not. NVIDIA will design it internally And then say, here it is, you can plug into it. I think you're taking a different philosophy. You're co-designing together. So let's dig into that. What specifically are you doing? And then I want to understand if there are trade-offs of that. So let's start with what you guys are co-designing together. What does that involve?

Soni Jiandani

>> So Omar, why don't you talk about what we did together on the scale-up network with UALink over Ethernet? Maybe that's a good place to start.

Omar Baldonado

>> I think that's a good one. So if we look at scale-up within the rack or potentially across multiple racks, you look at what are the workloads that are going to run at scale, certainly inference, but also certain parts of training. You want to say these parallel parts of the job run within a at scale domain. So we work on like, what types of parallelisms in a job, when you talk about a huge tens of thousands, hundreds of thousands of GPUs, you want to know what parts are running inside a at scale versus running on a scale out. And what do you need to optimize memory access, storage access within that part of a job. So I think some of it's sharing how, what's important to our workloads, And when I say our workloads, there are many different kinds of workloads that we have. Saying, hey, that's what we need to support in a at scale. The other part around at scale is the operational model. And that actually goes across the whole network, but within the at scale, because you want them all acting as a unit as quickly as possible. You want to know if there's any failure or if there's any interruption, how do I make sure that nothing is interrupted from a job perspective? And that's why getting a good operational model for scale out was something we also worked together on.

Dave Vellante

>> Okay, I'm going to take a deviation here. I thought you needed InfiniBand to do at scale. How do you get around that conventional wisdom?

Soni Jiandani

>> You needed InfiniBand to do scale out.

Dave Vellante

>> Yeah, I said at scale.

Soni Jiandani

>> Scale up.

Dave Vellante

>> I said at scale, I meant scale out, yeah.

Soni Jiandani

>> You needed it, but that was a few years ago. Because if you take a look at the latest numbers, Ethernet has surpassed InfiniBand in terms of the build -out of AI systems And the scale -out networks. More than 50 % of all scale -out networks are moving in the direction of Ethernet. So I think the customers are voting with their dollars. And it allows you to build today, for the frontier models, some of the largest scale -out networks are built off of Ethernet today. So, there again, we co -innovated together, because as you think about connecting hundreds And thousands of systems, racks, together over a scale -out network, you want the adaptability of while Ethernet becomes your transport, you want to have the ability to run custom protocols on top. So we were the first company to introduce AI NICs that are fully programmable for the scale -out networks for 400 gig. As part of Helios, we are the first AI NIC that's fully programmable for 800 gig. Why do you want to have the custom protocols or enhanced protocols like UEC or MRC to run in your scale -out NIC so that you can run your scale -out networks for AI in a much more efficient way, in a much more economical way? So when you use features like multi -plane, you're able to do it with almost a 30 % lower total cost of ownership because you're deploying one -third less switches in that infrastructure, but you're driving And delivering all the things over lossless Ethernet, or lossy fabric, which only was possible at a much smaller scale on InfiniBand.

Omar Baldonado

>> Okay, you would agree? We are big fans of Ethernet So we've worked together on Ethernet at scale, this is through OCP, the ESUN effort, but then also Ethernet for scale out. We've been using that And driving that, even as we built some InfiniBand network, but we're pushing And pushing on Ethernet as, this is what we're investing in for larger And larger networks. Ethernet, building larger And larger requires you to work very closely with the NIC to understand how are you going to handle varying distances. Because when we talk about scale out, there's scale out within a building, And I think I mentioned it, but once you leave a building, because you're trying to bring capacity together as fast as possible, And I need 100 megawatts here, 200 megawatts there, And trying to bring this all together, what happens is the transport And the NIC help you stitch this all together. Because if I want to pull together a gigawatt as fast as possible, I can't just design it from scratch. We do that too. But I also want to be able to bring stuff together as quickly as possible that's existing. And the NIC And transport help you do that.

Soni Jiandani

>> And that's why we partnered also on the custom protocol with Meta.

Omar Baldonado

>> Okay.

Soni Jiandani

>> On the AI NIC that is running the Meta custom protocol to bring the heterogeneous pool of GPU resources into being much more accessible to themselves as opposed to somebody telling, Santosh telling Mark Zuckerberg, you should have told me two years ago before you wanted a one gigawatt data center.

Omar Baldonado

>> On this custom transport, by the way, so we have been working with the programmable NIC nature, the programmable AI NIC that AMD has. we've been looking at that as we need custom transport. How do we design that with the teams, with NIC teams? And I can't go into too much, but more will be coming on that in the next little while. But we also need that to make things scale well.

Dave Vellante

>> Okay, And I think to your point on Ethernet And the market deciding, All the hyperscalers are using Ethernet And they have for a long, long time. I want to come back to this notion of co-innovation. You have to move much faster than Moore's Law. Yes. Okay, And so the competition would argue that you're building server farms, not AI factories, And that you won't be able to keep pace. What would you say to that? What makes it, server farm versus AI, I don't know what all that means, but I want to hear from the customer what the requirements are, but let me start with you, Soni, what makes this an AI -ready infrastructure, And are you able to keep pace with blowing away Moore's Law, whatever that metric is?

Soni Jiandani

>> So I think if you look at the pace of innovation in networking for AI has to move at AI speed. You can only do that if you have programmable infrastructure. That theme was brought up also by Lisa. It's highly programmable. You can move at AI speed if the devices that you are building, whether it is for scale out networking like our AI NIC, or your data processing unit, your DPUs for front end networks, are fully programmable. That's how we can move at the speed of the network. So doubling bandwidth every 18 to 24 months, going from 400 gig to 800 gig, we did it in one year. Now going from 800 gig to 1 .6 terabit, We're going to do it in another 12 months

Dave Vellante

>> Okay, so you're on a 12 -month cycle for that.

Soni Jiandani

>> We are on a 12, 18 -month cadence where you're doubling the speeds. But then comes the whole notion of custom transports. The ability to adapt with KVCache And have more capabilities to bring in storage in a disaggregated but yet in a shared way, accessible as a resource. Offloading the CPUs by taking on more tasks in your DPUs. All of that is only possible if you're programmable. Otherwise, you have to spin new ASICs in a highly dynamic environment. So in order to keep pace with AI speed And build AI factories, you need underlying infrastructure that's highly programmable. And that's a foundational element. That's a moat that AMD has for front -end networks, for back -end networks, And an Ethernet -based fabric that can be ubiquitous, including both the flexibility of UAL And UAL over Ethernet. And doing it not alone, but doing it with our ecosystem in a tightly co -developed manner And moving at the speed that the whole world is moving with AI.

Dave Vellante

>> And this is what you need, you need compute, actually you're arbitraging compute now, but we'll see where all that goes as you start to build up your frontier models, but what's your take on that, Omar?

Omar Baldonado

>> So I'll come back, I agree that there's so much that we need to work on to keep up with AI speeds. I will say that I've been at Meta for a long time, And the AI needs over the last six or seven years have been forcing us into a much faster cadence. Our team used to design the fabrics, And we would roll out a new fabric every few years, but now it's like, I need to roll out a fabric practically every year to handle the new use cases And the new capabilities. Otherwise you're introducing new bottlenecks. That's exactly right. It's a perpetual game. So it's a little bit like, the bottleneck is here. How do I make sure that technology is ready? The bottleneck's over there now. How do I make sure the technology is there?

Dave Vellante

>> Okay, where do these benchmarks come from?Let's dig into that. I'm seeing all these kinds of, I was like, wow. They are not pulling any punches here. So, okay, are they based on published specs? You got a rubric somewhere, that I don't know about? Is it, how are you?

Soni Jiandani

>> There must be a rubric for us to contrast And compare these benchmarks And for us to put up on stage.

Dave Vellante

>> Well, they published a lot of stuff out there. Okay, so.

Soni Jiandani

>> No, I think that, as Lisa mentioned, she put qualifiers on all the tests that were conducted per benchmark, And these are all open models. People that have access to Helios will be able to run those same benchmarks. So there wasn't much detail left out in my mind in terms of the benchmarks that we published.

Dave Vellante

>> Okay, so, it's funny, I wrote a piece the other day that, again, I said that all you have to be is the indispensable second platform, but I think your ambitions appear to be higher. So, okay. So let's move on here. So when you think about, we talked about scale across networks, from your standpoint, what are the most important attributes that you need from AMD, from a customer perspective? I wonder if you could talk about that, Omar.

Omar Baldonado

>> Yeah, yeah. So I think scale out, when you're building big scale-out clusters right now, this is where you're trying to bring together 100 megawatts here, 200 megawatts here, you're trying to bring together buildings or whole regions to be networked together. I think this support for a transport that is resilient to failures, can react quickly, because an AI job doesn't wait as other jobs may have been able to do Other jobs are like, hey there's a timeout, let me try again, it takes a little while. But AI jobs want to like, I've got 100 ,000, or however many GPUs, they got to stay in lock step. So having transport that is very adaptive And very quick to respond to conditions is critical. And so then working with AMD And their NIC technologies has been great to make sure that those use cases are addressed for us. Scale across is a tough problem because you can't cheat distances, physics is real, speed of light is real. And then you can't put as much bandwidth as you would want. I can build the whole fiber plant infrastructure from scratch And I know exactly what I'm going to put. Yeah, but when you're connecting across kilometers.

Soni Jiandani

>> At distances.

Omar Baldonado

>> I can't do that. And so that's where the software transport layer comes in to help.

Dave Vellante

>> Actually, can you explain that a little bit further? How does, talk about the magic sauce of that software transport layer. How does that get around that bottleneck?

Soni Jiandani

>> So I think if you look at, we have talked extensively about programmability And we co -developed a protocol called MRC. Multipath Reliable Connection, working with, And we co -developed it not alone, we co -developed it with OpenAI And Microsoft And a few other players in the industry. It is ultimately about increasing the performance, driving up the resilience on Ethernet as a protocol that you co -develop, And have the efficiency to run it across your scale -out networks that can also be extended to longer distances. Same work we did with Meta. We said Ethernet is a common factor for scale -out networks. Meta had unique needs to bring the heterogeneous pool of compute together. The example you gave, 100 megawatts here, another 200 megawatts, let me pool that. So I have a heterogeneous environment, nobody only has an all Vera Rubin environment or an all Helios environment How do I pool these resources together? That is where the magic, the secret sauce is in that definition of the transport, implementing it in hardware, which is highly programmable. So you're delivering at the speeds of the network, 800 gig case in point, And having the ability as that protocol evolves to keep evolving it in this programmable hardware rather than saying, Omar, you need to wait for my next generation chip because that was hardwired in it.

Omar Baldonado

>> Got it. Okay, And go ahead, Omar.

Omar Baldonado

>> So I will say that that is what we've, something that we've been focusing on in our transport research, right, And development. It has been, if you think of scale out only, scale out is more homogeneous because you know the distances, it's bounded, the delays are all kind of expected because you designed a building a certain way. But when you go scale across, it is heterogeneous distances, it is not equal bandwidth, things will fail differently, And you have to react differently. So that's something where I think we've been spending a lot of time to understand that to go faster on larger And larger clusters.

Omar Baldonado

>> And recovery, presumably is different.

Soni Jiandani

>> Recovery is key. Something goes wrong. How do I load balance? How do I optimize the performance using load balancing? How do I behave in congestion? It's congestion, networks are congested

Omar Baldonado

>> And it's different, scale out, how you handle those problems recovering in a scale out in a building is different than how you handle those problems in a scale-out.

Dave Vellante

>> And your point about homogeneous,homogeneous infrastructure kind of went out with mini computers.

Soni Jiandani

>> Exactly.

Dave Vellante

>> In the 70s. Okay, are we talking about traditional, when we talk about scale across, Are we talking about traditional synchronous distances or does it have to be compressed because of the complexities? Or whatever, 200 kilometers, are we talking about?

Omar Baldonado

>> Yeah, yeah. So there are a couple of things. A scale across, you can work on it at different layers And you have to work on it at different layers because at the job layer, let's say PyTorch or RCCL, which we've worked with AMD on, you handle distances And failures differently. Because you don't want to do a matrix multiply, tensor parallelism, across 200 kilometers. Doesn't make any sense. So you got to make sure that at the job level, you're only putting the types of operations that you want to scale across. And then, once you've handled that.

Omar Baldonado

>> And that's, sorry, that's a routing problem?

Omar Baldonado

>> That's actually a, it's a PyTorch And job scheduling And placement problem. because again, when you have a bunch of GPUs talking to each other within a at scale, they're going to all do some matrix multiplication very, very fast, you need super low latency. You don't want to do that across.

Dave Vellante

>> No, that's a job control.

Omar Baldonado

>> Yeah, okay, yeah, let me control that.

Dave Vellante

>> So I interrupted you, carry on. So once you solve that, carry on.

Omar Baldonado

>> Once you solve that, so you get the right workloads that you expect working across a scale-out, then you need to say, okay, but what happens if there's a failure? What happens if one link goes down between these two buildings. Then you have to say, okay, how can I quickly detect that? Can I tell everybody, make sure the job doesn't just restart?

Soni Jiandani

>> Time out or the whole system is now sitting idle because the job timed out

Omar Baldonado

>> So it's really this multi-layer problem of, let me make sure the jobs And placement are doing it properly. And then let me make sure that the transport is handling things properly. Then you even go layers. Is there good load balancing? Is it responding quickly? It's actually super fun.

Dave Vellante

>> Is Jitter a different problem in AI? And is that something that you have to deal with?

Soni Jiandani

>> Yes, at scale, you have to focus on making sure not only are you blazingly fast, but your throughput is very high, because you are also synchronizing memory, shared memory, it's like a pool of memory for all these high speed GPUs. Jitter is very important, low latency, And a consistent Jitter is very important, because these are all synchronized workloads, so you need to make sure that the network is very resilient. That's why reliability And availability is quintessential. Because you'll be running these jobs for a week, And you want system availability at all times.

Dave Vellante

>> Right, right, it is.

Soni Jiandani

>> So reliability is very important, as well in addition to Jitter.

Dave Vellante

>> Availability, because it's such an expensive resource, And you want to keep that resource busy.

Soni Jiandani

>> You want to keep the resource busy, And you want the job to be completed, which may take a few days or a week or two weeks.

Dave Vellante

>> Lisa talked about this, I think 2026 is kind of the crossover year where 60 % of the compute is now going to inference versus training. So that changes things. We were talking about Jeetu's chart that he showed with single prompt chat bots is very spiky. Now with agentic, you're painting the whole thing red. What kind of challenges does that bring in? And what kind of innovations do you need to support that?

Soni Jiandani

>> So I think at least looking at it through the lens of the innovations, we are working again with customers And in our own engineering place within our engineering team is, the wave of agentic is context bound. So you need vast quantities of KV cache data that needs to be shared across servers And racks And storage systems. There our DPU becomes a context infrastructure processor, where we have the ability with NVMe over Fabrics to treat disparate pools of storage as a shared pool, where we are accelerating that within the DPU, And then we are also having the ability to keep this whole KV cache management And take it on as a task that we're offloading into the DPU. Therefore, the ability now for me to keep all this context And KV cache local in the DPU allows the GPUs to have direct access into the DPU And access that context. I free up the CPU to go focus on agentic workloads And the maximized density for agents per CPU. So the DPU for the front end is also taking that role on. So I think much like the DPU for the front end has offloaded software -defined networking, has delivered inline security Now, you should come to expect that it'll play a bigger role in taking care of that third leg of the stool, which is storage And KV cache management.

Dave Vellante

>> And that storage hierarchy's becoming much more memory intensive.

Soni Jiandani

>> That's right.

Dave Vellante

>> And then the persistence layer kind of gets pushed down a little bit. Is that another bottleneck? I think about destaging the cache. Used to be a bottleneck. I don't know if that's a network problem. Is that a... I think everything's a network problem.

Omar Baldonado

>> But this is the great thing. So actually, agentic, we are, obviously we're very invested in that personal superintelligence, a personalized agent for you. But it is the ultimate network problem because I think if I look at AI before agentic, we tried to segment it into, this is a pre-training problem of networking. So you had certain patterns. Or this is a disaggregated inference, prefill decode, that's a certain problem. But when you look at agentic, it brings it all. It brings, I need optimized CPU access, I need optimized storage access, I still need to do inference, distributed inference. So all this has to play together in an agentic deployment, And the networking, there's five different networking problems that come together that have to be solved when you interact with an agent. It's not just this one-shot prompt, let me just get an answer. It's like, let me load your history, but you didn't talk to me for six hours, so where did your history go? Let me bring it in.

Soni Jiandani

>> That's why the access, it's context based, so you need to have the ability to have this information at the speed that you're running the network. And therefore, the network takes that role on, leaving the GPUs And the CPUs to actually take on more agentic work.

Dave Vellante

>> So much is changing so fast. This has been like a master class in networking, I love it. I was writing down some things the other day of the assumptions that we were making about the industry that were totally wrong. Like, oh, rag -based chatbots are going to be amazing And drive a lot of value. There's going to be one dominant player in AI. Token costs, they don't matter. Tokens will be free. US frontier models, they're going to own the world. Everything's going to go to the cloud. All of that, you can just throw it away because it was wrong.

Omar Baldonado

>> Exactly.

Dave Vellante

>> So my question is how do you guys, maybe you can see around corners, but you have to architect because you can't predict exactly what the next trend is going to be. So that makes your job really interesting And challenging. How do you think about that? How do you architect for that?

Soni Jiandani

>> So again, what's exciting for me, at least I've been in networking forever as you know, what's exciting for me is the way we are going about it is with an open ethos And a strong partnership with customers like Meta. No one company can figure this out alone. As you're building AI infrastructure for the next five years, having the opportunity to co-design, co -develop, think about where the next generation of workloads will land, gives you the opportunity not just to bring the best of assets that we have, but also to bring the best of assets with our customers along. And having that lever is, in my humble opinion, a huge vantage point because ultimately the market votes with their dollars for systems that are highly innovative, but that are highly open.

Dave Vellante

>> All right. Omar, I'll give you the final word.

Omar Baldonado

>> All right, I think a couple things. On the how do we look ahead, we stay very close to our AI researchers across the company, And in fact across the ecosystem, And try to understand And predict, hey, where does the networking infrastructure need to be? So that's how we try to anticipate If I look at our fleet, we run a multi -generational fleet, we're running some things in large scale operations, but I'm also looking ahead to five, six years, what innovations are we going to work on there? So it's all in play simultaneously, so that's the way we look ahead is to make sure we understand what the researchers are looking for in the model perspective, use case perspective, And then try to anticipate And translate that into. I love the partnership And being open And working at all these different layers. Networking is super fun. I think it's super challenging all this stuff to do.

Soni Jiandani

>> Before coming here, David, we were just talking about that next generation platform, the use cases.

Dave Vellante

>> Yeah, yeah, amazing.

Soni Jiandani

>> So the engineers from his team And our engineers are sitting on this floor in a conference room talking about that next generation. Trying to predict what that looks like. What the use cases are.

Dave Vellante

>> Yeah, right, okay.

Soni Jiandani

>> Exactly.

Dave Vellante

>> Guys, amazing, we're seeing Meta, such a transformation. whether it's Metaverse or now leading on AI, Dave Brown's joining Meta from AWS, that's a huge, huge pickup. We're seeing AMD transform from a chip company into a full systems company, networking, huge part of that. Soni And Omar, thanks so much for coming to theCUBE.

Soni Jiandani

>> Thank you, it's a pleasure.

Dave Vellante

>> Thank you, thank you very much.

Omar Baldonado

>> Pleasure.

Dave Vellante

>> Thank you for watching, this is Dave Vellante. We'll be right back from Moscone West, AMD's Advancing AI, I'm Dave Vellante, you're watching theCUBE. Excellent. From the biggest stories to the most complex,

Soni Jiandani, AMD & Omar Baldonado, Meta

search

(INTRO)

Dave Vellante

>> Hey everybody, welcome back to Moscone West. My name is Dave Vellante. You're watching theCUBE's live coverage of AMD's Advancing AI. I hear 14 ,000 people registered for this event. Amazing keynote this morning by Lisa Su. I am super excited. We're going to go deep into networking with a long time CUBE friend, Soni Jiandani. She's here, she's Senior Vice President And GM of the Networking Technology And Solutions Group (AMD) at AMD. I'm joined by Omar Baldonado, Senior Director at Data Center And AI Networking at Meta. You've heard of them. Folks, great to see you. Thanks so much for spending some time with us.

Omar Baldonado

>> Great to be here, thank you.

Dave Vellante

>> Thanks for having us.

Dave Vellante

>> Great keynote by Lisa. she talked about how AI demand is absolutely exploding across, she laid out the workloads. AMD, amazing. Chip company turning into a full -scale systems company with rack -scale systems like Helios. We're going to get into that, really at the center of all this. She talked about 35 quadrillion tokens consumed every month, And that's an increase of 158x in just two years, but the real thing I took away was the curve is steepening. So the demand is going to be there for a little bit. Don't freak out too much about the AI bubble. It will be okay for a little while here, but I want to start by just asking you to talk more about the importance of networking in the whole context of AI systems, generally And Helios specifically. Maybe Soni, you could start off.

Soni Jiandani

>> So I think some of the best statements that I heard from Lisa And our customers today were, the unit of deployment used to be a rack, it's now the data center. So you have to think about all things coming together, which is building system level approaches to a data center, not just to feed the beast, the GPU, but also the CPU is now a first class citizen with agentic AI, And the role of the network is only increasing, because if you have to deliver these systems at scale with performance, with high availability, And this type of token consumption of 158X in two years, the network is foundational to deliver a balanced approach to processing those types of tokens And delivering the results that the customers want with the evolution towards these large frontier models And agentic AI, the network couldn't be more foundational. And that's how we have been able to innovate as part of Helios, where we are now claiming a 15 % higher throughput when compared to our competitors that have been delivering rack scale designs with Helios, as well as 30 % more tokens that we can generate per dollar, which ultimately is the metric for customers.

Dave Vellante

>> I'm going to push you on those benchmarks. We'll come back to that.

Soni Jiandani

>> Sure.

Dave Vellante

>> Because those were bold statements. But I wanted to add, Omar, one of the other key attributes And AMD's playing this card hard And the marketing standpoint is openness. I was there at the first OCP conference. You guys were very instrumental. I don't know if you were with Meta at the time, but very instrumental in setting that foundation for open infrastructure. even today the innovations like warm water cooling. that is changing the game. But I wonder if you could talk about, from your perspective, the importance of things like rack scale, the data center, is that how you think about it?

Omar Baldonado

>> Yeah, no, absolutely. Absolutely, we do think, And we do think of it as a data center scale problem And even beyond. I think Santosh And Lisa talked about that this morning. But let me go into the openness first.

Omar Baldonado

>> Yeah, please.

Omar Baldonado

>> With the OCP, we were founders of OCP, co-founders of a number of other companies, And in fact we helped start the OCP Networking way back when, And it was because we saw that networking needs the openness, because it's about plugging things together, And so many different technologies across the stack have to interoperate And work together, so our approach was to do it openly. And I think that's actually carried into the partnership with AMD on Helios. Be as open as we can on all the different parts of the network. So let's talk about some of the challenges in networking.

Dave Vellante

>> We were talking off camera, it used to be when you had spinning disks, spinning rust I call it, that was the bottleneck And you took some of the pressure off the network, that's changed with flash storage And new storage architectures, memory architectures, KV cache, that bottleneck is gone, so it pushes all the pressure onto the network. So let's discuss those bottlenecks, And then let's bring it to Helios, some of the innovations that you've had to deal with to address those challenges.

Soni Jiandani

>> So I think the beautiful part about our architecture is that it's an open architecture, which gives us an opportunity not only to partner with customers And the people that are leading the AI frontier models directly, And this has been a relationship over the many years where we have innovated to lead to the point we are at on Helios, by having an open platform you have the ability to see where the puck is landing by working with these large frontier AI model companies. Ultimately you care about driving the outcomes And having an open platform allows you to innovate alongside your partners, where you're not going at it alone. So the pace of innovation in an open platform like Helios becomes much faster. It's the force multiplier where you have multiple people coming And solving it together. You're looking at it at a data center scale. I think if you look at the network across Helios, whether it's across front -end networks where DPUs have a very, very important role to play, or across scale -out where the AI NICs become important, or even in the scale-up environment where you are bringing together dozens if not hundreds of GPUs in a large scale-up domain where you want to deliver memory as a shared resource, as a pool of resources, where your GPUs are acting as one big GPU, without the network participating And you innovating in that network, none of that is possible.

Dave Vellante

>> Okay, I'm really going to push you guys on this.

Soni Jiandani

>> I will go into that.

Omar Baldonado

>> I really want to dig deep on this And understand,

Omar Baldonado

>> but Omar, you had a comment.

Omar Baldonado

>> No, I was saying, if we talk about the network being under pressure And having to deliver, there are so many different ways to cast that problem. The networking in the rack, you can bring different technologies And co-design that together. Networking within a building, we can have a different discussion And different technologies. Networking across buildings, between data centers, that's yet another level of discussion. And so networking is not a single faceted problem, there's so many different domains that we have to work on together. And within Helios between the at scale, the NICs, even at the software layers, we're working together with AMD.

Soni Jiandani

>> And if you want I can give you a couple of examples. And at scale, we partnered very closely with Omar And Meta to make sure that the Ultra Accelerator Link (UALink) that becomes the fabric to bring together all the GPUs along with the large domain of memory which we enable, we have 50 % more memory than our competitor. How do we bring high availability attributes, because now the failure domain is dozens of GPUs, acting as one with a lot of memory. And these jobs are running for weeks, if not months. So you want to be in a position where it's a highly available fabric. And you want to recover dynamically from link layer failures, packet drops, packet corruption. So we partnered very closely with a company like Meta to ensure that at a software layer, not only with the hardware that we are delivering on Helios, but from a software layer, we are able to automate the recovery so the jobs are not interrupted, whether it's distributed inferencing workloads or large models for training.

Dave Vellante

>> Okay, so I want to ask how you guys work together, because when you hear the term extreme co-design from your competitor, we all know who you're talking about. When I first heard that term, I thought it was okay, the ecosystem, John Furrier And I talk about this all the time, yeah, it was an ecosystem play, it's not. NVIDIA will design it internally And then say, here it is, you can plug into it. I think you're taking a different philosophy. You're co-designing together. So let's dig into that. What specifically are you doing? And then I want to understand if there are trade-offs of that. So let's start with what you guys are co-designing together. What does that involve?

Soni Jiandani

>> So Omar, why don't you talk about what we did together on the scale-up network with UALink over Ethernet? Maybe that's a good place to start.

Omar Baldonado

>> I think that's a good one. So if we look at scale-up within the rack or potentially across multiple racks, you look at what are the workloads that are going to run at scale, certainly inference, but also certain parts of training. You want to say these parallel parts of the job run within a at scale domain. So we work on like, what types of parallelisms in a job, when you talk about a huge tens of thousands, hundreds of thousands of GPUs, you want to know what parts are running inside a at scale versus running on a scale out. And what do you need to optimize memory access, storage access within that part of a job. So I think some of it's sharing how, what's important to our workloads, And when I say our workloads, there are many different kinds of workloads that we have. Saying, hey, that's what we need to support in a at scale. The other part around at scale is the operational model. And that actually goes across the whole network, but within the at scale, because you want them all acting as a unit as quickly as possible. You want to know if there's any failure or if there's any interruption, how do I make sure that nothing is interrupted from a job perspective? And that's why getting a good operational model for scale out was something we also worked together on.

Dave Vellante

>> Okay, I'm going to take a deviation here. I thought you needed InfiniBand to do at scale. How do you get around that conventional wisdom?

Soni Jiandani

>> You needed InfiniBand to do scale out.

Dave Vellante

>> Yeah, I said at scale.

Soni Jiandani

>> Scale up.

Dave Vellante

>> I said at scale, I meant scale out, yeah.

Soni Jiandani

>> You needed it, but that was a few years ago. Because if you take a look at the latest numbers, Ethernet has surpassed InfiniBand in terms of the build -out of AI systems And the scale -out networks. More than 50 % of all scale -out networks are moving in the direction of Ethernet. So I think the customers are voting with their dollars. And it allows you to build today, for the frontier models, some of the largest scale -out networks are built off of Ethernet today. So, there again, we co -innovated together, because as you think about connecting hundreds And thousands of systems, racks, together over a scale -out network, you want the adaptability of while Ethernet becomes your transport, you want to have the ability to run custom protocols on top. So we were the first company to introduce AI NICs that are fully programmable for the scale -out networks for 400 gig. As part of Helios, we are the first AI NIC that's fully programmable for 800 gig. Why do you want to have the custom protocols or enhanced protocols like UEC or MRC to run in your scale -out NIC so that you can run your scale -out networks for AI in a much more efficient way, in a much more economical way? So when you use features like multi -plane, you're able to do it with almost a 30 % lower total cost of ownership because you're deploying one -third less switches in that infrastructure, but you're driving And delivering all the things over lossless Ethernet, or lossy fabric, which only was possible at a much smaller scale on InfiniBand.

Omar Baldonado

>> Okay, you would agree? We are big fans of Ethernet So we've worked together on Ethernet at scale, this is through OCP, the ESUN effort, but then also Ethernet for scale out. We've been using that And driving that, even as we built some InfiniBand network, but we're pushing And pushing on Ethernet as, this is what we're investing in for larger And larger networks. Ethernet, building larger And larger requires you to work very closely with the NIC to understand how are you going to handle varying distances. Because when we talk about scale out, there's scale out within a building, And I think I mentioned it, but once you leave a building, because you're trying to bring capacity together as fast as possible, And I need 100 megawatts here, 200 megawatts there, And trying to bring this all together, what happens is the transport And the NIC help you stitch this all together. Because if I want to pull together a gigawatt as fast as possible, I can't just design it from scratch. We do that too. But I also want to be able to bring stuff together as quickly as possible that's existing. And the NIC And transport help you do that.

Soni Jiandani

>> And that's why we partnered also on the custom protocol with Meta.

Omar Baldonado

>> Okay.

Soni Jiandani

>> On the AI NIC that is running the Meta custom protocol to bring the heterogeneous pool of GPU resources into being much more accessible to themselves as opposed to somebody telling, Santosh telling Mark Zuckerberg, you should have told me two years ago before you wanted a one gigawatt data center.

Omar Baldonado

>> On this custom transport, by the way, so we have been working with the programmable NIC nature, the programmable AI NIC that AMD has. we've been looking at that as we need custom transport. How do we design that with the teams, with NIC teams? And I can't go into too much, but more will be coming on that in the next little while. But we also need that to make things scale well.

Dave Vellante

>> Okay, And I think to your point on Ethernet And the market deciding, All the hyperscalers are using Ethernet And they have for a long, long time. I want to come back to this notion of co-innovation. You have to move much faster than Moore's Law. Yes. Okay, And so the competition would argue that you're building server farms, not AI factories, And that you won't be able to keep pace. What would you say to that? What makes it, server farm versus AI, I don't know what all that means, but I want to hear from the customer what the requirements are, but let me start with you, Soni, what makes this an AI -ready infrastructure, And are you able to keep pace with blowing away Moore's Law, whatever that metric is?

Soni Jiandani

>> So I think if you look at the pace of innovation in networking for AI has to move at AI speed. You can only do that if you have programmable infrastructure. That theme was brought up also by Lisa. It's highly programmable. You can move at AI speed if the devices that you are building, whether it is for scale out networking like our AI NIC, or your data processing unit, your DPUs for front end networks, are fully programmable. That's how we can move at the speed of the network. So doubling bandwidth every 18 to 24 months, going from 400 gig to 800 gig, we did it in one year. Now going from 800 gig to 1 .6 terabit, We're going to do it in another 12 months

Dave Vellante

>> Okay, so you're on a 12 -month cycle for that.

Soni Jiandani

>> We are on a 12, 18 -month cadence where you're doubling the speeds. But then comes the whole notion of custom transports. The ability to adapt with KVCache And have more capabilities to bring in storage in a disaggregated but yet in a shared way, accessible as a resource. Offloading the CPUs by taking on more tasks in your DPUs. All of that is only possible if you're programmable. Otherwise, you have to spin new ASICs in a highly dynamic environment. So in order to keep pace with AI speed And build AI factories, you need underlying infrastructure that's highly programmable. And that's a foundational element. That's a moat that AMD has for front -end networks, for back -end networks, And an Ethernet -based fabric that can be ubiquitous, including both the flexibility of UAL And UAL over Ethernet. And doing it not alone, but doing it with our ecosystem in a tightly co -developed manner And moving at the speed that the whole world is moving with AI.

Dave Vellante

>> And this is what you need, you need compute, actually you're arbitraging compute now, but we'll see where all that goes as you start to build up your frontier models, but what's your take on that, Omar?

Omar Baldonado

>> So I'll come back, I agree that there's so much that we need to work on to keep up with AI speeds. I will say that I've been at Meta for a long time, And the AI needs over the last six or seven years have been forcing us into a much faster cadence. Our team used to design the fabrics, And we would roll out a new fabric every few years, but now it's like, I need to roll out a fabric practically every year to handle the new use cases And the new capabilities. Otherwise you're introducing new bottlenecks. That's exactly right. It's a perpetual game. So it's a little bit like, the bottleneck is here. How do I make sure that technology is ready? The bottleneck's over there now. How do I make sure the technology is there?

Dave Vellante

>> Okay, where do these benchmarks come from?Let's dig into that. I'm seeing all these kinds of, I was like, wow. They are not pulling any punches here. So, okay, are they based on published specs? You got a rubric somewhere, that I don't know about? Is it, how are you?

Soni Jiandani

>> There must be a rubric for us to contrast And compare these benchmarks And for us to put up on stage.

Dave Vellante

>> Well, they published a lot of stuff out there. Okay, so.

Soni Jiandani

>> No, I think that, as Lisa mentioned, she put qualifiers on all the tests that were conducted per benchmark, And these are all open models. People that have access to Helios will be able to run those same benchmarks. So there wasn't much detail left out in my mind in terms of the benchmarks that we published.

Dave Vellante

>> Okay, so, it's funny, I wrote a piece the other day that, again, I said that all you have to be is the indispensable second platform, but I think your ambitions appear to be higher. So, okay. So let's move on here. So when you think about, we talked about scale across networks, from your standpoint, what are the most important attributes that you need from AMD, from a customer perspective? I wonder if you could talk about that, Omar.

Omar Baldonado

>> Yeah, yeah. So I think scale out, when you're building big scale-out clusters right now, this is where you're trying to bring together 100 megawatts here, 200 megawatts here, you're trying to bring together buildings or whole regions to be networked together. I think this support for a transport that is resilient to failures, can react quickly, because an AI job doesn't wait as other jobs may have been able to do Other jobs are like, hey there's a timeout, let me try again, it takes a little while. But AI jobs want to like, I've got 100 ,000, or however many GPUs, they got to stay in lock step. So having transport that is very adaptive And very quick to respond to conditions is critical. And so then working with AMD And their NIC technologies has been great to make sure that those use cases are addressed for us. Scale across is a tough problem because you can't cheat distances, physics is real, speed of light is real. And then you can't put as much bandwidth as you would want. I can build the whole fiber plant infrastructure from scratch And I know exactly what I'm going to put. Yeah, but when you're connecting across kilometers.

Soni Jiandani

>> At distances.

Omar Baldonado

>> I can't do that. And so that's where the software transport layer comes in to help.

Dave Vellante

>> Actually, can you explain that a little bit further? How does, talk about the magic sauce of that software transport layer. How does that get around that bottleneck?

Soni Jiandani

>> So I think if you look at, we have talked extensively about programmability And we co -developed a protocol called MRC. Multipath Reliable Connection, working with, And we co -developed it not alone, we co -developed it with OpenAI And Microsoft And a few other players in the industry. It is ultimately about increasing the performance, driving up the resilience on Ethernet as a protocol that you co -develop, And have the efficiency to run it across your scale -out networks that can also be extended to longer distances. Same work we did with Meta. We said Ethernet is a common factor for scale -out networks. Meta had unique needs to bring the heterogeneous pool of compute together. The example you gave, 100 megawatts here, another 200 megawatts, let me pool that. So I have a heterogeneous environment, nobody only has an all Vera Rubin environment or an all Helios environment How do I pool these resources together? That is where the magic, the secret sauce is in that definition of the transport, implementing it in hardware, which is highly programmable. So you're delivering at the speeds of the network, 800 gig case in point, And having the ability as that protocol evolves to keep evolving it in this programmable hardware rather than saying, Omar, you need to wait for my next generation chip because that was hardwired in it.

Omar Baldonado

>> Got it. Okay, And go ahead, Omar.

Omar Baldonado

>> So I will say that that is what we've, something that we've been focusing on in our transport research, right, And development. It has been, if you think of scale out only, scale out is more homogeneous because you know the distances, it's bounded, the delays are all kind of expected because you designed a building a certain way. But when you go scale across, it is heterogeneous distances, it is not equal bandwidth, things will fail differently, And you have to react differently. So that's something where I think we've been spending a lot of time to understand that to go faster on larger And larger clusters.

Omar Baldonado

>> And recovery, presumably is different.

Soni Jiandani

>> Recovery is key. Something goes wrong. How do I load balance? How do I optimize the performance using load balancing? How do I behave in congestion? It's congestion, networks are congested

Omar Baldonado

>> And it's different, scale out, how you handle those problems recovering in a scale out in a building is different than how you handle those problems in a scale-out.

Dave Vellante

>> And your point about homogeneous,homogeneous infrastructure kind of went out with mini computers.

Soni Jiandani

>> Exactly.

Dave Vellante

>> In the 70s. Okay, are we talking about traditional, when we talk about scale across, Are we talking about traditional synchronous distances or does it have to be compressed because of the complexities? Or whatever, 200 kilometers, are we talking about?

Omar Baldonado

>> Yeah, yeah. So there are a couple of things. A scale across, you can work on it at different layers And you have to work on it at different layers because at the job layer, let's say PyTorch or RCCL, which we've worked with AMD on, you handle distances And failures differently. Because you don't want to do a matrix multiply, tensor parallelism, across 200 kilometers. Doesn't make any sense. So you got to make sure that at the job level, you're only putting the types of operations that you want to scale across. And then, once you've handled that.

Omar Baldonado

>> And that's, sorry, that's a routing problem?

Omar Baldonado

>> That's actually a, it's a PyTorch And job scheduling And placement problem. because again, when you have a bunch of GPUs talking to each other within a at scale, they're going to all do some matrix multiplication very, very fast, you need super low latency. You don't want to do that across.

Dave Vellante

>> No, that's a job control.

Omar Baldonado

>> Yeah, okay, yeah, let me control that.

Dave Vellante

>> So I interrupted you, carry on. So once you solve that, carry on.

Omar Baldonado

>> Once you solve that, so you get the right workloads that you expect working across a scale-out, then you need to say, okay, but what happens if there's a failure? What happens if one link goes down between these two buildings. Then you have to say, okay, how can I quickly detect that? Can I tell everybody, make sure the job doesn't just restart?

Soni Jiandani

>> Time out or the whole system is now sitting idle because the job timed out

Omar Baldonado

>> So it's really this multi-layer problem of, let me make sure the jobs And placement are doing it properly. And then let me make sure that the transport is handling things properly. Then you even go layers. Is there good load balancing? Is it responding quickly? It's actually super fun.

Dave Vellante

>> Is Jitter a different problem in AI? And is that something that you have to deal with?

Soni Jiandani

>> Yes, at scale, you have to focus on making sure not only are you blazingly fast, but your throughput is very high, because you are also synchronizing memory, shared memory, it's like a pool of memory for all these high speed GPUs. Jitter is very important, low latency, And a consistent Jitter is very important, because these are all synchronized workloads, so you need to make sure that the network is very resilient. That's why reliability And availability is quintessential. Because you'll be running these jobs for a week, And you want system availability at all times.

Dave Vellante

>> Right, right, it is.

Soni Jiandani

>> So reliability is very important, as well in addition to Jitter.

Dave Vellante

>> Availability, because it's such an expensive resource, And you want to keep that resource busy.

Soni Jiandani

>> You want to keep the resource busy, And you want the job to be completed, which may take a few days or a week or two weeks.

Dave Vellante

>> Lisa talked about this, I think 2026 is kind of the crossover year where 60 % of the compute is now going to inference versus training. So that changes things. We were talking about Jeetu's chart that he showed with single prompt chat bots is very spiky. Now with agentic, you're painting the whole thing red. What kind of challenges does that bring in? And what kind of innovations do you need to support that?

Soni Jiandani

>> So I think at least looking at it through the lens of the innovations, we are working again with customers And in our own engineering place within our engineering team is, the wave of agentic is context bound. So you need vast quantities of KV cache data that needs to be shared across servers And racks And storage systems. There our DPU becomes a context infrastructure processor, where we have the ability with NVMe over Fabrics to treat disparate pools of storage as a shared pool, where we are accelerating that within the DPU, And then we are also having the ability to keep this whole KV cache management And take it on as a task that we're offloading into the DPU. Therefore, the ability now for me to keep all this context And KV cache local in the DPU allows the GPUs to have direct access into the DPU And access that context. I free up the CPU to go focus on agentic workloads And the maximized density for agents per CPU. So the DPU for the front end is also taking that role on. So I think much like the DPU for the front end has offloaded software -defined networking, has delivered inline security Now, you should come to expect that it'll play a bigger role in taking care of that third leg of the stool, which is storage And KV cache management.

Dave Vellante

>> And that storage hierarchy's becoming much more memory intensive.

Soni Jiandani

>> That's right.

Dave Vellante

>> And then the persistence layer kind of gets pushed down a little bit. Is that another bottleneck? I think about destaging the cache. Used to be a bottleneck. I don't know if that's a network problem. Is that a... I think everything's a network problem.

Omar Baldonado

>> But this is the great thing. So actually, agentic, we are, obviously we're very invested in that personal superintelligence, a personalized agent for you. But it is the ultimate network problem because I think if I look at AI before agentic, we tried to segment it into, this is a pre-training problem of networking. So you had certain patterns. Or this is a disaggregated inference, prefill decode, that's a certain problem. But when you look at agentic, it brings it all. It brings, I need optimized CPU access, I need optimized storage access, I still need to do inference, distributed inference. So all this has to play together in an agentic deployment, And the networking, there's five different networking problems that come together that have to be solved when you interact with an agent. It's not just this one-shot prompt, let me just get an answer. It's like, let me load your history, but you didn't talk to me for six hours, so where did your history go? Let me bring it in.

Soni Jiandani

>> That's why the access, it's context based, so you need to have the ability to have this information at the speed that you're running the network. And therefore, the network takes that role on, leaving the GPUs And the CPUs to actually take on more agentic work.

Dave Vellante

>> So much is changing so fast. This has been like a master class in networking, I love it. I was writing down some things the other day of the assumptions that we were making about the industry that were totally wrong. Like, oh, rag -based chatbots are going to be amazing And drive a lot of value. There's going to be one dominant player in AI. Token costs, they don't matter. Tokens will be free. US frontier models, they're going to own the world. Everything's going to go to the cloud. All of that, you can just throw it away because it was wrong.

Omar Baldonado

>> Exactly.

Dave Vellante

>> So my question is how do you guys, maybe you can see around corners, but you have to architect because you can't predict exactly what the next trend is going to be. So that makes your job really interesting And challenging. How do you think about that? How do you architect for that?

Soni Jiandani

>> So again, what's exciting for me, at least I've been in networking forever as you know, what's exciting for me is the way we are going about it is with an open ethos And a strong partnership with customers like Meta. No one company can figure this out alone. As you're building AI infrastructure for the next five years, having the opportunity to co-design, co -develop, think about where the next generation of workloads will land, gives you the opportunity not just to bring the best of assets that we have, but also to bring the best of assets with our customers along. And having that lever is, in my humble opinion, a huge vantage point because ultimately the market votes with their dollars for systems that are highly innovative, but that are highly open.

Dave Vellante

>> All right. Omar, I'll give you the final word.

Omar Baldonado

>> All right, I think a couple things. On the how do we look ahead, we stay very close to our AI researchers across the company, And in fact across the ecosystem, And try to understand And predict, hey, where does the networking infrastructure need to be? So that's how we try to anticipate If I look at our fleet, we run a multi -generational fleet, we're running some things in large scale operations, but I'm also looking ahead to five, six years, what innovations are we going to work on there? So it's all in play simultaneously, so that's the way we look ahead is to make sure we understand what the researchers are looking for in the model perspective, use case perspective, And then try to anticipate And translate that into. I love the partnership And being open And working at all these different layers. Networking is super fun. I think it's super challenging all this stuff to do.

Soni Jiandani

>> Before coming here, David, we were just talking about that next generation platform, the use cases.

Dave Vellante

>> Yeah, yeah, amazing.

Soni Jiandani

>> So the engineers from his team And our engineers are sitting on this floor in a conference room talking about that next generation. Trying to predict what that looks like. What the use cases are.

Dave Vellante

>> Yeah, right, okay.

Soni Jiandani

>> Exactly.

Dave Vellante

>> Guys, amazing, we're seeing Meta, such a transformation. whether it's Metaverse or now leading on AI, Dave Brown's joining Meta from AWS, that's a huge, huge pickup. We're seeing AMD transform from a chip company into a full systems company, networking, huge part of that. Soni And Omar, thanks so much for coming to theCUBE.

Soni Jiandani

>> Thank you, it's a pleasure.

Dave Vellante

>> Thank you, thank you very much.

Omar Baldonado

>> Pleasure.

Dave Vellante

>> Thank you for watching, this is Dave Vellante. We'll be right back from Moscone West, AMD's Advancing AI, I'm Dave Vellante, you're watching theCUBE. Excellent. From the biggest stories to the most complex,