The Networking for AI Summit | Praful Lalchandani, Jon Green & Bharath Ramesh, HPE Networking

Clips
More from The Networking for AI Summit

Praful Lalchandani

VP PLM

HPE Networking

Jon Green

CTO of Networking

HPE Networking

Bharath Ramesh

AI Performance Engineer

HPE Networking

play_circle_outline Transforming Connectivity: HPE Networking Team's Role in Accelerating AI Solutions for Enterprise Deployment

play_circle_outline Unique demands of AI factories compared to traditional enterprise data centers.

play_circle_outline Intent-based automation and self-driving networking to simplify operations and deployments.

play_circle_outline AI factories require resilient networks that can self-heal and minimize downtime.

play_circle_outline Enhancing AI Networking: HPE and Juniper's Strategic Integration Boosts Operational Efficiency and Infrastructure Collaboration

Info
Transcript

Praful Lalchandani, Jon Green & Bharath Ramesh, HPE Networking

Praful Lalchandani

VP PLM HPE Networking

Jon Green

CTO of Networking HPE Networking

Bharath Ramesh

AI Performance Engineer HPE Networking

AI factories move from pilot to production in this in-depth conversation hosted by theCUBE Research’s Bob Laliberte. He’s joined by Praful Lalchandani, Vice President of Products for Data Center Business Unit, HPE Networking; Bharath Ramesh, Head of Product for AI Factory Solutions, HPE; and Jon Green, Chief Technology Officer and Chief Security Officer, HPE Networking. The panel examines why enterprise networks are now the foundation for training, tuning and large-scale inferencing. They detail reference architectures that scale from 16 nodes to hundreds of ... Read more

explore Keep Exploring

What is the current state of enterprises' transition from AI pilots to production scale deployments, and what demands does this shift place on the data center network? add

What is driving the trend of repatriation of workloads from public cloud to on-premise AI, and what challenges do clients face during this transition? add

What challenges does network congestion pose for AI applications, and how can enterprises address these challenges to enhance network performance? add

What considerations are important when building large-scale AI systems to ensure resilience and minimize downtime? add

What recent developments have occurred regarding the collaboration between Juniper and HPE, and what benefits does this integration offer to enterprises? add

bolt Powered by CUBE AI

Praful Lalchandani, Jon Green & Bharath Ramesh, HPE Networking

search

Bob Laliberte

>> Hello and welcome to the Networking for AI Summit. I'm Bob Laliberte, Principal Analyst at theCUBE Research, and I'm very excited to host today's discussion with the HPE Networking team. I'd like to start by welcoming our distinguished panelists, Praful Lalchandani, the Vice President of Products for Data Center Business Unit, Bharath Ramesh, Head of Product for AI Factory Solutions, and Jon Green, Chief Technology Officer and Chief Security Officer for HPE Networking. Welcome.

Bharath Ramesh

>> Thanks, Bob.

Bob Laliberte

>> Absolutely. Great to have all you here. And it's interesting because right now, we're really at this pivotal moment where enterprises are moving rapidly from AI pilots into that production scale deployments. And that shift is really putting new and urgent demands on the data center network, from building high performance fabrics for training and inferencing to simplifying operations with automation, to embedding security directly into the network. And so, that's what we're going to do today is really explore how HPE is enabling that next generation of AI-ready networks and how the integration of Juniper's capabilities further strengthens that vision. So let's just dive right in. Bharath, I'd like to start with you, as I mentioned, enterprises are shifting to these large-scale production deployments with AI. That transition is putting very different demands on the data center and especially the network fabric. From your perspective, what makes networking and AI factories fundamentally different from the traditional enterprise data centers?

Bharath Ramesh

>> Yeah, Bob, that's a great question. And we are indeed seeing this acceleration of workloads getting repatriated from the public cloud to on-premise AI because they're mission-critical or because the data gravity is within their network. And what we're seeing is when clients navigate this transition, they realize it's a huge chasm between building and operating it in the cloud and building and operating an AI platform within your premise. And the particular reason why this happens is because with the AI factories, you need to comprehend a huge spectrum of AI and non-AI workloads, each of which have unique dependencies on the network fabric, which if you leave unaddressed, essentially you have very expensive GPUs and very expensive users sitting, waiting for things to complete. And so the way I think about it is, look at the most demanding workloads. AI training, for example, you're taking a model, you're sharding it across hundreds of thousands of GPU nodes. Each node is kind of loading their piece of training data. They're using that to further refine model parameters. The nodes are checkpointing because you don't want to base an entire training run because a GPU failed or you run into some issue during the training run. And at the end of the training epoch, these nodes then have to integrate their progress and evaluate the model performance. So it's a very comprehensive cycle. It repeats again and again until the model converges to the desired result. This drives huge amounts of east-west traffic, which means your GPU-to-GPU connectivity within the cluster is absolutely critical, right? High bandwidth is a given, but it needs to be very low latency connectivity between these nodes. We can't do best effort anymore. We need to have guaranteed delivery of packets, techniques like NVLink help, but again, you're limited to a smaller scale, right? Eight GPUs to 72 GPUs. We often see things like InfiniBand being used for that part of the fabric, but we're also seeing Ethernet really starting to catch up, certainly on the performance front, but definitely on the economics and ease of management front. And then you look at the north-south fabric, I mean that's not immune to any of these performance demands as well, right? Because you have client traffic flowing through that network. You have traffic into storage going over that network. And we've seen things like 800 gig Ethernet and techniques like ROCEv2 and GPUDirect RDMA being used to bridge the performance gap, but that is also evolving as the demands of these workloads evolve. And that's just training. We haven't even talked about tuning, we haven't talked about the demands of scale-out inferencing and agentic AI and all that good stuff that's going to drive more complexity and demand to the network fabric. So in short, that's kind of a long way of saying we're seeing AI workloads really putting some new demands on the network and design and performance of the network. And we're super excited to have the Juniper team and their expertise on board to help us operationalize this for our customers.

Bob Laliberte

>> Yeah. And that's really the key is helping them get to that operational state more quickly as you mentioned as organizations are ramping things up, but the interesting thing is because it's so different, there's not a lot of skill sets out there. So I wonder if you could just quickly touch on also how you are enabling organizations to accelerate that adoption by the use of either reference architectures or blueprints that you're putting together.

Bharath Ramesh

>> Yeah, absolutely. So we have learned over the past few decades of deploying very large scale supercomputers, right? Because we bought SGI and Cray not too long ago. We know the operational constraints that customers operate into where they're investing in these very expensive clusters, they need the clusters that have not just very good uptime, but they also need to have very good utilization. And we have, through that experience of decades worth of building supercomputers and AI clusters, distilled that down into reference architectures that take you anywhere from 16 node, kick the tires kind of on-premise cluster, all the way up to hundreds of thousands of nodes when you're operating leadership class AI. And we have distilled that down into reference architectures that take you through not just the hardware building blocks, but the platform building blocks, the software and service building blocks that you need to build and operate this at large scale. So you can start at 80% of the solution and kind of iterate from there of starting from 0% every time. Because ultimately, we're a believers that the quicker we get customers onboarded and successful with large scale AI, the more successful they're going to be and then the more successful we're going to be as a result.

Bob Laliberte

>> Absolutely. It makes a ton of sense. Praful, I'd like to turn to you now because, obviously, once these are created, there's a lot of operational complexity and that's clearly one of the biggest obstacle enterprises have when they're deploying these AI clusters. The network teams in particular, as we mentioned, they really face a lot of steep learning curves, especially when compared to the traditional data center stacks. So how's HPE leveraging intent-based automation and self-driving networking concepts to simplify both the deployments in day-to-day operations?

Praful Lalchandani

>> Yeah, Bob, thanks for the question and happy to be here joining all of you. So as you said, the networking, the whole stack, the whole technology stack for AI clusters is widely different from what anybody is used to, right? If you're used to VMware stacks for general-purpose compute, the stacks that are prevalent for AI compute is very, very different. Starting from the fact that if you look at any general-purpose compute server, two ports of 25 gig used to be more than enough for any of your enterprise workloads. But you fast-forward to today and you're looking at Blackwell GPUs driving 800 gig per GPU, or you're looking at coming into the Rubin class of GPUs, you're looking at almost 1.6 terabytes per GPU. It's completely different ballgame. It's looking at seven to eight terabytes of bandwidth going to every particular node. And Bharat kind of touched upon the training aspect of it, training being a very widely distributed computing problem, but so also is inferencing. Actually, if you look at the inferencing paradigms of today, looking at a mixture of exports models, this is becoming increasingly a distributed computing problem. And most likely for enterprises, they're not going to be building large language models. Most likely, they're going to be consumers of those models. So anyways, my point is training or inferencing highly distributed, and that means that, as Bharat already alluded to, that the network needs to be highly performant. And what we have seen is that the congestion is the silent killer for AI application. First of all, very difficult to monitor, and once you even detect congestion in your fabric, you are basically in a situation now as an operator, you don't know what to do with that information because there are thousands and thousands of congestion metrics that are being exposed to you. So one thing that we have done for enterprises who as we know that they're going to be deploying more and more of the AI clusters as AI goes mainstream, is given them end-to-end visibility with Apstra intent-based automation system of congestion metrics, not just from the fabric alone, but extending all the way out to the NIC. And then we didn't stop there. It's not just enough to give you congestion metrics and congestion visibility. We also took a step forward into self-driving networks where we tell the operator, "Okay, now you don't have to pass through tons of information and figure out for yourself how you need to tune the network to maximize performance." We give the operator recommendations using our AIOps engine connected with Mist. We are leaders in that AIOps space, and we've applied that to this problem space as well, where we are giving the operator intel and recommendations of how to actually go tune it, and with the click of a button, they can actually go implement the recommendations that are coming out of our AIOps engine, which is basically the combination of Apstra and Mist. So that's how we are helping our customers in one way operationalize their clusters by eliminating congestion and maximizing the performance of their AI clusters.

Bob Laliberte

>> Got it. Got it. Now, that makes sense. I'm wondering if you have any examples of customers who were doing that and what they were doing before, and then how much time they saved. Because obviously, as you said, they're highly distributed, it's highly complex, so trying to troubleshoot that manually has to be incredibly time intensive. So I'm wondering if there's any examples that you might have that you are allowed to share of the impact?

Praful Lalchandani

>> Yeah, I would say in some cases, the time compression is from infinite to few minutes. And the reason I say infinite is that if you are a network operator, you know that if you see those ECN triggers popping up and your pause frames popping up, you look at it and you go, "Man, I don't even know what to do with it. Maybe I'll play around with my cost parameters, see if it improves." Sometimes it does improve, sometimes it doesn't, but it's an iterative process. In the end, you may leave your cluster at 80% performant and say, "You know what? This is too painful. I'm not even going to try doing this. It's running an 80% performance. It's good enough." But that's not what we want customers to do from their clusters. Again, GPUs are expensive. You want them to be utilized at maximum performance. So with what we have been able to give with automation, operators can bring that down to in a few minutes. Again, we are comparing with a human operator taking hours, days to do this and maybe sometimes just giving up to bringing it down to a few minutes.

Bob Laliberte

>> Excellent. Yeah, and that's certainly when you think about what you're paying for all those GPUs and so forth, you definitely want them at higher than 80% efficiency.

Praful Lalchandani

>> Yes.

Bob Laliberte

>> For sure. Yeah, absolutely.

Bharath Ramesh

>> Not to mention, Bob, the very expensive data scientists and machine learning engineers waiting on those GPUs.

Bob Laliberte

>> Yeah, correct. Yeah, exactly. Yeah, you don't want them knocking on your door. So we talked a lot about performance and certainly given the costs and so forth, that performance is absolutely critical, but also just as critical is a security aspect of it. And so Jon, I know security has to be built into AI infrastructure from day one, right? This is not something that you can really bolt on at a later time. So as enterprises are building out their AI factories, what are the most important network working security considerations they should prioritize?

Jon Green

>> Yeah, I'll give you a couple, and I think the answer, probably the most important answer is inherent in the question that you asked around AI factories. AI services I think are very much like general enterprise software in terms of how you can consume that. And so you can go to a service provider and you can say, "I need an AI application that does a particular type of data analysis and consume that as a service." And that's kind of the software as a service model, that's going to be multi-tenant. And so my concern in those situations is sort of, well, what security mechanisms did that service provider put in place to segregate my data? We've been talking a lot about performance, and one of the issues there, of course, is if you've got really expensive GPUs and you've got really expensive infrastructure, security historically gets in the way of that. It has a negative impact on performance sometimes. And so I think the temptation is going to be, hey, just deploy this without security built in. A second model of consumption is more like infrastructure as a service where you go to one of these neoclouds and you say, "I need to rent 16 GPUs or 24 GPUs or what have you." I still have kind of the same sorts of concerns about that because that is necessarily a multi-tenant environment, at least from the network perspective, and maybe you've got full access to GPUs at that point. The safest way to do this sort of thing is going to be in your own data center, your own colo facility, something that you own. And that's I think the advantage of what HPE bringing together the offerings from Juniper can offer is pre-configured rack systems that you can roll into one of these sites and it's under your full control. You can start to relax a little bit on security because of that, just because it's not a multi-tenant environment. It's something that's within your own control. That being said, AI is different than kind of standard enterprise systems in a really important way, which is observability of the data itself. You think about an AI model that's been trained that now I'm going to do inferencing through because of things like tokenization. The data, the programming of this is going to be really large vector spaces. It's going to be a bunch of numbers, and there's no real way to know what those numbers actually mean. So if the AI is producing a result that's anomalous, it's tough to go back and look and say, "Well, why did that result happen?" Because there's not code to work through. There's not kind of records out of databases where I can look and say, "Oh, the amount was incorrect here." So integrity becomes really, really important. In security, we talk about confidentiality, integrity, availability. A lot of people overlook that integrity idea and they say, "Well, confidentiality is the thing I care the most about." In the AI world, integrity is actually super important because when the output is wrong, it's tough to go back and figure out, how did that happen exactly? If you're not controlling that, you've got things like adversarial AI where I'm injecting malicious training data in the first place somehow, and so protecting all of that becomes even more important. But I'm going to say the most important thing that we can do is to actually get control of that infrastructure ourselves, and then behind that standard enterprise security controls are still really effective at doing what they do, but I want to see what's happening inside that cluster.

Bob Laliberte

>> I think that makes a lot of sense. And as you mentioned at the beginning, it's always been the age-old dilemma of performance versus security. If you add one, the other one goes down. But the reality is in today's day and age, you can't afford to do that, right? It's going to be both performant and secure. And so great advice, at least starting with it on-prem, bringing those factories, being in control, having that observability is certainly going to help with the security aspect as well. Very good. Praful, Let me come back to you. We've talked a little bit about training and inferencing already, and a lot of times, I think right now, at least initially, everyone was talking about training and thinking it was the heaviest lift, but inferencing is increasingly becoming a real challenge, especially a distributed computing challenge with all the agentic AI and the mixture of expert models that are happening right now, and all that's going to make the network fabric just as critical for the utilization as the GPUs themselves. How's HPE architecting the data center fabric to keep those GPUs fully utilized during the inference workloads?

Praful Lalchandani

>> Yeah, very good question, Bob. So by the way, HPE Juniper has been really been pioneering in this space for the last two years. Actually, we were the first OEM vendor in the industry to ship a 64 port 800 gig switch, again, to meet that insatiable demand that AI workloads demand, whether it is training or inferencing, but all networking experts will know that throwing bandwidth at the problem is maybe necessary but not sufficient for those ROCEv2 workloads. And the reason for that is Ethernet was never really designed to be congestion-free or lossless. It was designed to be best effort. Now, best effort works very well for TCP workloads. It doesn't really work well for RDMA workloads. Again, congestion causes drops and drops causes retransmissions during which GPUs are setting idle. So at Juniper, over the last two, three years, we've been innovating and developing a lot of capabilities under the umbrella of what we call as AI-optimized Ethernet. And we have delivered many different load balancing techniques that take Ethernet and give it a boost in performance going from best effort Ethernet to more predictable performance. These are techniques like dynamic load balancing or global load balancing or RDMA-aware load balancing that along with other congestion control techniques like DCQCN have given Ethernet the performance boost that is needed for those AI inferencing workloads that you talked about. But again, the second thing, Bob, you can allude it to this, that there is a skills gap, right? Enterprises can rest assured that the technology exists. Ethernet is up to the par. Many cloud providers are already deploying Ethernet at tens of thousands of GPU scales, so enterprises can rest assured that the technology exists, but there is that skills gap. And the skills gap is even in the world of Ethernet, you're not doing top of rack designs, you are doing rail optimized design. So there are a lot new concepts that you need to absorb while you are bringing up your AI clusters. And what Juniper did and HPE Juniper did was we wanted to simplify that experience for our customers as well with our validated designs that are anchored on either NVIDIA or AMD GPUs. Actually, you can go Google this, it's publicly available. You can Google for Juniper AI validated designs for data center, and you'll see these complete blueprints which tell you how to configure the NIC, how to configure the fabric, how to work with vector and VAR storage with our partners, how to work with various different types of NICs, whether it's a Broadcom NIC, or it's a NVIDIA NIC, or it's an AMD Pensando Pollara NIC. So we gave you these blueprints, but again, we went one more step further to make it easier for enterprises to deploy. And the way we did that is you don't need to be a CLI junkie or even have to read through this whole documentation to go deploy it. We codified the whole experience in abstract, which is again our intent-based fabric management platform. So once you've physically racked and stack your gear, you click a button and you will be able to deploy this entire fabric within minutes.

Bob Laliberte

>> Got it, got it. Yeah, that sounds like it's certainly going down the right path to enable organizations to accelerate their adoption. And when we had asked organizations why they wanted to use Ethernet over InfiniBand, the number one response was because we have it everywhere else. So as you said, being able to have that similarity in what they're doing, even though this might be a little bit different version of Ethernet than what they're used to, it's a lot more familiar to them. So clearly, an area where organizations want to go on. Quick follow up for you, as the models keep evolving and becoming more dynamic, where do you see some of the new stresses coming out on the network, and how are you thinking about that and making sure you're able to prepare for them?

Praful Lalchandani

>> Yeah, I think I'll come back to that point that even you made about inferencing. Inferencing, even three, four years ago, everybody used to say a low-end GPU is fine. I can sometimes even do it on CPUs. But we are seeing that the traffic characteristics, first of all, as we said, inferencing is becoming a multi-GPU communication problem, but the traffic characteristics are not always the same as what you would see with training. We are seeing different traffic characteristics and that sometimes requires different cost profiles to be deployed in the fabric. So what might work for a training model might not work for inferencing model A and then might not work for inferencing model B. And again, we don't want to leave operators to do the guesswork and figure out how to maximize performance. We are delivering the automation tools that become aware of what the model is, as well as then tune the fabric parameters. But one more thing that we have done for operators is that within the abstract dashboard, again, after I used to be full lifecycle management for fabrics, design, build, deploy, but increasingly we've been extending that to some capabilities that we call as job awareness. So a network operator, not even like IT manage, IT operator, but your network operator has visibility of which job is running where in my network. So my inferencing job is running on these particular GPUs that are connected to this part of the network. So some issue happens before even that trouble ticket comes in and says, "Hey, my job is running slow." You can actually pinpoint and say, "Yeah, I knew about that. I had optics failure in that part of the network and I know your job was impacted. I'm already on to it."

Bob Laliberte

>> Got it. Absolutely. No, that certainly makes a lot of sense. This has come up a couple of times in our discussion, so I wanted to open this up to all of the panel. Now that Juniper is part of HPE, we're getting a lot of questions. I'm sure you are as well, customers asking how this combination is going to create even a stronger foundation for AI. So I'm wondering if from each of your unique vantage points, how do you see the integration of Juniper enhancing the ability to deliver the AI-ready networking at scale, the network-enabled AI factories? I guess we'll start. Why don't we start with you, Jon? We'll start on the security side.

Jon Green

>> Yeah, sure. So Juniper was fairly pioneering in a lot of this AI. Before, it was always, "Oh, InfiniBand has got to be the thing that you used to get this done." Juniper set up a lab and kind of proved decisively that it can be done with Ethernet. They put six different types of NICs in these servers. They ran it six different ways and produce the test data to show that this can be done, and that's something that I haven't seen anybody else produce before. So that's pretty exciting to have that real world data. That lab is even available for customers to come in, and they've done this where customers bring in their own data and say, "How is this going to perform? And let's test this and tune it on the basis of my own data." So that's invaluable to have that type of expertise. You combine that with the rack scale server architecture, storage systems that are optimized for AI and that type of thing that come out of other parts of HPE. I think we're going to see consumption models that look a lot like the ISV world before where you would come to HPE and you would say, "Hey, I want to buy a rack scale SAP HANA system." It will come pre-configured. All of the tuning has been done for you. The software is pre-installed, it's ready to go. I think you're going to see the same sorts of things on particularly AI inferencing, because it's going to be tunable for a lot of different types of workloads. Being able to buy that and have it be pre-tested, pre-configured, set up and ready to go is going to be a huge time savings for an average enterprise customer. So I think that's the biggest value I see of bringing all of that together. You add in the high performance compute experience that HPE brings with the likes of Cray and SGI and some of those. AI is different, but it's another form of high performance compute workload, and so we've got lots of experience to bring to bear with that as well.

Bob Laliberte

>> Excellent. I think you make great points. And by the way, I've actually been to that lab and I've seen it in action, so it is impressive and it is a great asset for customers looking to bring on their own workloads.

Jon Green

>> It's loud though.

Bob Laliberte

>> It is loud. There's lots of moving parts in there. Absolutely. Bharath, why don't we turn to you next?

Bharath Ramesh

>> Yeah, sure, Bob. I always joke and say we're building AI factories, not AI shacks. So when we think scale, we think really big clusters. And when you get to a certain cluster side, it's not a question of, are you going to run into issues? It is a fact that you are going to run into issues. And then the question is, how do you deal with that in a way that is self-healing, resilient, so you don't have downtime and expensive GPUs and expensive users of the GPUs waiting around for you to fix problems? And I think that's really where I get really excited because with the huge investment Juniper has done and the AIOps and AgenticOps part of the network fabric, just tying that into the strength we have as HPE around building an operationalizing these very large clusters and tying it very closely with the workloads and platform management stacks that we see a lot of the customers using, I think really imputing that value in, so our customers spend more time being productive and less time waiting to fix problems. I see that as kind of the better together, the one plus one equal to three story with us and Juniper.

Bob Laliberte

>> Absolutely. No, I think that makes a lot of sense, and I know I'm certainly looking forward to seeing maybe some integration with the heterogeneous management and being able to support all different types of technology. I think that'll be interesting to see how that evolves. And Praful, take us home.

Praful Lalchandani

>> Yeah, first of all, I'm so excited that Jon can advocate for Juniper's AI lab, and I even forgot to mention about it during this. So it just goes to show that just within eight weeks, we are only eight weeks into the integration or maybe somewhere around that. And the folks at HPE can speak on my behalf, so thank you for that. But again, the cultures that both these companies are very forward-looking, let's just get it done. So I'm super excited to be part of the HPE umbrella. We've been talking to a team within HPE, Bharath and Jon about how we can bring this better together to the market really, really quick, because we don't want customers to be left kind of buying best of breed components and then trying to stitch them together and going through the process. But with what we can offer integrated, we bring our bench strength in networking. The HPE team brings their bench strength in compute and storage, and now customers don't really have to stitch this whole thing together. They get this fully integrated stack, and we are already on our way to build that solution. So eventually, I think what enterprises will get out of this is really the peace of mind that they don't have to stitch things together on their own.

Bob Laliberte

>> Excellent. Well, like I said, I think we're all looking forward to seeing what innovations come of this and the integrations that come from it and how that's going to accelerate the adoption of these AI factories and make it a lot more efficient to manage them over the long term. So, unfortunately, that's all the time we have. So I want to thank you Bharath, Praful and Jon for sharing all your insights.

Jon Green

>> Thank you.

Praful Lalchandani

>> Thank you.

Bharath Ramesh

>> A pleasure to be here.

Bob Laliberte

>> Absolutely. Great to have you on. I think this discussion really underscored why the network is just as critical to the GPUs and these AI deployments and how innovations like intent-based automation and self-driving fabrics are really going to help pave the path forward. So, again, appreciate everyone watching. Stay tuned for more Networking for AI Summit coming up.