Broadcom Delivers the Modern Private Cloud, APJ | How Broadcom & NVIDIA Are Powering the Modern Private Cloud with AI

More from Broadcom Delivers the Modern Private Cloud, APJ

Himanshu Singh

Director of Product Marketing | AI, Kubernetes & Cloud Management

Broadcom

Mark Chuang

Sr. Director Product Marketing

Broadcom Software

play_circle_outline Introduction of VMware Cloud Foundation (VCF) 9 and its private AI focus.

play_circle_outline Importance of data privacy and compliance in Private AI Foundation with NVIDIA.

play_circle_outline VMware Private AI Foundation: Key milestones and enterprise demand for high-performance, privacy-focused AI solutions with Nvidia

play_circle_outline Transformative Success: Case Studies of VMware Private AI Foundation and the US Senate Federal Credit Union's Innovative Implementations

play_circle_outline Resources available on VMware website for starting the private AI journey.

Info
Transcript

How Broadcom & NVIDIA Are Powering the Modern Private Cloud with AI

Himanshu Singh

Director of Product Marketing | AI, Kubernetes & Cloud Management Broadcom

Mark Chuang

Sr. Director Product Marketing Broadcom Software

Himanshu Singh, director of product marketing, AI and modern apps, for the VCF Division at Broadcom Inc., and Mark Chuang, senior director of product marketing for the VCF Division at Broadcom, join theCUBE’s Rob Strechay at the “Broadcom Delivers the Modern Private Cloud” event to explore how VMware Cloud Foundation 9 supports the evolving needs of enterprise AI. Their conversation highlights the growing role of the VMware Private AI Foundation, in collaboration with NVIDIA, in powering secure and compliant AI infrastructure.

Singh and Chuang discus... Read more

explore Keep Exploring

What is the strategic value of VMware Private AI Foundation with NVIDIA in relation to VMware Cloud Foundation (VCF)? add

What is the concept of private AI and its key components? add

What are the key milestones and achievements associated with the VMware Private AI Foundation and NVIDIA since its announcement? add

What are some examples of organizations that have adopted VMware Private AI Foundation with NVIDIA, and what are the key benefits of this solution? add

What are some recommended resources for learning about VMware's AI and ML offerings? add

bolt Powered by CUBE AI

How Broadcom & NVIDIA Are Powering the Modern Private Cloud with AI

search

Rob Strechay

>> Hello and welcome to theCUBE Palo Alto Studio for a special presentation with VMware by Broadcom, delivering the modern private cloud with the launch of VCF 9. To help us dig in, we really got some experts who are coming on here to help me unpack this all, with Mark Chuang, who's the senior director of product marketing. And Himanshu Singh, who's the director of product marketing from VMware Cloud Foundation division at Broadcom. Welcome on board, guys.

Mark Chuang

>> Thanks, Rob. Thanks for having us.

Rob Strechay

>> Yeah. I think the momentum that Broadcom has seen with VCF and VMware Cloud Foundation across the customers has really been noticeable and measurable. And I think there's some really good focus going on, especially in the stuff that you guys are here really trying to help customers better understand as part of their VCF journey. I think in particular, today we're going to be talking about VMware Private AI Foundation with NVIDIA, that's really powering AI on VCF 9. So again, this has been a journey. I think, again, we've been talking about this, the private AI and private AI on VCF for over a year now. And I think it's just been really exciting to see what customers are doing. But AI has become super important these days, and every enterprise is really leaning into AI. How does VMware Private AI Foundation with NVIDIA align with and enhance the strategic value of VCF in the broader portfolio? Why don't we start with you, Mark?

Mark Chuang

>> Yeah. With private AI, I think it's important just to even define what it is, because we all know about the general umbrella of AI. We all know about GenAI, unless you've been living under a rock. But private AI is all about data privacy, regulatory compliance, controlling sensitive information. But then ultimately, about controlling costs in this use case as well. And so with Private AI Foundation with NVIDIA, that builds on top of VMware Cloud Foundation. As that underlying platform, it's really about addressing this key use case and this key customer need for our customers.

Rob Strechay

>> Yeah. I think to me that is the key, is that how the power of VMware by Broadcom and NVIDIA coming together as part of this. What have been some of the key milestones and achievements that you've seen for VMware Private AI Foundation with NVIDIA since its announcement?

Mark Chuang

>> Yeah. It is a new offering, a joint offering from us, but it's actually built upon a decades-old, strategic relationship, partnership that we've had with NVIDIA, not just on the go-to-market side, but definitely the deep, hard engineering work there. This joint AI platform was introduced, it's been in market for about a year, and we've just seen very robust demand and interest from our customers. Because at the end of the day, they're like, "With all of the considerations around generative AI relative to privacy, performance, having choice in models, being able to manage to compliance and cost." They need all of this when they're thinking about, "How do I fine-tune my LLMs? How do I deploy retrieval-augmented generation workflows? How do I run inference workloads across all of this, but with also those key characteristics?" So that's where that deep partnership has led to this joint AI platform, and we're seeing very strong pickup from customers.

Rob Strechay

>> Yeah. And it seems like there's been a lot of customers in that way that have really been picking up on it. I think the US Senate Federal Credit Union is one of the ones that you guys have mentioned to me.

Mark Chuang

>> Yeah.

Rob Strechay

>> And a number of higher education institutions around the US as well, and I think that to me is one of the keys. What is really unique about VMware Private AI Foundation with NVIDIA?

Mark Chuang

>> Yeah. I think it first starts off with the fact that it's standing on the shoulders of just a long history of innovation that we've had. If you go back several decades, it really, literally has been decades now, we helped to unlock the benefits of virtualization for enterprises. And now we're unlocking the benefits of AI for those exact same enterprises. You mentioned US Senate Federal Credit Union. They really adopted our solution because they wanted to figure out how to automate a lot of their banking processes, like loan processing, but at the same time, proactively identify emerging needs of their customers. And in many cases, it was just like the everyday transactions of it. And so other examples, you mentioned education, because we do a lot of work with government agencies, including those three-letter agencies.

Rob Strechay

>> Acronyms, yes.

Mark Chuang

>> Not surprisingly, regulatory with education, healthcare and financial services. One in particular that comes to mind, a major higher education institute in the US. They were all about, "How do we improve the experience for our staff, as well as our students?" So with private AI enabling virtual, AI-enabled tutors, helping the staff with budgeting processes or analyzing benefits analysis. So just efficiency, cutting down the time to improve the complex analysis. And at the end of the day, that's a benefit for both students and staff, so seeing a lot of great results like that.

Rob Strechay

>> Yeah. I think you get it's with that base, and as you talked about, with VMware Cloud Foundation now really very robust. Covers a lot of ground for organizations and their infrastructure. What are some of the key, most important things that people should note about this release in the Private AI Foundation with NVIDIA?

Mark Chuang

>> Yeah, yeah. So I'm going to kick off with a little bit of high level, and Himanshu will go into more of the details here, for sure. But fundamentally, we want to enable organizations to be able to bring their AI models to where their data is. And then have the flexibility and choice to be able to choose different models depending upon what their future business needs may be. Okay, so that's what we're going to do at the end of the day. And so with this coming release, we really wanted to enhance key capabilities that are already there. So that on one hand, the infrastructure provider can have simplified management of all of these private AI workflows. And then on the other hand, really streamline the way that the data scientists or the MLOps people can deploy these different models. So at a high level, some of the key things that we'll be delivering. You'll see things around how do we make sure that we load balance these AI applications so that they have access to all the infrastructure resources to run optimally? How do we enable them to ensure high uptime by doing things like move very large language models in vMotion, which VMware invented? Do that, but with these giant language models so that they don't have any downtime. We're going to continue enhancing areas like, "How do we deliver all of these operational benefits, but at near bare metal performance levels?" Because if we do all of this, and with all the new stuff coming in 9.0, our analysis shows that we can deliver a total cost of ownership up to 90% lower than if you're running on public clouds. Up to almost 30% lower than even running on bare metal, so that's why we're so excited about what we've got coming out.

Himanshu Singh

>> Yeah, and if I can add to what Mark said. Well, when it comes to what's fundamentally driving us, whether it is this release or so far, it's been the idea of one, unlocking all the value you have based on your current investments. Mark said it's been decades when we unlocked virtualization for CPUs. We're bringing all that value to GPUs, to all the accelerators that customers are using in their environments for the modern workloads that we have now. And so the idea is we optimize the experience, and we optimize the cost that you have to basically look at for across your environment when it comes to AI workloads. So that's the driving factor. Now, as we had these conversations with customers about, "What are you really looking for?" Because at the end of the day, our roadmap is all driven by what is the customer's need now and into the future? So three key personas basically bubbled up to the top for us to really focus on, "Hey, what do we really want to focus on for this release?" First one is around your CXO level or decision-maker level persona. They're all about, "Hey, I want to make sure that my internal data, my intellectual property is safe, secure, any sensitive information, et cetera, doesn't get leaked out." So that's number one. So we're making sure that we're delivering on privacy and security in the release, and I can talk more about that as well. The second key audience is, well, the audience that we've always talked to for a couple of decades, which is the infrastructure team. And you want to make sure that they're able to have a simplified infrastructure management experience, so we are bringing in capabilities around that too. And the third key audience, which is unique to AI workloads in this case, is the data scientist. And for the data scientists, we want to make sure that they're able to really streamline the deployment of their language models, large language models, small language models, et cetera. So across these three different personas, if we can provide them what they really need. Then the organization essentially is enabled to be able to be successful and unlock all the value of AI, while making sure that you stay secure. Now, one thing I want to highlight, this thing came out of the private cloud report recently, and when we surveyed, I think it was more than 1,800 decision makers. One of the key stats that came out is when it comes to privacy and security, those decision makers, 92% plus of those, trust a private cloud environment for that as opposed to something else. I don't want to go into that. But that just tells you the focus that our customers' enterprises have in this space, and we're basically all gung-ho about delivering on that.

Rob Strechay

>> Yeah. And I think we see very similar from in our research, that a lot of organizations say that 85% of the data that they want to use in these models is actually on-prem. And it's in these other applications and other ways that they need to be in other data formats and things of that nature. That has to be playing towards the privacy aspect of it as well, as well as the infrastructure management because you want to make it simple. Everybody's looking for the easy button for AI and things of that nature. And again, I think talking to the personas is great because data science, that's the people who are actually building it and using the data and things like that. Are you finding that those three pillars seem to be driving a lot of the new features that are in this release?

Himanshu Singh

>> Absolutely, yeah. In fact, as Mark said earlier, it's about bringing that innovation, the simplification. What we want to do is we want to make sure that we are making AI as just another enterprise workload on VCF. We want to have consistent operations. We do not want to create new silos when it comes to AI. Mark knows, there's enough complexity in IT already and anything we can do to simplify it, I think customers appreciate that. So that's been a big, big focus for us. So let me give you a little bit more detail in terms of what's in this release specifically when it comes to being with Private AI Foundation with NVIDIA with VCF 9.0? That rolls off the tongue, right?

Rob Strechay

>> It's actually not that bad. I got to say for long names, it actually does roll off the tongue.

Himanshu Singh

>> Yeah, there you go, so yeah. Okay. So I talked about three key personas in line to that. So the first one is all about enabling privacy and security. So a couple of things that I want to highlight in that. The first one is a Model Gallery. Now the idea is as a customer, if you're trying to access a particular language model, you want to build something on it, well, how do you make sure that it's a secure model? How do you make sure that it's been trained on data that is obtained in a legitimate way? So you want to make sure that you have this Model Gallery, which is making available like a catalog of curated, safe, secure models. So the end user has all the choice they need for their specific use cases, but you also have the right level of governance and control. You have the role-based access, et cetera, so that's what the Model Gallery does. It's actually built on Harbor at the backend, and so that I think is going to be super valuable. And then when you talk to customers, they're like, "Yep, that's exactly what we need."

Rob Strechay

>> And just to put in there, for those who don't know, Harbor is actually one of the CNCF projects. And that's great to see that you guys are involved with that as well.

Mark Chuang

>> We continue to invest in it. And this is one of the biggest asks that we get back from our customers. Because I think what they realize is that what's happening with AI right now, it feels a little bit like when Kubernetes and containers first came on the scene, what, maybe five, eight years ago? And where everyone could just go out to Git and you could just download some random code, and you could load it into your environment and just start running it. That's pretty scary, and so that's what's happening with these AI models and this code right now. You can go out there, literally download almost anything you want. And so the governance piece that we're delivering as part of the Model Gallery, really helps to address this problem. Because otherwise, there's just way too much exposure out there.

Rob Strechay

>> Yeah. Well, I think again, organizations are really looking at it from, to your point on governance, they want to make sure that everybody's using the standard that's being out there. They want guardrails and guidelines from a governance perspective. But there's also the part, and I think you talked about this a little bit, maybe we can go a little bit deeper, is around the infrastructure management and simplifying that. To me, that's one of the things that to your point, when people look at the hyperscalers, they're either using a lot more GPUs than they know. Or they're using them but they're paying for them, they're not using them, and they have them on a lease and things like that. How are you helping simplify that infrastructure management?

Himanshu Singh

>> Yeah. There's a couple of things, and I'll get back to some of the security pieces, because there's some very exciting stuff there as well. But just from the idea of simplifying infrastructure management when it comes to AI workloads, a couple of new things that we're excited about. And it's simple stuff, but it's extremely valuable on a day-to-day basis. One is around this concept of vGPU profile visibility. So previously, and as an admin, you had to do a bunch of manual work to really figure out, "For this GPU, what are the vGPUs? How do I track it?" Now, we have a simple interface in vCenter where you can see all of that very easily. So just from a productivity perspective, it just simplifies that. On top of that, what we've also done is we've brought in something called enhanced GPU observability. So now you're able to see the GPU resource utilization, the GPU temperature, because GPUs will run hot and you want them to run hot, but not too hot, along with a variety of host-level metrics, et cetera. So it's all right there in one screen across your entire GPU footprint in your environment. And that just gives the infrastructure team, the admin team, a lot of value in being able to just get things done, have easy level of control and visibility across. It just gives them a lot of peace of mind to just manage it just like they do the rest of their environment.

Rob Strechay

>> Yeah. So I think we touched on a couple different personas in those last two. But I think one of the things you brought up was really the model deployment and basically helping people streamline that. How is this release really helping organizations with that?

Himanshu Singh

>> Yeah. So I talked about the data scientist persona, and you can think in terms of app developers, MLOps engineers in the same category as well. So there's a few things that they're looking for, which is at the end of the day, they want to have faster time to value. They want to get to market faster essentially, and so there's a few things we're doing there. So one is around just providing this model runtime overall. So a set of capabilities that you really need to be able to create, deploy your model endpoints, and just be successful in terms of getting to value faster. There's things like we have a data indexing and retrieval service that's come online, which basically helps with chunking and vectorizing of data. So you can take all those internal data sources and then make that data available. Whether you are doing fine-tuning as a use case, or you're doing RAG or retrieval-augmented generation in this case, that just simplifies some of these things. Another big one right now, which I think it's like the hot trend in AI right now, is agentic AI. That's the big word for 2025.

Rob Strechay

>> Absolutely.

Himanshu Singh

>> And so in order for customers to be able to build and deploy these AI agents, they're essentially trying to enhance their generative AI workloads, whether it is for any specific creative task or functional task, or enhancing the automation for a particular workload. So what we have is we have the framework, we have the guidance available to them as part of this agent-builder service. And you can leverage all the runtime capabilities and the services I talked about to be able to really build AI agents faster, and you can deploy that faster and you can enhance your AI workloads overall. One other one that comes to mind, and I went, "We've got such a massive list that I'm struggling to contain myself right now." There's just so much good stuff in this release.

Rob Strechay

>> It's a lot.

Himanshu Singh

>> Yeah, which the one I'm going to call out is the API gateway. So again, specifically to the data scientist audience, and there's really a problem it solves in their lives. So think of this, you've got this API gateway that gives you a nice, secure, stable interface to connect to your model endpoints, which basically means that you don't have to worry about unauthorized access and all that. Your access control is secured. On top of that, what you're also getting is the ability to now, with the API gateway, to manage the APIs at the other end. So even if a language model that you're using evolves, the APIs change, it doesn't break any integrations that you've built in. So you've got seamless integration and you've got consistent performance from your workloads. And that just makes it easier for the data scientists, and it's less worry for them on an ongoing basis. So again, just a few highlights, but super excited about the release. If we have time, I do want to call out a couple of things, which are actually huge on a slightly different space, and that's on the security side of things.

Rob Strechay

>> So I think you're going where I want you to go, which is I think one of the things, because Mark brought up three-letter acronym places and things like that. And a lot of times, they're looking to be air-gapped and other things. I would assume that this really leads into that as well.

Himanshu Singh

>> Yeah, that's exactly what I wanted to call out. So government obviously, any kind of organization that has sensitive data.

Rob Strechay

>> Or even sovereign clouds.

Himanshu Singh

>> Sovereign cloud as well, right?

Mark Chuang

>> Yeah, absolutely.

Himanshu Singh

>> So that's very important. So we're introducing support from air-gapped environments now with this release. So that's a huge ask from our customers, for sure, and so I'm really excited to bring that to market. And the other one you talked about from a sovereign cloud or even a CSP provider perspective, is something that's around multitenancy. So we're now able to support multitenancy for private AI workloads. So that you can think of, "Well, of course, you've got the CSPs who can now be able to offer GPU as a service to their customers." But even within an enterprise, you can actually now deploy and divvy things up based on different teams, different organizations. Have that role-based access and manage multitenancy internally that way, and that gives a lot of control in cost access and all that stuff within organizations. So we're seeing so much demand for this capability across the board, so we are very excited to bring that to market.

Mark Chuang

>> Because with that multitenancy in there, you're now able to take this super powerful but very expensive resource in your data center, these GPUs. Maximize your utilization because you want to squeeze all the work you can out of it. But now seamlessly and pretty transparently for the data scientists and the end users, make sure you wrap it in the policies so that it's not just a free for all out there. And so all of the things that Himanshu described, it's really been talking and understanding what are the biggest pain points that our customers have? Some of them literally require computer science, rocket science type of stuff, and others are just solving core problems that they have. Like with that data retrieval and indexing service, that was just to solve the problem of, "I've got to keep this vector database up to date, even as its data sources are constantly changing and evolving." So how do we make that just easier for people? So really been a great journey, since this is still so early on for a lot of enterprises, and just trying to address all of these major pain points for them.

Rob Strechay

>> Yeah. I'm glad you set me up for my final question, which was for those folks that are watching us today, where should they go to really learn more as they start their private AI journey?

Himanshu Singh

>> I would say the easiest thing to do is go to vmware.com/aiml-nvidia, that's where you'll get started, you'll start learning about what we're offering. There's a ton of resources depending on what you're looking for. You're looking for high-level information, looking for deep-dive content, you really want to compare us against somebody else. It's all in there, just go to vmware.com/aiml-nvidia and you'll have access. That's the best place to start.

Rob Strechay

>> I also know there's another one. There's a Cloud Infrastructure/Private AI Foundation NVIDIA as well. So I learned that earlier today, so I figure I'll throw that out as well.

Himanshu Singh

>> You're ahead of me.

Rob Strechay

>> Well, no, I got to do my research to be a good host here.

Mark Chuang

>> We all cross-link from each other, we make sure we point it back and forth.

Rob Strechay

>> Absolutely. Well, hey, thanks, Mark and Himanshu, thank you for coming on board. This has been great. I could sit here and talk all day about this. I love AI and private AI.

Mark Chuang

>> Very exciting feel. Thank you for having us, Rob.

Rob Strechay

>> And congratulations on the launch, this is huge. I know it's been a bit of time coming, so these big things take a bit, so congratulations.

Mark Chuang

>> We're very excited about it.

Himanshu Singh

>> Thank you very much.

Rob Strechay

>> And thank you for watching this episode of theCUBE from the Palo Alto Studios, a special presentation of VMware by Broadcom, delivering modern private cloud with the launch of VCF 9. See you soon.