VMware Explore 2024 | David McIntyre | Samsung & Arvind Jagannath | VMware by Broadcom

Clips
News
More from VMware Explore 2024

David McIntyre

Director Product Planning & Business Enablement, Device Solutions

Samsung

Arvind Jagannath

Sr. Product Line Manager

VMware by Broadcom

Samsung and VMware collaborate on CXL solutions for server memory expansion

Compute Express Link, or CXL, has only been in use for five years, yet it is already having an impact in connecting server components.The technology, introduced by Intel Corp. in 2019 and designed as an open-standard interface for high-speed communications, connects a server’s central processing unit with accelerator chips attached to the machine. As AI workloads increasingly place strains on server memory capacities, CXL has found appeal among firms seeking a performance boost.Samsung’s David McIntyre and VMware’s Arvind Jagannath talk with theCUBE about CXL solutions

play_circle_outline AI impacting storage hierarchy, memory tiering triangle with HBM, DRAM, storage class memory, NAND.

play_circle_outline CXL technology enabling fast memory tiering, reducing performance requirements, expanding memory capacity.

play_circle_outline CXL memory module devices combining DRAM and NAND for increased host capacity.

play_circle_outline Samsung's technology platforms (CMMH, CMMD, CMMB) for memory expansion, tiered memory, persistence.

play_circle_outline Focus on TCO benefits, system expertise, partnership with VMware for better utilization of virtual machines.

Info
Transcript

David McIntyre | Samsung & Arvind Jagannath | VMware by Broadcom

David McIntyre

Director Product Planning & Business Enablement, Device Solutions Samsung

Arvind Jagannath

Sr. Product Line Manager VMware by Broadcom

TheCUBE covers VMware Explore 2024 at the Venetian Conference Center in Las Vegas. Discussions by Samsung and Broadcom emphasize the importance of persistence and tiered memory in addressing constraints and improving performance. CXL is highlighted as a solution for memory expansion and pooling. Samsung and VMware are collaborating on CXL-enabled platforms for improved performance capabilities. Samsung's roadmap includes products to address server constraints. The conversation also touches on evolving memory hierarchy, cost considerations, and potential appli... Read more

explore Keep Exploring

What is the latest innovation in infrastructure around memory tiering that Samsung Semiconductor is bringing to the table? add

What potential benefits does CXL offer in terms of memory tiering and reducing performance requirements for customers? add

What is the significance of CXL in the tech industry and how is Samsung involved in it? add

What technology platforms has Samsung developed in terms of memory expansion and management? add

What are some ways in which companies can bring expertise to their customers in terms of fine-tuning systems and understanding the functionality of the entire system, including transitioning from large language models to large action models, exploring agentic AI, and collaborating with partners like VMware to optimize virtual machine utilization? add

bolt Powered by CUBE AI

David McIntyre | Samsung & Arvind Jagannath | VMware by Broadcom

search

Dave Vellante

>> Hi everybody. Welcome back to the Venetian Conference Center here in Las Vegas. You're watching theCUBE's coverage of VMware Explore 2024. I'm here with Rob Strechay, my co-host. John Furrier is also in the house. This is our 11th year at either VMworld or VMware Explore, a couple of years of COVID in between. AI and GPUs and high bandwidth memory and CXL and NAND, it's changing. AI is really changing the storage hierarchy. Particularly not even so much, we're talking about primary storage. We're talking about the top of the pyramid, if you will, and we're really excited to have David McIntyre here. He's the director of product planning at Samsung. He's joined by Arvind Jagannath, who's the senior product line manager for Broadcom. Guys, thanks for taking some time and coming on theCUBE. Some real hot innovations going on. The whole stack is being reformatted. We just had AI infrastructure leaders out of our Palo Alto office in conjunction with the New York Stock Exchange, and so much is happening in semiconductors and software. So really appreciate you guys coming on. David, you were walking by. We started having a chat.

David McIntyre

>> Yes, we did.

Dave Vellante

>> We were given a presentation today.

David McIntyre

>> Yep.

Dave Vellante

>> Set it up. What were you talking about? And then we want to bring Arvind in and understand where ESX plays and vSphere and all the innovations that VMware is bringing. So, please.

David McIntyre

>> So coming from the Silicon Valley and representing Samsung Semiconductor, I was bringing to you the latest and greatest in infrastructure around the memory tiering that you just mentioned, the triangle where you've got HBM at the top, DRAM, and then storage class memory followed by NAND and even hard disk drives today. And the closer you get to the top, you get highest performance, lowest latency. And then as you drop down in that pyramid, things get a little less expensive and you get that capacity boost as well. But anyway, that comes from AI and the opportunity, the surge in the compute elements that are somewhat constrained by memory, and you've heard of maybe the memory wall. And so Samsung is the leader in memory and storage, has the opportunity to push down that memory wall and make GPU and CPU resources and other compute resources working hand in hand. In fact, I put together a nice slide that shows the balance of compute, memory and storage together working harmoniously to get the best performance and drive the application requirements using that technology.

Dave Vellante

>> So it used to be pretty simple. You'd have main memory and then you'd have a spinning disk which had a cache in it, and then you had maybe tape in the back end. And the bottleneck then was always you had to de-stage the cache to the spinning disk. And so years ago when EMC came up with Symmetrix, it was as many drives as you could get. Remember they didn't use RAID-V because it was too much overhead. So they did mirroring. And that was state of the art. That is now ancient history. AI is changing everything. What you described is a much more granular hierarchy.

David McIntyre

>> Yes.

Dave Vellante

>> And one that is highly persistent. So you've still got the de-staging bottlenecks, but it's happening at much, much higher speeds. And you've injected persistence into the system through both NAND, I guess also CXL, and also the ability, if you pull the plug. So explain where we are today in that hierarchy.

David McIntyre

>> Right, you emphasize persistence, which is very, very important. And solutions have been out, NVDIMMs, the power goes out, you have electrolytic caps that can preserve the state of that DRAM. And then another complement to that is the memory tiering, you also mentioned it. How about if you put both main functions in one common platform? So you're addressing the triangle that we just walked through, but you're also providing persistence where persistence is needed. So the lights go out and you need to preserve your data, that is certainly one clear requirement for persistence. But also especially across enterprise, persistent rights that have to support service level agreements, SLAs. That's a policy requirement, and sometimes that gets overlooked. So by providing a new technology in a platform that provides persistence as well as contributing to that tiered memory triangle that we were talking about is pretty exciting stuff.

Dave Vellante

>> And that's a compliance thing in certain industries, like financial services perhaps?

David McIntyre

>> It absolutely is.

Dave Vellante

>> And is that an asynchronous right, or is it a synchronous right?

David McIntyre

>> Synchronous.

Dave Vellante

>> So there's a little bit of latency involved, but by law, I guess, or by policy, they have to endure that latency because that persistence is more important-

David McIntyre

>> As long as you're deterministic as well. That's right.

Rob Strechay

>> And there's things that have to happen in the platform. Like CXL has been around for about five years now. I think 2019 is when it formed up as an offering. So what's happening in the platform?

Arvind Jagannath

>> So what VMware is doing is we realized the potential of CXL very early. We had been playing with PCI for a long time now, and we also have a lot of expertise with memory in general because we were one of the leaders in VM migration, tracking pages and memory, et cetera. So now we are at a stage where CXL came to us at exactly the right moment. Obtain went away, but there was CXL, and then we have the great collaboration with Samsung now bringing their device. Now what CXL does is it has a lot of interesting features such as giving the CPU the ability to run instructions on the device directly, memory instructions on the device, and for the device to be able to access the host memory in a coherent manner. So what that gives us is a really fast memory tiering, which goes beyond software schemes. And I think we have all the expertise and know-how about how to make this fast and jitter-free for our customers. And our goal is to reduce performance requirements for our customers. In terms of applications, we don't want them seeing any performance loss. So CXL definitely achieves that for us.

Dave Vellante

>> Is this current hierarchy that you're describing, high bandwidth memory's in short supply, it's also superfast. Is this a bridge architecture or do you see this as more of a long-term, even when high bandwidth memory becomes more available, which I don't know how long that's going to take. But do you see this surviving that or is this a bridge or a hybrid? Can you help us understand that?

David McIntyre

>> I can comment on that, sure. So CXL itself really solves a problem on server constraints where you only have so many DIMMs that you can put in an individual server. Some servers can handle up to maybe five terabytes of DRAM, which is a huge amount of memory, but then it stops there. CXL provides, as Arvind is articulating, memory expansion, eventually memory pooling. It basically frees up these constraints so that now you can expand out with additional devices or even systems that are CXL enabled across fabric through orchestration layers, and the devices itself that you can expand beyond just the physical constraints of the server. So CXL, in fact, Samsung's on the board of directors of the CXL Consortium, and CXL has absorbed many other standards and consortiums as well. So it's living and thriving and it's starting to find its way into actual production deployments. And in fact, Samsung has now built a full portfolio of CXL enabled platforms and one is called CMMH. So CMMH stands for CXL memory module, and the H stands for hybrid. And Arvind knows this very well, the hybrid part includes DRAM as well as NAND flash all in one E3 device. We also have a variant of that in a AIC form factor type two called the CMMH VSA. And Arvind can tell us more about that because Samsung and VMware are partnered on that particular device.

Dave Vellante

>> So just to follow up. When I'm constrained in the physical server, I can scale out, is that right?

David McIntyre

>> Yeah, with CXL, you can.

Dave Vellante

>> With CXL-capable systems?

David McIntyre

>> Right.

Rob Strechay

>> And to me, and we'll get to the VSA part of it because that's very interesting to me as well, is that I would see this as a huge thing, especially as we go towards pooling. Because when you start to look at pools of memory being utilized, AI and checkpointing and being able to complete things, a lot of times it's how fast you can feed the GPUs. And a big constraint of that is how fast it can get from memory or from disk or what have you into that. This would seem like this is one of those, "Hey, it's the right time to have this solution come out." Are we reading the tea leaves that way?

David McIntyre

>> Absolutely.

Arvind Jagannath

>> Yeah, the other thing I would say which this solves is what we have seen across our customer base is memory tends to be utilized and consumed, but a lot of the customers report that they only use 10, 20% of their CPU capacity. How can we make the infrastructure efficiencies better? How can we reduce this imbalance that occurs between CPU, GPU, memory? And a CXL-based approach definitely helps us. And like you mentioned with infinite pooling-like ability, now you get infinite supply of memory to make sure that these resources are better utilized.

Rob Strechay

>> And how does the VSA play into that?

Arvind Jagannath

>> So the VSA actually provides a large capacity, and it also has extendability, meaning you can actually plug a VSI into multiple hosts. And we are architecting it to be extendable so that in the future, CXL also has the ability to hide the underlying infrastructure complexities, but still provide the ability to give more memory on demand even.

Dave Vellante

>> Where are we in the cost-per-bit equation? NAND obliterated so-called high-spin-speed disk. That was an oxymoron. But disk's still around. Hyperscalers use disk. Customers use it for cheap and deep. What's that? Hierarchical? So DRAM, HBM most expensive and then CXL, and then NAND and then spinning disk, right? And then I guess tape orders of magnitude lower, is that right? But where are we in say, the NAND to spinning disk equation? That's been bouncing around for a long time.

Arvind Jagannath

>> Yeah, NVMe and NAND is still considered pretty fast. I think in the industry it's really used for a variety of different applications, high speed applications. CXL actually brings the bridge between the DRAM and NAND a little closer. If you architect it well enough, then you can bring elements of CXL to speed up some of the NAND that's being used for CXL. So it definitely speeds up the application and brings a better value.

Rob Strechay

>> And to that point, to Dave's question around the whole cost aspect of it, it would seem like this can also help expand the life cycle and duty time of a server as well because you can go and bring pieces to that server.

David McIntyre

>> Yes. So you can add additional servers to the problem because you have this memory constraint. And talking with the financial services people here at VMware, that's an issue. As you talked about utilization. If we could improve that utilization ratio drastically by balancing the compute, memory and storage resources, you're getting a CapEx and OpEx boost.

Dave Vellante

>> Excuse me. Well, also the income statements ... So all the hyperscalers are now depreciating over six years, so that makes the income statement look a little better. You got to read the fine print in the financial statements. But this is an exciting time. HBM's still very much constrained. So I come back to the earlier question I asked is, do you guys see this architecture as evolving over time? Because the balance is the key. You mentioned CXL is like the load balancer, if you will, the balancing element of the stack. So the hierarchy has elongated and compressed over time, and it seems to be getting more granular now.

David McIntyre

>> Yes.

Dave Vellante

>> Do you see that continuing, I guess is the question?

David McIntyre

>> Yes. So in fact, you could argue that, especially for AI application, storage was the second tier cousin to DRAM, HBM was the favorite child. And now that's a great way to look at it. We're seeing that triangle compressing together. So HBM has the highest bandwidth, the best near CPU or GPU performance. And with DRAM, and working closely with NAND memory, in fact with the CMMH Samsung platform that I referred to, by putting both of those media types in one common platform, you're actually showing the host the cumulative capacity between the DRAM in that same platform as the NAND. So if I had 16 gigabyte of DRAM cache plus maybe four terabytes of NAND, altogether, it looks like a four plus capacity point that's now available to the host.

Dave Vellante

>> Oh, interesting. Okay. So that's just a virtual pool that they see. Now when we talk about AI, we throw AI into this big bucket. With private AI and the power law of gen AI that theCUBEResearch put out early last year, we see customers wanting to do things on-prem to the extent that they can get GPUs. There's also the inference piece of it. So how are you seeing the memory hierarchy and trends supporting the different AI workloads? Can you guys speak to that?

Arvind Jagannath

>> So we are looking into HBM as a potential tiering candidate. So because we have the expertise in tiering between different types of devices, HBM could potentially become a candidate between, for example, DRAM and HBM. So that way we can make sure that GPU gets additional memory because it's tiering between HBM and DRAM. And at the same time, we are trying to give more value to gen AI type of applications by reducing the CPU utilization issue. Now, we improve CPU utilization, give more CPU to the AI applications. So overall it is definitely more value that we bring.

Dave Vellante

>> Right. So let's see, what else should we be thinking about roadmap-wise? Can you share a little vision as to where you see this whole thing going without obviously divulging anything confidential?

Arvind Jagannath

>> Yeah, with CXL, I think we are one of the industry firsts which is looking into a type two that they call, which is really custom. And we are thankful to the collaboration with Samsung that we are working together on this new architecture. So that gives us an ability to not only extend within the memory realm, but also go beyond. For example, we can do VMotion improvements potentially. We could also do AI inferencing, because the accelerator has the ability to accelerate a lot of different use cases. So that's the vision we are-

Dave Vellante

>> The other question is, you see obviously Blackwell, we need bigger GPUs. You hear Jensen talk about that. Blackwell maybe ... Looks like not maybe, but it looks like it's delayed a bit, but it's coming. And then also, at the other end of the spectrum, you have folks working on non-monolithic designs, chiplets and things like that. How do you see the memory hierarchy? Is it irrelevant What approach it takes? Whether it's a huge monolithic GPU versus some kind of alternative architecture, you're seeing a number of new startups come out. How does that play into the way you guys think about supporting all these new workloads?

David McIntyre

>> That is definitely the latest dynamic. Large language models are very, very important to our industry, but now we're seeing more tuned language models to the size of the application to the specific requirements. So you don't necessarily have to have a trillion parameter model to do the job. So we see aligning our memory compute or GPU resources to best support those more custom ... Cloud-to-edge applications is certainly one, and edge doesn't need to run an LLM necessarily. It can have a more custom-sized model to do so. From Samsung's perspective, we've developed, as I mentioned earlier, a full portfolio of technology platforms. I emphasize the CMMH, which as you pointed out, Dave, features persistence and also tiered memory. We have a CMMD product, which is focused just on memory expansion alone, and we talked about that's a way to alleviate the server constraints. We also have, and this is the roadmap part that you asked about, the CMMB, which is the CXL memory module box. And so this is putting a mix or the same types of devices all in one box that can be managed with ... We call it Cognos. It's our orchestration layer. So now Samsung is not just providing a device but a complete system that's available. And then for the overall advantage on TCO, we're experiencing at the show that customers are really focused on the TCO CapEx, OpEx benefit these types of solutions offer.

Dave Vellante

>> Plus that system's expertise you can bring to your customers, help them fine-tune things and understand how that whole system works. So you've got large language models, you've got small language models. And we've put forth this idea that you've got LLMs going to LAMs, large action models. You can imagine everybody's talking about agents and co-pilots, but not a lot of people are talking about swarms of agents. Agentic AI is something we've researched a little bit, started to write about. We haven't done the research on what that means for the infrastructure, but that could bring new capabilities. And I'm sure you guys are thinking about the possible and supporting the future. This isn't something that you developed to retire a year from now. You're thinking in presumably 10 year cycles or longer.

David McIntyre

>> And even far shorter than that is the ability for Samsung to continue our partnership with VMware, where we can better improve the utilization of virtual machines that may not be deployed because they're constrained on memory resources. So this platform variance, it's a special use case for VMware where the CMMH with a VSA at the end for the VMware project, that allows us to best support virtual machines, which helps all of the great products that VMware deploys be better utilized.

Dave Vellante

>> Now, what was the reaction to your presentation today? What questions were you getting? Was this novel to the customers or were they saying, "Wow, this solves my problem"? What feedback were you getting?

David McIntyre

>> Pretty much appreciative that a lot of end users in the audience and the financial services lead for VMware, Steve Fusco, he was basically presenting the voice of the end customer. And I'm coming in as a Silicon Valley technology lead saying, "Okay, here's what we have in our toolkit that we can have to solve these problems." But having that voice of the end customer there really enables, "Okay, well this tier one customer has this specific criteria. These channel customers may have different required elements." And so being able to support and match up our technology with the end user experience in financial services was a great opportunity. And that was the Q&A that we were getting from the audience.

Dave Vellante

>> And Arvind give you the last word. Any hallway track, or if you're presenting, what are you hearing from folks?

Arvind Jagannath

>> Yeah, so I think, like I mentioned, memory tiering is one of our key pillars this year in terms of infrastructure enablement. I focus mostly on the infrastructure from a product management point. And vSphere, again is ESX's foundation to everything that we do. And I think we are definitely providing more value to VCF because we are improving our foundation. So we think that customers are getting more from VCF because we are improving the efficiencies and thus we are helping them reduce their costs.

Dave Vellante

>> Guys, love this conversation. Love to geek out with the Silicon Valley experts. Thanks so much for coming on theCUBE and helping us wrap up day one. Appreciate it.

Arvind Jagannath

>> Thank you.

David McIntyre

>> Thank you.

Dave Vellante

>> All right, this is it for the CUBE coverage of VMware Explore 2024. This is Monday, day zero, if you will. We'll be back tomorrow. Here through Wednesday. I'm Dave Vellante, he's Rob Strechay. John Furrier is also in the house. Go to siliconangle.com, thecuberesearch.com and theCUBE.net for all the coverage. Thanks for watching. We'll see you tomorrow.