Carmen Li, founder and CEO of Silicon Data, delves into the transformative realm of AI factories in this episode. Hosted at NYSE Studios by John Furrier of theCUBE, the discussion centers on "AI Factories - Data Centers of the Future," emphasizing infrastructure and performance in a swiftly evolving technological landscape.
Carmen Li shares insights from their leadership at Silicon Data and Compute Exchange, two firms revolutionizing AI infrastructure. With expertise in making AI systems more transparent and efficient, Li discusses trading in spot compute and the innovative approach of treating graphics processing units as commodities. The video also presents perspectives from theCUBE Research, with analysis by John Furrier as co-host.
Key takeaways include the importance of system designs in AI infrastructure, the role of transparency in industry growth, and the potential for financialization of GPU resources, as highlighted by Li. The conversation underscores the necessity for adaptable AI models, focusing on configuration and workload efficiency to maximize productivity and performance in a constantly changing market.
Forgot Password
Almost there!
We just sent you a verification email. Please verify your account to gain access to
theCUBE + NYSE Wired: The AI Factory - Data Center of the Future. If you don’t think you received an email check your
spam folder.
Sign in to AI Factories - Data Centers of the Future.
In order to sign in, enter the email address you used to registered for the event. Once completed, you will receive an email with a verification link. Open the link to automatically sign into the site.
Register for AI Factories - Data Centers of the Future
Please fill out the information below. You will receive an email with a verification link confirming your registration. Click the link to automatically sign into the site.
You’re almost there!
We just sent you a verification email. Please click the verification button in the email. Once your email address is verified, you will have full access to all event content for AI Factories - Data Centers of the Future.
I want my badge and interests to be visible to all attendees.
Checking this box will display your presense on the attendees list, view your profile and allow other attendees to contact you via 1-1 chat. Read the Privacy Policy. At any time, you can choose to disable this preference.
Select your Interests!
add
Upload your photo
Uploading..
OR
Connect via Twitter
Connect via Linkedin
EDIT PASSWORD
Share
Forgot Password
Almost there!
We just sent you a verification email. Please verify your account to gain access to
theCUBE + NYSE Wired: The AI Factory - Data Center of the Future. If you don’t think you received an email check your
spam folder.
Sign in to AI Factories - Data Centers of the Future.
In order to sign in, enter the email address you used to registered for the event. Once completed, you will receive an email with a verification link. Open the link to automatically sign into the site.
Sign in to gain access to theCUBE + NYSE Wired: The AI Factory - Data Center of the Future
Please sign in with LinkedIn to continue to theCUBE + NYSE Wired: The AI Factory - Data Center of the Future. Signing in with LinkedIn ensures a professional environment.
In this interview from theCUBE + NYSE Wired: AI Factories - Data Centers of the Future, Carmen Li, chief executive officer of Silicon Data and Compute Exchange, joins theCUBE’s John Furrier to unpack a thorny question at the heart of modern AI infrastructure – when the GPUs are “the same,” why do results vary so widely? Li explains how Compute Exchange is building a more transparent spot market for GPU clusters while Silicon Data brings Bloomberg-style pricing signals and third-party benchmarking into a market that too often runs on assumptions instead of evi...Read more
exploreKeep Exploring
What questions are being raised regarding the infrastructure and performance of AI factories?add
What is the purpose and benefit of the Compute Exchange in the context of the computing resource market?add
What are the key offerings and benefits of Silicon Data in the context of GPU processing and alternative data?add
What considerations should be taken into account when tuning data acquisition in a distributed heterogeneous environment?add
What is a GPU spot market, and how does it function?add
>> Welcome back, everyone. I'm John Furrier, your host of TheCUBE. We are here at TheCUBE's NYSC studio. Of course, this is our East Coast home. We have Palo Alto Studios as well, connecting Silicon Valley and Wall Street, tech and capital markets coming together. That's our NYSE Wired program, a Cube Original. And of course, the NYSC community is an open access network, feeding entrepreneurs, getting access to deals. We've got a Cube alumni here to talk about our AI factories here. Carmen Li. She's the CEO of two companies, Silicon Data and Compute Exchange all tied around data. And as we look at AI factories, one of the biggest questions that's come up over the past year and will be for this year is, what's going on with the infrastructure? What's the performance look like? Is it worth the money? Is it performing properly? Are those workloads hitting what they need to do from a performance standpoint? Because AI factors about data in and out comes out. Carmen Li is back on TheCUBE. Carmen, great to see you again.
Carmen Li
>> Good to see you, John. It's a great Happy New Year.
John Furrier
>> Yeah, happy new year. Silicon Data. Love the name as you know, Silicon Angle, pretty close to that. You're the data side on the Silicon and Compute Exchange, another company, two companies you're the CEO of.
Carmen Li
>> That is right.
John Furrier
>> Love the dual CEO company thing, but there's synergy between them. First, explain Silicon Data and Compute Exchange because it's hand in hand, but different in the right ways.
Carmen Li
>> You're 100% right. So we're creating this ecosystem while bringing so much transparency, efficiency to the compute world. Compute Exchange is exchange layer for actual spot compute trading. So think about if you need GPU clusters, 200 nodes of B200, let's say, for next three months, there's a few ways you do it. You can call 70, 75 neoclouds provider yourself or go to Exchange. Everything's more transparent, we have all the rubrics servicing on the performance differences, pricing, terms, support level. So, it's much easier for our startup of the world to procure GPUs. We love compute change, brings so much efficiency and transparency, and price advantages to ecosystem. Silicon Data is a Bloomberg for GPUs. So we publish indices, which the GPU A100, H100, B200, which is available at Bloomberg terminal and terminal. We also publish alternative data sets, benchmarking tools, which we'll talk about, so they can mark that we have seen very interesting results across 20 neocloud providers. And lastly, we also provide defend application tools on top of that. So, it's a great two sister companies.
John Furrier
>> Yeah, yeah. I like the Exchange. I think that's going to come into the demand curve. Silicon Data, you mentioned it's like the Bloomberg terminal for what's going on on benchmarks. I wrote a post about Lambda, thanks to your great report. Seeking the Holy Grail, Why AI Performance is Now a Systems Problem. Obviously, that's in context some of our reporting around AI factories. But you had put out an independent benchmark on GPU platforms with your SiliconMark capabilities. This was an amazing data. Of course, it turns into media because it was high quality data. But you're measuring the internode bandwidth and latency in this case. But this is the kind of level of detail that gets into the weeds a bit, but it's important because it's like a car engine or an airplane engine. You want to have a health meter on what's going on in the system, depending on what it's designed to do.
Carmen Li
>> That's exactly right. So one of the things we observe in the marketplace is nowadays, people treating H2100 the same across all different providers. Which you can't blame people because there's no other transparencies available, no data points, to your point. So we're the only third party benchmark services which we're surfacing, not just the flaw performances, but the memory hierarchy numbers from D2Ds to HBMs to L2 cache, all different memory bandwidth to the users. And then we draw things independently and very efficiently. And then we actually have a paper coming out very soon talk about all the findings.
John Furrier
>> And the reason why I call it seeking the holy grail, it's a little bit of an old school reference to the Monty Python seeking the holy grail, which they never found by the way, but it was a seeking. But I think the system's angle on this is huge because the number one thing we're seeing in this AI infrastructure that you're also reporting is the AI infrastructure build out and the developer capabilities. And that they're all converging into be frictionless, scalable, low latency, high velocity. But the problem is that these system designs aren't a one trick implementation. I can buy a big monster Blackwell rack with Nvidia if I have that use case. But if I'm an enterprise, I might have that and maybe a couple other different configurations based upon the workload. So, this opens up kind of the design side. So, I think there's a huge market for the design side. And I want to ask you, as you look at the data, do you see the same thing? And how are you guys tuning your data acquisition to kind of understand? Because it might be, "Hey, I'm going to have a couple GPUs here, more compute over here. I'm going to change my memory architecture because it's less expensive for what I'm doing. But I'll still pay the big bucks for the training and inference and reinforce learning over here."
Carmen Li
>> trade off.
John Furrier
>> So, there's a trade off, but it's a distributed heterogeneous environment. So, you have to have analytics.
Carmen Li
>> Your intuition is 100% right. I think unless you have a limited pool of capital forever, you can keep getting the greatest and latest, the best interconnect everywhere, the best energy, best cooling. Usually people facing trade off. And sometimes it's for the right reason, right? You don't need the lowest latency for every workload. You don't need the highest HPF for every single thing. It depends, is it training, is it inferencing, what we're looking to do? So, your intuition is 100% right. We see a big variance within the whole new eco cloud, including hyperscalers with them in the whole pool as well. So even for the same GPU configuration, the variance we observed across 6,300 tests is about 37% variance. It's really high. You arguably probably paying the same amount, but you're getting a very big difference in performances. You might not even know that. You might know, "Hey, I'm supposed to finish my cycle at this time. Why is it delayed? What happened to it? What's the causation? Is it the neocloud provider? Is it GPU itself? Is it memory? What is going on?" So, lack of transparency really hindered the whole industry growth.
So, what we try to do is bring that before you start your workload. So you have understanding the trade off. You understand this is how much I'm paying, this is what it's supposed to get. So, everything is a lot more about calculated suggestions.
John Furrier
>> Well, I love Silicon Data. In fact, on that story I wrote, I went out and compared that to our AI engine. And I got some great quotes from Nvidia Senior Vice President, Gil Shainer who captured this clearly, he said on TheCUBE, "Distributed AI only works when thousands of accelerators behave like one supercomputer. That requires low, jitter, deterministic interconnects, otherwise the GPUs are just waiting on the data." In your report, it showed Lambda, for instance, really did well on the PCI-
Carmen Li
>> PCIE....
John Furrier
>> PCIE connections. Again, in context with Gil at Nvidia, who's an engineer and architect, he's saying, it's not just about the GPUs, although they're in the GPU business, they're engineering systems.
Carmen Li
>> It's very .
John Furrier
>> So if you don't think about these interconnects, the performance could really plummet and be inefficient. The expensive GPUs are just waiting around when they need to be fed.
Carmen Li
>> Exactly. So Lambda especially outperform in the host of device categories. So what I thought was pretty good is what it did is, especially if you have a lot of data need to moving outside the whole ecosystem moving to the GPU. So for example, heavy load inferencings, inference on large language models, or real time, right? Host to device connection, PCI is very critical. Obviously you can do SXM, but it's all become dollar per token, right? What's your trade off for that? And then there's other things does well. So we test everything from the flops to, as I said, different hierarchy of different memories. We also noticed sometimes people are actually getting better, right? Doesn't mean you always stay where you are. People make an effort. They realize we're not the neocloud provider. It's pretty interesting. They start working with us. They realize that, "Hey, I didn't know where... we didn't do well in this sector." They are really low in one particular memory hierarchy. So, they fixed it and then we test again. It's actually not at the medium. So for them, sometimes even for the provider, it's not very clear.
John Furrier
>> Yeah. And the data helps. Where do you see this going? Because I see big promise in how you're doing that. And we'll get to the Exchange in a second, but on Silicon Data, what's next for you? What are some of the things people are resonating with? What's the traction point? And then where does it go next?
Carmen Li
>> So many things. The market growing so fast, which is exciting. We have many design houses now. We have NVIDIA which is obviously the leader. We have AMD, which I'm excited about their newest generation. We have SambaNova, Cerebras, we have all the independent design houses, and AWS TPUs. So with all design house chips, you need a way to compare them side by side. You need to have a common inputs and common output to say, "Hey, for this particular set, here's the commercial type, which we understand per GPU per hour, roughly. And here's why they are different. Which one should you pick for your use cases?" It's not one best thing fits all. It's about education, about awareness, about transparency. So, that will be-
John Furrier
>> So I interviewed Michael Dell and also the CTO of Dell Technologies, John Roese, a year and a half ago. And they were the first ones, John Roese in particular, the CTO, who identified that the AI factory certainly would be relevant. And that was obvious to us when we talked then, now it's clear. But one of the things he brought up in talking to his customers in the enterprise is an AI factory implies like it's a server in the old days. But it's not a server. It's like a bunch of servers and it's a system component, as you guys are measuring. And the number one question for enterprise is, how do I configure these? What do I load on them? Now, they know their workloads and they're going to plan for AI native workloads. So this idea of tuning is coming up a lot in the enterprise. So I think what you're doing is very relevant there. What's your reaction to that and how do you see enterprises adopting? Because this is where they're going to need to have, "Hey, I'm going to buy a car for this road. I want the big fast race car for the track." It's almost like you can't-
Carmen Li
>> I don't know if it's the right answer, John. You're probably a better person to answer this question. I just feel like right now there's newer chips coming out every year. I know people are excited about Vera Rubin and people are still trying to get a B300 up and running. So people, do you need B300? Maybe we see a huge jump in H-100 prices in the last few weeks. I don't know if you noticed. It's about 10%, which is quite a lot for a very mature chip. So you can see demand supply definitely are still shifting, shifting around. You might not need the latest and greatest. And then think about, to your point, system level, right? Not just the GPU configuration, it's about the libraries. Which version are you using? Is it to 28? Is it 30? When do you switch? Is it configuration? Is the interconnect? It's a lot of thought. It's a lot of things you have to test and fine tune. If I were a large enterprise, what I would do is really try to leverage neoclouds as much as I can or hyperscalers. So then you can test and verify before you invest a lot of money because those servers are millions of dollars.
John Furrier
>> Actually, you brought up a good point. I want to touch on that because I think I've been interviewing a lot of neoclouds and the hyperscalers, all of them. You know who they are. Also been interviewing their customers. It's the Metas of the world. Even Amazon and hyperscalers are buying from neoclouds. So, what's interesting is that they look at the available GPUs and systems as an aggregate. And so they don't really care too much about just CoreWeave, they want, "I want to get it from wherever I can." So there's almost an abstraction of, how can all the neoclouds work together? So that brings up the point of if you have an observation space for your data across the neoclouds, you can help provide direction for say a Meta or even a hedge fund.
Carmen Li
>> Yes, the utilization, the capacity. So-
John Furrier
>> And let them compete on their value.
Carmen Li
>> Exactly. So, your costs are not the same. Should it be the same? Maybe you shouldn't. And then there's also regional differences, right? I don't know if you start a big buildup in the greater self.
John Furrier
>> Yeah.
Carmen Li
>> Right? Maybe they need so much GPU for latency reasons. So, we definitely see a lot of movements. So, we do a really good job at tracking inventories globally from Compute Exchange. So obviously for the clients, right, when they RFQ a particular reserve request to compete change, we have to match them up with the relevant data centers and neoclouds. And then not only we need to know, do you have 100 H100 nodes available? It's more about, "Hey, is this compliant? Hey, what kind of support do you provide?" Hey, can they, to your point, stitch up with other providers to form a bigger cluster, because no one hey had enough? It still sellers market today. So that's something we try to figure out, how can we help our clients to match up the right compute resources?
John Furrier
>> And that's on the Compute Exchange side.
Carmen Li
>> It's on the Compute Exchange side.
John Furrier
>> Because that solves the problem of, my workload has these parameters to them or requirements.
Carmen Li
>> And they change too.
John Furrier
>> It's not just give me some GPUs. Does it fit my workload?
Carmen Li
>> The data is the qualities, the coolings and comes down to operation systems. Well , what kind of Linux? Ideally, it should be uniform so you have a better experience.
John Furrier
>> So if I get this right, correct me if I'm wrong. So what I hear you saying is, you got a Compute Exchange for understanding what's going on in the map environment, Silicon Data to verify those configurations
Carmen Li
>> SiliconMark.
John Furrier
>> SiliconMark. Okay, SiliconMark. Explain that. So, SiliconMark is what? That's on the Exchange side or Data?
Carmen Li
>> It's the data size. Think about SiliconMark almost like the Moody's for securities, but that's the Moody's for GPU clusters. So, we do branch mark on actual hardware and then we track the UID. So depreciation schedule, we know exactly the same GPU cluster. When it depreciates, the flops will decrease. When the CPU gets swapped out, the UID will change. So, we check out the machine level information and then we provide fair market value assessment. So by year three, the market is trading H100, say 30 cents or a dollar, your machine may be much better than average three year vintage, you should be trading higher. So, we provide that fair market value assessments. The Exchange is purely exchange where transaction happen, the matching happen, either rental or retail.
John Furrier
>> So SiliconMark is in Silicon Data, but you also do benchmarking. That's kind of the Moody's.
Carmen Li
>> Yeah, we .
John Furrier
>> Verifying, that's almost putting the stamp of approval, certifying if you will, kind of what that is. Okay, good. So, they all work together. So I have to ask you, what's it like being the CEO of two companies? It's kind of one company, but it's two different companies, right?
Carmen Li
>> It's two very different companies. One is like-
John Furrier
>> By the way, I love that, by the way.
Carmen Li
>> Oh, thank you, thank you.
John Furrier
>> Most people are like, "Whoa, you can't be CEO of two companies."
Carmen Li
>> I don't think it's easy. One is like a and the other is like Bloomberg. It's so much synergies. Obviously, before I commit to this dual structure though, I have concerns. I have two kids and I have two companies. It's not the easiest thing. The thoughts I really have is, is the whole thing worthwhile? Now I'm five month in, I'm so happy. I mean, the decision is I bring so much more value to both sets of customers. Not only do I see global data, global movement, global volatility, creating financial products, benchmarking, really contextualize with actual deal flows. I know exactly who is asking what kind of GPU when. I know exact pain points, real time physical pain points people experiencing for neocloud side. From our customer side, it is so important to have that transparency and bring the synergies. Now everyone benefit... Well, I'm the only one sort of unfortunately sacrificing sleep a little bit and phone time with my kids.
John Furrier
>> I know you've been doing a lot of traveling, but my point is I think that's okay because they have such synergies.
Carmen Li
>> It's a lot of synergies.
John Furrier
>> You get the holistic, but they're also different competencies, but they play well together. So it's almost good to have separate companies, maybe separate investors, or you can maybe grow one faster.
Carmen Li
>> Exactly. One is a marketplace, right?
John Furrier
>> Yeah.
Carmen Li
>> It's own challenges and upside. One is a very traditional, in my mind, data company, right? Data financial products and intelligence layer, right?
John Furrier
>> So, what's your vision? When we last talked last year, I love this whole financial play of the commodities, not commodities, but like the GPUs.
Carmen Li
>> The , yep.
John Furrier
>> It's like tech meets finance, which is our wheelhouse. What's your vision on? Do you see that continuing? And what's the state of the current market? Are GPUs up?
Carmen Li
>> I think it's accelerating. I think, oh, it's a great poly market. poly market. So I think this is accelerating. It's just not growing, it's beyond the linear projection I have in my mind. We are planning to launch futures options on top of our indices, the market needed. I have so many financiers, technology companies, so all the ecosystem participants express interest. And at the same time, we have token. A lot of people are going to care more about token prices fluctuation versus GPUs because they're one degree removed from GPU themselves. So we have a token indices, token products, token data sets. And at the same time, we're helping people think through, "Hey, what's your long-term strategy with a total cost of ownership?" There's a lot of moving pieces, but financialization is definitely going to be very critical.
John Furrier
>> What's been the feedback from, say, people who care about this? Because I can see clearly risk management, pricing, investment like private equity companies, hedge funds. There's a lot of bets going on in the sector and it's kind of a new motion-
Carmen Li
>> It is....
John Furrier
>> new mechanisms.
Carmen Li
>> Next-
John Furrier
>> What's the-...
Carmen Li
>> 18 months, right?
John Furrier
>> What's been reaction?
Carmen Li
>> I think with the biggest trade of our century away, the next 18 months and off, we know the options expiring a year and a half and two years, right? So, people want clarity. If you really go down too deep, people want clarity. What's the depreciation schedule should be? Are those things really worth nothing after a certain amount of years? Who decides that? What kind of volatility are we looking to see? How is it going to correlate with any other asset classes? Things will only be clear once we have more data, more ways for people to express opinions. Because right now you can launch on NVIDIA, but what else can you really do? It's almost tell you, "Hey, there's no all your futures." The only thing you can do is launch your . It's self-sufficient.
John Furrier
>> One of the things I saw with Burry's comments about AI and Nvidia, which I thought he got all wrong, was he was basically looking at the GPU as an asset, like a house. But if you look at the older versions of NVIDIA stuff, they're being used in other use cases.
Carmen Li
>> L40s is still in use. Yep.
John Furrier
>> So, I think the reuse is a new variable. What's your reaction to that? Have you seen that in the data?
Carmen Li
>> Yes. So obviously, we capture all different chips from consumer grade, so RTX of the world, to older chips, L40s. I mean, today L40s, some hyperscalers will charge you 40, 50 cents on an hour, L40s. You imagine that?
John Furrier
>> Yeah.
Carmen Li
>> Right? So, there are different use cases of different GPUs. As I said, today we're still experiencing somewhat sellers market. People want to do whatever they can get their hands on.
John Furrier
>> Carmen, I'm really impressed by your vision, but also you're executing. I have to ask you a kind of question to define what a spot market is relative to GPUs for folks that don't know that aren't in the weeds. Because they hear, this GPU spot market or these kinds of new markets. Explain very elementary for the folks that aren't in the industry or in... They may be in business, they're super smart. They read the Silicon Angle, New York Times, maybe not know all the stuff you know. What is these markets? What is a GPU spot market?
Carmen Li
>> It sounds complicated. It's actually, at the core, it's very 1800s. Think about today, if you're a farmer, you try to sell corn, right? You go to and say, "I want to sell corn?" And people say, "I want to buy corn. I want to buy corn from you for the next three months." So it's actually corn transacted in Chicago, right? So compute change is the actual GPU resources getting transacted, right? You get what you pay for. You get what you deliver. It's actually transaction either through reserve rental rate or through actually people say, "Hey, I want to get rid of E100s." You may be like, "Hey, I can use E100."
John Furrier
>> So, it's the current price of GPU-
Carmen Li
>> The actual GPUs....
John Furrier
>> cycles that you might need.
Carmen Li
>> You can actually get your hands on the resource itself. On the other side, Silicon Data's published indices think about all the future indices, all the indices that people publish, which you can't really get GPU by looking at indices. It's a market demand supply metrics, right? So, it's kind of different. The future is obviously behave like futures. You're trying to hatch your volatility for the forward.
John Furrier
>> I've heard NVIDIA and Jensen say this. I've talked to other founders like Andrew from . It's like electricity.
Carmen Li
>> Exactly.
John Furrier
>> What's the price of electricity today?
Carmen Li
>> You turn it on, right?
John Furrier
>> And I'm using it. So developers can just come in, interface into an abstraction, could be a marketplace, and that pricing will be reflected. The underlying pricing powered by its Silicon Data and Compute Exchange.
Carmen Li
>> Yeah. It's a perfect analogy. So electricity today, they have spot market as well as futures market. Depends on what we need to do. You and me, we don't really care. We just turn on the switches and things just work, right? Think about the .
John Furrier
>> But someone has to track it.
Carmen Li
>> Someone has to track it, exactly.
John Furrier
>> Don't leave the lights on.
Carmen Li
>> Exactly. Turn the lights off. The building manager will care, the factory manager will care. They have to procure electricity, the data center manager will care. So, they're the one doing the spot market. I wanted this how much gigawatts today. They're the one buying the electricity.
John Furrier
>> I think you're ahead of the curve on this in a good way. And I'm glad you're getting out front. In the cloud computing market, obviously the rise of cloud, we saw exponential growth, great ecosystem. And then as it, not mature, but as it got kind of more settled in production workloads, FinOps became important, observability. So all the instrumentation around, okay, what's the economics? We get how it drives value, but now operating costs become a huge thing. This is early days for the AI infrastructure.
Carmen Li
>> It is exciting times where we can hopefully ship the system be more efficient. The only way to make things efficient is being transparent about ideally everything from commercials to your terms and services, to a support, to the action machine itself.
John Furrier
>> Carmen, great to see you again. Thanks for coming in and participating as an expert in our AI factory series. And congratulations on the two companies you're continuing to run and-
Carmen Li
>> Thank you....
John Furrier
>> we'll keep following you. We'll see you on LinkedIn and X and social handles. And thanks for coming in.
Carmen Li
>> Thank you so much, John.
John Furrier
>> Okay. I'm John Furrier, host of TheCUBE. This is our AI factory series where the leaders come in and talk about what they're working on. But really they're driving this infrastructure that will power in agentic infrastructure and ultimately AI, physical AI and physical factories. You're starting to see the digital and physical world come together. And AI infrastructure is going to move to autonomous, all going to be instrumented, measured, priced, consumed, and developed on. Really super exciting future for AI. We're doing our part to bring it to you. Thanks for watching.>> Clear.
John Furrier