theCUBE + NYSE Wired: CMO Leaders | JB Baker, Scaleflux

Clips
More from theCUBE + NYSE Wired: CMO Leaders

JB Baker

CMO

ScaleFlux

play_circle_outline Addressing memory wall with technologies like CXL

play_circle_outline ScaleFlux's role in AI infrastructure efficiency

play_circle_outline Challenges in managing reads and writes in AI workloads

play_circle_outline Future workloads require innovation in data access

play_circle_outline ScaleFlux: Innovating Silicon Development for Rapid Chip Innovation Operations with Expertise

Info
Transcript

JB Baker, Scaleflux

JB Baker

CMO ScaleFlux

JB Baker, the chief marketing officer at ScaleFlux, joins theCUBE's John Furrier to discuss the innovations at ScaleFlux, an AI infrastructure startup. Baker's expertise in optimizing AI and data center infrastructure is highlighted, touching upon efficient data utilization in power-demanding environments. TheCUBE Research team delves into the strategic role of storage and memory in enhancing AI operations, alongside analysts providing insights into how these solutions impact the broader AI ecosystem.

Key takeaways from the conversation include the c... Read more

explore Keep Exploring

What technologies is ScaleFlux working on to break down the memory wall in AI infrastructure and improve the efficiency of data transfer to GPUs? add

What is the potential impact of controllers saving power at both the component level and system level in a storage array? add

What are the different types of workloads in AI and how do they affect the balance of reads and writes on storage drives? add

How should CMOs think about AI generally when planning for the future and taking advantage of available technology? add

What was the background and experience of the core team members who came from Fusion, and how has their expertise enabled efficient and high-quality silicon development for theCUBE? add

bolt Powered by CUBE AI

JB Baker, Scaleflux

search

>> Welcome back everyone. To theCUBE here in our Palo Alto Studios. I'm John Furrier, your host of theCUBE. We are here for theCUBE in the NYSE Wired community series of CMO leaders leveraging AI, building AI. JB Baker's here, the CMO of ScaleFlux, hot growing AI infrastructure startup company, growing really fast, enabling all the AI. JB, great to see you. Saw you last night at the reception that Brian Baumann, founder of NYSE Wired and team put together. Good to see you in studio. Thanks for coming in again.

JB Baker

>> Yeah, appreciate the opportunity here.

>> So CMO leaders, obviously this is a program around how CMOs are seeing value with AI. You guys are enabling it with ScaleFlux. Let's get into a little bit about what you guys do and then we can kind of connect the dots on where that fits into the AI implementations for CMOs.

JB Baker

>> Sure, yeah. For ScaleFlux, we're in the storage and memory arena and the core of what we do is we help make the whole AI and data center infrastructure more efficient, more effective, help you get better utilization out of those power and dollar hungry GPUs and CPUs by keeping them fed with the data and just streamlining that whole data pipeline to balance the access to data and the storage of data with the growth that's happened in processors.

>> We had an opportunity to listen to you guys give a talk at the NYSE. Brian had an event. You guys presented. And it was interesting. A lot of the alpha nerds were in New York really loving the presentation because it's about powering everything right now. You're seeing all the innovations. The DeepSeq style shows, okay, you can be clever, software is going to be coming in, open source is driving it. But the AI infrastructure's still on that one last mile of the journey. That's my view and our view, is that you're going to see a lot continuing to get that performance squeezed out of it. And once that hits, we're predicting a tsunami of apps, a Cambrian explosion of new kinds of apps, maybe ones we've never seen before, and so that means the underlying infrastructure is changing radically. You guys are doing it. What are you seeing right now that's happening that people could look at? Because a lot of the CMOs we talked to, they can go after opportunities, but still more coming. And they're looking at this as a platform opportunity. What do you guys see as that next breakthrough moment in the AI infrastructure?

JB Baker

>> For generating content for marketing or-

>> No, just in general, under the infrastructure. Once that power source comes in, what's next with ScaleFlux? What are you guys working on now that's going to continue to push the water through the pipes, so to speak?

JB Baker

>> Okay. Well, with ScaleFlux, what we're trying to help do with the AI infrastructure, as I mentioned, streamline that data pipeline. What we've seen over the past several years is the capability of the processors to consume data has far exceeded the ability of memory and the capacity for memory to deliver data to the GPUs. So this is the memory wall, is what you'll see in the industry. And so we're working on the technologies to help break down that memory wall, like CXL, to provide more physical memory attached and more bandwidth attached. Innovations in the storage domain and the NVMe SSDs to better align the flow of data with the type of the IO size that GPUs require to run their tasks. There's a mismatch today, and so there's wasted bandwidth. And so it's, how can we improve the effective payload of the data that gets transferred into the GPUs so that they get more utilization?

>> What's the core problem that you guys solve? And you market the solution. Obviously your sales motion is really technical. You guys integrate it in. What are some of the challenges that you guys help people overcome?

JB Baker

>> We help them overcome basically the efficiency of that whole AI infrastructure and data center infrastructure. We talk about AI a lot, but it's not just that, right? It's your standard mission-critical applications are also boosted by leveraging more advanced SSD technology that helps reduce the latency so that your applications are more responsive, you can do more transactions, you can handle more users within a given amount of infrastructure. And over the past couple of years with the massive boom in AI, what you really, really start to see is that massive increase in how much power gets consumed per processor, per rack. And so getting more work done per watt is crucial to being able to scale your applications and your capabilities.

>> I was going to bring that up. I'm glad you brought up power because everyone talks about speed. Speed does win. We've been covering that. We saw news just this week Cerebras with Perplexity, Le Chat. So you're seeing a lot of work being done on speed. So IO packets are moving faster, data's moving faster. Great. Continue to push the envelope on that. But power is where the lever matters. Expand on the power piece. I think power efficiency is the key constraint. This is a huge point. It's kind of nuanced, but everyone talks about power. This much power is on the rack, this much power is in the facility. So power is a huge constraint in the design on the systems. Why is it important and what do you guys do as different?

JB Baker

>> Well, boy, where to start on why it's important? From the global level, it's important just from the demand that's on power in data centers and to support the growth that's planned for data centers. There's not enough of it in our grid. And then bringing on new power plants and more power generation capacity is a long haul, right? You can build out a data center faster than you can get all of the approvals to build a new power plant. Though you see now with many of these data centers, they're starting to plan to build a power plant adjacent to it to supply it, whether it's going to be traditional fossil fuels or even nuclear power plants as well. So there's that global level piece. And then at the local level of go, dive right all the way down to the processor, when you've got a GPU that's running hundreds of watts as opposed to your laptop's CPU that's running single digit watts, maybe 10 watts, there's a massive challenge in one, getting the power down to that, two, getting the heat that's generated off of that. And then so you've got to innovate in the cooling capabilities and you've got to make sure that you're getting as much work done out of that power that's sent to the GPU as you can.

>> Scope the magnitude of the power efficiency. Give us an example of where we are today, what you guys do because you have a variety of products. You mentioned SSD. You have the system on a chip. What's the capacity uptake or... Not uptake. The scope of benefit?

JB Baker

>> Sure.

>> Try to scope that for us.

JB Baker

>> I got to think here a little bit. So at the component level, we're selling NVMe SSDs, or the controllers that go into NVMe SSDs. And in a storage array, your one or two U array, you might have a couple dozen of these drives. In a rack, now you're talking hundreds of those. So having a controller that can save power at hundreds of spots within a rack, now you're saving hundreds of watts. So there's that aspect just at the component to component level. Then there's the system impact of that controller saving power and delivering more data to the GPU. Now the GPU that was going to consume all those watts anyway, it's able to get more work done. It's not-

>> So more capacity.

JB Baker

>> Yeah, more capacity or more efficiency out of that power that you expended. Because it's not like we're going to say, "Oh, that our capability reduces the power that the GPU is going to consume," but we can reduce the amount of power that a job takes to consume by helping you with the efficiency of the GPU so that it gets work done faster.

>> Dave Vellante and I always talk about how benchmarks are always, depending on who you talk to, everyone's number one at something in the AI world. Job completion is a huge stat people tend to throw around now. So I want to get your thoughts on getting the job done in terms of the completion. But you mentioned SSDs and the storage array so I want to talk about that for a second. So we've seen in the LLMs and the foundation models that in the classic storage world, reads and writes were always talked about. In some cases there's more reads and more writes depending on the use case.

JB Baker

>> Yes.

>> So this becomes a factor. How do you guys view that and how does that factor into some of the advances in terms of the software and controls that you guys have to manage reads and writes? Because if you're training, that's a different behavior than inference.

JB Baker

>> Yes.

>> Talk about that dynamic, because I think this comes into the design side pretty important.

JB Baker

>> Yeah, absolutely. AI is not just one workload, right? And depending upon what you're doing, whether you're building the model, training the model, you're running inferencing, you're running RAG, there's a different balance of reads and writes and a different type of reading and writing, whether it's going to be sequential, which is just streaming the data as fast as you can off of the drives into the local memory for training, for example, or you may be reading and writing back and forth randomly to the drives. And each of those puts a different pressure on the media itself. If you are just doing sequential reads, basically every drive can fill the pipeline. But once you start going after these very small random accesses, that's where it gets to be challenging for the drives and there's a mismatch between how the flash behaves and how the GPUs want to access the data. So that's an area still for innovation. And then also as you start to do the mix of reads and writes, how quickly the SSD can write the data becomes tremendously important because as you're writing the data now you're blocking the ability to read from the NAND at the same time, and so you have to wait for the writes to complete to start doing your reads again.

>> That's where the software comes in, and this has become a thing. So I got to ask you about the workload because you guys are building out on a longer horizon, thinking about the next workloads coming. What do you guys see? What are people designing around on the workloads? What's state-of-the-art today? Table stakes, and then where's the puck going? As they say, "Skate to where the puck is going," to quote Wayne Gretzky. You guys are ahead of the curve right now. You got nice progression on the product roadmap. I love the system on the chip. I think that's phenomenal. Love the software innovation you guys have. What's the workload you guys see coming that you're preparing for? Can you share any data on that, or workloads, generically?

JB Baker

>> Yeah. I won't say specifically, but in general, what we do see is an evolution in the type of AI workload and how that is going to access its data and how it interacts with both memory and storage. And so that's where inferencing has been the... Or sorry, model training of the big LLMs has been the thing that stood out, right? Chat GPT is an LLM. Llama is a big LLM. Even DeepSeq is an LLM. And those were very, very heavy on streaming reads off of the data. Or sorry, off of the media. And as we have evolution into more of the inferencing and the...

>> Reinforced learning or whatever.

JB Baker

>> Yeah. The reinforced learning, the RAG workloads where those are going to be not just consuming data, but then generating data that has to be written, and accessing the media in a different manner, these small, as I mentioned, these small random reads and writes, that's going to change how the fundamental architecture of the controllers and how you access your data. And so that's where we're heading is mixing to an -

>> You got to be careful not to give up too much.

JB Baker

>> I know.

>> Yeah.

JB Baker

>> Well, NVIDIA presented about this mismatch. They presented multiple times. They presented back at OCP. It's a public thing out there. And then they held a kickoff session in December talking about where they see that there is this mismatch and asking the industry to help them solve that problem. Well, of course...

>> It helps them. NVIDIA has been great at putting out, setting the agenda of what accelerated computing looks like. There's also been efficiencies in how things are configured. We've been covering that, what's around the chips. Memory, SSDs, flash, NAND. And then now you've got the whole, okay, what software runs on it? And then as you said, you got to be prepared for the diverse set of at any given time, this kind of read pattern, this kind of write pattern, or mix of those, a mixture of access to the data is coming, and as models integrate. So we're starting to see that too. Okay. Let's take this back to the CMO piece because I think that's part of the conversation here. CMOs are going to take advantage of what's available today, but also try to plan for what's coming. So assume that you guys will continue to thunder along and do your innovations. As a CMO, your fellow CMOs out there watching are trying to put plans together. "Okay, I'm going to have a platform operation side, some sort. I'll have..." They have a lot of data. It may not be mission-critical other than proprietary data on customers, for example, but they're going to have to go out there now and knock down some wins. And there's plenty of use cases. We covered a lot them. As a CMO, how should they think? You're a CMO. How do you think about AI generally? Take your ScaleFlux hat off, put on your CMO hat and comment because the best CMOs are going to lean into this.

JB Baker

>> Yeah. Well, I got to put on preface my comments with, I'm a very B2B focused person. Our products, we're going after major data center customers, Fortune 500 type companies, people spending hundreds of millions a year on their infrastructure. So that's my focus.

>> You're very targeted.

JB Baker

>> Yeah. But for AI, we are already starting to, we utilize it to help us with drafting content. It's not great for generating final content, but helping you draft.

>> Do some heavy lifting.

JB Baker

>> Yeah. And just getting off of that blank sheet of paper and into a first draft to tear apart. We were leveraging it to help us generate more engaging images and video. And I definitely, looking forward, I haven't dug too deeply here yet, but I'm looking forward to utilizing it to help us with rapid testing and rapid refinement of core messaging. Because a critical thing for us marketers is, are you getting the click-through rates for any of your advertising? Are people staying on your website and getting educated and then engaging with your sales team? I tell you, that is the hardest thing in this attention economy is keeping somebody's attention long enough to get them willing to talk to your sales team so that you can generate some revenue.

>> Well, you're in an ideal content marketing opportunity because you have thought leadership, you have education, in your motion, education, and then support of that educational progression. It's huge. You guys have a very narrow target window of customers that'll be leaning in and learning with you.

JB Baker

>> Right, right. And now reaching those people enough times as well-

>> Get their attention.

JB Baker

>> Is challenging.

>> Yeah. It's like once you get their attention, "Well, we succeeded. Now what?" So you have to kind, this is where AI, I think could help. Once you know you get it, you can connect to it.

JB Baker

>> Yeah. Yeah.

>> What's new for you? What are you most excited about as an industry participant? Obviously B2B focus, ScaleFlux. You got a visibility into what's happening in the AI infrastructure. What are you excited about?

JB Baker

>> That's a tough one. Well, just this rapid pace of evolution here is, that's why you got into the high-tech industry to begin with. But I am very interested to see where we're going to be able to go from a marketing perspective on the content generation and digging into the data better to say, "Okay, these are the pain points," in the words that our customers use and feeding that back in to help our messages more resonate so that we get... I've talked to folks and there's the, you've got your shotgun marketing where you're guessing, but then being able to know that based upon the response rates we've had from these different tests, that absolutely, these are two or three key metrics that our users are going to care about and arming the sales folks with those to be much more effective and drive efficiency into marketing just as we drive efficiency into the infrastructure.

>> Well, JB, great to have you on the program. Final minute we have left, talk about ScaleFlux's team, because you guys are a very impressive team of industry veterans. We're in a market where it's highly compressed product cycle times, so software obviously a big part of it. You get the system on the chip, you got the SSDs, you got a great product mix. Talk about the team and the pedigree and the background. Give a peek into the inner workings of the innovation engine that is ScaleFlux.

JB Baker

>> Yeah. A lot of our core team came out of Fusion, which was a startup 15 years ago where they paved the way for Flash.

>> We started 15 years ago with theCUBE.

JB Baker

>> Yeah. They were the first really to make Flash as a ubiquitous storage media to accelerate the data center infrastructure. So we've got that. There's a lot of people with that mindset. And then our co-founders and CEO, Hao Zhong and his team there, they have really built a very efficient and high quality silicon development team such that we can scale from one chip to another chip to another chip and reduce the R&D it takes us to get to market with a chip. Having spent years at Intel and LSI and Seagate and seeing how big these development teams are, when I was talking with Hao and we were doing some roadmap planning things, and they were saying how much it was going to cost to develop a chip, I was like, "You're crazy. That'll never happen. Yeah, that's impossible." But they've done it. So it's really impressive what they've been able to do in terms of efficiency.

>> It's interesting you bring up that whole way it used to be, and now. Much different world. It's so fast. And that's great for the innovation. It's phenomenal. Well, great to have you on, JB. Good to see you. Thanks for joining the CMO Leaders series.

JB Baker

>> Great. Great to see you too.

>> With NYSE Wired. I'm John Furrier here with theCUBE in Palo Alto. I'm your host, and of course, the CMO Leaders as part of theCUBE and the NYSE Wired community of putting together here across both Silicon Valley and Wall Street. Thanks for watching.