RAISE Summit 2025 | Vipul Ved Prakash, Together AI

Clips
More from RAISE Summit 2025

Vipul Prakash

Co-Founder & CEO

Together

play_circle_outline Unlocking AI Potential: Together AI's Cloud Solutions for GPU Optimization in Large-Scale Training and Inference for Generative Workloads

play_circle_outline Insights into current trends in hardware and software integration for AI applications.

play_circle_outline Growth in demand for specialized cloud services catering to AI-native companies.

play_circle_outline Innovations in architecture, including NVLink and memory coherence for efficient AI processing.

play_circle_outline Transformation of software engineering driven by AI models in code development.

Info
Transcript

Vipul Ved Prakash, Together AI

Vipul Prakash

Co-Founder & CEO Together

In this RAISE Summit interview, Vipul Ved Prakash, co-founder and CEO of Together AI, sits down with theCUBE’s John Furrier to unpack how an “AI acceleration cloud” is redefining supercomputing access. Prakash explains why generative AI demands both raw horsepower and hardware-software co-optimization, sharing that Together AI now fields its first Grace Blackwell 200 cluster delivering 1.4 petaFLOP per rack and is adding tens of thousands of NVIDIA GPUs each quarter. He details how software tuning cut DeepSeek model inference from USD 8 to 55 cents per millio... Read more

explore Keep Exploring

What does Together AI do, when was it founded, and what are its key focus areas and technologies? add

What advancements have occurred in computer hardware for AI over the past few years? add

What are the key differences between AI computing and traditional CPU computing in terms of architecture, workload, and customer base? add

What are some notable innovations in architecture within the industry? add

What significant changes are anticipated in the field of software engineering due to advancements in AI and GPU technology? add

bolt Powered by CUBE AI

Vipul Ved Prakash, Together AI

search

>> Welcome back everyone to theCUBE's live coverage here in Paris. I'm John Furrier, host of theCUBE. Two days here at the RAISE Summit in Paris where all the leaders are coming together to talk about the future of AI and all the infrastructure going on, all the enablement, the value creation, value extraction, this market is hot, the data is going to be unlocked. That's the core story here and the value. We've got a great guest, Vipul Ved Prakash, co-founder and CEO of Together AI, hot company, couple of years old, providing the kind of capabilities that's going to help everyone kind of go that next level. Vipul, thanks for coming on theCUBE. Appreciate it.

Vipul Prakash

>> Thanks for having me. - So first of all, we were chatting

>> before we came on camera, I want you to explain what you guys do because we're in a market where there's always the picks and shovels conversation, but there's a real appetite for horsepower because if everything's open and accessible, the thesis is you need more horsepower. So you're in that business and explain what you guys do, when you started, some of the stats and we'll get into it.

Vipul Prakash

>> Sure. Yeah. So Together AI was founded almost three years ago now. When I return we have our third year anniversary on Thursday. And really we described Together as an AI acceleration cloud. We accelerate workloads from large scale training of models to large scale inference. Now, we really believe that generative AI will be the digital mechanism of the world. This is really early days of this technology and, as you said, it requires incredible amount of horsepower, but it also requires optimization of these workloads, which are really computations happening in the GPU. We excel at that. We do a lot of hardware, software, co- optimization to really extract as much out of GPU computing as possible.

>> Yeah, we're in an age now where you're seeing a lot of CapEx investment, a lot of adoption on the enterprise side, certainly the hyperscalers and what you guys are doing, your provisioning gear. What are some of the key things that are happening now in the hardware? Because the hardware- software integration is impacting robotics, it's impacting enterprise, it's impacting how people are, one, spending their money to provide the application developers who are coming in and saying, hey, I'm going to move from SaaS evolve to agentic, which is going to feed more reasoning, more horsepower is going to be needed, more data. Everything's a data AI show these days. So what's the configurations that are hot right now? What are some of the things going on in your world as you guys tune the machines and then what do I got to tune on the software side because that's what we're seeing?

Vipul Prakash

>> Yeah. I would say this has been one of the sort of fastest areas of growth in silicon in the last few years. Just three years ago when we started, the best GPUs were these GPUs called 3090s produced by NVIDIA really for gaming use cases. And today you have these massive supercomputers. We just brought up our first Grace Blackwell 200 cluster in Memphis, which is incredible. It's got 1.4 petaFLOP of computer on a single rack. So what's happening is that in AI, these computations are really supercomputer computations. So all the machines in a data center, they participate in a single training computation. So you can think of these massive data centers as a single computer, really. And hardware design has progressed to make the interconnectivity between servers, between the chips really, really fast. And as Jensen says, you can transfer the entire data off the internet in a few seconds -

>> You guys are AI accelerated cloud?

Vipul Prakash

>> Yes. - You're a cloud service.

Vipul Prakash

>> And we just focus on generative AI workloads.

>> Got it. Okay. Talk about the open source angle here.

>> We're seeing a lot of activity in open source.

>> Meta has done a great job. You're seeing a lot more developers coming in. Inference and fine-tuning is a hot area.

Vipul Prakash

>> Correct. - Talk about the AI acceleration.

>> You got the hard where you guys building the capabilities. Open source is driving a lot of action. The fine-tuning custom models, this is kind of the progression we're seeing. What's your thoughts on that?

>> Yeah, we are big believers in open source.

Vipul Prakash

>> We provide a model platform that has 200 open source models that are ready to use and you can use them directly as they are or you can fine tune them or do reinforcement learning on them to adapt them to your use cases. And this produces and gives you a lot of control over how these models behave for your particular application and also provides the best sort of price performance possible.

>> So on the inference and fine-tuning, I mean last year was the year of the killer app as inference. We're seeing a lot more about multi-step. We saw DeepSeek came out. We're seeing a lot more software innovation taking advantage of the hardware step-ups in capabilities.

Vipul Prakash

>> Yeah. Now you have reasoning and test time compute and inference, which really allows models to think about what they're saying and sort of self-correct themselves, which has massively reduced the sort of hallucination problem that the native AI models have. And you're seeing larger and larger models being built and they're also being built more efficiently so that they can run, at inference time they're activating only a small number of parameters in the model so that they can do more inference, pass more tokens in real time.

>> So tell about your business. I mean obviously we're seeing the CoreWeave, these clouds that are specialized like what you're doing, having great uptake because the demand for... How do I say this? Price performance. People don't want to just write a blank check. They want to get their arms around the data. They want to put the use case. They want to go the end-to-end workflows. They want to get that intellectual property unlocked. So they're going out and probably coming to you. Who are your customers? What's the use cases? Where's the uptake?

Vipul Prakash

>> Yeah. One thing I would say is that AI computing is fundamentally quite different and the cloud architectures for AI are also very different. It's really different from the data center up. The amount of power density per rack is very, very different from CPU computing. The type of silicon, the type of software layers that sit on top of the silicon and then the type of workload. So it's very different framework and a different product. Which also then motivates different pricing and packaging. Our customers now obviously AI native companies are some of our top customers. I would say about 80% of our business today is AI native companies. Some of these companies are growing 10% VT, their growth rate is immense. They're companies which are doing generative media, these are companies that are doing robotics, these are companies that are designing new forms of language models. And then you have a layer of applications on top of this from customer support to healthcare to back office automation, data processing, all these applications are finding incredible users with generative AI.

>> It's interesting. We're living in an era now where

>> developers never really had supercomputing as a service. It's like, wow.

Vipul Prakash

>> You could just snap into these clouds and get supercomputers.

>> This is phenomenal and this is kind of going to change

>> the entrepreneurial landscape. You're seeing startups come out of the woodwork.

>> Absolutely. - Then you go to the enterprise,

>> they haven't seen this kind of performant IT in generations.

Vipul Prakash

>> Dell's all in on the AI Factory, I saw NVIDIA with them doing a thing. But they're just hardware. You got to have software to run on. We've got to have a stack. This is where we're seeing on the enterprise side a real gap-

Vipul Prakash

>> You need orchestration.

>> What's the gaps? - You need observability.

Vipul Prakash

>> There's also a lot of failure in this hardware, it runs at a very high thermal load. So that sort of the fault tolerance layer is really, really important. Being able to mix and match workloads. This is very expensive hardware at the end of the day and you're trying to really get maximal performance and utilization out of it. When your inference is not running, you want to be training and doing RL. When your inference loads are higher, you want to make those sort of smaller during the day. And all that sort of orchestration becomes really important. And then you have data provenance, security, those threats have to be woven into computing-

>> Vipul, that's a great point, and the hallway conversation

>> here at this conference raised on it and the same thing was at HumanX and these other events, we go to hundreds of events, it's coming back but here more acutely is that operations. It's not just the prototype, how do I get into production?

Vipul Prakash

>> Right. And keep it running. Right.

>> Keep it running because all the things you're mentioning

>> are little in the weeds, details that matter. Governance, compliance, data strategy, harmonization of the data, semantic layer. We had Snowflake on earlier. So these are operational systems that are being retooled-

>> And they're completely new.

Vipul Prakash

>> They're being retooled and they have to be really designed from first principles for AI computing.

>> How is that playing out? I mean now you also got the crypto wave. We cover a out a lot of the NYSE, programmable money, some disruption to the financial institutions. There's a lot of decentralization. So I mean, I might want have... I mean this is distributed computing at the end of the day. So if I'm going to tap the cloud, it's got to be native to what my architecture is going to look like. How do you view that and how are customers thinking about that? Are they still scratching the surface? What's the sentiment? What's the progress?

Vipul Prakash

>> I would say one of the things that we are seeing is definitely this acceptance of a term neo-cloud, which has now become a category, this acceptance of we need a new kind of computing. And this computing is best provided by companies that are specializing in it. The AI data center landscape is very decentralized just because-

>> Power.

Vipul Prakash

>> You can't get that much power in one place. We operate, I think, around 17 data centers today, but that number is going up. We've just announced building two gigawatts of capacity in Europe that will be across multiple data centers and multiple partners. So you are seeing the sort of decentralization of infrastructure that's happening and then you have to kind of bring it all together, have protocols and security policies and compliance policies that can let you operate in this decentralized cloud with confidence.

>> Talk about that European announcement because we're in Paris, so we're in Europe, you guys are expanding. Talk about the news.

Vipul Prakash

>> Yeah, there is a huge scarcity of, one, just infrastructure and compute in Europe. We think the world is going to move towards more and more sovereign systems in AI countries want to have really a say in the composition of models and AI and the culture it represents. AI is going to become more and more part of a workforce. So there is sort of this desire to have more control over, sovereign control over the digital workforce. So we see sovereign AI as a big trend over the next 10 years and we are building really sovereign infrastructure that can be controlled by companies and sovereign nations.

>> Vipul, that's a great point. One of the things that I've been saying on theCUBE is sovereign cloud, sovereign AI, kind of the same thing in my mind these days, is most important story that no one's talking about. We think in the second half of this year and going into next, we predict that sovereignty as a discussion is going move way beyond the current old way, oh, GDPR, blah, blah, blah. It's on country sovereignty, which is not a country, it's the telco, it's the infrastructure. All that has to be run like a distributed computing paradigm, and with all the compliance it's a complex system.

Vipul Prakash

>> It's a complex system.

>> So okay, so how do you architect the change on the tech stack and the infrastructure you guys are doing and marrying it from day one with compliance in mind? Because a lot of people just go, I'm doing some stuff in country. Like, oh, I got all this compliance and all these check boxes. I'm oversimplifying it. But that has to be built in from day one. What's your vision on sovereign cloud, sovereign country, competitive advantage, geopolitical landscape? I mean, it's all melting pot.

Vipul Prakash

>> It is. I think it's fundamental and essential and it has to happen. At Together we really kind of designed the platform from the day one for a very decentralized operation. So our data centers are completely sort of independent of each other. There's a top level routing layer that will route traffic to each of these data centers and this is really becoming... Because there is not a lot of cloud baggage and you're building a cloud from scratch, we've been able to create this cloud system which can live in a sovereign location and operate by sovereign rules. You're right that every country has different rules and different laws of land and you sort of have to follow those and build those into those systems.

>> You guys use the term, I like this, that's right,

>> regulation ready AI infrastructure.

>> Right. - What does that mean?

Vipul Prakash

>> I think data is becoming really the most important asset.

>> Your intelligence is commoditizing

Vipul Prakash

>> and the data that then operates and prompts this intelligence is going to become more and more valuable resource over time and this data has to then follow the regulations that countries are putting into place through their sort of democratic process. It's really becoming amplified in AI computing and it really started as this very open grassroots technology. But now that it's being deployed into very sort critical contexts, we see data regulation as being something that AI really has to adopt.

>> I'm smiling because you guys at Together AI are nerding out big time on the infrastructure, which is really cool in itself. We can talk about that. But what I'm really more interested in is you get the best gear, so everyone wants the next NVIDIA. How many NVIDIA servers do you have? Scope the gear you have in terms of the goodness because and what it's doing to the business and the step function change because we're starting to see... I mean Jensen's very transparent. I'm publishing a five-year roadmap. I want my customers on one side to know the supply chain's solid, know what's coming. He's got an ecosystem development. So NVIDIA's doing a good job. Got to give them props on that. How many NVIDIA GPUs do you have? Give us some of the stats.

Vipul Prakash

>> So what I can share is we are adding tens of thousands of GPUs every quarter right now and we are mostly building Blackwell. So there's a B200s, the Grace Blackwell 2000s and very soon the B300s and Grace Blackwell 300s. We have almost 200 megawatts of capacity this year in the US and plans to build gigawatts of capacity over the coming years.

>> Are your engineers salivating when they know 300s coming and it's like-

Vipul Prakash

>> People are very excited about

>> it. - ... is like a fight for the loading dock?

Vipul Prakash

>> There literally is like Slack fights on who's getting access to it first.

>> It's good people management skills.

>> Do better, your review didn't come out very good, you're low on the list. You're low on, as they say, the totem pole. But take me through just the impact in all seriousness, all kidding aside, there are performance gains. Talk about some of the things that you're seeing. Can you share any data around just observations around how much of a change these things have to how you do business?

Vipul Prakash

>> Yeah, it's quite incredible when you put it on a sort of timescale. I'll give you an example. When DeepSeek released their model, it was maybe like five months ago now things move pretty quickly. It costed about $8 per million token to serve it at cost. And over these five months we have done software optimizations that have brought the sound to 55 cents a million tokens and with the Blackwell architecture it's going to drop again by 3X or 4X. And this really opens up, in some ways you're seeing this cost of AI drop, but this opens up lots of new use cases on top because some applications are just not economically rational unless the cost of AI is at a certain threshold. So as we are making per token cost lower and making AI cheaper, you're seeing this effect of more and more users that's ramping up.

>> Yeah. One of the hot areas that we cover we love and certainly important is AI factories. Jensen talks about it. Michael Dell talks about it. You got gear, hardware on one side. You're talking about software innovation. DeepSeek I think was a great example of what you can squeeze out of a factory or configuration. So I have to ask you, what are some of the innovation things that you see happening in the industry that's notable around architecture? High bandwidth memory, a big part of it. Getting close to the density of the compute. Making sure that data data's moving in and out. And you've got network fabrics.

Vipul Prakash

>> And network fabrics and things like NVLink that NVIDIA has created, which connects now 72 chips together, this was eight chips till the release of Grace Blackwell. There's also ARM CPUs on these devices which have their own memory and there's memory coherence between the CPU and the GPU. So it really opens up... It will take maybe 18 months to really start utilizing these architectures. They're really kind of solving fundamental problems in neural network training and inference that we know about. Now that these architectures exist, we can go and build a software, build the neural network architectures that really sort of can utilize this hardware.

>> I just have to ask you, I'm sure you get this question a

>> lot, but the common refrain we hear on theCUBE all the time is the script has flipped. So I have to ask you what's changed fundamentally that goes against conventional wisdom of data center engineering, software engineering. As those systems become more dense and capable and flexible because now you can look at workload characteristics and kind of define the computer, which is the data center as you pointed out. I love that. We agree with you on that. What's different? No, that's wrong. That was the way we did it before. What's flipping?

Vipul Prakash

>> I would say one thing that's very different is the value that's coming out of the GPU. If you produce a token and that's the value. So in some ways the application logic and the layers and layers and layers of software that we were building in the CPU era, that's all sort of moving into, I describe it as moving into the latent space of the model. And then it really becomes about how well can you teach in AI system what you want it to do and then how efficiently and scalably you can run this thing. I mean, we think that there'll be miles and miles of data centers serving models in years to come. So I think a lot has changed and I expect that for example, software engineering in 12 months it would be a completely transformed field. We are writing maybe 80% of the code at Together using AI models today. And that is changing so rapidly that we think both the quantity of systems that we can now develop, the speed at which we can develop and the commoditization of software is almost complete. I mean, I think of it as software singularity, it's already here and this is going to happen to a lot of other fields.

>> All right, so your vision for the future, you're a relatively small company with about 200 people. What's the vision for your business?

Vipul Prakash

>> As I said, we think generative AI is the new digital mechanism of the world and we want to be a leading infrastructure cloud provider for this era.

>> Awesome. Vipul, thanks for coming on theCUBE. Really appreciate it.

Vipul Prakash

>> Thank you so much. - Congratulations on your success.

>> We'll be tracking you. Love the GPU stories. I'm John Furrier, all the horsepower is getting faster, supercomputing's being democratized. Again, we're getting all the data here in Paris, sharing with you on theCUBE. Thanks for watching.