Clips
News
More from NVIDIA GTC 2024

Martin Yip

Head of Product Marketing, EC2 Compute and Networking

AWS

The Nvidia and AWS blueprint for a GPU-powered future

Over time, materials such as coal, steel and crude oil have served as the global economy’s backbone. Today, that mantle falls on the silicon chips that power enterprise-critical servers and graphics processing units.Enter Amazon Web Services Inc. and its Ceiba project. Underpinned by Nvidia GPU tech, it aims to enhance control, security and performance as companies embed AI deeper into their operations.“We have a great partnership with Nvidia, and we are bringing a lot of their new GPUs to the cloud,” said Martin Yip (pictured), head of product marketing, EC2 compute and networking services at AWS

play_circle_outline Discussion on the pace of innovation in the GPU era

play_circle_outline Partnership between AWS and NVIDIA for building a supercomputer in the cloud

play_circle_outline AWS's Nitro technology and its role in Project Ceiba

15:15

16:37

Info

Martin Yip | AWS

Martin Yip

Head of Product Marketing, EC2 Compute and Networking AWS

(light music)

>> Hi everybody, we're back at GTC 2024. We're here in San Jose. San Jose Convention Center downtown. Yeah, it's still rocking at GTC. This is Dave Vellante. John Furrier is here and Martin Yip is in the house. He looks after EC2 and networking, product marketing at AWS. Martin, thanks for coming on "theCube."

>> It's great to be here, thank you for having me.

>> Yeah, we got the little popup cube. We're running and we're gunning and it's like kind of like, you know, unconferences, this is like the uncube. So we're just going to sort of wing it a little bit. But of course we know AWS, we had you guys on, we've been following you for years and years and years. What do you make of this conference? I mean, we're entering a new era, it feels like.

>> We absolutely are. It's an amazing conference. Love to see all the innovation. Loved hearing how Jensen sees the future in his keynote and just loving talking with everyone, networking. This is the first GTC that's been in person since the pandemic, so it's great to meet and see everyone again.

>> There's a lot of talk, I would even say confusion. "Well, NVIDIA has GPUs, but AWS has GPUs too" and it's like, "Yeah, huh." So you guys have been making silicon for a long, long time. It gives you a competitive advantage. You started with Annapurna, even before you bought Annapurna. Then we saw Graviton, we saw Trainium, we see Inferentia. So you guys have been at this for a while. What are you doing specifically with NVIDIA? And then let's talk about what you guys are doing in Silicon.

>> Yeah, we are deepening our partnership with NVIDIA. We have a great partnership with NVIDIA. We were the first to bring GPUs to the cloud, quite a while ago, 13 years I believe. And we are bringing a lot of their new GPUs to the cloud. One of the big announcements is that we are working with them on something called Project Ceiba, which is building a supercomputer for them in the cloud, using their GB Blackwell, Grace Blackwell architecture, bringing that platform to AWS for the use of NVIDIA's R&D.

>> So then Blackwell was, you know, the big star of the Monday keynote. Of course, you know, Jensen was sort of tongue in cheek, but he was saying, "Sorry, sorry, Grace Hopper, you know, we love you, but this guy is big." So it's quite amazing to me to see the pace of innovation here. I mean, it was just like last week I felt like everyone was trying to get their hands on, you know, the previous generation.

>> Yeah.

>> You guys shouted out with some other cloud providers, as you know, in line to get the first orders. So how should we think about the progression of, I mean, we know for years we watched Moore's Law and we saw the innovation there, we're at a much steeper innovation curve in terms of absolutely performance.

>> Absolutely. >> It's insane. Of course, you know, energy consumption is a really big consideration. How are you guys thinking about all the dynamics and the complexity of this new era?

>> Well, it's absolutely great what NVIDIA's doing,. If you saw some of the performance numbers that Jensen shared, he was showing more than 2X gains in performance over the Hopper architecture, as well as lowering the energy efficiency, lowering the energy usage of the GPUs while increasing performance. So that's just amazing. I think it's going to unlock a lot of new use cases. We're going to see an explosion in new applications and customers figuring out how generative AI is going to fit into customer experiences, whether it is building new models, making it easier for customers to do something like create an image or summarize a text or something else that we haven't thought of yet. It's going to be interesting in the next couple years.

>> We wrote a piece several years ago talking about AWS's secret weapon, Nitro, and then it sort of Graviton, and then these other silicon chips that we talked about. I wonder if I could ask you about Nitro.

>> Yeah.

>> I mean it was really groundbreaking. Gave you such an advantage in terms of control, optionality, security, I mean, on and on and on.

>> Yeah.

>> And we've had a number of deep dives and very impressive. And it really is a competitive weapon in that it allows you to both lower your costs, bring in other silicon suppliers, bring in your own silicon, and continuing to drive costs down. Now we enter the AI era, is Nitro ready?

>> Nitro absolutely is ready. In fact, project Ceiba is going to be in part bringing the best of NVIDIA with the best of AWS, including things like Nitro, EFA, Key Management System. So, is absolutely going to be a part of that Project Ceiba, super computer that I talked about. You know, in addition to lowering cost, bringing more choices to customers, it also helps with performance quite a bit. In terms of our ability to offload virtualization functions onto dedicated hardware, ultimately freeing up resources, as well as increasing performance for things like networking and storage. So it's going to be tremendous. Security is one aspect that you didn't talk about. And one of the things that Nitro enables is we have this capability called Nitro Enclaves that allows you to have trusted execution environments. And that's going to be huge in terms of when we think of Gen AI, we are going to think about security as well and making sure that the customer's data is secure. You can obviously have security with encryption at rest and in motion, but then when you're computing on the data, you can't encrypt it. So then therefore you need this trusted execution environment and that's where Nitro is going to play in all this.

>> Yeah, and classic virtualization, you know, too many people can get to the data.

>> Yeah. >> And even AWS can't get to the data in in this situation. So that's definitely a huge breakthrough. So what's different? I mean the cloud obviously changed the way that we thought about IT, took away all that undifferentiated heavy lifting that became a famous phrase. How much of that applies to the AI era, the GPU era, and what's different about the era in terms of the workload, the characteristics of the workload, and how are you guys and your customers adjusting?

>> Yeah, no, the benefits of the cloud, the main reason why people move from on-prem to the cloud is very much the same reason why customers want to access the GPUs in the cloud as well, which is the scalability, the reliability, the security, and the availability, the global availability. You know, the way we build our data centers is very different from other cloud providers. But that leads to the reliability again, that I talk about and the availability. You know, I think what is different is just you could see very quickly what the impact of generative AI is going to be, whether it's through building and training your own model to deploying it, to using applications that have generative AI capabilities built in. It's very clear to see what some of the benefits are very quickly.

>> How would you characterize AWS's compute strategy? I mean, I would say is obviously a lot of optionality.

>> Yeah.

>> There's not a vendor that you really don't work with. You obviously Nitro and Graviton, et cetera,

ARM-based designs. >> Yeah.

>> You're using Intel, using AMD, you've got NVIDIA, et cetera, et cetera. But help people understand your strategy? How would you characterize it?

>> Yeah, it's really about offering the broadest and deepest choice for customers. And it's really up to what the customers need for their application workloads. Some workloads are compute intensive, others are memory or storage intensive, and others require an accelerator, like a GPU. So it's really about giving customers the choice to get the level of performance in at the cost and security that they need.

>> Do you think AI can help us choose the right processor?

>> Absolutely. >> I mean, a kind of tongue-in-cheek question, but I'm actually kind of serious.

>> Absolutely. And one of the things that we have built recently is we have something called Q and we have an instant selection tool that will absolutely help customers pick the right instance for them. We have over 750 instances now, which is a lot. But then based on a customer's workload and the characteristic of the workload, we can absolutely help them pick the right instance that will fit their needs. And then we have Compute Optimizer, which is kind of as your workload's running, we will assess and analyze how you're using the infrastructure and make recommendations to you based on what your workload needs are again, and help you optimize again what the infrastructure you need is.

>> So Amazon's got a very easy to understand, even though it's very rich and capable and detailed AI stack starting with the infrastructure at the bottom where that's where the silicon is. And then I think the middle layer is AI tools.

>> We call that Bedrock >> Bedrock would be in there. And then the top of the stack is the applications which would be Q.

>> Q and as well as Code Whisperer.

>> Right.

>> And many the other AI tools as well.

>> Code Whisperer is the coding assistant.

>> Yes. >> Okay. And so, in thinking about where all that goes and thinking about compute and all the different optionality of silicon you have, I'm curious as to what you're seeing from customers in terms of training and inference. I asked Adam Solinsky, actually John asked him, I was listening in to the recording, Adam about this, about training versus inference and he says, "You know, we got a lot of training to do."

>> Yeah.

>> And that's happening in the cloud, but more and more people are talking about inference. Is it different infrastructure, the same infrastructure, sometimes? It depends?

>> Yeah.

>> How are your customers sort of adjudicating between training and inference and what types of chips or compute they're choosing.

>> Yeah,, I mean the characteristics and the needs of training an inference are quite different. We have instances, our P series are mostly targeted towards training workloads. And then we have our G series as well as Inferentia which is more for inference. So, P series and training for training. G series and Inferentia for inference. And the characteristic is quite different. With training you need, of course, powerful processors and powerful compute, as well as very fast network. You need large clusters of compute, aggregate compute. With inference it's more about near realtime responses. So you need really low latency. You know, it would really not be a good experience if someone asked a question or had a query and it took minutes or even hours to get a response, right? You want it in real time so you need that low latency there.

>> And Bedrock is all about LLM optionality.

>> Yes.

>> And I presume that's it's horses for courses from the EC2 standpoint, right? It's the right tool for the right job. But are you discerning any patterns, now that you know, GA Bedrock. In terms of matching the LLM characteristics with the EC2 instance the choices that you have there? Is there clarity there amongst customers or is there still a lot of experimentation?

>> I think there's still a lot of experimentation. I think Adam always says, you know, "There's no model to rule them all." So really, you have to figure out what you want to do and then try out different models to see which one works best for your needs. So if you're creating an image, it's very different for an image-generation model, versus a translation model, versus something that will write a essay for you.

>> Hm. All right, give us the final word here. You know, you guys are customer obsessed.

>> Yes.

>> So give me the customer angle. In thinking about the customers that you talk to, what do they need in this AI era and how is AWS helping them?

>> Yeah, you know, I think it's still really, really early in this AI era, especially Gen AI. Customers are still trying to figure out what they need, honestly. And they're coming to us to ask the questions. The biggest thing is try things out. Make sure to not just trial the infrastructure, but also talk to the experts to help you deploy it. Gen AI, conceptually it's simple, but under the hood it's quite complex and you need to work with the right experts also to help you. But the big thing is that it's still really early on and customers are still trying to figure these things out and we're here to help.

>> Yeah, the data from our partners from ETR shows that two-thirds of the customers say they want to get ROI inside of 12 months?

>> Yeah.

>> That's very aggressive. I don't know. I said last question, do you think 2024 is going to be the year of AI ROI or do you think is maybe it's going to get pushed into latter part of this year or maybe into next year?

>> I think it's going to be very much like the internet boom that we saw in the early 2000s where there'll be a mad rush of companies trying to figure out what this internet's about. Some of them won't make it, but for those who do, it'll be an amazing thing. And, you know, regardless, the industry's going to change because of it. And we're going to see a lot of great applications, a lot of new use cases, a lot of new companies pop up to support it.

>> So you want to experiment? The cloud's a great place to do that. Martin, thanks so much for coming on "theCube." Appreciate it.

>> Awesome, appreciate it. >> Good to have you. All right, keep it right there, everybody. This is Dave Vellante for John Furrier. We're live, we're not live actually, but we will be on-demand, GTC 2024. You're watching "theCube." (light music)

Martin Yip | AWS

(light music)

>> It's great to be here, thank you for having me.

>> Yeah.

>> Absolutely. >> It's insane. Of course, you know, energy consumption is a really big consideration. How are you guys thinking about all the dynamics and the complexity of this new era?

>> Yeah.

>> I mean it was really groundbreaking. Gave you such an advantage in terms of control, optionality, security, I mean, on and on and on.

>> Yeah.

>> Yeah, and classic virtualization, you know, too many people can get to the data.

>> How would you characterize AWS's compute strategy? I mean, I would say is obviously a lot of optionality.

>> Yeah.

>> There's not a vendor that you really don't work with. You obviously Nitro and Graviton, et cetera,

ARM-based designs. >> Yeah.

>> You're using Intel, using AMD, you've got NVIDIA, et cetera, et cetera. But help people understand your strategy? How would you characterize it?

>> Do you think AI can help us choose the right processor?

>> Absolutely. >> I mean, a kind of tongue-in-cheek question, but I'm actually kind of serious.

>> We call that Bedrock >> Bedrock would be in there. And then the top of the stack is the applications which would be Q.

>> Q and as well as Code Whisperer.

>> Right.

>> And many the other AI tools as well.

>> Code Whisperer is the coding assistant.

>> Yeah.

>> And that's happening in the cloud, but more and more people are talking about inference. Is it different infrastructure, the same infrastructure, sometimes? It depends?

>> Yeah.

>> How are your customers sort of adjudicating between training and inference and what types of chips or compute they're choosing.

>> And Bedrock is all about LLM optionality.

>> Yes.

>> Hm. All right, give us the final word here. You know, you guys are customer obsessed.

>> Yes.

>> So give me the customer angle. In thinking about the customers that you talk to, what do they need in this AI era and how is AWS helping them?

>> Yeah, the data from our partners from ETR shows that two-thirds of the customers say they want to get ROI inside of 12 months?

>> Yeah.

>> So you want to experiment? The cloud's a great place to do that. Martin, thanks so much for coming on "theCube." Appreciate it.