Cloud AWS re:Invent Coverage | Dave Brown, AWS

Clips
News
More from Cloud AWS re:Invent Coverage

Dave Brown

VP, Elastic Compute Cloud (EC2)

AWS

Dave Brown talks Trainium2: AWS’ secret weapon for generative AI leadership

Prior to Amazon Web Services Inc.’s re:Invent conference this week, Dave Brown, vice president of compute at AWS, shared a glimpse with me into the future of cloud computing. During an exclusive podcast interview, Brown unveiled the company’s newest silicon innovation — Trainium2 — and delved into how AWS is redefining the infrastructure landscape to meet the burgeoning demands of generative AI.For years, AWS has been a driving force behind enterprise cloud computing, but as generative AI reshapes industries, the stakes have never been higher

play_circle_outline Overcoming Systems Challenges: GPU Clusters and Water Cooling in Data Center Infrastructure

play_circle_outline Focus on hardware innovation and custom silicon for performance improvements

play_circle_outline Emphasis on continued innovation across AWS services

play_circle_outline Excitement and opportunities in applying generative AI to new areas

play_circle_outline Role of partners like Nvidia in providing hardware for AWS

play_circle_outline Balancing Supply Chain Challenges with Customer Feedback Loop for Custom Silicon Success

Info
Transcript

Dave Brown, AWS

Dave Brown

VP, Elastic Compute Cloud (EC2) AWS

John Furrier is at AWS headquarters in Seattle covering Reinvent, joined by Dave Brown, who oversees Compute for AWS. They discuss the emphasis on performance and developers focusing on chip-level efficiency. The conversation highlights AWS history, successful innovations like the Graviton chip, and challenges in hardware cycles. Brown talks about advancements in Trainium2 and the democratization of high-performance computing. AWS simplifies access to chips and offers affordable options for GPU clusters. Brown discusses SageMaker's role in managing AI jobs an... Read more

explore Keep Exploring

What are some challenges that enterprises face when trying to build their own potential, particularly in terms of systems complexity and power consumption? add

What role has custom silicon and hardware innovation played in recent advancements in technology? add

What are some of the innovative advancements happening in AWS outside of generative AI, including the announcement of improvements to Kubernetes? add

What new opportunities and advancements have you seen in the field of generative AI? add

What companies are close partners with AWS and how do they collaborate on bringing technology together, particularly with Nvidia hardware? add

What is the feedback loop from the market regarding customer silicon partnerships, specifically with Nvidia, and how does this tie into the software advantage? add

bolt Powered by CUBE AI

Dave Brown, AWS

search

>> Hello, welcome to theCUBE's coverage here in Seattle. I'm John Furrier, host of theCUBE, really digging into the Reinvent coverage. We're here in the Seattle headquarters of AWS. The building's name is Reinvent. We're going to get a preview and dig deep into the news and analysis at Reinvent. Dave Brown, who runs Compute for AWS, great to see you CUBE alumni. We have a prop you're going to bring up here. Good to see you.

Dave Brown

>> Yeah. Great, John, it's great to be here. Thanks for having me.

>> Well, thanks for coming to visit us at Reinvent. Reinvent the building.

Dave Brown

>> This is a home game for me.

>> It's an away game for me. It's a home game for you guys as they say in sports. You're in the hottest area right now. You oversee what is the most in demand right now, which is I need horsepower to run stuff, train stuff, make inference, semiconductor businesses. You're seeing the custom silicon coming out. There's a real emphasis... And by the way, developers are going down to the machine level to get advantages at the chip level. I haven't seen that. It's like reverse. It's like they're supposed to go up the stack. No, they're going down because small ball or small code is very efficient because it's not to maximize resource, it's actually to get productivity performance.

Dave Brown

>> Well, performance is everything at the moment, as we've seen with large language models both in the training and the inference. You're right, developers are getting all the way down to the silicon and starting to think about exactly what they need to do to get the code there and get the performance, but it's a story that sort of follows on from something that we've seen at AWS really since the very start. And one of the things that has driven us as a cloud provider has been how do we give customers more performance for every dollar spent? It's a very simple formula. Internally, we call it price performance, but it really is that. It's really a follow on from Moore's Law. Even though Moore's Law isn't really a thing anymore in the way that it was defined, but customers do expect more performance for every dollar spent. It's been an algorithm that has driven us to not only innovate at the hypervisor level, but all the way down into the silicon level. And that's what we're seeing a lot of today is on there.

>> I remember when you guys, a couple of game changing moments for AWS history was, one... A couple of notables was obviously the CIA thing when IBM sued that opened up the whole CIA, "Wow, cloud is actually bigger." And when you guys started putting out your numbers and it was infrastructure as a service, when you started breaking them out from the Amazon earnings. Everyone was like, "Wow, there's a real business there.' It's mostly infrastructure as a service. It was the bread and butter or the core services. And then the past layer, the higher level services just continues. It grew big time. So we saw the enablement, it was disruptive enablement because it was creating disruption in a good way. It was enabling disruption.

Dave Brown

>> And the interesting thing about sort of, as you say, we put out our numbers and people were like, "Hey, wow. This is actually a growing business and a big business," was if you go back to 2006, 2007, 2008, there was a lot of debate about whether the cloud could really deliver the performance that was needed to actually run a lot of what enterprises at the time called bare metal performance. Could the hypervisor actually deliver performance that you could run all of these enterprise scale applications on? And it was through the innovation... It's not where we started in 2006 with our compute service, but through the innovation that we did and our willingness to go to any level of the stack that really allowed us to continue to win these customer workloads. We're still doing that today. If you look at our latest chips that we put in out there, it's the same journey of understanding what customers need, working out how do we innovate all the way down to the silicon and then putting something out there that gives them better price performance.

>> Yeah. And we've been saying for many years on theCUBE, hardware matters when everyone's like, "Don't talk about speeds and fees, talk about solutions." I'm like, "No, no, no. Hardware matters because performance... The game is one of the best graphics cards. The crypto guys wanted the GPUs to mine Bitcoin. Now you need GPUs to run the GenAI." So clearly, we are in a hardware accelerated model or accelerated computing and you guys have always led the pack in the infrastructure side. And the old guard was always building longer cycles to build stuff. So you guys shortened that down and innovated. So two questions for you, this year, how are you seeing those cycles of release of new chips, one, happening? And then the innovation, the value coming out of it, can you share your thoughts on that, how GenAI has impacted the price performance, which is the speed and fees, but also the cycle times to get it shipped out better hardware?

Dave Brown

>> Yeah. I mean, cycle time in hardware is challenging because any mistake and you're adding a minimum of six months to the schedule. We've seen that recently with some other companies, but it's something that when we started our journey with Nitro all the way back in 2014, 2015, we started to get really good at building silicon and knowing that that silicon is actually going to be correct before we tap it out. We were actually able to use AWS to do that. So given that we had this enormous compute service, we were able to use that compute to actually validate the CPU or the chip that we were building at the time. That became very true of Graviton as well. If you remember, first Graviton was 2018, 2019. Then we very quickly, in our fourth generation, we announced last year at Reinvent, it's now in instance types being used by customers growing really well, but those are four generations in really about six years. And so it's every 18 months or so. And more importantly is our ability to be successful with that silicon, to build the right thing that customers are going to need, to get it right the first time because it's really mistakes that slow things down. Now, one of the things that Matt just announced at Reinvent today is our Trainium2 Accelerator, which we spoke about last year, but it's now generally available to customers to actually go and run their ML workloads on Trainium2. And while it's the same recipe that we've used all along, price performance, how do we improve the hardware to give that level of performance at a lower price, going after generative AI workloads and with Trainium2... I actually have one of the chips with me here today-

>> And it's called T2 in shorthand?

Dave Brown

>> Yeah, we call it Trn2. Trn2 is the name of the instance, and so you'll see instance types called Trn2. T2 we actually have already, that's one of our early instances. Had we known we were going to have Trainium, we may have saved the letter. So that's the chip. It's our most complicated chip that we've built, and a Trn2 instance actually has 16 of those chips built into the instance that are all connected with our Neuralink protocol. So you can treat all 16 of those accelerators as a single memory-

>> You're going too fast here.

Dave Brown

>> Am I going too fast?

>> Yes. What is on this chip? I can see...

Dave Brown

>> Yeah. So that's obviously all the chiplets and the various bits of technology that go in to making sure that we can deliver the processing power that's required to do generative AI, which is a lot of-

>> High bandwidth memory, all the-

Dave Brown

>> There's definitely HBM memory and that that goes into this, so it's got all the latest technology that's built into that. And making sure that we can run all of these

>> Okay. Scope the performance gains here on price performance because that's a concern people... We're up in the era, zero interest rates, everyone knows from the startup bubble. They don't have as much cash, but they want to be profitable.

Dave Brown

>> That's right.

>> People don't want to break the bank, but they want to push the envelope on training, they want to push the on GenAI, but they want to have good visibility and cost. What's the price performance of that chip?

Dave Brown

>> You're absolutely correct. I mean, if generative AI is going to be successful in the long run, we do have to make it more price performance. We have to improve the price performance. Folks need to be able to run inference or workloads at a lower cost. And so if we use Trainium1 as the baseline, this is actually four times just raw performance, faster than Trainium1. So it's a significant improvement of Trainium1. We think this compared to other types of accelerators that we have available in AWS will give you about 30 to 40% better price performance, and that's something that we're actually starting to see with some of our customers, folks like Poolside and Adobe that have been testing these chips. I actually seen about that level of price performance about 40%. And when the numbers are small, 40%, it's like, okay, I'll save a few thousand dollars, but when the numbers get really, really large, 40%-

>> Give a use case example of someone who would take advantage of that

Dave Brown

>> Well, many folks, if you look at somebody training a large language model, they're often using many thousands of these chips and it's working very, very tightly coupled across our Elastic Fabric Adapter network. And so a 40% saving on a cluster of that size is pretty significant.

>> I was putting together a research note here, but we wrote a blog post on SiliconANGLE and the title was a little bit of link bait, which we don't like to do, but we wanted to get people's attention, we said Jamie Dimon's the new competitor to Sam Altman. And the context of it was Jamie Dimon obviously runs JPMorgan. If you look at the data that they have from a petabytes perspective is actually bigger than OpenAI. So our thesis was, "Hey, what is he going to do? One, he's got a property, he's not going to want to put it into OpenAi, but he's also got the cost challenge. What's his strategy? How does he train all that? How does he make that data put to work because that's a lot of intellectual property?"

Dave Brown

>> I think JPMC, they're a great AWS customer as well, there's folks like that in the financial industry and then just more broadly across the industry. I think many of those enterprises are looking to say what is their generative AI strategy? I think Jamie Dimon, I'm sure, will be a very special case and will do something very interesting with the data that he has, but a lot of customers are starting to say, "How do I take the data that I have? Do I need to build my own model?" We've worked with folks like Databricks that have actually gone and built their own model on AWS and Anthropic's doing that. Some of them are saying, "Hey, there's models like Llama out there that are actually really, really good and definitely good enough for what I'm doing. So how do I take something like a Llama model and fine tune that model using something like Trainium and being able to ultimately get to running inference?" And speaking about those models, latency, again, coming back to where you started the conversation, being able to run Llama with the best possible latency and the highest throughput in terms of what they call tokens per second. That's something that we actually think that Trainium2 is going to be the best available on the market for those

>> I'm smiling because it's almost full circle like old school AWS. even say old school, but one of the benefits of AWS was I want to run my software from what I'm building and I got to put it on a server and I would go to get and some storage and I'm off to the races. Llama was very popular for developers, but they have to host it somewhere.

Dave Brown

>> Yes, exactly.

>> You've got to host that.

Dave Brown

>> Exactly. And you want to give them choice, and so I think that's been our strategy with custom silicon. We have our own custom silicon, but we still support Nvidia, right? We have Graviton, but we still support AMD and Intel. And we partner very closely with those partners in making sure that their software and their silicon runs best on AWS. And so the customers want that choice when they bring a Llama model.

>> So I'm doing a little White Research note on GenAI. I want to get your thoughts on a paragraph since that you're here, since you're an expert. It's called Democratizing Super Computing: High Performance Computing for All Category. The recurring theme in the market is democratization of HPC, which essentially was reserved for the Boeing doing the big wing design, the hundreds of thousands of compute hours needed to get in line. It's like time sharing. It's like, "Can I have my turn?" It's tough. Enterprises now see the vision of computing options for on the cloud and distributed edge, certainly with computer vision, multimodal all coming down the pike. The ability to deploy large scale GPU and CPU clusters is now a reality, and this is what people want to know. How do I stand up? Because you're seeing creativity come for these big opportunities that weren't gettable like an entrepreneur seeing an opportunity and say, "Hey, I'll index the biosphere." Who would've thought about that? Yeah, good idea, but not possible years ago. Now, you can do things like that. "Hey, I want to solve a complex problem. I'm going to need to stand up a supercomputer."

Dave Brown

>> Exactly.

>> Take me through that. How do I do that?

Dave Brown

>> Yeah. Well, the first problem is getting access to the chips that you need. And the last couple of years with all the supply chain up and downs that we've seen on a number of different markets, that has been one of the largest challenges for many customers. So when you talk about democratizing access to chips has been how do I get access to those? And we've seen a number of very large players get a large percentage of the chips as well, which means that there's been less chips available. I'm just talking generally in the industry. We've been thinking about that and we've actually... Last year at Reinvent, we announced capacity blocks, which is actually a feature of which allows you to get access to these chips in a way that says I don't need to actually reserve this chip for its full life cycle, like five years or three years. I can get it for a week, I can get it for a day next week Tuesday. And so by making sure that that capacity is available and can be shared across the market, it's actually allowed a lot of startups and a lot of very large enterprises to get access to this technology when and where they need it. And so our capacity blocks offering is doing incredibly well and I think change the way that most enterprises consume GPUs. The second part of democratizing it is how do you get the cost of the place where it's affordable? And really the cloud has been doing this. I'm thinking all the way back to 2012, the New York Times Index, the entire catalog for $200 on AWS, something they just could never have done. But this idea of do I have this compute power to do these larger and larger tasks or things that would just never have been possible, that's where innovation in custom silicon and competition in the market is really getting to a place where you can use something like capacity blocks to get access to a Trainium that lets you do so much more for the dollars that you're spending that would've otherwise been possible-

>> So you mentioned just doing that archiving of that index would've been troublesome years ago. Now, it's instant. This brings up the case of build it, they will come mentality. There's two sides of the build that they will come kind of coin. One is you build it and no one shows up. It's like you just built something and it's like, "Wait, where's the customers?" Now, you guys in the cloud have a different scenario because you have the horsepower. So if you build, there's demand for obviously GPUs. But for enterprises, they're trying to build their own potential. Talk about the systems challenge because there's a buy-build kind of mindset going on now around, "Hey, I'll just stand up a bunch of GPUs and put them on a rack. Oh, I don't have enough power. Or, Hey, I didn't use Ethernet there, I used InfiniBand there." There's a lot of expertise that goes into building these systems.

Dave Brown

>> Yeah.

>> I mean, you're talking about complex systems. These are clusters. These aren't like a server on a motherboard and throw a board in for an IO adapter. It's not a PC

Dave Brown

>> And that is one of the places that I think we've really been able to differentiate what AWS offers. The complexity of these deployments is just increasing at an enormous pace. We are now doing water cooling in our data centers, which is something we've avoided for a long time. But really as you get to the next generation of chips which are using more than a thousand watts per accelerator, which is crazy in itself from a power consumption point of view, you really cannot cool them with air anymore. Now, you need to start getting into water cooling, which in our data centers is a complicated thing to do, but even just keeping the system stable-

>> How about cooling on the chip? There's a trend-

Dave Brown

>> It's water cooling... You mean-

>> I've see some MIT activity where they're putting actually on the chip itself, programmable sensors that say, okay, cool, this processor

Dave Brown

>> Okay, interesting. All the way down to that level? And there's going to be technologies like that. In the innovations, we're doing a number of different things. The problem is the power has gone up so high that now your thermal dynamics become a problem and you've got to find alternative ways to do it. So we're looking at all of that and we're building that, but in terms of complexity, these are the most complicated systems that we have to build. And trying to do that in your own data center is an incredible challenge. The other thing is once you have the whole thing up and running, you will see failure rates. So you will see accelerators that fail, you'll see servers that fail, power supplies, whatever it might be, optical failures. And your workloads are so sensitive to any sort of hardware failure. The workloads, these large training clusters actually struggle with any sort of failure. And so you really want to keep those failures to be as small as possible and then make sure that the whole system can recover quickly if you do see a failure. And that's a place that we've just excelled on AWS and often had customers say, "Hey, you're the most stable place to run." And we work hard to make sure that it's true.

>> Amazon's always had people say, "Oh, it's never going to work," and it always works. " was never supposed to be unsecure. Cloud's not secure." Okay, it's the most secure. Andy Jassy said on his earnings that there's been big growth in the GenAI scene, some great uptick. I'm imagining that it's only going to be better as you start getting in with the better chips and the better infrastructure because I want to get back to the scenario, say JPMorgan Chase or a they have all this data. I want to start turning on these large scale clusters. Similar user experience, go to the console and just say give me a bunch of GPUs. What's the system build look like? If I'm a system architect and I say, "Hey, I want to index all this stuff, I want to do these systems," is it console-based? Do you have to do a little thinking? What's the-

Dave Brown

>> It's actually very straightforward. With capacity blocks and EC2, we're able to give you that capacity very, very easily. And with the launch of Trainium, you're going to be able to go straight to the console and actually get those Trainium servers as part of capacity blocks. Now, the good thing about that is when we give you a capacity block, it's actually already using a high-performance Elastic Fabric Adapter network. So you know that you're getting those... It's four servers. You know they actually network together and they're available tightly integrated. We also have SageMaker. Now, SageMaker is a service that runs on top of EC2, and what it's doing is actually making it easier for you to run, whether it's a training job or an inference job. So it's gone a little higher up the stack so that we'll take care of much more than just a straight EC2 instance, and customers are enjoying that. That makes your models more stable when you're actually training-

>> I noticed that they reposition, not reposition, but made Bedrock the flagship for the models.

Dave Brown

>> We changed our stack. We've always had three layers. We said, "SageMaker is no longer the middle layer, it's now the infrastructure layer," which I think makes more sense.

>> It's a shim layer between EC2... And because it was valuable, it was complicated. Some said it was hard to use, but let's just abstract it away,

Dave Brown

>> If you look at a customer use case, if a customer's just using EC2 and let's say and using Kubernetes on EKS, they may be struggling with underlying issues with nodes, they may be struggling with some of the stuff with the training model. SageMaker just takes care of all of that for you and makes you more productive.

>> Explain SageMaker as an infrastructure element and its role now for people who might get confused because SagMaker, I thought that was my interface to AI, but it really was just infrastructure. Explain what it is in infrastructure and how it relates to-

Dave Brown

>> You think of SageMaker as something that has some knowledge about the job that you're running, and so it's a little bit further up the stack, it knows the type of job you're running, you can describe the job, you can provision the underlying EC2 instances, which is what it's using, and then it's actually managing that job for you. So if there's a node failure or problem with your cluster, SageMaker's actually doing some work to make it better, and so it's just a little bit higher up the stack.

>> So it's like the role of an orchestrator or a scheduler would be kind of like it's that... It's mission control?

Dave Brown

>> Exactly. Because it understands the job, it's able to think about how the infrastructure is set up currently, any failures so that you can complete the job. And then you've got Bedrock in the middle layer. Now, this is where I think you're going to find the vast majority of our customers is just using Bedrock. There's no need with a lot of the models like Claude with Sonnet and Haiku that's available and-

>> So platform engineering is really SageMaker, Bedrock's more for the user/developer/business person?

Dave Brown

>> Yeah. I think you're going to find the vast majority of enterprises and customers are probably going to use a model in Bedrock, be able to use RAG, fine tune the model. Bedrock's got some great features, but if you need to get down into those details, it's all available in EC2 and all available in SageMaker as well. And it speaks back to what you were saying earlier with sort of infrastructure as a service, you get access to all the layers of the stack. So really you know what? No matter what problem you have, there's the speed option in something like Bedrock, but then there's also the, "Hey, I want to build this myself" option with something on EC2.

>> I think I might've coined a new term called business as code earlier in another interview, but I liked that concept because infrastructure as code brought in DevOps and API's connected everything. That's a wonderful thing that creates the cloud. Now, you have GenAI with data connectors that's native to the application, not just API.

Dave Brown

>> I'm assuming the business's code is the top layer of that stack

>> Yeah, it's the write me a SQL query because I don't want to call. I can just take care of it-

Dave Brown

>> Exactly. And that is by far the simplest place to get where you've got deployed.

>> It's interesting, I was saying what the cloud did is change the labor equation that brought in the 10x engineer and all the goodness of the provisioning being automated away with you guys as you created that 10x engineer. And that's continued to power through with the value you're doing. But now you have a 10x business person because now you think about... And people are getting along. It's no more fighting. It's like, "Hey, I love you. You're my teammate, I love developers. Hey, can you have the LLM help me do this business problem? Yeah, no problem." So there's a lot of harmonization going on at all layers of the stack. What's the big innovation in your area that you can say, "Hey, with GenAI, this has made us better, it's forced us to go faster?" What are the key things that changed your roles and your team's work?

Dave Brown

>> So we've done a few things. I think some of them are customer visible. One of the big ones we went after last year, really two things. One of them was just providing customers with more insights into their network topology. And as you know, networking was one of the hardest places. If you've got a misconfiguration somewhere or something creeps in over time, knowing exactly what that is-

>> We're hearing about LLM routing now, model routing. Networking is moving up the stack as a concept.

Dave Brown

>> So we built a service called Access Analyzer using automated reasoning and mathematics. It creates a mathematical model of your VPC and able to reason about the VPC. And so with this incredible reasoning engine, we introduced generative AI to that to say, "Okay, can I ask just any natural language question about my network?" And so now you've got this sort of real-time debug capability that allows you to deeply understand your network and debug any issues. Talking about as code, we did console to code or console as code. So you can take anything in the console and actually generate whether you want the command line tools, you want the actual code for that. You maybe did some sequence in the console. We can now give that to you that you can go put in cloud formation or do whatever. Internally, we are finding uses for generative AI all over the place. And it's a journey I think similar to what you were talking about with Q. we're using that internally and we're making a lot of progress even in our metrics. When you do see an issue somewhere, how does generative AI help you actually debug what that issue is? So, yeah, a lot of-

>> We're seeing a huge tech party going on, I'm telling you, and you guys even have a site called PartyRock internally. I mean, everyone's happy right now. You got the application, the users are happy, you've seen a new user interface, the business productivity, business as code is happening. But if you look at in your area, and this is a consistent thing I'm seeing for the past year, it's almost like a freak out to me because I feel like I'm in 1995 because it feels like the '90s because networking and compute and hardware, the area that you're in, is not only transformative, it's exciting. It reminds me of all that excitement of, "Hey, a new router came out, a new switcher. Oh, my god, a new processor." So the hardware enablement is driving so much change. And at that time, that revolution was productivity like PC.

Dave Brown

>> Absolutely. Well, I think we are, and we've seen this, it's tied in with a lot of the custom silicon we've done, we are at a point now where the hardware really, really matters again. We kind of got a little bored of hardware for a while in the early 2000s when Moore's Law kind of died. Everybody was like, "Okay. Well, we're not going to see that sort of innovation again." I always say that Graviton played a big part in saying, "Hey. No, you can get 40% performance improvement, 30% performance improvement generation over generation." And then obviously what's happened with generative AI and this whole new area of accelerators, it's a super exciting time.

>> Yeah. We've had chats earlier on on theCUBE career when you were on earlier when you were in the product side doing product work, James Hamilton also talked about it. The Nitro was the beginning of a mindset shift of, "Hey, we have all this gear, let's..." I'm obviously over simplifying. You probably tell the story better, but that became really the directional... It's like what with Lambda was. I talked to Werner about this on my interview two days ago. It's like Lambda changed the game because we already had the components. People were doing it, it's just let's just abstract... New things happen. I think Nitro and then Annapurna, which Dave and I say is the best acquisition in the industry, VMware and Annapurna are probably the two biggest deals ever done for the price and value-wise. I mean, you guys got major. So those two points really kind of set the stage for this unleashing.

Dave Brown

>> I always go back much further. I go back to 2007. EC2 is just about the GA, we've been around for about a year and customers are saying, "Hey, virtualization is never going to work for us." And the thing back then was actually jitter. So we called it jitter and we had a meeting every single week on how do we reduce jitter, whether that's on EBS for storage-

>> Jitter on the networking side

Dave Brown

>> Anything. So EBS, networking... And that's the thing that was killing us back in the day was you'd have great performance, but you'd have these excursions of really high latencies. And the journey of solving jitter, we realized we had to go to hardware. And when we realized we had to go to hardware, we found Annapurna Labs and we started to build these NICs, smart NICs, with ARM processors. And then when you had the ARM processors, we got really good at coding for ARM. And when we got good at coding for ARM, we said, "Could this be a system processor?" And that's where Graviton came from, and then ultimately Inferentia. So you can draw this line, it's just innovation doesn't happen in a... It's not a eureka moment always. It's this constant focus and innovation happens over many, many years.

>> Yeah. And Steve Jobs said it brilliantly. You never end up where you started, but you can kind of get the direction. And also GenAI has the same playbook where like Lambda and and some of the things you mentioned is that for customers, they just have to get in, knock down some winds, and then you'll learn and, okay, your aperture increases-

Dave Brown

>> And I always tell customers, with all this new technology, the best thing you can do is get hands on keyboard and start playing. Just find some use case. A lot of folks will typically tend to try and do all the estimation, "How long do we think this is going to take us?" They really have no idea.

>> cost.

Dave Brown

>> Exactly. They're trying to answer all those questions. Give yourself a budget, get something that you can start playing with a reasonable amount of money, something that's scoped and just start playing with it. And I've seen that with my teams by the way. When we started playing with generative AI, that's where the networking feature came from. We didn't set out to build a networking feature. Some team just started to play with generative AI and eventually, we had a networking product in relatively short amount of time. And so I think customers would see the same thing.

>> Yeah. I mean, the slogan build on, as Werner's been saying and Reinvent is a big takeaway for the market to look at. Again, hardware and custom silicon's hot, you guys are doing your job well. Thank you for that. The industry thanks you. But the data layer's got to come too. And so SageMaker and Bedrock will kick in. What's your big takeaway for Reinvent? What should people take away from the performance increases the work you're doing? What would you say is the big takeaway for your work perspective?

Dave Brown

>> I think certainly on the generative AI side, with what we've doing with the launch of Trainium2, and you'll see a number of benchmarks in Matt's presentation of how we've been able to move the needle on both performance and price. It's just amazing. It's incredibly exciting and I think it's opening up generative AI to more and more of our customer base. Obviously, I'm sure you chatted to Swami and there's so much happening at his layer as well. But then outside of generative AI, Peter's keynote, you're going to see a lot about just the sort of innovation that's continuing to happen whether it's the different layers of the compute stack, and we're going to be announcing some incredible things around Kubernetes and what are we doing there to make that just continues to be easier to run and easier to use on AWS.

>> It's like the new Linux.

Dave Brown

>> It is the new Linux. Yeah.

>> It's boring and reliable like it should be.

Dave Brown

>> Yeah, exactly. And so there's so much innovation just happening broadly across AWS. Now, it's always fun to talk about generative AI, but I think you're going to hear a lot from us outside of generative AI as well about everything we've been doing.

>> I always joke, someone else said this, and I can't take credit for it, everything that happened in Star Trek will be invented. I said, "What about the transporter " so it's like this is a old school Star Trek. Engine room I need more power.

Dave Brown

>> Exactly. Well, that's true.

>> You're in the compute business and this is an area of more power in terms of performance, but also less power on efficiency. That's a big deal. How should customers think about the new Trainium2 in terms of, "Okay, I could go?" What's the go plan? What's the go bag so to speak in saying, "Okay, I can use this app for this. It's now gettable?" What workloads, what use cases, does this fit nicely that would give someone confidence?

Dave Brown

>> I think the first thing is they're going to get the benefit from Trainium2 right at the gate just by using the latest Anthropic models, the latest Llama models. The performance on Bedrock is going to be better than anywhere else just because it's using Trainium2 behind the scenes. And so that's the easy lift, right? Get the benefit of Trainium2. AWS has done all the work behind the scenes. You get the best latency and the best performance and best accuracy. A customer looking to actually execute one of these available in capacity blocks, available through SageMaker, and you can start using that.

>> So you're saying Anthropic's got a nice edge with you on Trainium?

Dave Brown

>> We're working very closely with them. And so they've been putting all of their models to actually run really, really well on Trainium and obviously working to optimize that stack together. How do we get that to be just the best place, best performance to access an Anthropic model?

>> So Bedrock, it's a nice lift here too.

Dave Brown

>> Bedrock is just so great to use and so easy to use because it does all the heavy lifting for you.

>> You like Bedrock?

Dave Brown

>> I do. I think it's a really good model. I actually think that that's where most customers are going to probably spend a lot of their time just using models that have been built and put in their data. You get so much value so quickly from that. You can certainly get to the lower layers of the stack. And I love it when you want to use EC2, and I love it when you want to use SageMaker, but I think having those different building layer is super important for-

>> How about the jitter and latency between Bedrock and the chips because that's going to be a concern for folks to make sure that that's covered.

Dave Brown

>> As we've done with every one of our services, I was talking about EC2 all the way back in 2007, we focus on that. And so that's our problem to make sure that we're giving you the very best latency and the very best performance. And if we're all seeing jitter and latency, it's going to be a problem with customer adoption. And so you know we're going to be spending a lot of time getting that right.

>> By the way, you were working on that at the time. You were the lead on that. By the way, I used EC2 in 2007 before they had custom domain support. They had that long string URL and I had to go to RightScale at the time was the only company that would provide any kind of custom domain. That was early days.

Dave Brown

>> Yeah. I remember that. There was Thorsten at RightScale. I was on the other side of that, working with RightScale, making sure we run this cloud.

>> I'll never forget that. I was like, "Oh, my God, this EC2 thing is hot. This is the best thing I've ever seen." It's a server of

Dave Brown

>> There wasn't even an instance like that.

>> I'm flexing my little knowledge of AWS going back to the early days, but that was a game changer moment for the industry. Again, we're just reliving glory here. But what it meant was... I was doing a startup at that time, and I think SiliconANGLE was at the beginning of that time SiliconANGLE, and I was trying to get this stuff done and I'm like, "I didn't have to go buy a box and provision it and dial up and do remote access."

Dave Brown

>> Find the colo, find the data sets.

>> "I got to load Linux on it. Oh, damn." Back then, you'd load Linux, provision it. So GenAI is going through the same thing and you now have Silicon tied to performance layers. So the new GenAI stack is tied to the performance here. That's the key strategy here for customers to look at, right?

Dave Brown

>> Yeah. I mean, we have to build to innovate at all levels of the stack if we're going to give customers the performance at the right price. That's the important thing, right? It's performance at the right price. And I think the biggest thing that we could do for our customers right now is make sure that we can give them generative AI at a lower price. And that's what we're hoping to achieve.

>> What about people who say, "Hey, I'm going to build my own clusters on premise and I'm going to put Ethernet, Fabric and have InfiniBand. I'll build my own"?

Dave Brown

>> Yeah. I think it's going to be interesting. I know a lot of folks are looking to do that for many, many different reasons. Some of the most sovereignty reasons where they believe they need to be on-prem in certain locations. I think it's an incredibly hard lift to go and do that. We've-

>> What would be some of the areas of concerns that if you were going to point that out, if you were a consultant and say hey-

Dave Brown

>> Well, firstly, just the power, getting enough power to go and do that. The amount of power that we use in the space, it's a lot of power in certainly in smaller data centers to be able to run a meaningful number. Obviously, the networking and then also just dealing with failures. It's just the whole how do you get these things to run really, really efficiently? Where our numbers are... We just have teams of people focused on identifying every single little fault that happens and making sure it never happens again. And going back to suppliers to say we think we found this. At scale, at our scale, you see things that you would never ever see in a smaller data center.

>> Yeah. Scale gives you a great advantage.

Dave Brown

>> And so you're able to go fix all these things and get it to a better place for your customers. So just stability is generally that. But we have actually-

>> You say wait till you do your first root cause analysis exercise.

Dave Brown

>> It's really, really hard. And we've spent 18 years at it now and we've gotten better. But it's been a lot of hard work and we still meet on it every single week with my teams, just making sure that we're at the right place in terms of performance.

>> Okay. So I have to ask the question because I was a big fan of this, but it kind of went in a different direction. But Outpost was, I think timing-wise, a little bit off. But if you look at now, now I got compute, the compute power you have now and form factor, edge is huge. And you got computer vision, which is a killer app too on multimodal cameras. So image recognition, this is like a dream scenario for compute, I mean, in the sense of eating more data. I mean, compute loves data.

Dave Brown

>> And that's kind of where I was actually going to go because a lot of those use cases, I said digital sovereignty and customers want to be in their own data center sometimes for latency. And Outpost actually is a product that's allowing us to do that. And so a great example of Outpost is actually the NASDAQ where we are actually running NASDAQ markets today on Outpost in a data center in New Jersey. And we had to be there for latency reasons, but we do have GPUs available, we will have accelerators available in Outpost if folks need those in edge locations.

>> I can see some scenarios where you have such scale that it's almost a barrier to entry for a competitor or alternative. I'll give you an example. What came out of the pandemic was Amazon Connect. Why? Because people weren't working anymore. That became a great solution, it saved everyone's butts basically because it was a lifesaver for many, Connect. And now that's a full service and the scale that GenAI is kicking in to Connect is going to look at... That's something that's hard to do at that scale. So I can imagine the compute farm, if you will, compute capabilities of Amazon could be like Connect. Why not be providing the edge?

Dave Brown

>> Yeah. Well, Connect is a great example actually of the benefits that using the cloud really brings because prior to Connect, and we still have customers today, but they have some sort of PBS on site and they've got a team of people that manage it, it's hard to run and you don't see the innovation, you don't see the features that you really get. So with Connect, customers say, "Hey, let's move all of this to the cloud, let's just run on Connect. And then you get all this innovation and it's more highly available and easier. And so we've had some really great success stories with that.

>> I mean, Amazon has all this learnings. Even on the retail, you can infrastructure and say, "Here's your retail box." And it's got all the talks to the call zone to the cloud and runs a high performance edge supercomputer. You can do the takeout stuff and then visual. This is really kind of gettable.

Dave Brown

>> Lots of room for innovation. You didn't even mention Kuiper, satellites in space now.

>> The world's round. That's like space. It's like outer edge.

Dave Brown

>> Exactly.

>> It's the outer edge.

Dave Brown

>> Exactly.

>> What's exciting these days, you got a great job, it's a tough job. You have a tough department, you have a lot of pressure, but what's exciting you in this job? It's super valuable.

Dave Brown

>> I think the opportunity is enormous, right? So we've gotten to... What we do and how we do it hasn't really changed. We're just getting to work with some new silicon, generative AI has given us a new area to go and apply really the same things we've been doing all along to this new area. And I think it's the customer innovation, seeing customers starting to kick the tires on generative AI a few years ago, starting to see some of the great use cases that are out there now and customers are really starting to use. And then obviously knowing about what Trianium2 can do and the benefits that I think customers are going to get, whether they're using it in Bedrock, SageMaker, EC2. And then Trainium3. I don't know if the name was a secret, but the next one's Trainiu3. We're going to be talking about that as well number, which is going to give us just even more performance. And so there's just no end in sight for the level of innovation that we're bringing. And internally, we just continue to run as fast as we can.

>> My last question for you, I know you got to go, appreciate your busy time here in theCUBE, what about the role of partners in your world? I mean, ecosystem partners. You got Nvidia's out there, you got your own providers, and you got Broadcom out there, you've got Ethernet and Ethernet, you've got the I know you use Infiniband, but Ethernet's the bread and butter. It's where the actual-

Dave Brown

>> Yeah. We don't use InfiniBand.

>> You don't use InfiniBand at all?

Dave Brown

>> No.

>> But Ethernet?

Dave Brown

>> So we use Ethernet and then we actually have a protocol called Elastic Fabric Adapter, EFA, that we developed over many years now. We use that for all of our

>> So there's other suppliers out there and supply chain's not infinite. You mentioned that earlier. There's constrained supply. You guys build your own. What's the role of you guys versus, say, the Nvidias, the Broadcoms and everyone else out there that are making silicon, in-custom silicon?

Dave Brown

>> Firstly, folks like Nvidia are a very, very close partner of ours. The way I think about Nvidia is I want to say how do I make Nvidia hardware? How do I make AWS the best place to run Nvidia hardware? And so we partner very closely with them in bringing our technology close to their technology. I don't know if you saw Project Saber, we announced it last year at Reinvent, and that's getting very close to deployment right now with the new Blackwell GPUs, that's a 20,000 GPU cluster on AWS. So we've been very excited about that. But from a supply chain point of view, the last couple of years have shown us just how many challenges supply chain can have. Even now, I always get asked questions about, "Hey, what's the next challenge for the supply chain, whether it's some sort of geographical issue or it might be just a shortage in some area or a whatever it might be?"

>> Horde those chips and build your own to get differentiation, right?

Dave Brown

>> Yeah. The thing with supply chain is you just have to be ready, really good at it. And again, it's another scenario with scale helps. And so the scale of AWS means that we're... Man, we've been running supply chains since we launched Amazon back in the day, right? It's a supply chain business. And so we've benefited a lot from the Amazon supply chain. And then obviously as AWS, we've had to build our own supply chain. And we've learned a lot over the last five years and all the challenges we've seen.

>> A big theme here at Reinvent this year is going to be customer loops, feedback loops. It's been part of the DNA of AWS. What's the feedback loop coming in from the market on the custom silicon, your silicon partnering with Nvidias of the world? What are they telling you? Tie the software advantage here. What's the big feedback loop?

Dave Brown

>> I mean, the first thing is everything we do, we do not want to build products you mentioned earlier, build a product that nobody wants. We do not want to do that.

>> Build and they'll come.

Dave Brown

>> We do not want to do that.

>> hoarding the GPUs saying, "Hey, I'm going to hold you hostage. Come to me."

Dave Brown

>> Maybe. We want to build products, we want to build custom silicon, we want to have software that really solves the customer problem. And then our belief is if we're solving a customer problem, they're going to come. And so there's a lot of excitement around Trainium. We have a lot of excitement to run Nvidia on AWS and what's happening there. We want to make sure we can support all of those workloads. Whether a customer chooses Nvidia or they choose Trainium on AWS, we want to be the best place to support those workloads.

>> Awesome. Well, Dave, thank you for coming on theCUBE. Appreciate it.

Dave Brown

>> Yeah, great. Thank you for your time.

>> Good to see you. Congratulations and love this new chip.

Dave Brown

>> Yeah, thank you.

>> And I get my workloads on there, and I want to get all of our CUBE data. We only need Trianium1 probably for congratulations. Thanks for coming on.

Dave Brown

>> Great. Thanks for your time.

>> Okay. This CUBE coverage here, we're in Seattle at the headquarters of AWS called Reinvent's the name of the building. I'm John Furrier, your host of theCUBE. Thanks for watching.