Name: Accelerate your data lake to lakehouse journey with Amazon S3 and Amazon SageMaker
Uploaded: 2025-03-14T15:00:00.000Z
Duration: 21 min 49 s

Clips
More from Digging Into AWS Pi Day

Andy Warfield

VP & Distinguished Engineer

AWS

play_circle_outline AWS Pi Day event: data lake to lakehouse journey with Amazon S3 and SageMaker

play_circle_outline Dominance of Parquet format in data lakes, advantages, challenges, and need for richer functionality

play_circle_outline Revolutionizing S3 with Iceberg Tables: Enhanced Schema Evolution and Table Abstraction for Efficient Data Management

play_circle_outline Integration of Iceberg with S3, support for Iceberg catalog API, and client libraries like Duck DB

play_circle_outline Collaboration between storage and analytics teams, de-stratification of the stack, focus on performance

16:47

Info
Transcript

Accelerate your data lake to lakehouse journey with Amazon S3 and Amazon SageMaker

Andy Warfield

VP & Distinguished Engineer AWS

Join Andy Warfield, vice president and distinguished engineer of AWS Storage, as TheCUBE explores the transformative insights from AWS Pi Day. This engaging session hosted by Rob Stretchay, managing director of TheCUBE Research, delves into advancements in Amazon S3, the shift towards structured data management, and the evolving landscape of data lakes and lakehouses.

In this video, Warfield shares expertise on AWS's history with S3, highlighting the growth of data storage and the rise of open table formats such as Apache Iceberg. The discussion addr... Read more

explore Keep Exploring

What is the special Cube event discussing, and who is joining for the discussion? add

What are the advantages of the widespread adoption of Parquet for storing tabular data in S3? What are some of the challenges that have led to announcements regarding dealing with Parquet files at petabyte and exabyte scales? add

What are some key features and benefits of using Iceberg as a table abstraction layer on top of Parquet? add

What aspect of Iceberg's development has been remarkable in terms of its speed as an open-source project and standard for working with tables? add

What are some observations on the current collaboration and advancements in data management within the teams at our company? add

bolt Powered by CUBE AI

Accelerate your data lake to lakehouse journey with Amazon S3 and Amazon SageMaker

Rob Strechay

>> Hello, and welcome to this special Cube event where we're digging into AWS Pi Day, and announcements around the data lake to lakehouse journey with Amazon S3 and Amazon SageMaker. I'm Rob Stretchay, Managing Director with the Cube Research. And in this episode, we're going to explore the accelerating enterprise adoption of data lakes and lakehouses, the rise of open table formats, such as Apache Iceberg, which is getting all of the press right now and leading the way, and how organizations are finding smarter, more efficient ways to manage their exploding data assets. So let's examine and uncover what's next in the data-driven enterprise. To help me do this, I'm joined by Andy Warfield, who's the VP and distinguished engineer with AWS Storage over at Amazon and the Amazon storage team. Welcome on board.

Andy Warfield

>> Thanks, Rob. Thanks for having me on.

Rob Strechay

>> Yeah, it's great to talk to you. We haven't talked in quite some time. And S3 has really had a long storied 19 year history. We've seen an incredible growth, where it's storing over 400 trillion objects, which just boggles the mind if you think about it. Can you help us understand, and share how customer data volumes need to evolve, and what trends you're seeing regarding organizations with petabyte data and exabyte scale storage requirements, which is just also mind-boggling?

Andy Warfield

>> Yeah, for sure, Rob. I mean, I think a thing that's remarkable throughout all my time working on S3, is how S3 over 19 years has really followed where customers have taken us on stuff. And we've always built forward to the directions that we get pulled, and customers are always pulling us into surprising and interesting areas. And so, with the announcements that you saw at re:Invent at the end of last year, and that you're seeing today with Pi Day, you are really seeing this kind of fascinating shift, I think, in S3, toward supporting not just traditional objects, but also through the launch of S3 Tables and S3 Metadata, to supporting structured data on top of S3, deeply integrating with analytics tooling, and building a set of functionality around understanding the data that's stored in S3.

Rob Strechay

>> Yeah, no, I think so. And I think, again, breaking it down even further, because we get a lot of data nerds who get on, and I kind of fall into that category myself. Parquet has really been the dominant format for storing tabular data in S3. I mean, again, it's really been where people have been at. It's open with the Apache Foundation supporting it. What are the advantages that you've seen in the widespread adoption? And what are some of the challenges that have really led to some of these announcements that you're making, with dealing with Parquet files that are at that kind of scale, that petabyte and exabyte scale?

Andy Warfield

>> Sure. Well, I mean, Parquet has been around not quite as long as S3, but not far off. I think it's about 13 or 14 years old.

Rob Strechay

>> Yes.

Andy Warfield

>> Right? And it's kind of been the bread and butter of how data lakes are built, and how people have stored data, originally with Apache Hadoop and then Spark. And so, you know this, the Parquet file format allows you to build a table as an object. And it has all sorts of internal optimizations inside it that allow queries to run a lot more efficiently, on top of tabular data that's stored in that format. And so what we saw, over the 13 or 14 years of Parquet being stored, was tools like Apache Hadoop built connectors to be able to store Parquet in S3. The cloud really stepped in as an awesome base for building data lakes. We saw the width of throughput that S3 could offer being a real accelerator for building huge analytics jobs, where you have literally thousands, tens of thousands of compute nodes in some cases, accessing data and Parquet in S3. And people building workflows where, for log data, for all sorts of business data, you have producers just dropping those Parquet files into S3. And so, there was this point where what we were seeing with Parquet was a really attractive end-to-end sort of analytics space in storage. But, people were starting to get a little bit, not frustrated, but maybe desiring of a richer set of storage functionality for that data.

Rob Strechay

>> Yeah, no, I think so. I think, again, being not only a data nerd, but a storage nerd way back, I look at this and say, "This to me was one of the evolutions." And I think one of the big evolutions, and kind of, really, I guess you could say accelerated a lot of people competing, not at the storage layer, but more at the compute layer around this, is really Iceberg, and Iceberg Tables, and table format. How does that really play into this now, and what you guys are doing, and what you've seen, and how that's transformed S3 as well?

Andy Warfield

>> Sure. Well, I mean, the thing that you call out about the diversity of engines is totally right. And it's kind of been the premise of Parquet, initially, and now Iceberg, on top of S3. The thing that has really, I think, motivated a lot of the idea behind data lakes, and now the lakehouse idea, is I'd like to be able to store my data in a format that allows me to continue to choose what tools I bring to work with it, and doesn't limit my team's abilities to make their own calls on how they work with the data. And so, with Parquet, what we were seeing was enormous data sects building and growing, but it wasn't really a first-class table primitive. And what we saw in the original development of Iceberg, as an open table format, coming out of Netflix originally, through Apache, was a desire, on top of S3, to go further with `what a table was in the storage layer. And so Iceberg takes Parquet and it builds a really nice metadata layer on top of it that gives you a first-class table abstraction. It allows you to do schema evolution, adding, removing columns. Allows you to do mutations on the table by inserting and deleting things. And it gives you this sort of snapshot in time travel ability. And so we, over about three or four years, saw a lot of quick growth on Iceberg in particular, to the point that a lot of the customers that were really using Iceberg at scale were coming to us and going, "Can you make this easier?" Right? Because, ultimately, Iceberg was a file format on top of S3. There's a whole bunch of work that, just working against that file format, customers were having to do to manage it on their own, in terms of reclaiming things that were deleted, compacting the lots of small Parquet objects that Iceberg creates to get performance back, and so on. And so that was really the thinking behind creating this first-class S3 primitive, and S3 Tables, and really launching a managed Iceberg offering.

Rob Strechay

>> Yeah, I think that, to me, was a key announcement, back at re:Invent in 2024. It feels like yesterday, even though it was a few months back. How have you seen that really, fundamentally with the customers, shifted how they approach, really, S3, and how they're building their data lakes out? Is that really, again, helping them get more out of their analytic data and their assets, their data assets?

Andy Warfield

>> Absolutely. I mean, I think we were really pleased with the warm welcome that the S3 Tables announcement got. We had customers being really, really excited about it. And we've just had, the adoption has been really remarkable to watch, and we've had a lot of strongly opinionated votes for new features and improvements on it, which is awesome. Because the team, they worked crazy to get S3 Tables out the door, and they really have not taken their foot off gas, in terms of addressing the things that the first round of customers have really flagged. And so, in the period of time since, like you said, it's just yesterday, the re:Invent launch last December, here we are in March-

Rob Strechay

>> But that's what you guys are known for, is innovation. So I mean, what-

Andy Warfield

>> The thing on that, as a total aside, is we are absolutely known for innovation. We really drive to move fast. But on the storage services, we're also super careful. And durability and security are the first order of business on everything, and there's lots of reasons to go slower. And it's been really, really cool, at S3 scale, to see launch like this continue to just have momentum. So we've done stuff, like we've launched improvements on all of the APIs to make table creation simpler, to allow schemas to be specified up front, really making sure that we're filling out any gaps between what was there and what was in the Iceberg catalog API spec. We've just actually doubled down on our support for the Iceberg Catalog spec. This is a little bit of a nerdy piece, Rob, but I think you'll get a kick out of it.

Rob Strechay

>> I mean, trust me, the people who are watching understand catalogs and understand why they're so important. And I know we'll talk about that a little bit later, but this is, help people understand, because get into the nerdy stuff here. This is great.

Andy Warfield

>> Okay, for sure. Well, I mean, one aspect that's been remarkable, in terms of an illustration of how fast Iceberg is moving as an open source sort of project, and as a standard for working with tables, is that when we first started talking to some of our largest Iceberg customers, so folks like Netflix and Pinterest, early on in fleshing out what S3 Tables was going to look like as a product, all of the connections into Iceberg were through a set of libraries, like code for each different storage service that it talked to. And so there was a connector for Glue, there was a connector for Hadoop based storage, and so on. And you had to go in, make sure you had the most recent version of Iceberg to use those connectors. And features meant redeploying and stuff like that. And so the Iceberg folks had a proposal to build a REST API on top of it, this Iceberg REST catalog API. It's not exactly a technical catalog in the traditional sense, it's more like a data path thing. It's much lower down the data path there. But at the time that we started working on it, it was such an early bit of a proposal inside the Iceberg community that the customers we were working with discouraged us from implementing that, because they weren't sure where it was going to land, and they weren't sure if it was even going to be ratified all the way through. And so, we charged at building an S3 connector. And by the time that we launched, which was, I don't know, maybe like, certainly less than a year after those initial guiding conversations, the IRC protocol was established enough that we'd wished that we'd implemented it for launch. And so, that was one of the biggest things the team's done, is they've gone through, and soup to nuts, we've built in IRC support, which means that all of the tools that are emerging for Iceberg, the client libraries, like Pi Iceberg, and Polars, and Duck DB, which we can talk about more, are all able to just work with S3 Tables out of the gates.

Rob Strechay

>> Yeah, I think that, to me, is that has to be one of the most exciting things for the customers, is that, really, the major benefits of S3 Tables delivering to customers is helping them evolve how they look at their data lakes. Is that really what you're seeing out of the customers? Because you guys work backwards, and like you said.

Andy Warfield

>> Absolutely. And when we work backwards on this thing, one thing that I think is super important to call out, in terms of how the S3 team thinks about it, and then how all of like AWS is really working with Iceberg and S3 Tables, is from a storage perspective, Iceberg kind of evolved for analytics. It emerged for these integrations with tools like Spark. And so it's heavily shaped by database engines, which is an important critical aspect of why these tables exist. However, when you build it as a storage primitive, when we build it alongside objects in S3, customers actually want a primitive that's as easy and universal to use as objects themselves. So they definitely want to do analytics, but they also want this table to be a data type that they can program to, that they can build a storage for their apps, that they can fit logs into, and that there's a really end-end story, from application and data production, all the way through to doing rich analytics. And so, as an example, we've been working with Duck DB, right, which is a client-based engines. Really, really popular as a way of connecting all sorts of database and data handling applications and data. And Duck didn't have support that was all the way there for Iceberg, but we know that customers love using Duck as a connector for Tables. And so we sponsored the Duck open source team and we've been developing really closely with them. And today, Duck is launching Read Path support as part of a set of incremental steps towards really supporting Iceberg, as a super easy to use backend for storage under Duck's engine.

Rob Strechay

>> That's awesome. And I think also, what we've been hearing is that there's some integration that really happened with SageMaker, and SageMaker Lakehouse. And that combination seems to be really looking to aim and solve some of the data management challenges that those data scientists, and analytics, and data teams have been facing with getting at the right data, and understanding where the data lives, and how to get at it. What are some of those integrations like, and why S3 Tables and SageMaker Lakehouse make a lot of sense together?

Andy Warfield

>> Well, Rob, this is kind of one of the most exciting bits of, I think, what we're doing right now. And we're still early and we're still learning a ton in this direction, but I think a thing that I've seen as a really big change in how all of our data and analytics teams have been approaching things over the past year, is just like I mentioned for S3 Tables, we're really trying to create this very, very simple first class experience with working with Tables. When we stepped back and looked at the analytics stack in AWS, what we found was we had all of these individually great products, Athena, Redshift, Blue Data Catalog, and Lake Formation. All these things were individually very, very effective, but the user experience was kind of like going store to store to these different things, product to product, and working across them. And what we really heard from customers was they wanted a much more developer-friendly, analytics-centered experience on working for all these things. And so, rather than continuing to press on these individual things as ends to themselves, today we're GA-ing, launching SageMaker Unified Studio. And what Studio does is it ties together all of these things into a single developer experience. And that experience, it's a very consumer developer experience. It's not inside the AWS console. It has its own console that you can log into directly with user credentials. It shifts to have this really, really cool primitive, which is like the studio project primitive, where you can create a project and you bring data assets into the project. You can do analytics development in SQL, with Q, doing natural language to SQL Query generation. You can use the entire set of SageMaker development tools, in terms of integrating with a lot of the AI facilities that have been built in there, like Guardrails and so on. And just in one UI, you have this ability to work across all of these individual products, and also work across other developers in your org.

Rob Strechay

>> Yeah, it would seem like this brings together a lot of the different assets into SageMaker Lakehouse, with S3 Data Lakes, the new S3 Tables, Redshift, Managed Storage, or other third party stuff like you brought up with Duck and things like that. It would seem like this is really a key to helping simplify the experience, like you said, with Unified, of driving that forward and making it so that people can find their data and bring it together.

Andy Warfield

>> Absolutely. And so what you see is Tables being able to automatically register in the Lakehouse, and show up in there. And you just see a really deep focus on developer workflows around managing how data is shared, managing how it's governed, but ultimately getting as fast as possible between the path of data being generated, and being able to ask questions, and be curious, and build applications on top of it.

Rob Strechay

>> It would seem that this is also bringing storage and analytics closer together. What's your thought on how people have been managing it, and how this simplifies that going forward?

Andy Warfield

>> I think you're totally right on that, both in terms of how our teams talk to each other, but also with S3 Tables, how our teams talk to partners around federation and integration of data. I think you're just seeing this really cool, I don't know what to call it, like a de-stratification of the stack. And then, I'm seeing analytics folks deeply looking at how Iceberg is built, and object-level performance characteristics of it, to make sure queries run super fast. That's both internally, and with all of the partners that build engines on top of S3. And similarly, the storage teams, I think, have learned more about engines, and about these structured data workflows in the past 12 months than ever before. It's just really, really interesting to see everybody working together, in terms of integrating all these things.

Rob Strechay

>> I love that. So closing thought, what's your vision for how S3 Tables, SageMaker Lakehouse come together and really support organizations' journey as they're really focusing on being data-driven, particularly balancing performance with cost, efficiency, and you brought up the whole governance thing as well, which is very critical to organizations?

Andy Warfield

>> Is a great question, but I'm not going to take the bait on it, in that I kind of feel like on this one, we got where we are on this integration, and on both the Tables launch and the Studio launch, by really just listening to where our customers were pulling us. And so I don't know that there's a big grandiose vision that we have at this point. I think it's very much around just continuing to listen. And I think we feel like we're early, but we've made a fundamental change on this space. And I think it's going to keep moving, like you said before, really, really fast.

Rob Strechay

>> Well, Andy, I love it, because again, and I said this many times, I don't think AWS and Amazon get enough credit for working in the open source community, and working with others, and partners. And you guys are definitely doing that in S3, and working across the different services to bring a solution to organizations. I really appreciate that. I mean, especially with SageMaker Unified, and SageMaker Lakehouse, and S3 Tables. I am loving this stuff. I was one of the people who was a big fan of it when it was announced back in almost the beginning of winter now. But it's really great to see it coming out, and I really appreciate you coming on board today.

Andy Warfield

>> Well, thanks for taking the time to chat. It's always awesome to talk.

Rob Strechay

>> Thank you all for joining us again. We're going to be digging in, getting another piece of pie on Pi Day here, talking more about SageMaker Unified Studio, S3 Tables, SageMaker Lakehouse, and all how this is going to support your journey to analytics and better lakehouses and data lakes. Stay tuned and we'll be back.