Name: Zhitao Li, Uber
Uploaded: 2024-10-01T21:30:00.000Z
Duration: 20 min 23 s

Clips
News
More from Ray Summit 2024

Zhitao Li

Director of Engineering, AI & Model Infrastructure

Uber

Three insights you might have missed from Ray Summit

The rapid rise of generative artificial intelligence is driving significant changes in the enterprise landscape, and companies such as Anyscale Inc. are at the forefront of this transformation. With the complexity of AI infrastructure increasing, businesses are turning to innovative solutions to help them scale AI efficiently

Uber’s journey to hybrid cloud AI with Anyscale Ray and Kubernetes

Artificial intelligence is already part of catching a ride, as Uber Technologies Inc. continues to upgrade its platform with scalable GPU clusters.Building upon its history of containerized workloads, Uber has adopted Anyscale Ray to run clusters with greater speed and scale, according to Zhitao Li (pictured), director of engineering, AI and model infrastructure at Uber. Uber’s Zhitao Li talks about upgrading Uber’s AI platform

play_circle_outline Challenges of building an AI platform and the benefits of using Ray

play_circle_outline Benefits of using Ray for GPU training and distributed Python computation

play_circle_outline Importance of multi-agent systems for solving complex gen AI problems

play_circle_outline Managing relevance and recency of data for efficient machine learning operations

play_circle_outline Future engineering priorities for Uber including hybrid cloud setup and open sourcing innovations

Info
Transcript

Zhitao Li, Uber

Zhitao Li

Director of Engineering, AI & Model Infrastructure Uber

Anyscale's Ray Summit in San Francisco brings together top engineers to discuss machine learning, generative AI, and the platforms driving these advancements. Uber's director of engineering, Zhitao, talks about their use of Ray as the core AI engine, allowing them to scale their platforms. With Ray, Uber can run over 20,000 model training jobs monthly, saving on costs and enabling scalability. They have evolved from predictive ML to generative ML, optimizing resource sharing in Kubernetes clusters. Uber plans to enter a hybrid cloud setup, focus on efficiency... Read more

explore Keep Exploring

What technology has allowed Uber to efficiently run machine learning model training on GPU native clusters in a maintainable and scalable way? add

What are some of the things that enable you to do better with regards to model training jobs on your clusters and data centers? add

What are some challenges companies like Uber may face when developing and launching products using gen AI technology? add

What are some examples of the data aspects that need to be included in the productionization of an ML platform beyond just the model itself? add

What are some of the key areas that Uber is focusing on in terms of their technology and data management strategies? add

bolt Powered by CUBE AI

Zhitao Li, Uber

>> Hello, welcome back to theCUBE's live coverage here in San Francisco for Anyscale's Ray Summit. I'm John Furrier, host of theCUBE with Savannah Peterson. We're breaking down all the action here, where all the top engineers are here, really setting the agenda for the future of machine learning, generative AI, the platforms that are going to power the next generation. We got Zhitao here, he is the director of engineering of AI and model infrastructure at Uber. We know they are doing some really good stuff. They're one of the organizations at the forefront of AI. They have powerful AI and ML platforms. They use Ray, their core product at Anyscale as the core AI engine. Zhitao, thanks for coming on theCube. Thanks.

>> Thank you.

>> Okay. So you guys, first of all, are featured all the time on SiliconANGLE and theCUBE, Uber's infrastructure team, their engineering team, phenomenal work. The app, people, places, and things. Really is state of the art in my opinion, in our opinion on the research team, of what the platform of the future is going to be. So you guys are a good proxy for all the cool engineering work. So, building an AI platform is hard. You're here at Ray Summit, I'll see that Ray scale a product for Anyscale. They're scaling lot more practitioners here, open source, they had users, the growth of the product, just overall good stuff for the company. What's it like to build an AI platform? What are some of the engineering things you're looking at? What's the core problem you're solving?

>> Thank you. So, we have been doing AI platform for about eight years. We have this amazing platform called Michelangelo, which, the lineage actually started in 2016. We were one of the earliest teams which productionized the concept of feature store and then bring feature store to production. And later, as the rest of the world started to revolutionize machine learning, from linear regression decision tree into deep learning, we essentially recognize the challenges of our older stack. It works well by running on top of Spark ML and then wiring everything together using Spark Transformers. But deep learning really calls easy access to GPUs, which brings in unprecedented competition power to the world. The workflow. The workflow also calls for very easy access of distributed Python computation. All the deep learning custom code are written in Python. We really hit the nail on the spot of offering the great flexibility and easy access to Python code and great access to GPUs across the life cycle of the workflow. That is why we adopted Ray a few years ago and the journey just go amazingly from there.

>> All right, so explain to me what your life was like before Ray and then after Ray. And if you weren't using Ray today, what were some of the things that you had to deal with? Take us through that progression.

>> Thank you. Yeah. Before Ray, we were operating distributed Spark clusters that tries to run machine learning model training on the inside, but it's very difficult to retrofitted those things to run on the GPU native cluster, managing the life cycle of the workload. And it's very difficult to fully containerize everything. Uber love the containers. And the fun fact, I was one of the first engineers building containers for Uber back to 2014. It's almost a decade of history of the containers. So these containers provide the best isolation of the workload and allowed us to utilize the GPUs and run heterogeneous computation cluster in the most maintainable and scalable way. So with Ray we are able to run many clusters. I believe we are doing more than 20,000 model training jobs every month on our clusters and in our data centers, and this would not be possible without Ray.

>> What are some of the things that... Does that enable you to do better? I mean, obviously, great value. What does that enable you guys to do now?

>> It allows us to actually evolve the platform before generative AI into the generative AI. We actually published a blog post a few months ago called Machine Learning Platform From Predictive ML To Generative ML. We are so lucky that we end up using one platform as I keep extending it and solving skills that we can't even imagine in the past. These days, our model is often even bigger than the data set we used to solve. But we are able to keep extending the platform and the incremental innovations, and we keep supporting the users to evolve and allow them to expand their workload by all their magnitude directly. It doesn't mean we're talking about 50%, 80% increase. They come to us and say, "I want to 10X my model. I want to do 100X my model. How do I do that?" We are happy to say, our infrastructure stays on the frontier and allow this scalability problem to mostly solve, so they can keep using the well laid path and go from there.

>> Yeah. First of all, I love the name Anyscale. I wrote a tweet about that because it's a play on words. But you're really getting at setting the table for what could be unlimited upside on engineering, that multiple step function advantages. What's the coolest thing that you've done that you could share that's come from that? Because now that you've got the freedom to scale, what pops out? What kind of benefit? Share some cool things.

>> I think one of the cool things we actually managed to solve is how to elastically sharing resources in Kubernetes cluster. We moved all the GPU trainings into Kubernetes and we run 100% of the and we operate the cluster. Teams come to us with requesting reserved capacities in the cluster, but we actually have an amazing mechanism to allow teams to borrow resources from between each other when they're not running their training job at this moment. This actually allowed us to satisfy the training, need a much smaller sum of the GPU costs than what teams believe they do. And this save a lot of infrastructure costs for Uber. We can sustain the basis growth from here.

>> Talk about the cost side of it, because obviously everyone goes crazy over the cost of GPUs. I heard a stat today, this morning, from someone in the hallway, that said they know a company that their friend works at. It's always a friend that works there. They're going to spend 10 billion just this year on GPUs. Obviously, might work at Meta, who knows, maybe even Uber. But there's a huge cost on the hardware, but there's also the cost of the infrastructure. What are some of those cost points that you guys want to stay ahead of to create that value? Just take the innovation deliverable to be faster than the cost increase.

>> That is a great question. I think my personal motto and something I hope my team can do for Uber is making sure the cost of machine learning grows slower than the value we create for our company so we can sustainably doing this next couple of years. When we get to the concrete cost, everybody know GPUs are not cheap. I don't have to recite the numbers from public cloud providers. But I think we want to unlock innovation. If we look at the things in time-trend, things will get cheaper by themselves and they'll have these amazing peers in the industry, including 80 skill people who keeps coming up with the great news of something is 10X cheaper, 50, 90% cheaper than what is six or 12 months ago. So what you do is helping people to launch so they unlock the business gains fast, making sure we understand the cost, making sure periodically revisit this. And as an infrastructure provider and platform provider, keep working close with the stakeholder and the innovators so that we bring down the cost while they gradually roll out the features for rest of the world.

>> And you guys just do a lot of work over that. Can't reiterate enough how impressed you guys are on the platform. It really is, to me, the future. Because all the things you'd mentioned, Kubernetes, all these concepts of borrowing resources, loading them around, reminds me of my old operating system days where you got to look at the system architecture. It's a systems play, there's costs involved in the inherent costs and people, the resources, the costs, the GPUs and whatnot. So that's one angle I want to get your thoughts on. And then, two, the data sets are emerging, where you have multimodal. Again, that, combined with the fact that you guys are dealing with multiple kind of third party, you got drivers, you got eats, you got cars, you got, I mean, all kinds of different systems, all running real time. So you get the systems opportunity, challenge and opportunity, and then the multimodal data. What's your thoughts around that? Share your insight.

>> So, really excellent about our unstructured data and the multimodal data. Uber team doing computer vision, natural language processing, and a hybrid of unstructured and unstructured data. They have no problem to use it directly. When we get to tabular data, the way we handle is we create abstractions for ourselves. So we convert some of tabular data into unstructured Parquet files and making sure we process these things under as intermediate representations, so that we hide all this complexity from our users. We operate one shared ML platform and solve both a tabular data that is pretty much existed before Uber even think about ML platform before 2014. And all the rising use cases of computer vision, video, all those new multimodal cases, allowed us to gracefully enter the generative era because gen AI is all about techs.

>> You guys definitely were in good position for the gen AI. I like you're talking about predictive and gen AI, but is predictive going away or that still hangs around and still functional? Because in some cases they work together, there's use cases for predictive and there's use cases for generative. How do you guys look at that dynamic?

>> I don't think they're going away anytime soon. There are special corners of machine learning and subdomains, which a smaller model is doing just fine. And the smaller models allow them to be inferred extremely fast. It's lesser about cost, but more about the latency. We're very committed to reduce the latency as much as we can, so that the Uber app delivers the best user experience. These small models are doing just fine there. There are some development of borrowing new research takeaways on this smaller model and this scale up. But I think in the foreseeable future, predictive and smaller model will just be there.

>> I want get your thoughts because you're in the engine room, you're making all the horsepower, home machine learning, scale. What's right around the corner in the industry right now, on the app side, is the agentic systems. Okay. As you look at agents, you say, "Okay, I get that. That makes a lot of sense." Retrofit workloads end-to-end with agents. How do you view that agent opportunity? Because if you get the work done at the data layer, infrastructure layer, the agents is the next logical piece. Because you got the data management, data is unstructured, semi-constructed, structured. You guys look at that as data. You manage that well. Now agents come in. Where do you see that playing on the ops side, engineering ops, and then also how does that render itself in the app? Any commentary there?

>> Yes. I personally think multi-agent is definitely one of the direction that we need to use to solve the most open-ended and the challenging problems on the gen AI aspect. My takeaway is that it's pretty easy to build a first prototype using JAI and then get something out. The challenge come from not only shipping the high-quality and high-accurate of the application, but giving the developers a well lit path so they can keep polishing and improving their qualities as with every quarter. Because nothing stays immutable. For companies like Uber, once we launch something, we need to stay committed to the quality and keep increasing it. So I think as things getting more complicated in advance, a lot of the use cases will have a need of multi-agent, and that's where I believe the future of gen AI will look like.

>> Well, since you brought it up, I want to just... If you don't mind asking about. You mentioned cost, which we talked about. In machine learning and AI, one of the topics that comes up a lot is latency, which you guys are perfecting the latency, obviously, you got to get that as fast as possible. But recency and relevance of data is a technical problem. How do you view that? How do your engineers managing that quality of the data? Because you can train content and trained data, multi-modal or language, but then you got to figure out, okay, what's relevant now and recent?

>> Yeah. I think this is where the productionization of ML platform goes beyond the model. There's also data aspect that needs to be productionization, there's a near real-time features. Michelangelo has internal support of a lot of near real-time features using various real-time technologies like Flink and Kafka. We also in the process of productionization, near real-time handling for the gen AI workloads. For example, allow people to regularly update their embedding spaces, vector database, all these things. The good thing is a lot of patterns are similar of our pre-gen AI world, so we get to move a lot of our learnings from that era and apply them in the new world. This is where a set of engineer have working all these years. We benefit from the expertise out of this.

>> Okay, so we're here at the Ray Summit, so Anyscale's growing company. Obviously, they're doing well. They got a new CEO, founders out there getting back in on the product side, which is great news to see. It's always hard to run a company as a founder, so I can see that. How would you describe their platform for the folks watching? What category is it in? How would you describe it? Why is it awesome? What's your experience with the platform?

>> So, Uber has mostly in the open source part of the Ray, but we do love the open source version because it offers, like I said, very easy distributed Python environment. The interactive experience with notebooks in the recluster just proves its value a lot by helping people to help them to debug their training workload and have a deep understanding inside their recluster. We love these aspects. And we just started looking at the platform offering, so I think as time goes, we'll have more learnings about that.

>> Yeah. What's next for Uber? Have you had to talk about the engineering priorities around Uber, what you guys are working on, again, people, places and things? You guys solving a lot of platform problems? And again, shout out to you guys. I think you were the first implementer of data lakes, if I remember correctly. Move from commoner store and redoing the streaming architecture. You guys have done a lot of work on that. Now you're here. What's next for you guys?

>> I would say, number one, Uber is entering hybrid cloud setup. Evolving the data lake, especially ML aspect of the data lake in the multi-cloud, multi-region, multi-provider environment, is going to be an interesting challenge. The second is, again, going back to efficiency. Keeps driving up the efficiency, driving down the cost, and removing all the frictions for our users. Some of the downside users has to break the highest level of traction to use a lower level offering for their early prototypes. We understand this is the maintenance cost for fast innovation, but you want to gradually move them back to highest level of traction so that they can reduce their frictions and we can support more use cases with our engineers. The last but not least, we are always thinking about open source. We want to open source the great innovations, those elastic resource sharing stuff. Some of the tools we develop for responsible AI like explainability on top of Ray, some of workflow stuff, and we want to always look around about great innovations in open source and absorb them in our platform.

>> Well, Savannah and I and Rob Strechay will be at KubeCon, so we'd love you to come in and share that open source with the KubeCon TNCF. I know they would love that. Again, that's orchestrating a lot of value there. Love having you guys on theCUBE. Your content's phenomenal. Again, you guys are at the forefront of the AI wave. My final question for you would be, for the folks that are trying to start thinking about, okay, have the old IT environment, the classic enterprise are realizing it, the boardrooms are saying, "Hey engineers, let's get engineering some new foundational sets of services." Okay, so you got a backend revolution happening around gen AI, merging predictive with gen AI, but the front end's also changing. I've never seen this in my career where both front end and back end, you got to do both. You got to get the workflow and the process nailed down and get the back end. What's your advice for folks who are looking at building AI platforms? What should they be thinking about? What are some best practices from your experience with your team and your work that they could take to heart as they look at the 20 mile stair for the future? What should they do?

>> My number advice is always embrace open source, embrace openness. I was at this tech advisory with yesterday. The challenges, at pinpoint, are strangely similar. We are all trying to solve a set of similar problems at different companies. So open source will speed up everybody's journey solving these things faster. And we will avoid the mistakes of solving something by result. Simply not knowing alternative exists. My second advice is, meet where your users are. Try to think about how they do things, what's most natural to them. This is probably going to be Python, Pandas APIs, analytical notebooks. Try to meet where they are and offer something that they can learn, even before joining your company. That is maximizing their productivity. Machine learning engineers are expensive, they're very hard to hire, so let's maximize their productivity on day one.

>> Well, there's a lot of people who want to make some good bank and be an engineer like you. What would be your advice for folks watching that are at the university right now? Because right now, computer science curriculum is rapidly, just in the past four years, adopting, and trying to stay on that rocket ship. There's a lot of young people watching, maybe work for Uber or another company. What should they be thinking about if they're at university, if they're a Python developer, or have natural engineering skills? What would be advice for those people?

>> I would suggest students to find a good balance between chasing the latest development and also pay some attention to the computer science fundamentals. The fundamentals are still there. I think this generation of AI is really enabled by the best system engineers who has got their expertise before gen AI was even a concept.

>> So systems thinking?

>> Yes, systems thinking.

>> Operating systems, compiler design, is there a certain lane?

>> Distributed systems, system designs, all the fundamentals of what computer science students should be learning their four years, or six years if .

>> As I always say in the theCUBE, I wish I was 25 again. If you're in computer science or even any engineering field, there's a huge amount of systems opportunity to build these platforms, and so congratulations on your success and thank you for sharing your insight with theCUBE community. Appreciate you.

>> Thank you.

>> All right. I'm John Furrier, host of theCUBE here in San Francisco for Ray Summit with Savannah Peterson, bringing you all the live action from the floor here at Ray Summit. Anyscale; fast-growing company, great platform. A lot of tier one engineering users building AI platforms. Companies like Apple, Uber, the biggest names with the biggest staffs changing the future here in San Francisco. Thanks for watching.