AWS Summit NYC 2024 | Vijay Karunamurthy | Scale AI

theCUBE.net

info Help
Sign in
Register

Clips
News
More from AWS Summit NYC 2024

Vijay Karunamurthy

Field Chief Technology Officer

Scale AI

The crucial role of data in Scale AI’s gen AI platform: Transforming enterprises and public sector solutions

The importance of data in realizing enterprise artificial intelligence goals has put AI data quality at the forefront of discussions. Ensuring high-quality AI data is now vital for a wide range of enterprise applications, from innovative development projects to insightful business intelligence.Scale AI’s Vijay Karunamurthy discusses the importance of quality AI data.Scale AI Inc., which provides labeled data used to train AI applications, strongly advocates for a data-centric approach

play_circle_outline Scale AI's focus on trustworthy AI systems and data-centric AI approaches.

play_circle_outline Three customer buckets: research labs, enterprise, public sector with agent-driven workflows.

play_circle_outline Simplifying deployment and evaluation of gen AI applications with Scale AI platform

play_circle_outline Achieving Seamless Model Deployment with Amazon: Strategic Partnerships and Integration with AWS Services

play_circle_outline Meeting customers where they are in gen AI adoption journey

Info

Vijay Karunamurthy | Scale AI

Vijay Karunamurthy

Field Chief Technology Officer Scale AI

search

>> Welcome back, everyone, to the theCUBE's coverage here in New York City for the AWS Summit. This is theCUBE's exclusive coverage. I'm John Furrier, host of theCUBE. We're here with Vijay, Field CTO at Scale AI, hot company, is growing great. Great to have you on theCUBE.

Vijay Karunamurthy

>> Thank you. Yeah, really excited to be here.

>> So we met, yeah, your CEO, I think at Dell Tech World, another event. You guys have been growing fast. Take a minute to explain what Scale AI does first for people who don't know what you guys are doing.

Vijay Karunamurthy

>> Yeah, definitely.

>> It's been a great success story. Go ahead.

Vijay Karunamurthy

>> Yeah. Alexandr Wang, our CEO, he started the company eight years ago. Even from the early days of Scale, we were recognized as a leader in building trustworthy AI systems. Some of the first use cases we tackled were self-driving and autonomy. how do you make sure the self-driving car stops at a crosswalk if a pedestrian's crossing the crosswalk? The key insight that Alexandr brought to bear is the idea of data-centric AI. If you find the right data that can help you reach that safety milestone, examples of pedestrians, maybe people on bicycles crossing that crosswalk, that data's worth its weight in gold for training the AI model and hitting that safety milestone in three months or six months, instead of waiting nine or 12 months to get there. So we took this approach for computer vision, for LiDAR, for all these cases, and then we ended up partnering with some cutting-edge research labs. One of them was a little startup known as OpenAI. And so running up, they had GPT-3, and we were partnering with them on this data workflow called reinforcement learning. That ended up powering ChatGPT, which was one of the first gen AI workflows to work. Human beings can interact with the model and reliably get certain answers, and ask the model to follow instructions and expect those instructions to be obeyed. Reinforcement learning is a key piece in making that possible. So today, we have three big buckets of customers. One are the OpenAIs of the world, the research labs that are building the next cutting-edge models. We're helping make those models more trustworthy across a range of new capabilities, like writing code or doing multimodal data. And the second is really one of the most interesting ones. It's enterprises, private companies that are actually adopting gen AI. They have really unique needs because they have private data. They have data that needs to be kept in a VPC in Stockholm, or in a VPC on the East Coast, and it can't leave the bounds there, or if it does, there has to be really strict role-based access control around how that data is used, even if you have derived data from that data. So making gen AI work in that environment has been a big challenge, and we've built this gen AI platform around that. And then the third big bucket of customers is the public sector. We have a range of agent-driven workflows, it's called Agent Donovan, but it helps people with really mission-critical objectives accomplish their work by talking to that commander in the way that they expect to be talked to.

>> Give me an example of Donovan real quick.

Vijay Karunamurthy

>> Yeah. A great example would be if you're an agency and you have a supply chain issue and you ask a model, "Find me all the warehouses around Chicago that could help solve this problem in the next six hours," with the model today, you might get a block of text as an answer. You might say, "Hey, here's five different cities," and some of those might be accurate, some of those might be from the wrong data. But what Agent Donovan does is it actually gets the breakdown answer, it verifies each one of those using multiple models in the loop, and then rather than just giving you a block of text, it'll actually go and plot it out on a map and show you the links in the chain of that supply chain so that you can visually understand, someone that's the responsible individual for the supply chain, this is where I can make changes, this is something I can do in the next 10 minutes to actually drive an outcome. That ends up being really important in timely, mission-critical objectives. So we're using it across a range of the US government. The Department of Defense has an office called the CDAO, the Chief Digital AI Officer. They're one of the first to use our models across a range of different use cases. But we're evaluating gen AI with that lens. How can you actually drive actions, rather than can just up-

>> So how do you keep up? How do you stay up at night? You don't get any sleep with all that action going on. You got the OpenAI's provider, I call it the hyperscale AIers, the hyperscale AIers, that might not be the right term, but the big AI providers.

Vijay Karunamurthy

>> Yep.

>> You got, just in general, the enterprise is a wide-open thing, and public sector. I mean, you guys got a lot of business going on here.

Vijay Karunamurthy

>> Definitely, yeah. It's been a breakneck two years. But one interesting thing, I think the pace of innovation on that hyperscaler side has been so fast. They've been adding new capabilities, like multimodal capabilities or the ability to do voice outputs, that just weren't there a year ago. Enterprises have been struggling to keep up. And one challenge we've seen is if you talked to some of these companies last year, they maybe had 100 different prototypes of gen AI applications they were just getting off the ground. Maybe each team that understood its data really well, they had their own their own data silos, they would build a gen AI application against their database and they would just have it plug in there. And they're realizing this year that there are two big bottlenecks to actually make that drive ROI. One is trustworthiness of the answers. If the model can't get the right data from the right part of the organization, it just can't give a reliable answer. If it doesn't have the most latest research reports I've written on Apple or NVIDIA as a stock, and you're giving research reports based on 10 years ago, you're going to be talking about GPUs used for gaming or the iPhone 9. That's a big problem if you're talking to a customer.

>> So stale data and hallucinations are a big factor-

Vijay Karunamurthy

>> A huge problem....

>> in those lack of or gap, data gaps.

Vijay Karunamurthy

>> Exactly.

>> That's called data gaps, right?

Vijay Karunamurthy

>> Yeah.

>> Whatever people call it.

Vijay Karunamurthy

>> It's a model understanding where to get the right data, how to execute SQL or other analytics languages in order to access that data, and then understanding when it doesn't just have the data, it needs to answer a question, and telling a human operator, "Hey, I don't have access to the right data."

>> You mentioned the LiDAR example. I want to dig into that because I think this is where it comes down to where I get excited by some of the coolness of this. Okay, I know what a bicycle looks like in the bike lane in San Francisco. Waymo's there, I'm powering Waymo. I'll store that because that's important to me because I have to know that image. Okay, I store that. I see some people walking around. Okay, I get that. That's not real-time information. That's historical data. You mentioned the staleness. You've got to keep feeding fresh information into the model to keep it fresh. There's a freshness. It's like the Budweiser, "Don't open after this date," kind of thing, but that's fact. So you've got historical data. This is important, what to keep track of. How do you guys look at that? How should people think about that? Because now they're getting all this data access, so unstructured data is piling up like you read about. SQL data is almost a system of record kind of context. But similar with that, I got a bicycle, I need to store that. And I got this real-time streaming data coming in. What do I do?

Vijay Karunamurthy

>> Yeah, you need constant, 24/7 online evaluation of gen AI applications, which is a new mindset that people didn't have before. In the past, if you wanted to evaluate a model, and let's say that model is a legal model that's going to summarize a legal document for you, you'd go and find your lawyer, your in-house lawyer, who'd say, "Hey, did this model get the right output? What feedback do you want to give the model so that it can improve the prompt or some other part of the application?" But really, what you need is 24/7. If people are asking legal questions and they're your employees, you need to have a range of different scenarios that you're testing and you're observing and you're seeing how the model's changing over time. A great example, if you're a multinational corporation, the laws in Switzerland and Germany are very different, even though customers are going to ask in German about both those countries. So you want to make sure just asking in German doesn't mean you're defaulting to German law. You need online evaluation to make sure you're not breaking those. Without that, as your data architecture changes and these data pipelines are breaking down and assembling new ones, you're just going to run into new hallucinations because the data just isn't there to support it.

>> It's interesting. It's almost a false summit you're climbing if you think that's the answer, you don't rethink it. Because ultimately, you think you're having success and all of a sudden, wait a minute, I'm way over my skis on this one.

Vijay Karunamurthy

>> Yeah, definitely.

>> I got to now go back and rearchitect. So this is the fear everyone has in the enterprise, and this is important. So I have to ask you, what do you do to solve that? Matt Wood announced on the stage, models, well-integrated models. Okay, but we've been doing data marketplaces for over a decade, so data exchanges have happened, but model exchanges haven't happened. So if you don't have a model, you're not in pole position at all to compete.

Vijay Karunamurthy

>> Totally, yeah.

>> So what do you do? Is it compete on models or what?

Vijay Karunamurthy

>> Yeah. Models can be some of the most important IP that some of these organizations have, especially when you fine-tune smaller models. You can make those models really accurate against your business or the way you talk to customers. You can have a model that chats with customers in the right tone of voice. But ultimately, your real milestone of how you measure your performance is can you unlock your people? Can you unlock all this organizational expertise that I have, and have them drive improvements over time? And ultimately, you can't hold any advantage that you can't observe. If you have an advantage in terms of your model being better or your data being better organized, if you can't observe that and demonstrate that through monitoring, you're just not going to hold that advantage.

>> We were riffing earlier on the analysts session. I was saying to Zeus and Sarbjeet who was here that the entrepreneur equation 20 years ago was, "Wow, I'm going to disrupt this category by renting my house out," or, "I'll start this micro SMS app," or whatever it did. And I just put the credit card down and spin up some compute, rather than buying a server at super micro, and see if it worked. Right?

Vijay Karunamurthy

>> Right.

>> That was the SaaS, that was the way the iPhone generation happened and SaaS happened. That is no longer the entrepreneurial case. The entrepreneurial case now is there's a complex app development environment that involves a lot of cloud. The cloud was just emerging. Amazon just happened to be at the doorstep and the place where this is awesome for entrepreneurship that the alternative is no risk, so there's zero risk. Well, credit card risk, 50 bucks. There's risk now because you have... But the entrepreneurs are looking at the data models and I could flip a model out there and change a category.

Vijay Karunamurthy

>> Yeah, absolutely.

>> So you're saying that if I'm doing data exchanges the wrong way, and I'm going down the path where there's going to be hallucinations, I get to the wrong spot, the technical debt is killing you. So I could come in and be a model developer and take over a category potentially. Do you agree?

Vijay Karunamurthy

>> Yeah. I worked at YouTube years ago, and I remember the early days of YouTube, it was exactly that disruptor mindset. You put down a credit card, you get a bunch of servers, you rack them up at Equinix, and suddenly, you're serving video traffic to millions of people around the country. The world today is very different. So these foundation models, they require easily $10 million now, $100 million of investment and data and infrastructure and monitoring and observability in order to just get these models out the door. But then once you leverage those models, there's a range of things you can do just based upon that initial investment that's just really powerful. So we're seeing a lot of firms, and those firms that normally we would consider to be the ones to be disrupted, they're actually leading the way in figuring out how can I build these centers of excellence within my company, have really smart AI engineers, and other stakeholders, maybe an AI ethics officer, collaborating together to see those opportunities forward? So we're seeing a lot of really groundbreaking innovation happening in these regulated industries where data governance has always been part of the question, understanding the lineage of your data, understanding how mature that data is, these are just fundamental parts of how they've done business for the last 10 years. And now they're applying the gen AI stack on top of that.

>> What about the entrepreneurial activity? Where do you see that happening?

Vijay Karunamurthy

>> Yeah, a lot of interesting stuff.

>> Given the high cost to do a model. I mean, to be a consumer model, that's going to be a high bar.

Vijay Karunamurthy

>> Yeah, definitely. To do a consumer model, it requires a lot of investment, so there's only a few players in that space. But there's a range of startups that are trying to do really interesting things based upon agents and other activity, really building to the use case. So if you have a legal use case, let's make a model that talks to lawyers in the language that they need to be talked to. If you have an education use case, we've seen some really interesting investment there in ed tech where talk to students where they are in the journey. If someone doesn't understand photosynthesis, they're a first-year college student, talk to them differently than the way you would someone who's already done organic chemistry and knows everything they're not going to know.

>> Yeah, it's interesting. You could be a domain-specific expert in, say, legal and get a small, little beachhead feature, and then end up aggregating data for the whole vertical.

Vijay Karunamurthy

>> Totally, yeah.

>> Potentially, right? I mean, I'm just making this up, but that could be a trajectory.

Vijay Karunamurthy

>> Data's unbelievably valuable, but maybe the bigger use case for data's actually just evaluating how the models are doing today. Without the right data, you can't figure out how the models are performing, you can't move forward towards a goal. So having the right data about your specific use case, that's worth its weight in gold right now, I think.

>> Yeah, data observability's a hot area. What's your view on that? Are we not there yet? Are some people doing it? Are you guys doing it?

Vijay Karunamurthy

>> Totally. The first thing is you have to do it. You have to have data observability.

>> Now!

Vijay Karunamurthy

>> Your data's changing over time. You're going to introduce bias, You're going to introduce data drift, all these issues if you're not observing your data. Data is the lifeblood of any gen AI application. And applications are changing in how they retrieve data overtime. We're starting to see this really important pattern that we do in financial services where the model writes the SQL and the SQL goes and executes and gets the data, and then it's transformed in a way that's useful to human analysts. You could trade a PowerPoint presentation or a chart or a graph out of that data. You need to understand how that SQL's executing. Does it know about the schema of your database? Is that schema changing over time? If you don't do that, you have this incredibly brittle application that works today and doesn't work tomorrow.

>> What's your big use case in enterprise? If people watching here want to see Scale AI, give you guys a look-over, what's the big use case that you guys are doing the most action with the enterprise?

Vijay Karunamurthy

>> Definitely. SQL's a big one. It's very painful to have a data analytics org be the bottleneck for everyone else in the company. If they want a question about how many customers bought hot dogs on the Fourth of July, and you have to wait three weeks for a data analyst to write the SQL query, that's really painful. And so having models generate SQL and generate accurate and reliable SQL, it empowers the data analyst team. It's a force multiplier for them to do their work a lot better.

>> Yeah. Well, the big models will you who won the Nathan's Hot Dog Eating Contest, for sure. We saw that on TV.

Vijay Karunamurthy

>> Yeah. Was it Kobayashi?

>> We'll sidebar that. Okay, so that's cool. So this is important because this now saves time. So the querying, the SQL thing, is going to be very key.

Vijay Karunamurthy

>> Yeah.

>> What's the fine-tuning equation? Here, a lot of people ask questions. It's the most common question I get is, "What do I do fine-tuning on? My data's already trained. I don't need to... What do I... And where's reasoning and reinforced learning, when does that apply? I don't know when to apply what to the data at what stage of wherever my journey is."

Vijay Karunamurthy

>> Yeah. When you have those questions with a customer, it's important to take a step back and figure out how do you evaluate what success is? And do you have the evaluations in place? We take this approach called hybrid evaluation where you have models that are always going to be important in evaluating other models because they can do things that human beings just can't do at scale. A model can translate your question about the law in German to Swiss German and ask that same question and see if there's a different answer. So that model-assisted evaluation is important. But expert oversight is incredibly important as well, bringing in all this knowledge that you have about the law, or about biology, and bringing that together at scale is incredibly important. But once you get that evaluation firm in place, then you can see what are the really important interventions I could do to make a difference this week? Is it fine-tuning a model? Maybe that takes two to three weeks. Is it adding a new guardrail on top of the model? Maybe I can do that in an hour. Could I just change up the prompts? Without evaluations, you don't know if you're moving towards your goal and you don't know how far away that goal is, but once you do know that, you can figure out what are the different interventions I could do to try to get alignment of how-

>> What's the advice you have for two use cases? One use case is you got a company, it's got a small language model, it's fairly trained, it's unique, so it's okay. What do I do with this model? What do I interact it with? How do I engage another model? And then two, mindset of the organizations who want to bring in this new culture of thinking, systems thinking, prompt thinking, to start the trajectory of culture shift inside their organization.

Vijay Karunamurthy

>> Totally. I mean, most models today in the enterprise are useless without data, and that's why there's this big story about data gravity when it comes to gen AI. You can have a small model and you can deploy it closer to where your data and where that compute lives, that ends up being much more valuable than having a large model that can't access the right data because of access control procedures you have. So we have lots of companies where maybe for some regulatory reason, they have data that sits in a VPC in Stockholm, and they have a separate set of data that sits in a VPC in Berlin, and those two datasets shouldn't be conjoined together in order to answer a question, but you can answer independent questions about each one of them. If you can have a smaller model that lives in those two different locations and is able to do a lot of knowledge and reasoning about what's local to it, that ends up being incredibly valuable and unlocks a lot of opportunities for you as well.

>> So Vijay, what we're talking about here is essentially platform engineering for data.

Vijay Karunamurthy

>> Yeah, absolutely.

>> I mean, is that data engineering that? I mean, is there a category? What's your view on this because this is huge. I mean, this is a huge shift. I mean, it's a platform shift.

Vijay Karunamurthy

>> Totally.

>> I mean, you see that, clearly. Your company's living it, but other companies are just trying to grok platform engineering with Kubernetes and cloud native. Now I'm like, "What? I got to reset my entire data analytics stack? That's not my department."

Vijay Karunamurthy

>> Right, right. And the first thing that gen AI did, even five years ago, was start to unlock the value of unstructured data. Before, if you had research reports, it was really hard to make sense of them or put them in a database and retrieve them. You can unlock all that value. But now we're seeing, even structured data, you're able to unlock all this value by understanding the data silos in your organization and finding ways to answer questions securely against that data. costs, maybe if you're a large bank and you've acquired 15 regional banks in the last 10 years, each of those might store customer data on different databases. Now you can build data pipelines that answer questions about your entire business in a way that you just couldn't possibly do before. So we see a lot of value in that. These data platforms are being built in part because you have this new tool that suddenly unlocks all this value.

>> Yeah, it's interesting. I see a couple of different use cases. We do talk with our research team in theCUBE. A couple scenarios I want to run by and get your reaction. One is here's an environment that is completely not positioned for gen AI. They're just old legacy. They never invested. They outsourced it to the hills. They got some token projects that'll look good. Then you got the companies that are actually doing really well in the modern devops factor, but don't yet have that data connected. And then ones that are actually investing. Okay. Let's talk about the first two. This is where you start getting into, okay, you need a full retrofit. It's like the bar takeover show. It's like, "Okay, shut it down. We're going to reopen in two months," or whatever that is. They have to rearchitect their entire thing because now, they have legacy storage, networking and compute, and the gen AI apps that they want to do, we'll have to train with different characteristics. I might need more reads on this and writes on that, and for instance, training are two different things. You talk about reinforced learning, that's ongoing. That's going to have a different ratio of reads versus writes and compute requirements. And network paths. You said VPCs, I'm going to send only these networks. So you talk about that systems going to be completely flipped upside down. What's your reaction to that? And how should people start thinking about this? Is it a mind-blowing thing, or it's reality, I think. Do you think it's reality? And if so, what do people do?

Vijay Karunamurthy

>> One of the most painful roles to be in right now is doing budgeting and planning for your infrastructure today at a large company because right now, the challenge you have is you have all these different prototypes of gen AI applications, and the roadmap for those carries you out maybe six months, maybe nine months, maybe 12 months, but there's a huge variability in you knowing when is this application going to get in production? What region of the world do I need to deploy this application in? What performance characteristics do I really need to have? Am I using a large model? Am I using a small model? The ML engineers are telling you, "I just don't know yet. I'm going to find out maybe a week before I deploy this, which one of those is better." That's a really challenging spot to be in. So one of the key reasons people use our Scale AI gen AI platform is it's just really simple to one-click deploy that to all of your VPCs, wherever they're located. They all operate under the same security principles. And then you're able to evaluate where you are this roadmap for each one of those applications, and make those assumptions actually much more tangible. Would a small model fit here? Would it give me more reliable answers? Can I get 75% of the way to production just using that small model alone or do I need to sometimes leverage a larger model to answer more complicated questions? And then you can actually do budgeting and planning about where those GPUs need to sit, what sort of top-of-rack configuration do you need? Hopefully, you're on AWS and you're leveraging Bedrock and SageMaker and so that's a little different-

>> All right. So now we're at... Thank you very much for a great conversation. Let's get into this what do you guys do with Amazon? How do they consume it? Are you in the Marketplace, I assume?

Vijay Karunamurthy

>> Yeah.

>> What's the interaction? What's the relationship with them? What's the coolest thing you're doing with AWS?

Vijay Karunamurthy

>> Yeah. Part of our announcement today is a multiyear strategic partnership with Amazon. A lot of Amazon customers that use Bedrock, use SageMaker today, a lot of them have asked for better evaluation of their models so they can assess whether a custom fine-tuned model is working here, whether a large model made from Anthropic or another provider is better suited for that application. They can use our gen AI application engineering platform at just one-click deploy to all of their use cases-

>> In bedrock today?

Vijay Karunamurthy

>> In Bedrock, yeah.

>> Available, I mean, instantly?

Vijay Karunamurthy

>> Yeah, yeah. And everything's billed through your AWS account, so you can use your AWS credits for there and help you with your budgeting and planning hopefully going forward using the insights you got there. But some of the decision-makers that have been asking for this, one is folks that are coming in to a chief AI ethics officer role, or a chief AI officer role. They have a lot of questions over how do I responsibly deploy AI? How safe are the AI applications I have out there?

>> Yeah, can I explain it?

Vijay Karunamurthy

>> Yeah, totally. Yeah. Can I explain why this cybersecurity incident happened against this gen AI application? And it's something really serious. I leaked internal data, but I can't explain why it happened. That's a bad spot to be in. So having access to this application on the Marketplace today, I think that's going to really empower a lot of those stakeholders to do that work.

>> You know, it's interesting. Software supply chain's been a conversation we've been having a lot on theCUBE. And for the first time on a keynote, I heard the words data supply chain, not in reference to lineage, In reference to, "Okay, I got to start tracking the data coming in." Again, you talk about observability challenges. Again, this is just another example. It's a whole new wild, wild world.

Vijay Karunamurthy

>> Yeah, totally. There are all sorts of attacks that could potentially come up through data in the worst-case scenario. And if you're a bank, you have all sorts of foreign governments that want to infiltrate your systems and get information out, so it's important to stay ahead of that. We do a lot of what's called red teaming where we use our platform and we try to think in the way a malicious state actor, maybe North Korea, I won't name other ones but North Korea's a good one, how they would go about hacking a gen AI application, what sort of changes in the prompting or other things would they do to try to get access to the internal data? And then you stay ahead of the game. You really know. We actually, we spent time talking to the US government and we presented to the Homeland Security Committee on red teaming and what it really means because it's important for the US government to spread this knowledge, help protect sensitive data. That's important for all the markets.

>> Yeah. I mean, we're projecting a Cambrian explosion of gen AI apps, the infrastructure, middleware, data, all has to be looked at and rethought through with a mindset with generative AI in mind, right?

Vijay Karunamurthy

>> Totally.

>> It's generating. It's run time.

Vijay Karunamurthy

>> Totally. Yeah, yeah.

>> We've see this movie before in distributed computing. It's called the edge, tactical edge. Vijay, if you could summarize your relationship with Amazon, the news, and what you guys do, what would be the bumper sticker? What would be the soundbite that you think that users should walk away with as a perception of what they want to take away?

Vijay Karunamurthy

>> Yeah. The key part of the strategic partnership is it's really meeting customers where they are. Every customer's on this different journey towards adopting gen AI. Some of them have successfully deployed prototypes into production. Others are really feeling bottlenecked by those trust and safety issues that get out there. So using our platform, it's an easy way to just get observability into where your data sits, how it's informing the answers you're getting out of gen AI, and really what it's going to take to accelerate the adoption of this technology. We're hopeful that, as part of this, customers don't just think that there's a one-size-fit-all answer to gen AI questions or one single data architecture's going to work for everyone. Instead, it's what's the way to get the observability into that? What's working today? And what do I need to invest in?

>> And you guys are going to fit right into the architecture of AWS and all those higher-level services.

Vijay Karunamurthy

>> Totally. Yeah, VPCs, the role-based access control you have in place, EKS, all these other technologies that financial services companies have used. We fit in very well with that in protecting it.

>> As CTO, what are some of the Amazon customers saying about this opportunity?

Vijay Karunamurthy

>> Yeah. It's been huge. Our key customers right now have been financial services, healthcare, insurance, education. All those businesses, they're just transforming. They're going to look very different five years from now than now. So anything you can do to get ahead of that, accelerate that transformation in six months, nine months, is just-

>> It's a land grab. I love this market. Vijay, thank you for coming on.

Vijay Karunamurthy

>> Thank you. Thanks, John. Yeah.

>> All right. Scale AI, making the difference here, changing the game out in Bedrock, bringing observability in. Again, the whole world's going to change. Reinforced learning right around the corner. Stay tuned for more coverage. We're bringing you all the data here on theCUBE. I'm John Furrier, host of theCUBE. Thanks for watching.

Vijay Karunamurthy | Scale AI

search

>> Welcome back, everyone, to the theCUBE's coverage here in New York City for the AWS Summit. This is theCUBE's exclusive coverage. I'm John Furrier, host of theCUBE. We're here with Vijay, Field CTO at Scale AI, hot company, is growing great. Great to have you on theCUBE.

Vijay Karunamurthy

>> Thank you. Yeah, really excited to be here.

>> So we met, yeah, your CEO, I think at Dell Tech World, another event. You guys have been growing fast. Take a minute to explain what Scale AI does first for people who don't know what you guys are doing.

Vijay Karunamurthy

>> Yeah, definitely.

>> It's been a great success story. Go ahead.

Vijay Karunamurthy

>> Yeah. Alexandr Wang, our CEO, he started the company eight years ago. Even from the early days of Scale, we were recognized as a leader in building trustworthy AI systems. Some of the first use cases we tackled were self-driving and autonomy. how do you make sure the self-driving car stops at a crosswalk if a pedestrian's crossing the crosswalk? The key insight that Alexandr brought to bear is the idea of data-centric AI. If you find the right data that can help you reach that safety milestone, examples of pedestrians, maybe people on bicycles crossing that crosswalk, that data's worth its weight in gold for training the AI model and hitting that safety milestone in three months or six months, instead of waiting nine or 12 months to get there. So we took this approach for computer vision, for LiDAR, for all these cases, and then we ended up partnering with some cutting-edge research labs. One of them was a little startup known as OpenAI. And so running up, they had GPT-3, and we were partnering with them on this data workflow called reinforcement learning. That ended up powering ChatGPT, which was one of the first gen AI workflows to work. Human beings can interact with the model and reliably get certain answers, and ask the model to follow instructions and expect those instructions to be obeyed. Reinforcement learning is a key piece in making that possible. So today, we have three big buckets of customers. One are the OpenAIs of the world, the research labs that are building the next cutting-edge models. We're helping make those models more trustworthy across a range of new capabilities, like writing code or doing multimodal data. And the second is really one of the most interesting ones. It's enterprises, private companies that are actually adopting gen AI. They have really unique needs because they have private data. They have data that needs to be kept in a VPC in Stockholm, or in a VPC on the East Coast, and it can't leave the bounds there, or if it does, there has to be really strict role-based access control around how that data is used, even if you have derived data from that data. So making gen AI work in that environment has been a big challenge, and we've built this gen AI platform around that. And then the third big bucket of customers is the public sector. We have a range of agent-driven workflows, it's called Agent Donovan, but it helps people with really mission-critical objectives accomplish their work by talking to that commander in the way that they expect to be talked to.

>> Give me an example of Donovan real quick.

Vijay Karunamurthy

>> Yeah. A great example would be if you're an agency and you have a supply chain issue and you ask a model, "Find me all the warehouses around Chicago that could help solve this problem in the next six hours," with the model today, you might get a block of text as an answer. You might say, "Hey, here's five different cities," and some of those might be accurate, some of those might be from the wrong data. But what Agent Donovan does is it actually gets the breakdown answer, it verifies each one of those using multiple models in the loop, and then rather than just giving you a block of text, it'll actually go and plot it out on a map and show you the links in the chain of that supply chain so that you can visually understand, someone that's the responsible individual for the supply chain, this is where I can make changes, this is something I can do in the next 10 minutes to actually drive an outcome. That ends up being really important in timely, mission-critical objectives. So we're using it across a range of the US government. The Department of Defense has an office called the CDAO, the Chief Digital AI Officer. They're one of the first to use our models across a range of different use cases. But we're evaluating gen AI with that lens. How can you actually drive actions, rather than can just up-

>> So how do you keep up? How do you stay up at night? You don't get any sleep with all that action going on. You got the OpenAI's provider, I call it the hyperscale AIers, the hyperscale AIers, that might not be the right term, but the big AI providers.

Vijay Karunamurthy

>> Yep.

>> You got, just in general, the enterprise is a wide-open thing, and public sector. I mean, you guys got a lot of business going on here.

Vijay Karunamurthy

>> Definitely, yeah. It's been a breakneck two years. But one interesting thing, I think the pace of innovation on that hyperscaler side has been so fast. They've been adding new capabilities, like multimodal capabilities or the ability to do voice outputs, that just weren't there a year ago. Enterprises have been struggling to keep up. And one challenge we've seen is if you talked to some of these companies last year, they maybe had 100 different prototypes of gen AI applications they were just getting off the ground. Maybe each team that understood its data really well, they had their own their own data silos, they would build a gen AI application against their database and they would just have it plug in there. And they're realizing this year that there are two big bottlenecks to actually make that drive ROI. One is trustworthiness of the answers. If the model can't get the right data from the right part of the organization, it just can't give a reliable answer. If it doesn't have the most latest research reports I've written on Apple or NVIDIA as a stock, and you're giving research reports based on 10 years ago, you're going to be talking about GPUs used for gaming or the iPhone 9. That's a big problem if you're talking to a customer.

>> So stale data and hallucinations are a big factor-

Vijay Karunamurthy

>> A huge problem....

>> in those lack of or gap, data gaps.

Vijay Karunamurthy

>> Exactly.

>> That's called data gaps, right?

Vijay Karunamurthy

>> Yeah.

>> Whatever people call it.

Vijay Karunamurthy

>> It's a model understanding where to get the right data, how to execute SQL or other analytics languages in order to access that data, and then understanding when it doesn't just have the data, it needs to answer a question, and telling a human operator, "Hey, I don't have access to the right data."

>> You mentioned the LiDAR example. I want to dig into that because I think this is where it comes down to where I get excited by some of the coolness of this. Okay, I know what a bicycle looks like in the bike lane in San Francisco. Waymo's there, I'm powering Waymo. I'll store that because that's important to me because I have to know that image. Okay, I store that. I see some people walking around. Okay, I get that. That's not real-time information. That's historical data. You mentioned the staleness. You've got to keep feeding fresh information into the model to keep it fresh. There's a freshness. It's like the Budweiser, "Don't open after this date," kind of thing, but that's fact. So you've got historical data. This is important, what to keep track of. How do you guys look at that? How should people think about that? Because now they're getting all this data access, so unstructured data is piling up like you read about. SQL data is almost a system of record kind of context. But similar with that, I got a bicycle, I need to store that. And I got this real-time streaming data coming in. What do I do?

Vijay Karunamurthy

>> Yeah, you need constant, 24/7 online evaluation of gen AI applications, which is a new mindset that people didn't have before. In the past, if you wanted to evaluate a model, and let's say that model is a legal model that's going to summarize a legal document for you, you'd go and find your lawyer, your in-house lawyer, who'd say, "Hey, did this model get the right output? What feedback do you want to give the model so that it can improve the prompt or some other part of the application?" But really, what you need is 24/7. If people are asking legal questions and they're your employees, you need to have a range of different scenarios that you're testing and you're observing and you're seeing how the model's changing over time. A great example, if you're a multinational corporation, the laws in Switzerland and Germany are very different, even though customers are going to ask in German about both those countries. So you want to make sure just asking in German doesn't mean you're defaulting to German law. You need online evaluation to make sure you're not breaking those. Without that, as your data architecture changes and these data pipelines are breaking down and assembling new ones, you're just going to run into new hallucinations because the data just isn't there to support it.

>> It's interesting. It's almost a false summit you're climbing if you think that's the answer, you don't rethink it. Because ultimately, you think you're having success and all of a sudden, wait a minute, I'm way over my skis on this one.

Vijay Karunamurthy

>> Yeah, definitely.

>> I got to now go back and rearchitect. So this is the fear everyone has in the enterprise, and this is important. So I have to ask you, what do you do to solve that? Matt Wood announced on the stage, models, well-integrated models. Okay, but we've been doing data marketplaces for over a decade, so data exchanges have happened, but model exchanges haven't happened. So if you don't have a model, you're not in pole position at all to compete.

Vijay Karunamurthy

>> Totally, yeah.

>> So what do you do? Is it compete on models or what?

Vijay Karunamurthy

>> Yeah. Models can be some of the most important IP that some of these organizations have, especially when you fine-tune smaller models. You can make those models really accurate against your business or the way you talk to customers. You can have a model that chats with customers in the right tone of voice. But ultimately, your real milestone of how you measure your performance is can you unlock your people? Can you unlock all this organizational expertise that I have, and have them drive improvements over time? And ultimately, you can't hold any advantage that you can't observe. If you have an advantage in terms of your model being better or your data being better organized, if you can't observe that and demonstrate that through monitoring, you're just not going to hold that advantage.

>> We were riffing earlier on the analysts session. I was saying to Zeus and Sarbjeet who was here that the entrepreneur equation 20 years ago was, "Wow, I'm going to disrupt this category by renting my house out," or, "I'll start this micro SMS app," or whatever it did. And I just put the credit card down and spin up some compute, rather than buying a server at super micro, and see if it worked. Right?

Vijay Karunamurthy

>> Right.

>> That was the SaaS, that was the way the iPhone generation happened and SaaS happened. That is no longer the entrepreneurial case. The entrepreneurial case now is there's a complex app development environment that involves a lot of cloud. The cloud was just emerging. Amazon just happened to be at the doorstep and the place where this is awesome for entrepreneurship that the alternative is no risk, so there's zero risk. Well, credit card risk, 50 bucks. There's risk now because you have... But the entrepreneurs are looking at the data models and I could flip a model out there and change a category.

Vijay Karunamurthy

>> Yeah, absolutely.

>> So you're saying that if I'm doing data exchanges the wrong way, and I'm going down the path where there's going to be hallucinations, I get to the wrong spot, the technical debt is killing you. So I could come in and be a model developer and take over a category potentially. Do you agree?

Vijay Karunamurthy

>> Yeah. I worked at YouTube years ago, and I remember the early days of YouTube, it was exactly that disruptor mindset. You put down a credit card, you get a bunch of servers, you rack them up at Equinix, and suddenly, you're serving video traffic to millions of people around the country. The world today is very different. So these foundation models, they require easily $10 million now, $100 million of investment and data and infrastructure and monitoring and observability in order to just get these models out the door. But then once you leverage those models, there's a range of things you can do just based upon that initial investment that's just really powerful. So we're seeing a lot of firms, and those firms that normally we would consider to be the ones to be disrupted, they're actually leading the way in figuring out how can I build these centers of excellence within my company, have really smart AI engineers, and other stakeholders, maybe an AI ethics officer, collaborating together to see those opportunities forward? So we're seeing a lot of really groundbreaking innovation happening in these regulated industries where data governance has always been part of the question, understanding the lineage of your data, understanding how mature that data is, these are just fundamental parts of how they've done business for the last 10 years. And now they're applying the gen AI stack on top of that.

>> What about the entrepreneurial activity? Where do you see that happening?

Vijay Karunamurthy

>> Yeah, a lot of interesting stuff.

>> Given the high cost to do a model. I mean, to be a consumer model, that's going to be a high bar.

Vijay Karunamurthy

>> Yeah, definitely. To do a consumer model, it requires a lot of investment, so there's only a few players in that space. But there's a range of startups that are trying to do really interesting things based upon agents and other activity, really building to the use case. So if you have a legal use case, let's make a model that talks to lawyers in the language that they need to be talked to. If you have an education use case, we've seen some really interesting investment there in ed tech where talk to students where they are in the journey. If someone doesn't understand photosynthesis, they're a first-year college student, talk to them differently than the way you would someone who's already done organic chemistry and knows everything they're not going to know.

>> Yeah, it's interesting. You could be a domain-specific expert in, say, legal and get a small, little beachhead feature, and then end up aggregating data for the whole vertical.

Vijay Karunamurthy

>> Totally, yeah.

>> Potentially, right? I mean, I'm just making this up, but that could be a trajectory.

Vijay Karunamurthy

>> Data's unbelievably valuable, but maybe the bigger use case for data's actually just evaluating how the models are doing today. Without the right data, you can't figure out how the models are performing, you can't move forward towards a goal. So having the right data about your specific use case, that's worth its weight in gold right now, I think.

>> Yeah, data observability's a hot area. What's your view on that? Are we not there yet? Are some people doing it? Are you guys doing it?

Vijay Karunamurthy

>> Totally. The first thing is you have to do it. You have to have data observability.

>> Now!

Vijay Karunamurthy

>> Your data's changing over time. You're going to introduce bias, You're going to introduce data drift, all these issues if you're not observing your data. Data is the lifeblood of any gen AI application. And applications are changing in how they retrieve data overtime. We're starting to see this really important pattern that we do in financial services where the model writes the SQL and the SQL goes and executes and gets the data, and then it's transformed in a way that's useful to human analysts. You could trade a PowerPoint presentation or a chart or a graph out of that data. You need to understand how that SQL's executing. Does it know about the schema of your database? Is that schema changing over time? If you don't do that, you have this incredibly brittle application that works today and doesn't work tomorrow.

>> What's your big use case in enterprise? If people watching here want to see Scale AI, give you guys a look-over, what's the big use case that you guys are doing the most action with the enterprise?

Vijay Karunamurthy

>> Definitely. SQL's a big one. It's very painful to have a data analytics org be the bottleneck for everyone else in the company. If they want a question about how many customers bought hot dogs on the Fourth of July, and you have to wait three weeks for a data analyst to write the SQL query, that's really painful. And so having models generate SQL and generate accurate and reliable SQL, it empowers the data analyst team. It's a force multiplier for them to do their work a lot better.

>> Yeah. Well, the big models will you who won the Nathan's Hot Dog Eating Contest, for sure. We saw that on TV.

Vijay Karunamurthy

>> Yeah. Was it Kobayashi?

>> We'll sidebar that. Okay, so that's cool. So this is important because this now saves time. So the querying, the SQL thing, is going to be very key.

Vijay Karunamurthy

>> Yeah.

>> What's the fine-tuning equation? Here, a lot of people ask questions. It's the most common question I get is, "What do I do fine-tuning on? My data's already trained. I don't need to... What do I... And where's reasoning and reinforced learning, when does that apply? I don't know when to apply what to the data at what stage of wherever my journey is."

Vijay Karunamurthy

>> Yeah. When you have those questions with a customer, it's important to take a step back and figure out how do you evaluate what success is? And do you have the evaluations in place? We take this approach called hybrid evaluation where you have models that are always going to be important in evaluating other models because they can do things that human beings just can't do at scale. A model can translate your question about the law in German to Swiss German and ask that same question and see if there's a different answer. So that model-assisted evaluation is important. But expert oversight is incredibly important as well, bringing in all this knowledge that you have about the law, or about biology, and bringing that together at scale is incredibly important. But once you get that evaluation firm in place, then you can see what are the really important interventions I could do to make a difference this week? Is it fine-tuning a model? Maybe that takes two to three weeks. Is it adding a new guardrail on top of the model? Maybe I can do that in an hour. Could I just change up the prompts? Without evaluations, you don't know if you're moving towards your goal and you don't know how far away that goal is, but once you do know that, you can figure out what are the different interventions I could do to try to get alignment of how-

>> What's the advice you have for two use cases? One use case is you got a company, it's got a small language model, it's fairly trained, it's unique, so it's okay. What do I do with this model? What do I interact it with? How do I engage another model? And then two, mindset of the organizations who want to bring in this new culture of thinking, systems thinking, prompt thinking, to start the trajectory of culture shift inside their organization.

Vijay Karunamurthy

>> Totally. I mean, most models today in the enterprise are useless without data, and that's why there's this big story about data gravity when it comes to gen AI. You can have a small model and you can deploy it closer to where your data and where that compute lives, that ends up being much more valuable than having a large model that can't access the right data because of access control procedures you have. So we have lots of companies where maybe for some regulatory reason, they have data that sits in a VPC in Stockholm, and they have a separate set of data that sits in a VPC in Berlin, and those two datasets shouldn't be conjoined together in order to answer a question, but you can answer independent questions about each one of them. If you can have a smaller model that lives in those two different locations and is able to do a lot of knowledge and reasoning about what's local to it, that ends up being incredibly valuable and unlocks a lot of opportunities for you as well.

>> So Vijay, what we're talking about here is essentially platform engineering for data.

Vijay Karunamurthy

>> Yeah, absolutely.

>> I mean, is that data engineering that? I mean, is there a category? What's your view on this because this is huge. I mean, this is a huge shift. I mean, it's a platform shift.

Vijay Karunamurthy

>> Totally.

>> I mean, you see that, clearly. Your company's living it, but other companies are just trying to grok platform engineering with Kubernetes and cloud native. Now I'm like, "What? I got to reset my entire data analytics stack? That's not my department."

Vijay Karunamurthy

>> Right, right. And the first thing that gen AI did, even five years ago, was start to unlock the value of unstructured data. Before, if you had research reports, it was really hard to make sense of them or put them in a database and retrieve them. You can unlock all that value. But now we're seeing, even structured data, you're able to unlock all this value by understanding the data silos in your organization and finding ways to answer questions securely against that data. costs, maybe if you're a large bank and you've acquired 15 regional banks in the last 10 years, each of those might store customer data on different databases. Now you can build data pipelines that answer questions about your entire business in a way that you just couldn't possibly do before. So we see a lot of value in that. These data platforms are being built in part because you have this new tool that suddenly unlocks all this value.

>> Yeah, it's interesting. I see a couple of different use cases. We do talk with our research team in theCUBE. A couple scenarios I want to run by and get your reaction. One is here's an environment that is completely not positioned for gen AI. They're just old legacy. They never invested. They outsourced it to the hills. They got some token projects that'll look good. Then you got the companies that are actually doing really well in the modern devops factor, but don't yet have that data connected. And then ones that are actually investing. Okay. Let's talk about the first two. This is where you start getting into, okay, you need a full retrofit. It's like the bar takeover show. It's like, "Okay, shut it down. We're going to reopen in two months," or whatever that is. They have to rearchitect their entire thing because now, they have legacy storage, networking and compute, and the gen AI apps that they want to do, we'll have to train with different characteristics. I might need more reads on this and writes on that, and for instance, training are two different things. You talk about reinforced learning, that's ongoing. That's going to have a different ratio of reads versus writes and compute requirements. And network paths. You said VPCs, I'm going to send only these networks. So you talk about that systems going to be completely flipped upside down. What's your reaction to that? And how should people start thinking about this? Is it a mind-blowing thing, or it's reality, I think. Do you think it's reality? And if so, what do people do?

Vijay Karunamurthy

>> One of the most painful roles to be in right now is doing budgeting and planning for your infrastructure today at a large company because right now, the challenge you have is you have all these different prototypes of gen AI applications, and the roadmap for those carries you out maybe six months, maybe nine months, maybe 12 months, but there's a huge variability in you knowing when is this application going to get in production? What region of the world do I need to deploy this application in? What performance characteristics do I really need to have? Am I using a large model? Am I using a small model? The ML engineers are telling you, "I just don't know yet. I'm going to find out maybe a week before I deploy this, which one of those is better." That's a really challenging spot to be in. So one of the key reasons people use our Scale AI gen AI platform is it's just really simple to one-click deploy that to all of your VPCs, wherever they're located. They all operate under the same security principles. And then you're able to evaluate where you are this roadmap for each one of those applications, and make those assumptions actually much more tangible. Would a small model fit here? Would it give me more reliable answers? Can I get 75% of the way to production just using that small model alone or do I need to sometimes leverage a larger model to answer more complicated questions? And then you can actually do budgeting and planning about where those GPUs need to sit, what sort of top-of-rack configuration do you need? Hopefully, you're on AWS and you're leveraging Bedrock and SageMaker and so that's a little different-

>> All right. So now we're at... Thank you very much for a great conversation. Let's get into this what do you guys do with Amazon? How do they consume it? Are you in the Marketplace, I assume?

Vijay Karunamurthy

>> Yeah.

>> What's the interaction? What's the relationship with them? What's the coolest thing you're doing with AWS?

Vijay Karunamurthy

>> Yeah. Part of our announcement today is a multiyear strategic partnership with Amazon. A lot of Amazon customers that use Bedrock, use SageMaker today, a lot of them have asked for better evaluation of their models so they can assess whether a custom fine-tuned model is working here, whether a large model made from Anthropic or another provider is better suited for that application. They can use our gen AI application engineering platform at just one-click deploy to all of their use cases-

>> In bedrock today?

Vijay Karunamurthy

>> In Bedrock, yeah.

>> Available, I mean, instantly?

Vijay Karunamurthy

>> Yeah, yeah. And everything's billed through your AWS account, so you can use your AWS credits for there and help you with your budgeting and planning hopefully going forward using the insights you got there. But some of the decision-makers that have been asking for this, one is folks that are coming in to a chief AI ethics officer role, or a chief AI officer role. They have a lot of questions over how do I responsibly deploy AI? How safe are the AI applications I have out there?

>> Yeah, can I explain it?

Vijay Karunamurthy

>> Yeah, totally. Yeah. Can I explain why this cybersecurity incident happened against this gen AI application? And it's something really serious. I leaked internal data, but I can't explain why it happened. That's a bad spot to be in. So having access to this application on the Marketplace today, I think that's going to really empower a lot of those stakeholders to do that work.

>> You know, it's interesting. Software supply chain's been a conversation we've been having a lot on theCUBE. And for the first time on a keynote, I heard the words data supply chain, not in reference to lineage, In reference to, "Okay, I got to start tracking the data coming in." Again, you talk about observability challenges. Again, this is just another example. It's a whole new wild, wild world.

Vijay Karunamurthy

>> Yeah, totally. There are all sorts of attacks that could potentially come up through data in the worst-case scenario. And if you're a bank, you have all sorts of foreign governments that want to infiltrate your systems and get information out, so it's important to stay ahead of that. We do a lot of what's called red teaming where we use our platform and we try to think in the way a malicious state actor, maybe North Korea, I won't name other ones but North Korea's a good one, how they would go about hacking a gen AI application, what sort of changes in the prompting or other things would they do to try to get access to the internal data? And then you stay ahead of the game. You really know. We actually, we spent time talking to the US government and we presented to the Homeland Security Committee on red teaming and what it really means because it's important for the US government to spread this knowledge, help protect sensitive data. That's important for all the markets.

>> Yeah. I mean, we're projecting a Cambrian explosion of gen AI apps, the infrastructure, middleware, data, all has to be looked at and rethought through with a mindset with generative AI in mind, right?

Vijay Karunamurthy

>> Totally.

>> It's generating. It's run time.

Vijay Karunamurthy

>> Totally. Yeah, yeah.

>> We've see this movie before in distributed computing. It's called the edge, tactical edge. Vijay, if you could summarize your relationship with Amazon, the news, and what you guys do, what would be the bumper sticker? What would be the soundbite that you think that users should walk away with as a perception of what they want to take away?

Vijay Karunamurthy

>> Yeah. The key part of the strategic partnership is it's really meeting customers where they are. Every customer's on this different journey towards adopting gen AI. Some of them have successfully deployed prototypes into production. Others are really feeling bottlenecked by those trust and safety issues that get out there. So using our platform, it's an easy way to just get observability into where your data sits, how it's informing the answers you're getting out of gen AI, and really what it's going to take to accelerate the adoption of this technology. We're hopeful that, as part of this, customers don't just think that there's a one-size-fit-all answer to gen AI questions or one single data architecture's going to work for everyone. Instead, it's what's the way to get the observability into that? What's working today? And what do I need to invest in?

>> And you guys are going to fit right into the architecture of AWS and all those higher-level services.

Vijay Karunamurthy

>> Totally. Yeah, VPCs, the role-based access control you have in place, EKS, all these other technologies that financial services companies have used. We fit in very well with that in protecting it.

>> As CTO, what are some of the Amazon customers saying about this opportunity?

Vijay Karunamurthy

>> Yeah. It's been huge. Our key customers right now have been financial services, healthcare, insurance, education. All those businesses, they're just transforming. They're going to look very different five years from now than now. So anything you can do to get ahead of that, accelerate that transformation in six months, nine months, is just-

>> It's a land grab. I love this market. Vijay, thank you for coming on.

Vijay Karunamurthy

>> Thank you. Thanks, John. Yeah.

>> All right. Scale AI, making the difference here, changing the game out in Bedrock, bringing observability in. Again, the whole world's going to change. Reinforced learning right around the corner. Stay tuned for more coverage. We're bringing you all the data here on theCUBE. I'm John Furrier, host of theCUBE. Thanks for watching.