theCUBE + NYSE Wired present the East Coast AI Leaders Executive Series | Rebecca Qian, Patronis AI

Clips
More from theCUBE + NYSE Wired present the East Coast AI Leaders Executive Series

Rebecca Qian

Co-Founder & CTO

Patronus AI

play_circle_outline Patronus AI's Mission: Ensuring Trust and Reliability in Accurate AI Models

play_circle_outline Need for use case-specific evaluation and expertise in AI evaluation

play_circle_outline Integrating Traditional Software Engineering Principles for Effective AI Evaluation and Overcoming Data Challenges in Development

play_circle_outline Enhancing Agent Evaluation Plans and Scaling Processes for Future Enterprise Deployments of Gen AI Solutions

play_circle_outline Emphasis on responsible AI development from day one

Info
Transcript

Rebecca Qian, Patronis AI

Rebecca Qian

Co-Founder & CTO Patronus AI

TheCUBE is covering events in New York City, specifically at the NYSE's East Coast CUBE Studio. Patronus AI, led by co-founder and CTO Rebecca, focuses on AI evaluation to ensure system trustworthiness in light of the rapid deployment of AI in various scenarios. With experience from Meta, Patronus AI is addressing issues like model hallucinations and bias in AI outputs. Companies turn to Patronus AI for evaluation before deploying AI systems live. The process involves identifying blind spots and exposing system vulnerabilities through red teaming exercises. O... Read more

explore Keep Exploring

What are some ways in which powerful evaluation can empower enterprises and AI development teams in delivering accurate and reliable AI outcomes? add

What is important to consider when evaluating different use cases for a specific project or product? add

What principles are being applied in computer science and software engineering when integrating Patronus evaluators into alerting systems for security risks? add

What are some key considerations and challenges that need to be addressed in the development and implementation of agentic systems and evaluation at scale in the context of shifting towards gen AI deployments in enterprises? add

What are some benefits of reliable automated evaluations in AI development? add

bolt Powered by CUBE AI

Rebecca Qian, Patronis AI

search

>> Welcome back, everyone to theCUBE's coverage here in New York City. This is our East Coast CUBE Studio here at the New York Stock Exchange. It's our new location. We're opening up the New York area, creating a subnet, a super, super node as we call it, a point of presence access point. We're connecting our Silicon Valley, Palo Alto Studio here with New York creating global community with the NYSE Wired community. Rebecca is here, co-founder and CTO of Patronus AI covered extensively on SiliconANGLE, recently funded, formerly with Meta. Congratulations. Rebecca, thanks for coming on theCUBE.

Rebecca Qian

>> Thank you.

>> Thanks for coming on.

Rebecca Qian

>> Thank you for having me.

>> I love this location. And you're in New York, right?

Rebecca Qian

>> We are in New York.

>> Talk about the company. We'll get into sort of the details, love what you guys do. Talk about the origination and what you guys are working on.

Rebecca Qian

>> Yeah, absolutely. So Patronus is the leading AI evaluation company. So we make sure our alums don't go off the rails. And as you see, ever since ChatGPT was released, people have just been rushing to deploy AI in every scenario imaginable. People keep coming up with new ways to use language models that... As someone who's been working AI research for years I hadn't even anticipated before. It's a lot of creativity. And we just make sure that you can develop AI while still trusting your system and making sure that it's doing what it's supposed to.

>> And you guys started when? About 18 months or so?

Rebecca Qian

>> Yeah.

>> Not even two years yet.

Rebecca Qian

>> Yeah. So March 2023. We publicly launched last fall. So it's coming up about a year.

>> You got modest funding, which is good. So you're disciplined and producing a lot of products. So you, guys are making some real headway. Tell me about what you did before. What got you here? You mentioned Meta. What were you doing at Meta? Because obviously Meta is well-known. They're doing amazing work right now, the open source model. So love the AI mojo at Meta. What were you doing there? How'd that translate into this venture? Because it seems like you just hit the ground running.

Rebecca Qian

>> Yeah. So my co-founder, Anand and I were both at Meta and he worked on AI explainability, looking at this more from a explainability and data science perspective. I was working on the research side of making language models more robust. And so some of the work I did is I led a team that trained the first open source language model with a fairness objective. So making sure that models were not discriminating against protected categories like women and minorities, et cetera, which is super important work and still very relevant in 2024. And then we just saw that models were hallucinating, that they were leaking sensitive information, that they were easy to jailbreak. And we continue to see that and now we're helping enterprises with that.

>> Just on a personal note, that must've been so much fun. I mean, playing with the early versions at scale, I mean, the data that Facebook has is just incredible, data set. So that must've been really rewarding.

Rebecca Qian

>> Yeah, absolutely. I remember the moment ChatGPT was released. I think it was late 2022. The first thing that Anand did was send it to me and we were like, "Okay, let's try to break this thing." We have a lot of examples. So actually when Llama was released, we also had a tweet that went viral, and we've had a number of these where we were just showing model failures and some of those actually went viral. For example, we put out FinanceBench, which was the first large scale financial benchmark, and it had questions and answers from what real world financial analysts would provide. And turns out that GPT and all the LLMs are not very good at it. They have about a sub 50% accuracy. And then so that-

>> That was a great tweet, by the way. That was awesome. Well, this brings us back to today. I mean, now that hype is now two years in. I love the word super cycle. Some people don't like it, but to me, I remember these super cycles, they're a good 10-year changing of the guard. New things are happening and we're just at the beginning of it. I ask people this question a lot in the past year, what does resilience look like in generative AI? Because that concept has been around in computer science mostly around security and servers where it's like, okay, ransomware or recovery, disaster recovery. It's a resilience question. It's a pretty well known storage problem. But when you go to gen AI, if something happens, what's the rollback? What's the recovery? What goes on there? There's not a lot of good answers there yet because no one is actually experienced. Kind of Armageddon version of ChatGPT. So there's a lot of focus on this. How would you describe how a company can be resilient? You got the Mad Money set behind us. Always fun here. Closing bell, Mad Money always going on here at NYC. How would you define resilience as people start to look at it? Because it's paying the bills, it's all the ethics stuff, we say that's all good. But now when you're in the technical computer science phase of this and the real engineering of language models, how they work together, the fusion of different integrations, then you're like, "Wait a minute. How did I get here? Making a meal? Why didn't I write the recipe down?" How would you look at resilience peaks?

Rebecca Qian

>> I love that term resilience. I think that's exactly how people should be thinking about their AI and making analogies to traditional software systems, because we've figured this out in security. For example, man in the middle attacks are now classic computer science security 101 course material. But at some point we didn't have a concept of security and there were a lot of browser-based attacks. Right? Pen testing has become commonplace, but at some point, companies were not shelling out money to undergo pen testing and compliance tests. Right? So we need to approach AI development in the same way. And right now it does feel like the wild west where companies are still so early in their journey towards developing resilient AI systems.

>> Rebecca, that's a great point. I mean, I was at a conference... Mandiant had a conference, mWISE which is now part of Google, but these guys are threat intelligence. They're really hardcore and they love AI, but they're also practically, "It's got to work." just maybe red team agents try to do some cool shit. But at the end of the day they're like, They're like, "Look AI is just another application for us. It's AppSec review. It's different. It's a different app review because it's more going on that's less So it's generative. So how do you guys look at that? You guys are providing that service now and then kind of a way of, "Hey, think of it... about reliability." How do you look at this app, new category of applications from a review standpoint? How do you attack that problem? Can you share your thoughts on that? Because again, it is kind of another application. It's kind of not at the same time.

Rebecca Qian

>> No, absolutely. It's super exciting for us. A lot of us come from a background of AI evaluation research, and we know that these are open research challenges. So not all of it is solved, but we are at the forefront. And like you said, you can use the same mentality of resilience and security, but these are a new set of challenges. And it's fundamentally because LLM inputs and outputs are unbounded. And so you have your network security issues, but you also on top of that have issues of what if the model hallucinates? What if the model outputs something that's toxic or biased? What if it's leaking sense of information? What if it's going off-topic? And this is a very common one, and you might laugh, what if it's recommending a competitor's product? Or what if it's giving you a discount where it shouldn't? Right?

>> Yeah. I mean, this is a data problem. And also just bad LLM, I mean at that point. All right, so talk about your products. I want to get into your company. I'm fascinated by you're taking, I won't say observability. It's not bad word for it. But you're taking more of evaluating the efficacy of the LLMs to achieve an outcome in the application. So whether it's direct LLM integration, how are you guys leaning into the market now from a value proposition? And what are some of the examples of work that you're doing?

Rebecca Qian

>> Yeah, absolutely. So our mission is to empower enterprises and companies, AI development teams to deliver accurate and reliable AI. So in that sense, powerful evaluation enables you to achieve outsize outcomes. So from an outcomes perspective, we've helped companies reduce hallucinations by 91% of their system. We've helped them ship applications and continue to monitor and mitigate failures in live outputs. We've enabled evaluation at scale, mitigating toxic and biased outputs. So all of this enables you as an AI development team to have confidence in your systems than you. If you have customers or users of your product, they also have confidence.

>> All right. So when you engage with an enterprise... By the way, enterprise, we just did a study... Not study. We just was talked about a study with IBM from McKinsey. 1% of enterprises are actually using AI. So it's a huge market opportunity in the enterprise, but also there's workflows and there's domain expertise. And these domain expertise have specific nuances, identity. There's always knobs and switches in these apps. There's weird stuff going on in the app. So you got to know that domain expertise. So how do you look at the process mining, because you can't just throw AI at the enterprise because it needs to be trained, but who's training the enterprise? I mean, every enterprise is different. So beauty is in the eye of the beholder depend upon where you're at. If you're at Citibank, that's not a healthcare company or that's a whole different vibe on what I'm looking at and what I want to lock in on.

Rebecca Qian

>> So what you're touching on is super important and what a lot of people don't realize, which is evaluation is very use case specific, and there's no one size fits all, and it requires a lot of expertise. And so that's where we do have an advantage because we've been doing this for a long time and we are helping companies like Citibank and healthcare companies with their evaluation and they're seeing these outcomes. And that's because we know, okay, so say you're developing a chatbot for customer service. Maybe you care about the chatbot responding in a certain way. Maybe you care about the output being grounded in your internal documents. So it's not giving incorrect information. Maybe you care about the tone of voice. But if you're an education company and you're developing an AI assistant to provide educational information, maybe summarizing content, what you care about is a totally different set of preferences.

>> It might be more entertaining, engaging.

Rebecca Qian

>> Exactly. Engaging us.

>> All kinds of tone.

Rebecca Qian

>> Right. But then going back to the Citibank use case engaging us is probably-

>> Or airline. "I just want to get in my flight. I don't need to be engaged. my booking." I mean, this is a key thing again. So how do you guys advise. Or take me through a typical cool engagement that you guys do? What's happening? Give me the day in the life of what you guys do.

Rebecca Qian

>> Oh, absolutely. So the first step is a company comes to us, they say, "We have an AI system. It can be an agent, a RAG workflow, and we want to ship it to production. And I just don't feel ready to ship it. I don't know what it's going to do."

>> I don't know what I don't know.

Rebecca Qian

>> And testing. "Our people are breaking internally. How do I gain confidence here?" So then we sit down with our team, and this is very specific to their use case. So it's like, "Okay, so here's a set of maybe 10 categories that we care about and we recommend to you. Do you think we should add anything here?" And so sometimes people will say, "Okay, so I don't want it to recommend this competitor, or I have internal, this company policy, or maybe I'm in a regular, for example, healthcare regulated

>> Don't tell people not to do. Don't talk about my competitors. And they'll be specific about that.

Rebecca Qian

>> Right. But we work with them because in some cases they might not realize that there's a blind spot. And so we point out the blind spots too.

>> What is a weird example that... You don't have to name a customer's name, but give me an example of something where the customer was confident but calls you guys in, then you guys expose something, "Whoa."

Rebecca Qian

>> Oh, all the time.

>> Share some examples. You don't have to name customers or that will give it away. But we're trying to understand the scope of the misfires or miss configurations with the AI.

Rebecca Qian

>> Oh, for sure. Happens all the time. It's also called red teaming where we will purposely break their system. And sometimes it can be something as simple as if you rephrase the question and say, "Oh, ignore the system, prompt, the model will then succumb to your attack." So it's very easy to

>> So prompt reject kind of stuff.

Rebecca Qian

>> Yeah, absolutely. We've shown that a lot. And so an example of a company is there's a company developing a finance assistant, and then we are able to catch hallucinations in the assistance outputs. And oftentimes it's really difficult for even a human to figure this out. So we put out this benchmark finance bench rate, and it was based on SEC filings. And when you look at it, even for a human to go and review an SEC filing, it could be like 40 pages. It takes a long time. So our platform, what it does is it produces a score on whether there was a hallucination or not but also produces an explanation. So the human just has to read the explanation and go, "Oh, okay, so this makes sense."

>> It's like finding the needle in a stack of needles. You can look at everything.

Rebecca Qian

>> Yeah, exactly.

>> And use software saying, "Hey, I can sift through all this to get to... I see a pattern in the algorithm saying this is going to be hallucination."

Rebecca Qian

>> Exactly.

>> That's predictive analytics on mess. Is that kind of the concept there?

Rebecca Qian

>> Yeah. So you can think of it as needle in the hay stack. It can just be one word that's wrong or one different... that we actually put out a paper for links where we showed this where even just changing one number, our model can catch that. And we trained proprietary models to catch hallucination, to tech misaligned outputs for different evaluation use cases.

>> Rebecca, it's fascinating. I love this conversation. I mean, this is really groundbreaking part of the industry right now. It's a fun time, but also there's a lot of work to get done. I have to ask you, as you look at this, do you see any patterns around how people are managing their data? Because I can almost imagine that there's some computer science principles that were on the shelf for 20 years that could come off. You mentioned man in the middle and other things. I mean, there's a lot of redo computer science principles coming to the table and new ones. Is data being looked at separately? Because the old concept was, "Hey, give me a pile of data and we'll do some data mining. Or no, it's actually let's segment the data and look at different sets, or what about models talking to each other?" Now, I have a trusted delegation relationship between two models that might complement each other or might not. They could blow up or whatever. But I'm saying not work or fail. What are some of the things that you're seeing around techniques around how to handle the data and then just any kind of computer science post that you can say, "This is coming back or this is-"

Rebecca Qian

>> Yeah, absolutely. I do think a lot of computer science and bread and butter software engineering principles are being applied. For example, we've seen engineering teams integrate Patronus evaluators into their alerting system, and they get integrations and alerts on security risks, which they do with their other non-AI systems as well. We've also seen people build CICD pipelines the same way that you unit test your software. You also want to get a tester of course, right? So absolutely those best practices.

>> So business must be really good then.

Rebecca Qian

>> Well, we always say out here. The journey is 1%. What's the way there?

>> What's your journey been like? You guys got a full team. Again, we covered you on SiliconANGLE. You got a lot of smart people in the company. What's the culture like? What's the journey like right now?

Rebecca Qian

>> It's amazing. It's been amazing to work alongside our team. We have researchers who are really passionate about AI evaluation. We have engineers are really strong in product and engineering, people who have built ML inference, pipelines at SageMaker, for example, or come from really top research labs. Just a really awesome team.

>> What's next now for you guys as you look out in this next second half of the year going into 2025? What's on the goals? What's on your agenda?

Rebecca Qian

>> Yeah. I would say in terms of our roadmap, we are always thinking ahead. So right now we're seeing a shift towards developing agentic system. So a lot of the discussion is how do we support evaluation for agents? So that's a big part of our roadmap. We're also seeing just this need for evaluation at scale, which becomes really challenging from an infrastructure perspective. How do you reduce the latency? How do you actually put in guardrails in place as these companies are shifting to a higher volume deployments of gen AI where like you said enterprise is still only 1% unlocked. For gen AI deployments, right? So when the 99% is ready, we will be there.

>> Yeah, it sounds like you guys got a great team to help these enterprises shepherd them through, get them ready to climb Mount Everest. They're like in the oxygen sea level right now. You got to get them up to the top. On that point, I think enterprises we're seeing in our research, a lot of low-hanging fruit, things they knock down, okay, tickets here, automate this workflow, get a taste for it, get a little wind under their belt. And then when they start thinking about the scope, two things happen. One, they go, "Wow, I got more opportunities with my data." Then they go, "Wait, are we actually set up for it?" And that brings up how do I deal with the data? So the question is, "Do you see a pattern on how successful companies or people that are directionally correct on how they're thinking about making their data horizontally scalable or at least scoped, guard-railed up properly so that it's harmonized with say, the application of the gen AI piece. Is there a technique, data lake, distributed training, distributed inference? Are you seeing any kind of methodology on how they handle governance? Is there a plate emerging that you see is a good direction for data management? Not in the old way, but in the new way?

Rebecca Qian

>> Yeah, absolutely. I would say there's data in your enterprise, and then there is data used for evaluation. It's super important to recognize that evaluation data sets. It's very important to get right. And you do want to work with a third-party because you have a lot of biases when you're constructing that. And so one example is if... A lot of times people will look at benchmark reports to decide if something is ready to be shipped. And then they'll realize the data set has issues with it, or it's not accurate, or it's not a diverse set. And maybe on one day you have a 90% accuracy and then you change the data that you add new subset, suddenly accuracy drops to 50%. And then you wonder what happened. So it's super important to look at your data. It's important to make sure you have good coverage, and it's better to work with a third party because when people are working with the same data themselves internally, they're going to bring their own biases in and they're going to see some of the blind spots in.

>> That's what you guys do. Well, I think the caution is don't rely on benchmarks. It's just the indicator. You got to do your own due diligence. I guess final question for you guys is if people watching or saying, "Hey, I love this rock star team of great people who have been there, done that are now paving the way for the future. I don't yet know what I don't know." What signs in their environment, tell them that they should call you guys? What are the leading indicators of, well, we should call them up. Is it because they're running through a POC? Is there a use case where they could look at their environment and say, "Okay, we should call you guys in now?" What are some of the examples with get me motivated to call you guys when I'm ready? Or is it more of anytime? Or what would be the example of a use case there?

Rebecca Qian

>> Yeah. So I would say, if you care about AI performance, you should call us

>> Check.

Rebecca Qian

>> Yeah. Because a lot of people just don't even realize what you can get out of your system if you have reliable automated evals. So it's really powerful and people just often don't know that they're even leaving performance and accuracy on the table. So that's something that we absolutely come in and help people realize. And sometimes you don't even realize that you would be able to save money on human review or just hours if you just have reliable automated evals. And then in terms of timing, I would say actually the earlier the better because it should be day one from your AI development cycle. So we've worked with companies different stages, and people get the most value when they think about a responsible AI development from day one as opposed to a later on when it's just reactive.

>>

Rebecca Qian

>> Next thing, you're in the news for the Air Canada incident.

>> Rebecca, great to chat with you. I'm so excited about your opportunity, and again, congratulations on your great team. Is there a culture that you would describe your venture? I mean, obviously you're on a great area, super exciting. I can see how rewarding to be to work there. What's the culture like there? I mean, every company has got the culture. They wake up and they eat machine learning for breakfast. What do you guys do? What do you guys do if you just had to describe your culture? I mean, everyone's got a cadence. There's always a DNA, and adventure. How would you describe your company's DNA?

Rebecca Qian

>> So we definitely ate machine learning for breakfast, lunch and dinner. Very much in our DNA. So we internally always say the best is yet to come. It's on our blogs. We've gone to karaoke and sang the song. So the best is yet to come.

>> Awesome. Well, congratulations. We'll see you soon. Thanks for joining us on our CUBE and NYSE Wired collaboration here as we open up our New York presence. And we'll certainly see you again and we'll keep covering you guys and we'll maybe see it on theCUBE interview again.

Rebecca Qian

>> Oh, thank you.

>> All right, thank you.

Rebecca Qian

>> This was fun.

>> All right. I'm John Furrier. Wrapping up day two of our media week. We're bringing in all the live action here on the floor. We had the bell kicked off, Mad Money people screaming down there. We had a lot of the AI leaders on the East Coast here. We also had a little space conversation with space event going on here at NYSE. And tomorrow is Venture Capital Private Equity Week. We'll dig into some of how AI is changing the landscape of returns. I'm John Furrier, your host of theCUBE. Thanks for watching.