Snowflake Data Cloud Summit 2024 | Shishir Mehrotra | Coda

Clips
News
More from Snowflake Data Cloud Summit 2024

Shishir Mehrotra

Co-Founder & CEO

Coda

Unlocking AI’s true potential: How Coda Brain tackles AI hallucinations and inaccuracies

In addition to opening a world of possibilities, ChatGPT’s emergence brought to bear several underlying issues with artificial intelligence.Those issues, which could range from data hallucination to bias, can severely affect enterprise operations. Preventing such an eventuality by streamlining data ingestion is Coda Project Inc.’s value proposition.Coda’s Shishir Mehrotra discusses Coda Brain’s approach to tackling AI hallucination.“Packs are the kind of extensions in Coda — they allow you to bring any other piece of information into Coda as well,” said Shishir Mehrotra (pictured), co-founder and chief executive officer of Coda

play_circle_outline The Power of Coda: Unlocking Efficiency in Document and Application Management for Companies

play_circle_outline Introduction of Coda Brain, a product developed in partnership with Snowflake Ventures

play_circle_outline Unlocking Efficiency: The Integration of Snowflake Cortex and Coda Packs in Coda Brain's Architecture

play_circle_outline Use of Snowflake as the underlying data warehouse for Coda Brain, offering a new data architecture solution

play_circle_outline Overcoming Data Integration Challenges: Leveraging Coda Brain for Confident, Informed Decision Making

Info

Shishir Mehrotra | Coda

Shishir Mehrotra

Co-Founder & CEO Coda

search

Rebecca Knight

>> Hello, everyone, and welcome back to theCUBE's Live coverage of the Snowflake Data Cloud Summit here in San Francisco, California. I'm your host Rebecca Knight, sitting alongside my co-host and co analyst, co-founder of theCUBE, Dave Vellante. Dave, it has been roughly a year and a half since ChatGPT was unleashed in the world and average consumers really could grasp what AI looks like and feels like, but we're still really now still defining the difference between consumer AI and enterprise AI.

>> It's true. I mean, the ROI on consumer AI is very clear. If you can build a bigger GPU cluster, you can sell more ads. Enterprises have to be a little bit more cautious. As I often say, they're hitting a lot of singles, meaning they're doing some nice little productivity hits, but the really big payback we haven't seen yet.

Rebecca Knight

>> Right. Indeed, we haven't. So, with that, I'd like to welcome our next guest to the show. His name is Shishir Mehrotra. He is the co-founder and CEO of Coda. Thank you so much for coming on theCUBE.

Shishir Mehrotra

>> Thanks for having me.

>> Good to see you, man.

Shishir Mehrotra

>> Yeah.

Rebecca Knight

>> So, why don't you start by telling our viewers a little bit about Coda?

>> And why did you start the company?

Rebecca Knight

>> Yes, exactly.

>> I always have to ask that question.

Shishir Mehrotra

>> Sure, sure.

Rebecca Knight

>> Love to have founders on.

Shishir Mehrotra

>> Yeah, happy to chat about Coda. So, maybe as context for everybody, we started Coda about nine years ago now. We build an all-in-one doc. It combines the best parts of documents, spreadsheets, presentations, applications, all into one single surface. And the promise is anyone can make a doc as powerful as an app. It's used by about 50,000 teams around the world, a few million users. Everything from taking notes in meetings to running huge applications. And why did we start it? We started because my view is that the line between documents and applications is very artificial. And if you look at most companies, they have all these applications on one side, but they spend all their time in documents, spreadsheets, slide decks, so on, and those tools weren't built for that. So, we thought we should start from scratch, build a new surface.

>> So, we run our company on what you just described. So, how could you help us get out of this hell that we're in?

Shishir Mehrotra

>> It's interesting, I think first off, when I started the company, everybody's first question was, "Do we actually need a new documents, spreadsheets?" And I think it's maybe not obvious to everybody that we've been living with the same model of thinking of those tools since the 1970s. Really, VisiCalc, WordStar, Harvard Graphics, that's what we're working with still. We've just advanced them through different UIs, from green screens to Windows, to mobile UIs to web and so on. But this idea of start from scratch, build a single surface I think totally changes how teams work. And there's a lot of different ways people use it, but the starting point is if you put it all in one place, you get this one home for your team to operate out of. No more copying and pasting it in different places, no more thousands of tabs to go look at. And then, people start building what we call their operating systems out of it, and run planning and decision making and your sales process and you do all your systems in it. So, it's really quite fun.

>> I'm really intrigued because we have a pretty complicated workflow. I mean, it's not rocket science, but when you think about when somebody wants to bring us to an event, there's travel that goes into a spreadsheet, there's teams that go there, there's all kinds of stuff. And we're jumping around, we're doing it now, we're jumping from the spreadsheet-

Shishir Mehrotra

>> Oh, I checked in on the spreadsheet, you sent me my questions in the doc, all those things are completely disconnected.

>> I go back to the PowerPoint with the show prep. And so, you would integrate all those into a single application, and so I don't have to context switch and go down to cul-de-sac and come back out?

Shishir Mehrotra

>> Yeah. And one of the key things and it sort of led to why we're here as well, is we actually not only integrate those surfaces, but we build a platform we call Packs. And so, Packs are the kind of extensions in Coda. They allow you to bring any other piece of information into Coda as well. And so, there's a few hundreds of these that are publicly available, and then thousands more of them have been built inside companies. So, you can take Coda and you can put not only your docs, spreadsheets and so on, but all your data from Salesforce or Jira or from your task-tracking systems or your analytics system. And you can pull it all into one surface that all the knowledge workers in your company can use to build data-informed decisions, meetings, whatever they need to do.

>> You know who I thought would have built something like this is Steve Jobs, seriously.

Shishir Mehrotra

>> Opened up.

>> All in one, right? But no.

Shishir Mehrotra

>> I mean think he tried. OpenDoc was definitely a path in that direction. I think there's something about this timeframe that people are all of a sudden comfortable. I think a lot of it comes from the web is that everybody who tried in the past, Jobs tried. There's actually a similar product at Microsoft at one point, tried to build a single document format. And all of them suffered from before you moved to the web, it's just very hard to make that work.

Rebecca Knight

>> So, it sounds as though this is a moment. And I know that Snowflake Ventures has recently invested in Coda. Tell us a little bit about that partnership and how you can use Coda Brain to help your customers turn data into action.

Shishir Mehrotra

>> Yeah, so I'm really excited about it. We just announced it here today. We're building a new product, it's called Cobra Brain. I call it the Coda-Snowflake-Coda sandwich. So, we're building it mostly with the Snowflake Cortex team. Sridhar talked it a little bit about in the keynote yesterday. And it came from a fun discussion I was having with the Snowflake team, Christian, Sridhar, so on, about how enterprise AI and consumer AI are different. And this is sort of where you started. And what we realized was we had a shared vision of how we're going to tackle that difference. In the consumer AI world, we're all used to this wonderful assistant, many of us learned on ChatGPT. It just feels amazing. You can talk to this, it feels like a person, it's everything on the internet.

Rebecca Knight

>> Kind of a dorky person, but yes.

Shishir Mehrotra

>> Kind of a dorky person. I'd like to call it an incredibly over enthusiastic intern.

Rebecca Knight

>> Who sometimes messes up, but yeah.

Shishir Mehrotra

>> Sometimes messes up, knows a lot about a lot of things, but nothing about you. And so, you take that and everybody's vision, you talk to any customer, you say, "What do you want in your company this year?" "I want that. I want ChatGPT for my company." Everybody wants the same thing. And then, you realize that, oh, actually the steps in doing that are not really just about building bigger models and so on. There's actually a bunch of really hard challenges. For example, when I come and say, "What's our vacation policy?" I expect it to know our vacation policy, not somebody else's. When I come and say, "Give me a list of customers that ordered with us last year or have spent more than this much money," I expect a tabular result with a data source. Probably the most fun example is I come in and I say, "Who's getting promoted next week?" I want everybody to get slightly different answers. And to build something like that, you have to start with a different approach. And so, that's what we're building. It's called Coda Brain. The sandwich is basically, we took Coda Packs because that's the basis of this system is that's the brain. So, you come to Coda Brain, either the admin or the user goes and sets up whatever systems you want to integrate into the Coda Brain. And one of the key things we do is we preserve all the permissions, so we know who has access to every different entity. Then in the middle stack is the Snowflake Cortex stack, and we basically use all of it. So, it's Cortex search, it's the Cortex Analyst, how they announced it today. And then, on top of that, we've built a RAG-based chatbot where you just come and ask questions and it gives you back responses.

>> And you're using, what, Snowflake's embedded vector database or-

Shishir Mehrotra

>> Yeah, that's right....

>> or are you getting access to all this stuff early?

Shishir Mehrotra

>> That's right. Yeah, we've been pushing the limits on all those different parts. If you think about the challenges here, the way it works, you come to Coda Brain, you ask a question. It's a RAG-based model. So, if people aren't familiar with the term, it's a multi-step process where you do retrieval augmentation generation. The hard part of retrieval starts with I need to pick a data source, I need to go check permissions. So, we use Snowflake Cortex search to pick the set of records. And then, just the underlying role-based ACL control to filter out what do you have access to and what do you not have access to? And then, we take that, we hand it to the LLM and we generate a response. We also do a whole other path. If you ask a question that we determine to be a structured question, we'll return you a table. So, we'll return you a live result from your Salesforce data set or Jira data set, whatever it might be. And we use Cortex Analyst to go generate the SQL for it. One of the neat tricks we do is we'll not only return the data, we'll return the filters in a way that you can understand them, so that you can have confidence in the data that was handed back to you. So, all of it is working together with... I think we use every part of the tech stack.

Rebecca Knight

>> So, what are you hearing from customers? What's the early customer... We know that Dave Vellante is intrigued, but-

>> I've got more questions.

Rebecca Knight

>> What else are you hearing?

Shishir Mehrotra

>> I mean, at a high level, that's the dream. So, everybody starts with first question is, "Can it understand my data?" All of it is not useful unless it integrates with my data and we integrate with about 500 different data sources, and then it's easy to add thousands more. So, everybody's got their list of how they want to do that. Next thing I want to know is can it handle both unstructured and structured questions? And it just turns out we don't ask that of ChatGPT that much, but inside a company, you often want a list, you want a data set, you want a chart. And so, I think that's really important. Probably, the next thing we get asked is permissions is how do you handle permissions? Almost everybody I've talked to has had some team that went off and built a RAG-based chatbot. I mean they even did it on stage in a couple of minutes.

>> I can show you theCUBE AI.

Shishir Mehrotra

>> Yes. And it's very cool, but the hard part is almost everybody builds one, and they either start with a really small data set that is shared with the whole company, it's your brand guidelines or your support book or so on, and then they pointed at something else and it leaks information. And now, somebody asks the question, and they shut the whole project down.

Rebecca Knight

>> It feels risky.

Shishir Mehrotra

>> It feels very risky. And so, I think that idea of I want a chatbot that actually understands my permissions is really challenging. And then, the other thing we're hearing a lot is, "Can you take that and not just answer questions but actually take action with what comes back?" And one of the big things we've done is you can take any result from Coda Brain and you can add it to a doc, and it immediately becomes a live query. And it's a normal table. You can go add columns to it. We have a bunch of capabilities in Coda where you can construct what we call buttons, which can take action. So, you can say a simple example I got from a customer was, "I want a list of users in every account. And as soon as some score gets above a certain number, I want to auto invite them to a webinar." And now, I just add the column, I add a button, I create what in Coda we call automation. And you can finish the journey of not just an assistant that helps me answer questions, but an assistant that actually takes a tedious task to make work out of my job.

>> So, obviously, you got a data platform here. There's a little bit of RPA-ish automation going on.

Shishir Mehrotra

>> Yeah, we get compared to RPA, no code, low code. We hear lots of those comparisons.

>> Okay. So, you said 500 data sources. So, again, coming back to me. So, emails, I got Google Sheets, Google Docs, I got all kinds of spreadsheets all over the place. Presumably, those are ground balled to second base.

Shishir Mehrotra

>> Right, that's the starting point. Yeah.

>> Right, okay. And so, what do I need? I need Snowflake. I need a AWS account, which I got. I need-

Shishir Mehrotra

>> So, we actually-

>> Do I not need Snowflake?

Shishir Mehrotra

>> It runs on Snowflake, but actually one of the interesting parts is we can use your Snowflake instance or we can use our own. So, that's actually quite interesting to many customers as I'm trying to get started here. I think that's particularly interesting to the Snowflake team, because one of the core insights behind this project is we'll take the data warehouse we create, the Coda Brain backend, and we'll actually make it available to your data team. So, your non-technical knowledge workers will come to this Brain UI. They can ask any question they like, but we'll actually take the underlying data warehouse and make it available to your data and developer teams as well. And the reason that's so important is I think that basically every company has to completely rebuild their data backends for being ready for RAG. And to do a good RAG-based chatbot, which is what everybody wants, you'd need your data warehouse to understand things at an entity level. It needs to be pre-encoded with embeddings, and it needs all the ACLs on every row, and that needs to be kept up to date on a constant basis. And nobody's data architecture starts with that assumption. So, part of the reason I think the Snowflake team is so excited about this project, they're very excited about any application of Cortex, but the other thing that happens is we show up, we say, "Actually we're to build you a new data warehouse is ready for your developer team to actually go build any chatbot or anything else you want to on that."

>> So, I'm trying to think of how I would do this without Coda. I could use a document database, maybe.

Shishir Mehrotra

>> Yeah. So, the hard part is you have to build every integration and synchronize for every system... So, I'll give you an example. So, if you want to synchronize in your support docs, that's straightforward. You take a bunch of PDFs, you load it to a column in Snowflake, straightforward. If you go to a system like Google Drives, you gave the example Google Docs, Google Sheets, so on, they actually have one API that's called the Meta API, where you can say, "Give me all the data, including the ACLs." You want to do the same thing with Salesforce. It's not very easy to do because in Salesforce you can come in and you can write permissions that say, "Reps and this team can only see orders that are less than a million dollars and non-government accounts," and blah, blah, blah. How are you going to know that that's a set of orders that can go to that set of people? So, you have to build every one of those connectors and then keep them up to date. So, I think what we're going to see is a lot of people are going to come in, they're very excited about it. Everybody wants ChatGPT for your company. Easy to train it on the universally-available data. Then, you'll go build some for some relatively easy to synchronize data sources that are very large, but then as you get to the tail, it's very hard to go build those things with the permission model in mind.

Rebecca Knight

>> So, hallucinations though, which are a huge headache, have you found ways to mitigate that?

Shishir Mehrotra

>> I think the hallucination problem, if we think it's big in the consumer world, it's enormous in the enterprise world. In the consumer world, honestly, we all kind of tolerate it. It's even kind of fun sometimes, there's some use cases for it. The enterprise world is completely not tolerable. So, there's a couple of things about hallucinations. First off, if you picture the Coda Brain UI, you come, you ask a question, we'll generally either give you an unstructured result or a structured result. We'll return you a paragraph, a sentence says, "Here's what we think," or return you a table. Each case how to deal with hallucinations has a slightly different approach. So, when you hand back an unstructured result, the typical way to handle that is citations. And I think the Cortex team has done a really nice job of interleaving how... One of the hard parts of the generation phase of RAG is you take all of these different documents, you pull back sometimes a multi-paragraph answer. And in each part, you don't just want, "Here's the multi-paragraph answer and here's all the data sources." You want it interleaved. "This one sentence came from here. This sentence came from here." That's really important. I think the Cortex team has a nice job of that and we plumbed that through. The other side if I hand you a structured data set, you want some confidence that you got the query right, you got the filter right. You understand where it's coming... I was talking to a customer earlier where the query they wanted was, "Can you give me all customer feedback about this feature?" Fairly reasonable thing to go ask. And it came back with a list, said, "That doesn't seem right." And you could immediately, in the Coda Brain UI will hand you back the data set, but we'll take the SQL, but will translate back to active filters, so you can go and change them. And they say, "Oh, I understand. It thought that it should look at this column, but it's actually wrong." And they went and changed. They asked again and they got a slightly better version, got better and better and better. I think it's you take hallucinations, you turn it to your advantage. "Explain to me what you're doing and let me go help you fix it." And so, that I think is the answer to how to handle hallucinations in the enterprise. The nice part is that usually in the enterprise there's a truth. So, if you can hand people where you came from, it's unstructured, here's my citation, it's structured, here's my query, here's my filters. I think we have a better answer-

Rebecca Knight

>> But it takes the human understanding that that's not quite right.

Shishir Mehrotra

>> Yeah, the human is part of the loop. We're not trying to get rid of the human. We're trying to enable the human. And same problem if you went to an analyst and you said, "Hey, can you give me the list of customers who asked about this feature?" Your analyst may misunderstand you too. So, I think this idea of you got back a report like, "That can't possibly be right." And you say, "Oh, you misunderstood me. I meant this feature. Look for this thing." That's I think what you want. And I think this idea of can we iterate with the user and the AI system? That's the loop I think you want for a product like this.

>> So, obviously, this is really interesting. And so, basically, like you said, everybody's built a RAG. It's kind of getting trite. They're not getting value out of it. So, how do you think this will change the ROI equation in enterprise?

Shishir Mehrotra

>> I would say most RAG applications I've seen so far don't get ROI because they get shut down, they don't reach scale. Because of this issue with data sources is I think the Snowflake team has made it so easy to do the middle of the sandwich. They've made it so easy to go and come up with the great embeddings, do this hybrid ranking model that Cortex is doing, has outperformed every benchmark we can come up with. The analyst can give you this text-to-SQL thing. But if you start with this tiny set of data, then the chances you're going to build something that's actually useful to your company are just not that obvious. I mean, the number of companies I've seen we built a RAG bot trained on our brand guidelines. It's like what ROI are you going to get out of, how many times does somebody need to know what are our brand guidelines? So, you have to get to the harder data sets. I need to get to Salesforce, I need to get to your email. I need to get to Slack. I need to get to our tickets. I need to get to our HR system. I need to be able to come and say, "How many employees left because of this reason?" Or, "How many employees did we hire with this background?" Then you get real value out of it. The second thing I'd say is everybody's focused too much on just the answer part. Now, I've got an answer, but I need to take that and repeat that process over and over and over again. And then, you need to connect it to a surface that allows you to take action, allows you to turn that into a repeatable thing, not a one-time thing. So, I think that's how we're going to get ROI at of RAG.

>> What about video?

Shishir Mehrotra

>> I mean video is a data source like everything else. I mean, this is a great example of where I think the work that I'm just so excited about, the pace at which the Cortex team is creating embedding models for anything. And so, this idea of can you chunk video? Can you get transcription out of it? Can you embed it so on? I mean it seems like almost every week there's a new thing the Cortex team has come up with on what they can do there.

>> But the previous problem you talked about, if I understand it, it's like having a consistent answer over time. One of the challenges we have, love to have you help us solve this is things change over time. We're ingesting video, transcribing that video. And then, we want to ask questions, "What were the top 10 takeaways from Snowflake Summit in 2024?" Boom. And then, if you ask that question, it's going to mix '23, '24. It might mix 2015 for all we know. It'll do that sometimes.

Shishir Mehrotra

>> Yeah, and the starting point of that is in video... And video is very hot, but you guys are a very specific example with the show. I think the one we see all the time is meeting recordings, and everybody wants their version of a chatbot over meeting recordings. I would say probably the most obvious example. And it's tricky because actually ingesting video and getting transcription is mostly a solved problem. At this point, that's not that hard. The hard part is who's supposed to have access to that? So, I should be able to come and say, "In how many meetings did we talk about Cortex?" I should get an answer, but only the meetings I was in and only the meetings I should have access to. And so, I think that's really the challenge of building an end solution. But interestingly, I think the core part of can I transcribe video and turn it into text and turn it into embeddings and can I rank it and can I do a vector search? That technology all exists. It's just packaging around it that hasn't-

>> But that's a confined data set, right? Imagine if you had to... Well, maybe you can solve this problem. Assume you have access to 1,000 meetings and you want to know, okay, what happened in those 1,000 meetings?

Shishir Mehrotra

>> That's what most companies are. These days, you record everything. Every Zoom meeting is recorded. It's not that confined. I mean, every day there's thousands of new meetings generated with thousands of different attendees. And in some cases, I want everybody to have access. And some cases I only want-

>> But I want the insights across meetings.

Shishir Mehrotra

>> That's right.

>> Right?

Shishir Mehrotra

>> That's right.

>> And you can help me do that.

Shishir Mehrotra

>> That's right. So, a meeting's a document just like everything else. And you go and you say, "This is who had access." Default policy is the people who are in it. And almost every platform, Zoom and so on, have some way to go change those ACLs. And so, we ingest Zoom recordings and we do this inside Coda all the time.

>> What were the biggest gripes that came out this week? And what are the best ideas that came out this week in all of our meetings? And you can solve that problem.

Shishir Mehrotra

>> For sure. I mean, I think the gripes tell me... Sales conversations are a really common one. "Tell me how many sales calls talked about feature X or talked about this conversation." But yeah, those are all-

>> And I don't need Snowflake, you're saying.

Shishir Mehrotra

>> So, you can bring it on Snowflake or you can use ours.

>> Okay. So, I do need Snowflake.

Shishir Mehrotra

>> Yeah, we run entirely on Snowflake.

>> Okay, 100%. Then, I misunderstood before.

Shishir Mehrotra

>> Yeah, sorry. I just meant that you don't have to already be using Snowflake.

>> No, got it.

Shishir Mehrotra

>> And I think part of the reason that's so exciting is we become a path to, I don't know if I want to be on Snowflake yet, then this is a path to say, "But I need a RAG-based backend. I'm going to use Cobra Brain to fill my Snowflake-"

>> So, I see Coda Brain, I don't see Snowflake?

Shishir Mehrotra

>> Until you ask for it, and then we'll give you the backend and you can query it directly.

Rebecca Knight

>> Shishir, you definitely have theCUBE's attention. So, thank you so much for coming on the show.

>> You got to go already?

Rebecca Knight

>> I know.

Shishir Mehrotra

>> Ran out of time?

Rebecca Knight

>> Yeah, this is almost a sales call here. So, there you go. Thank you so much for coming on theCUBE. Really appreciate it.

Shishir Mehrotra

>> Yeah.

Rebecca Knight

>> I'm Rebecca Knight for Dave Vellante. Stay tuned for more of theCUBE's live coverage of the Snowflake Data Cloud Summit. You're watching theCUBE, the leader in enterprise tech news and analysis.

Shishir Mehrotra | Coda

search

Rebecca Knight

>> Hello, everyone, and welcome back to theCUBE's Live coverage of the Snowflake Data Cloud Summit here in San Francisco, California. I'm your host Rebecca Knight, sitting alongside my co-host and co analyst, co-founder of theCUBE, Dave Vellante. Dave, it has been roughly a year and a half since ChatGPT was unleashed in the world and average consumers really could grasp what AI looks like and feels like, but we're still really now still defining the difference between consumer AI and enterprise AI.

>> It's true. I mean, the ROI on consumer AI is very clear. If you can build a bigger GPU cluster, you can sell more ads. Enterprises have to be a little bit more cautious. As I often say, they're hitting a lot of singles, meaning they're doing some nice little productivity hits, but the really big payback we haven't seen yet.

Rebecca Knight

>> Right. Indeed, we haven't. So, with that, I'd like to welcome our next guest to the show. His name is Shishir Mehrotra. He is the co-founder and CEO of Coda. Thank you so much for coming on theCUBE.

Shishir Mehrotra

>> Thanks for having me.

>> Good to see you, man.

Shishir Mehrotra

>> Yeah.

Rebecca Knight

>> So, why don't you start by telling our viewers a little bit about Coda?

>> And why did you start the company?

Rebecca Knight

>> Yes, exactly.

>> I always have to ask that question.

Shishir Mehrotra

>> Sure, sure.

Rebecca Knight

>> Love to have founders on.

Shishir Mehrotra

>> Yeah, happy to chat about Coda. So, maybe as context for everybody, we started Coda about nine years ago now. We build an all-in-one doc. It combines the best parts of documents, spreadsheets, presentations, applications, all into one single surface. And the promise is anyone can make a doc as powerful as an app. It's used by about 50,000 teams around the world, a few million users. Everything from taking notes in meetings to running huge applications. And why did we start it? We started because my view is that the line between documents and applications is very artificial. And if you look at most companies, they have all these applications on one side, but they spend all their time in documents, spreadsheets, slide decks, so on, and those tools weren't built for that. So, we thought we should start from scratch, build a new surface.

>> So, we run our company on what you just described. So, how could you help us get out of this hell that we're in?

Shishir Mehrotra

>> It's interesting, I think first off, when I started the company, everybody's first question was, "Do we actually need a new documents, spreadsheets?" And I think it's maybe not obvious to everybody that we've been living with the same model of thinking of those tools since the 1970s. Really, VisiCalc, WordStar, Harvard Graphics, that's what we're working with still. We've just advanced them through different UIs, from green screens to Windows, to mobile UIs to web and so on. But this idea of start from scratch, build a single surface I think totally changes how teams work. And there's a lot of different ways people use it, but the starting point is if you put it all in one place, you get this one home for your team to operate out of. No more copying and pasting it in different places, no more thousands of tabs to go look at. And then, people start building what we call their operating systems out of it, and run planning and decision making and your sales process and you do all your systems in it. So, it's really quite fun.

>> I'm really intrigued because we have a pretty complicated workflow. I mean, it's not rocket science, but when you think about when somebody wants to bring us to an event, there's travel that goes into a spreadsheet, there's teams that go there, there's all kinds of stuff. And we're jumping around, we're doing it now, we're jumping from the spreadsheet-

Shishir Mehrotra

>> Oh, I checked in on the spreadsheet, you sent me my questions in the doc, all those things are completely disconnected.

>> I go back to the PowerPoint with the show prep. And so, you would integrate all those into a single application, and so I don't have to context switch and go down to cul-de-sac and come back out?

Shishir Mehrotra

>> Yeah. And one of the key things and it sort of led to why we're here as well, is we actually not only integrate those surfaces, but we build a platform we call Packs. And so, Packs are the kind of extensions in Coda. They allow you to bring any other piece of information into Coda as well. And so, there's a few hundreds of these that are publicly available, and then thousands more of them have been built inside companies. So, you can take Coda and you can put not only your docs, spreadsheets and so on, but all your data from Salesforce or Jira or from your task-tracking systems or your analytics system. And you can pull it all into one surface that all the knowledge workers in your company can use to build data-informed decisions, meetings, whatever they need to do.

>> You know who I thought would have built something like this is Steve Jobs, seriously.

Shishir Mehrotra

>> Opened up.

>> All in one, right? But no.

Shishir Mehrotra

>> I mean think he tried. OpenDoc was definitely a path in that direction. I think there's something about this timeframe that people are all of a sudden comfortable. I think a lot of it comes from the web is that everybody who tried in the past, Jobs tried. There's actually a similar product at Microsoft at one point, tried to build a single document format. And all of them suffered from before you moved to the web, it's just very hard to make that work.

Rebecca Knight

>> So, it sounds as though this is a moment. And I know that Snowflake Ventures has recently invested in Coda. Tell us a little bit about that partnership and how you can use Coda Brain to help your customers turn data into action.

Shishir Mehrotra

>> Yeah, so I'm really excited about it. We just announced it here today. We're building a new product, it's called Cobra Brain. I call it the Coda-Snowflake-Coda sandwich. So, we're building it mostly with the Snowflake Cortex team. Sridhar talked it a little bit about in the keynote yesterday. And it came from a fun discussion I was having with the Snowflake team, Christian, Sridhar, so on, about how enterprise AI and consumer AI are different. And this is sort of where you started. And what we realized was we had a shared vision of how we're going to tackle that difference. In the consumer AI world, we're all used to this wonderful assistant, many of us learned on ChatGPT. It just feels amazing. You can talk to this, it feels like a person, it's everything on the internet.

Rebecca Knight

>> Kind of a dorky person, but yes.

Shishir Mehrotra

>> Kind of a dorky person. I'd like to call it an incredibly over enthusiastic intern.

Rebecca Knight

>> Who sometimes messes up, but yeah.

Shishir Mehrotra

>> Sometimes messes up, knows a lot about a lot of things, but nothing about you. And so, you take that and everybody's vision, you talk to any customer, you say, "What do you want in your company this year?" "I want that. I want ChatGPT for my company." Everybody wants the same thing. And then, you realize that, oh, actually the steps in doing that are not really just about building bigger models and so on. There's actually a bunch of really hard challenges. For example, when I come and say, "What's our vacation policy?" I expect it to know our vacation policy, not somebody else's. When I come and say, "Give me a list of customers that ordered with us last year or have spent more than this much money," I expect a tabular result with a data source. Probably the most fun example is I come in and I say, "Who's getting promoted next week?" I want everybody to get slightly different answers. And to build something like that, you have to start with a different approach. And so, that's what we're building. It's called Coda Brain. The sandwich is basically, we took Coda Packs because that's the basis of this system is that's the brain. So, you come to Coda Brain, either the admin or the user goes and sets up whatever systems you want to integrate into the Coda Brain. And one of the key things we do is we preserve all the permissions, so we know who has access to every different entity. Then in the middle stack is the Snowflake Cortex stack, and we basically use all of it. So, it's Cortex search, it's the Cortex Analyst, how they announced it today. And then, on top of that, we've built a RAG-based chatbot where you just come and ask questions and it gives you back responses.

>> And you're using, what, Snowflake's embedded vector database or-

Shishir Mehrotra

>> Yeah, that's right....

>> or are you getting access to all this stuff early?

Shishir Mehrotra

>> That's right. Yeah, we've been pushing the limits on all those different parts. If you think about the challenges here, the way it works, you come to Coda Brain, you ask a question. It's a RAG-based model. So, if people aren't familiar with the term, it's a multi-step process where you do retrieval augmentation generation. The hard part of retrieval starts with I need to pick a data source, I need to go check permissions. So, we use Snowflake Cortex search to pick the set of records. And then, just the underlying role-based ACL control to filter out what do you have access to and what do you not have access to? And then, we take that, we hand it to the LLM and we generate a response. We also do a whole other path. If you ask a question that we determine to be a structured question, we'll return you a table. So, we'll return you a live result from your Salesforce data set or Jira data set, whatever it might be. And we use Cortex Analyst to go generate the SQL for it. One of the neat tricks we do is we'll not only return the data, we'll return the filters in a way that you can understand them, so that you can have confidence in the data that was handed back to you. So, all of it is working together with... I think we use every part of the tech stack.

Rebecca Knight

>> So, what are you hearing from customers? What's the early customer... We know that Dave Vellante is intrigued, but-

>> I've got more questions.

Rebecca Knight

>> What else are you hearing?

Shishir Mehrotra

>> I mean, at a high level, that's the dream. So, everybody starts with first question is, "Can it understand my data?" All of it is not useful unless it integrates with my data and we integrate with about 500 different data sources, and then it's easy to add thousands more. So, everybody's got their list of how they want to do that. Next thing I want to know is can it handle both unstructured and structured questions? And it just turns out we don't ask that of ChatGPT that much, but inside a company, you often want a list, you want a data set, you want a chart. And so, I think that's really important. Probably, the next thing we get asked is permissions is how do you handle permissions? Almost everybody I've talked to has had some team that went off and built a RAG-based chatbot. I mean they even did it on stage in a couple of minutes.

>> I can show you theCUBE AI.

Shishir Mehrotra

>> Yes. And it's very cool, but the hard part is almost everybody builds one, and they either start with a really small data set that is shared with the whole company, it's your brand guidelines or your support book or so on, and then they pointed at something else and it leaks information. And now, somebody asks the question, and they shut the whole project down.

Rebecca Knight

>> It feels risky.

Shishir Mehrotra

>> It feels very risky. And so, I think that idea of I want a chatbot that actually understands my permissions is really challenging. And then, the other thing we're hearing a lot is, "Can you take that and not just answer questions but actually take action with what comes back?" And one of the big things we've done is you can take any result from Coda Brain and you can add it to a doc, and it immediately becomes a live query. And it's a normal table. You can go add columns to it. We have a bunch of capabilities in Coda where you can construct what we call buttons, which can take action. So, you can say a simple example I got from a customer was, "I want a list of users in every account. And as soon as some score gets above a certain number, I want to auto invite them to a webinar." And now, I just add the column, I add a button, I create what in Coda we call automation. And you can finish the journey of not just an assistant that helps me answer questions, but an assistant that actually takes a tedious task to make work out of my job.

>> So, obviously, you got a data platform here. There's a little bit of RPA-ish automation going on.

Shishir Mehrotra

>> Yeah, we get compared to RPA, no code, low code. We hear lots of those comparisons.

>> Okay. So, you said 500 data sources. So, again, coming back to me. So, emails, I got Google Sheets, Google Docs, I got all kinds of spreadsheets all over the place. Presumably, those are ground balled to second base.

Shishir Mehrotra

>> Right, that's the starting point. Yeah.

>> Right, okay. And so, what do I need? I need Snowflake. I need a AWS account, which I got. I need-

Shishir Mehrotra

>> So, we actually-

>> Do I not need Snowflake?

Shishir Mehrotra

>> It runs on Snowflake, but actually one of the interesting parts is we can use your Snowflake instance or we can use our own. So, that's actually quite interesting to many customers as I'm trying to get started here. I think that's particularly interesting to the Snowflake team, because one of the core insights behind this project is we'll take the data warehouse we create, the Coda Brain backend, and we'll actually make it available to your data team. So, your non-technical knowledge workers will come to this Brain UI. They can ask any question they like, but we'll actually take the underlying data warehouse and make it available to your data and developer teams as well. And the reason that's so important is I think that basically every company has to completely rebuild their data backends for being ready for RAG. And to do a good RAG-based chatbot, which is what everybody wants, you'd need your data warehouse to understand things at an entity level. It needs to be pre-encoded with embeddings, and it needs all the ACLs on every row, and that needs to be kept up to date on a constant basis. And nobody's data architecture starts with that assumption. So, part of the reason I think the Snowflake team is so excited about this project, they're very excited about any application of Cortex, but the other thing that happens is we show up, we say, "Actually we're to build you a new data warehouse is ready for your developer team to actually go build any chatbot or anything else you want to on that."

>> So, I'm trying to think of how I would do this without Coda. I could use a document database, maybe.

Shishir Mehrotra

>> Yeah. So, the hard part is you have to build every integration and synchronize for every system... So, I'll give you an example. So, if you want to synchronize in your support docs, that's straightforward. You take a bunch of PDFs, you load it to a column in Snowflake, straightforward. If you go to a system like Google Drives, you gave the example Google Docs, Google Sheets, so on, they actually have one API that's called the Meta API, where you can say, "Give me all the data, including the ACLs." You want to do the same thing with Salesforce. It's not very easy to do because in Salesforce you can come in and you can write permissions that say, "Reps and this team can only see orders that are less than a million dollars and non-government accounts," and blah, blah, blah. How are you going to know that that's a set of orders that can go to that set of people? So, you have to build every one of those connectors and then keep them up to date. So, I think what we're going to see is a lot of people are going to come in, they're very excited about it. Everybody wants ChatGPT for your company. Easy to train it on the universally-available data. Then, you'll go build some for some relatively easy to synchronize data sources that are very large, but then as you get to the tail, it's very hard to go build those things with the permission model in mind.

Rebecca Knight

>> So, hallucinations though, which are a huge headache, have you found ways to mitigate that?

Shishir Mehrotra

>> I think the hallucination problem, if we think it's big in the consumer world, it's enormous in the enterprise world. In the consumer world, honestly, we all kind of tolerate it. It's even kind of fun sometimes, there's some use cases for it. The enterprise world is completely not tolerable. So, there's a couple of things about hallucinations. First off, if you picture the Coda Brain UI, you come, you ask a question, we'll generally either give you an unstructured result or a structured result. We'll return you a paragraph, a sentence says, "Here's what we think," or return you a table. Each case how to deal with hallucinations has a slightly different approach. So, when you hand back an unstructured result, the typical way to handle that is citations. And I think the Cortex team has done a really nice job of interleaving how... One of the hard parts of the generation phase of RAG is you take all of these different documents, you pull back sometimes a multi-paragraph answer. And in each part, you don't just want, "Here's the multi-paragraph answer and here's all the data sources." You want it interleaved. "This one sentence came from here. This sentence came from here." That's really important. I think the Cortex team has a nice job of that and we plumbed that through. The other side if I hand you a structured data set, you want some confidence that you got the query right, you got the filter right. You understand where it's coming... I was talking to a customer earlier where the query they wanted was, "Can you give me all customer feedback about this feature?" Fairly reasonable thing to go ask. And it came back with a list, said, "That doesn't seem right." And you could immediately, in the Coda Brain UI will hand you back the data set, but we'll take the SQL, but will translate back to active filters, so you can go and change them. And they say, "Oh, I understand. It thought that it should look at this column, but it's actually wrong." And they went and changed. They asked again and they got a slightly better version, got better and better and better. I think it's you take hallucinations, you turn it to your advantage. "Explain to me what you're doing and let me go help you fix it." And so, that I think is the answer to how to handle hallucinations in the enterprise. The nice part is that usually in the enterprise there's a truth. So, if you can hand people where you came from, it's unstructured, here's my citation, it's structured, here's my query, here's my filters. I think we have a better answer-

Rebecca Knight

>> But it takes the human understanding that that's not quite right.

Shishir Mehrotra

>> Yeah, the human is part of the loop. We're not trying to get rid of the human. We're trying to enable the human. And same problem if you went to an analyst and you said, "Hey, can you give me the list of customers who asked about this feature?" Your analyst may misunderstand you too. So, I think this idea of you got back a report like, "That can't possibly be right." And you say, "Oh, you misunderstood me. I meant this feature. Look for this thing." That's I think what you want. And I think this idea of can we iterate with the user and the AI system? That's the loop I think you want for a product like this.

>> So, obviously, this is really interesting. And so, basically, like you said, everybody's built a RAG. It's kind of getting trite. They're not getting value out of it. So, how do you think this will change the ROI equation in enterprise?

Shishir Mehrotra

>> I would say most RAG applications I've seen so far don't get ROI because they get shut down, they don't reach scale. Because of this issue with data sources is I think the Snowflake team has made it so easy to do the middle of the sandwich. They've made it so easy to go and come up with the great embeddings, do this hybrid ranking model that Cortex is doing, has outperformed every benchmark we can come up with. The analyst can give you this text-to-SQL thing. But if you start with this tiny set of data, then the chances you're going to build something that's actually useful to your company are just not that obvious. I mean, the number of companies I've seen we built a RAG bot trained on our brand guidelines. It's like what ROI are you going to get out of, how many times does somebody need to know what are our brand guidelines? So, you have to get to the harder data sets. I need to get to Salesforce, I need to get to your email. I need to get to Slack. I need to get to our tickets. I need to get to our HR system. I need to be able to come and say, "How many employees left because of this reason?" Or, "How many employees did we hire with this background?" Then you get real value out of it. The second thing I'd say is everybody's focused too much on just the answer part. Now, I've got an answer, but I need to take that and repeat that process over and over and over again. And then, you need to connect it to a surface that allows you to take action, allows you to turn that into a repeatable thing, not a one-time thing. So, I think that's how we're going to get ROI at of RAG.

>> What about video?

Shishir Mehrotra

>> I mean video is a data source like everything else. I mean, this is a great example of where I think the work that I'm just so excited about, the pace at which the Cortex team is creating embedding models for anything. And so, this idea of can you chunk video? Can you get transcription out of it? Can you embed it so on? I mean it seems like almost every week there's a new thing the Cortex team has come up with on what they can do there.

>> But the previous problem you talked about, if I understand it, it's like having a consistent answer over time. One of the challenges we have, love to have you help us solve this is things change over time. We're ingesting video, transcribing that video. And then, we want to ask questions, "What were the top 10 takeaways from Snowflake Summit in 2024?" Boom. And then, if you ask that question, it's going to mix '23, '24. It might mix 2015 for all we know. It'll do that sometimes.

Shishir Mehrotra

>> Yeah, and the starting point of that is in video... And video is very hot, but you guys are a very specific example with the show. I think the one we see all the time is meeting recordings, and everybody wants their version of a chatbot over meeting recordings. I would say probably the most obvious example. And it's tricky because actually ingesting video and getting transcription is mostly a solved problem. At this point, that's not that hard. The hard part is who's supposed to have access to that? So, I should be able to come and say, "In how many meetings did we talk about Cortex?" I should get an answer, but only the meetings I was in and only the meetings I should have access to. And so, I think that's really the challenge of building an end solution. But interestingly, I think the core part of can I transcribe video and turn it into text and turn it into embeddings and can I rank it and can I do a vector search? That technology all exists. It's just packaging around it that hasn't-

>> But that's a confined data set, right? Imagine if you had to... Well, maybe you can solve this problem. Assume you have access to 1,000 meetings and you want to know, okay, what happened in those 1,000 meetings?

Shishir Mehrotra

>> That's what most companies are. These days, you record everything. Every Zoom meeting is recorded. It's not that confined. I mean, every day there's thousands of new meetings generated with thousands of different attendees. And in some cases, I want everybody to have access. And some cases I only want-

>> But I want the insights across meetings.

Shishir Mehrotra

>> That's right.

>> Right?

Shishir Mehrotra

>> That's right.

>> And you can help me do that.

Shishir Mehrotra

>> That's right. So, a meeting's a document just like everything else. And you go and you say, "This is who had access." Default policy is the people who are in it. And almost every platform, Zoom and so on, have some way to go change those ACLs. And so, we ingest Zoom recordings and we do this inside Coda all the time.

>> What were the biggest gripes that came out this week? And what are the best ideas that came out this week in all of our meetings? And you can solve that problem.

Shishir Mehrotra

>> For sure. I mean, I think the gripes tell me... Sales conversations are a really common one. "Tell me how many sales calls talked about feature X or talked about this conversation." But yeah, those are all-

>> And I don't need Snowflake, you're saying.

Shishir Mehrotra

>> So, you can bring it on Snowflake or you can use ours.

>> Okay. So, I do need Snowflake.

Shishir Mehrotra

>> Yeah, we run entirely on Snowflake.

>> Okay, 100%. Then, I misunderstood before.

Shishir Mehrotra

>> Yeah, sorry. I just meant that you don't have to already be using Snowflake.

>> No, got it.

Shishir Mehrotra

>> And I think part of the reason that's so exciting is we become a path to, I don't know if I want to be on Snowflake yet, then this is a path to say, "But I need a RAG-based backend. I'm going to use Cobra Brain to fill my Snowflake-"

>> So, I see Coda Brain, I don't see Snowflake?

Shishir Mehrotra

>> Until you ask for it, and then we'll give you the backend and you can query it directly.

Rebecca Knight

>> Shishir, you definitely have theCUBE's attention. So, thank you so much for coming on the show.

>> You got to go already?

Rebecca Knight

>> I know.

Shishir Mehrotra

>> Ran out of time?

Rebecca Knight

>> Yeah, this is almost a sales call here. So, there you go. Thank you so much for coming on theCUBE. Really appreciate it.

Shishir Mehrotra

>> Yeah.

Rebecca Knight

>> I'm Rebecca Knight for Dave Vellante. Stay tuned for more of theCUBE's live coverage of the Snowflake Data Cloud Summit. You're watching theCUBE, the leader in enterprise tech news and analysis.