MongoDB .local NYC 2024 | Suraj Patel | MongoDB & Brandon Duderstadt | Nomic AI

Clips
News
More from MongoDB .local NYC 2024

Suraj Patel

Head of MongoDB Ventures

MongoDB

Brandon Duderstadt

Founder & CEO

Nomic AI

How data visualization unlocks insights in the age of big data and AI

As data continues to be the backbone of enterprises, data visualization has become critical for the easier identification of outliers, trends and patt...

play_circle_outline Exploring MongoDB Local Event & theCUBE's Coverage on AI, LLMs, Nomic's Tooling, and Deploying Challenges

play_circle_outline Explanation of Nomic's visualization tool, Nomic Atlas, for exploring and analyzing data clusters

play_circle_outline Use of Nomic Embed, an open-source model, for improving RAG performance

play_circle_outline MongoDB Ventures and Nomic: Empowering Innovators in the Future of LLMs and AI

play_circle_outline Anticipation of simultaneous calls to multiple LLMs and the potential for more complex cognitive architecture modeling.

Info

Suraj Patel | MongoDB & Brandon Duderstadt | Nomic AI

Suraj Patel

Head of MongoDB Ventures MongoDB

Brandon Duderstadt

Founder & CEO Nomic AI

search

>> All right. Welcome back to New York City. We're here at MongoDB Local. You're watching theCUBE's coverage, wall-to-wall coverage, all the innovation that's all about AI, LLMs going up the stack, building new apps. Suraj Patel is here. He's the head of MongoDB Ventures, a CUBE alum, and Brandon Duderstadt is also here. He's the founder and CEO of Nomic, one of the portfolio companies of MongoDB. Gents, good to see you again.

Brandon Duderstadt

>> Thanks for having us.

>> You're very welcome. Brandon, let me start. I always ask founders of company, why'd you start the company?

Brandon Duderstadt

>> If we take a look at the journey, it really started a company called Rat.ai, which is where I was at before this, and that company was building transformers back in 2018 for use on radiology data. So we were working out the original Tensor2Tensor repo from Google And one of the things that we learned is if you want to be able to deploy LLMs in these incredibly high impact domains like medicine or defense or finance where you really have a vested interest in them being accurate, there's this massive data curation challenge that you have to overcome. So the way that at that company we were able to take that model and get it to performing better than radiologists was really taking a lot of time working with subject matter experts, really getting into the data the model was producing, the training data and going through this arduous curation process. What that really taught me was two things. One, the fact that we were able to get this model to outperform doctors at radiology meant these models are going to be everywhere in five years, and two, there needs to be some kind of tooling that lets both the machine learning engineer and the subject matter expert interact and collaborate on the massive unstructured data sets that power these things. So that's why I left to start really get into that domain specific model, two years ago, Cube Research put out the AI power law and we tried to think about power laws and how it's going to apply here and we took liberty, we basically said, Okay, you got size of model, but then you got model specificity on horizontalized. That's really what's playing out.Was that what you saw when you decided to do invest in Nomic?

Suraj Patel

>> It was actually a little bit of a different thesis when we invested in Nomic. We were thinking more generally. At the time, Brandon told you how he started Nomic and then he probably had some foresight into what was going to happen, but certainly the rubber hit the road with ChatGPT's release in 2022. When we met Nomic a few months later, we were seeing this phenomena of not necessarily subject matter experts, but software developers that really wanted to understand what's going into models, how are models being trained, and Nomic had this really powerful visualization that allowed someone that maybe doesn't day-to-day work in ML engineering or ML research to actually hover over different data points are going in to train the model, understand the relationship between those data points and curate that data. So we saw it from MongoDB perspective. Our persona has always been the software developer. We saw this as a really powerful tool to enable software developers to get dangerous with AI.

>> Right behind us, MongoDB has a little kiosk where you've got investment companies, your portfolio companies, and I walked by, I didn't see the branding of Nomic, but I saw this looked like a knowledge graph to me, this visualization. It was very cool. Explain what I was looking at.

Brandon Duderstadt

>> We call that the map, and that's part of our product, Nomic Atlas, which is distinct from MongoDB.Which is a fun little fun little game we always play. But in Nomic Atlas, every piece of your data set is a point on a scatter plot and two points are close if the content of that data is similar. So what you start to see are these emergent structures where we're able to group the data by its similarity, we're able to label those clusters with kind of semantics. You can think of them as landmarks, and we can apply a lot of different visualization techniques on top of it, letting you search it, filter that data, color it by different metadata. So suddenly you can detect these trends in your data, in these relationships that if you were just looking at it a table or maybe in something like a map plot lib visualization that's not interactive, it would be incredibly hard to discover. So one of my favorite quotes from one of our downstream customers SmarterX is they say, One of my favorite things about Nomic Atlas is I can find something new about my data within the first five minutes of using the product, and I can still discover new things about my data after using the product for hours.We think that really speaks to the power.

>> Then using that to build apps. One of the problems that we've had with our RAG was that ... somebody described this to me really resonated, is that these LLMs are like Swiss cheese. When the data's there, it's actually really good. When it's not, you've got all these holes and if the AI goes into those holes, it'll start grabbing from other data and that's when it really starts to hallucinate. I know others are working on that problem. So we've tried to figure out, Okay, how can we avoid either going into that hole or provide other data to fill those holes?Is that a problem that you're seeing and how do you deal with that?

Brandon Duderstadt

>> Yeah, so on the topic of RAG, there's a couple things here. One, that's certainly a phenomenon that we can visualize in the platform. If you take your RAG data set, you will put it into the map, and then you have your query stream streaming into the map. You'll see if there are queries that are appearing in locations where you don't have that context. If you're collecting ratings on that data, suddenly you can color by that and start to see, oh, the poorly rated things are in this area. There's no training data there, there's no RAG data there, We have to do something about that. But even going a step beyond that and getting into one of Nomic's other product offerings, which is Nomic Embed, one of the most important parts of RAG is retrieving the right data. One of the things that Nomic has done is we've released this model, Nomic Embed. It's the first truly open, that's open source, weights and data model that outperforms OpenAI's embedding models on both short and long context tasks. We've seen massive adoption of this thing. We passed a million downloads on Hugging Face and we're really excited about enabling the entire broader developer community to do RAG with their fully open source stack powered by the economics of that. Can you address that? You're getting better results. Are the economics also superior?

Brandon Duderstadt

>> One of the things that we think a lot about at Nomic, this goes into our third offering, GPT4ALL, is the unit economics of these models. I think what we're seeing right now are a lot of industries experimenting with the models so they're turning to something that's very general purpose, be GPT-4 or a Claude Opus. What I think you'll see is over time as organizations start to understand where in the flowchart of data through their organization these models fit, they're going to become more price sensitive to these things. So they're going to start turning to more specialized, more compressed, more edge deployable models. GPT4ALL comes into play is it's a platform that allows us to take these specialized models and compress them and run them either in the cloud on hardware that is more efficient or on the edge on a user device, and suddenly if you have a really well-specified operational domain, you can get the same level of performance as a GPT-4 or a Claude, but for a fraction of the cost.

>> Awesome. Thank you for that. Suraj, zooming out, can you just sort of describe MongoDB's sort of investment philosophy, and then specifically knowing what you've seen with Nomic over the last couple of years, how it applies there?

Suraj Patel

>> At a high level, MongoDB as a whole, our mission at the company is to empower innovators to use software and data to disrupt industries. So MongoDB Ventures, we look for founders that share in that mission. I think that when we met Brandon, as we mentioned, that was exactly what he was working on. He was thinking about the bottlenecks that, whether it's a software developer or ML researcher, as they're thinking about training a model, fine-tuning a model or putting a RAG system into production, what are the different ways that they approach this problem of what data sites should I use? How do I understand that data set? Then actually beyond that, they had also recently released a product called GPT4ALL, which had incredible open source adoption, allowed developers to run or actually anybody to run models locally on their machine. Again, it reflected on an incredible empathy for the developer andor the technical creator that really wants to bring AI into everyday applications. So that's why we invested in Nomic and super happy to do so.

>> I wonder if we could get your thoughts, both of you, on where you see LLMs going and AI going start with LLMs. There's a debate out there about whether these things become commoditized. You hear it on all talking head shows, and the premise is, well, because everything gets replaced, GPT-2 is who cares? GPT-4. So GPT-going to come out and nobody's going to care about GPT-4 anymore, so it's going to become one big commoditized bin. Others say, Well, no, as long as the innovation continues, it won't be commoditized.VC hat on, how do you think about it, and then ask the entrepreneur how he sees it.

Suraj Patel

>> I don't think the models are going to be commoditized completely. If we think about a world in which there's no post transformer architecture, if we're confined to transformer architectures, then GPU capacity is incredibly important and there's only a few players that will have the data sets and the GPU capacity to train a world-class frontier model. So there may be multiple players. Anthropics has done a great job with their latest cloud release, OpenAI always at the forefront. So there will be multiple players, but that list is not going to be 20 in our opinion, that list is going to be confined to half dozen or less.

>> Do you have a thought on that, Brandon?

Brandon Duderstadt

>> Yeah. The way I see this all playing out, I think of it as sort of concentric rings of intelligence. In the center most ring, you have models that run locally on your machine. These things are always going to be the least intelligent because they have access to the least number of FLOPS, but over time they'll become more intelligent. If we look at running Llama 3 on my MacBook now, it's a world better than running the original Llama release on my laptop just in terms of the quality of the outputs and so on and so forth. So you're always going to have something that you're going to want to be personalized running on your computer. Maybe there's sensitive data involved. Maybe you don't have the ability to communicate with a server, maybe it's just cheaper. That's almost always going to be cheaper. Then one ring out from that is going to be perhaps your organization's language model, which might be a fine tune or has RAG from your organization's context across maybe your company or your industry or what have you that you can call out to for certain kinds of problems. You're going to lose a little bit of that privacy and you're going to pay a little bit more for that, be able to bring in context that maybe you wouldn't have on the local machine. Then one ring out from that are going to be, I think, these big sort of AGI players, which are going to be the most expensive models where you have to send your data away and you have the least kind of data protections, but they are going to offer the most intelligence because they can utilize the most FLOPS. So the way I see this field evolving is we will always have these sort of concentric rings, and because of innovation, there will always be pressure that is taking the level of intelligence in the outermost ring and pushing it closer and closer to the device. So I think there's going to be room for all these players in the limit.

>> You mentioned FLOPS, and some people say, Well, FLOPS really isn't the issue. It's FLOPS per watt and things like thatwhich is, by the way, true. But I always say FLOPS matter because you look at the combinatorial effects of A6, now LPUs, and what you're able to do here, and as inference becomes the dominant workload ... I think last year at Mobile World Congress, MWC, Qualcomm I think showed a 7 billion parameter model running on a smartphone and I think they said to me in theCUBE that they'd be an order of magnitude more than that next year, which just blew me away. So that to me says there's going to be so much distributed power and the way in which applications are going to be written is going to completely change. First of all, do you agree and how do you see Mongo sort of playing in that world?

Brandon Duderstadt

>> I can almost guarantee you the model we're running is not a 7B model, but a quantized model. Those are very different things, and we can get into that if you want, but-

>> No, I get it, though. I get what you're saying. Right.

Brandon Duderstadt

>> I do think from the standpoint of if you look at the scaling loss for these models, it's obvious that if you can coordinate more compute towards a task, you are going to get a better end result. Even as the dominant paradigm moves the inference, we're seeing this idea of chaining multiple inferences together to achieve a better result. So if you look at, say, the idea of a validator prompt, which is not only asking a model for an answer, but then asking it to check its work, you're doing twice as many calls to the model, twice as many FLOPS, but suddenly the performance increases. So I don't see this idea of more FLOPS equals better outcome going away, and frankly, that's been the story of computing since the computer became a thing. As far as FLOPS per watt goes, I definitely see power becoming the next massive constraint here. It's going to be fascinating to watch the dynamics of the energy industry and the infrastructure for moving that energy over the next five years as organizations start to ramp up. The last kind of piece that I'll say here is the idea of synthetic data. One of the things that I think is going to really accelerate inference workloads is the fact that we see with things like DALL-E 3 for instance, the idea of recaptioning, that synthetic data can massively move the needle on the quality of these models and so now you have this massive inference problem before you can even train the next thing, and that's even going to further accelerate the demand for not only compute but for energy and so on and so forth.

>> Thank you. Thoughts on how MongoDB plays in that space?

Suraj Patel

>> I think people when they think MongoDB's AI play maybe default and think about our vector search, and certainly a lot of vector search workloads are related to AI. A lot of them are power and RAG architectures. But one of the things that Brandon touched on, which is this idea that when you put an AI application into production, there's multiple calls to an LLM or any type of AI model. There's responses and all of those calls and responses are recorded somewhere. I think the unanticipated things that we saw is that a lot of people are leveraging the fact that is Schema-less to record that type of data in MongoDB, and it's a new workload that's taking off quite quickly for us.

>> More real-time. Do you see a day ... because it seems like most applications today, the architecture is I'm calling one LLM at a time, and I don't think anybody's figured out how to do simultaneous calls to multiple LLMs. I don't even know what that world looks like. Do you see that happening? What's the technical challenge there? I wonder if could comment.

Brandon Duderstadt

>> I was talking to Harrison at Langchain and Jerry at Llama Index about this particular thing and so I was asking what kinds of agent or cognitive architectures are we seeing nowadays? Is it just a simple RAG with one LLM are we seeing more advanced things? Right now, I think we're in the very early days and they told me 90of the users are just LLM up to a vector database and that's kind of whole app. But those 10of users that are going beyond that and are today orchestrating many LLMs at different levels of sophistication, so maybe sometimes local, sometimes calling out to GPT-4 or a Claude and doing some of this more bespoke cognitive architecture modeling, they're seeing that start to pick up and they're seeing the real power behind those applications. So I think as developers get their hands on this technology and are enabled by some of the work that Mongo is doing now to actually have this stuff running on their machine in different dev environments and in the open source, they're going to be able to orchestrate increasingly complex apps that are going to be increasingly featureful and capable.

>> I think that's the thing that people sometimes miss when you listen to the All-In pod, they would go, It's so easy this stuffand you're going to be able to do more with far fewer developers, but to really get that big ROI, not so much the percentage, but the big numbers, the transformative ... Dev was talking about the waves and always happens, you hit the hype cycle or whatever, and you think about back in ... you were probably still in kindergarten, Brandon, but the dot com, you had Lycos, you had AOL was the granddaddy of them. So many of those are gone. Who ended up winning? It was Amazon, it was Google, it was obviously the social media folks that weren't even around in the early dot com days. I see something similar happening going to take some time. I know. Your investment horizon is oftentimes eight to 10 years to build a company. I still think it's going to take that long to solve some of these problems. You're just going to be able to do it maybe with a lot less resources or maybe you're going to need as many resources, but it's going to change the world in ways that we can't even imagine. Do you see the same dynamic playing out?

Suraj Patel

>> Yeah. We invest in infrastructure, so companies that are really enabling innovators to create the end user application. We love that area of the stack and we think that several winners will emerge from that area of the stack. If I think about the application layer, though, what's interesting is I think today we are seeing basically AI enable existing workflows. With every platform shift, you see this period of time where people are enabling existing workflows on the new platform and at some point we're going to get to a point where new workflows are created that are native to that platform. If you think about how long into the mobile cycle it took for an Uber to be created, which was really an experience that could only happen once there was mobile maturity, I don't think we're there yet in terms of AI applications and seeing truly AI native workflows yet. Certainly the other aspect that I think about a lot is the AI applications we are seeing today, they're making some tough math work. As Brandon was saying, if you're taking multiple calls to a model, now you have to figure out, Okay, do I host that model myself? Do I put it on the public cloud depending on how big the model is? Do I need a high performance GPU? Can I use the last gen GPU?All of these economic calculations, I don't think many of them have been figured out in a meaningful way yet.

>> I guess so many questions. First of all, Uber, that's a great example. We envision Uber for all that is real time people, places and things. Is all the action happening in the cloud today or is it actually creeping on prem? What are you seeing there?

Suraj Patel

>> Both. I think some of the model, LLMs that you talk to, they're finding interesting use cases in financial services, healthcare where people aren't comfortable sending calls outside of their four walls. Actually, Brandon and I were having a talk about this just yesterday. Maybe you want to touch.

Brandon Duderstadt

>> In terms of the industry, like VPC link?

Suraj Patel

>> GPT4ALL and the traction you're seeing in terms of local.

Brandon Duderstadt

>> Sure. One of the things that we see with GPT4ALL is the extreme of that, which is bringing the model to the end device. The entire sort of value prop of that tool chain is you can take something like a Llama 3 quantized to four bit or eight bit and run it at 20 tokens a second on a MacBook, and now suddenly you have the ability to use that if maybe you don't have wifi, maybe you have sensitive data, you don't want to pay for those GPU workloads, and so it comes back to this idea of the concentric circles of intelligence where I think in the near future, probably less than a couple of years, you're going to have language models burned into the actual chips of these devices. I certainly know Apple is investing in this idea with their M series processors and is super well positioned because of that, but I think that's going to become increasingly common.

>> Do you agree that you're going to need GPU power there to do that? Even in inference? It's simple matrix math, but there's a lot of it. Is that correct assumption that you are going to need GPUs at the edge?

Brandon Duderstadt

>> You're going to need something that looks somewhat like GPU. If you look at what Apple is doing, they have the integrated memory of things. It's not quite a GPU, but it's not quite not a GPU, and you're going to need something like that. But the ability to coordinate a massively parallel number of threads is really what you're looking for and I think we're going to see this whole spectrum of bespoke kind of end device hardware that does that.

>> Well, and you need a shared SRAM. That's one of the things we talk about all the time you need a big shared SRAM monolithic chip to do that? Or is this chiplet architecture with what I think a relatively slow asynchronous connection is going to do it? There's probably room for both. Topic for another day. Guys, thanks so much for coming to theCUBE. Really appreciate your time.

Brandon Duderstadt

>> Thanks for having us.

Suraj Patel

>> Thank you.

>> You're very welcome. All right, keep it right there. We're live at MongoDB Local from the Big Apple. You're watching theCUBE.

Suraj Patel | MongoDB & Brandon Duderstadt | Nomic AI

search

>> All right. Welcome back to New York City. We're here at MongoDB Local. You're watching theCUBE's coverage, wall-to-wall coverage, all the innovation that's all about AI, LLMs going up the stack, building new apps. Suraj Patel is here. He's the head of MongoDB Ventures, a CUBE alum, and Brandon Duderstadt is also here. He's the founder and CEO of Nomic, one of the portfolio companies of MongoDB. Gents, good to see you again.

Brandon Duderstadt

>> Thanks for having us.

>> You're very welcome. Brandon, let me start. I always ask founders of company, why'd you start the company?

Brandon Duderstadt

>> If we take a look at the journey, it really started a company called Rat.ai, which is where I was at before this, and that company was building transformers back in 2018 for use on radiology data. So we were working out the original Tensor2Tensor repo from Google And one of the things that we learned is if you want to be able to deploy LLMs in these incredibly high impact domains like medicine or defense or finance where you really have a vested interest in them being accurate, there's this massive data curation challenge that you have to overcome. So the way that at that company we were able to take that model and get it to performing better than radiologists was really taking a lot of time working with subject matter experts, really getting into the data the model was producing, the training data and going through this arduous curation process. What that really taught me was two things. One, the fact that we were able to get this model to outperform doctors at radiology meant these models are going to be everywhere in five years, and two, there needs to be some kind of tooling that lets both the machine learning engineer and the subject matter expert interact and collaborate on the massive unstructured data sets that power these things. So that's why I left to start really get into that domain specific model, two years ago, Cube Research put out the AI power law and we tried to think about power laws and how it's going to apply here and we took liberty, we basically said, Okay, you got size of model, but then you got model specificity on horizontalized. That's really what's playing out.Was that what you saw when you decided to do invest in Nomic?

Suraj Patel

>> It was actually a little bit of a different thesis when we invested in Nomic. We were thinking more generally. At the time, Brandon told you how he started Nomic and then he probably had some foresight into what was going to happen, but certainly the rubber hit the road with ChatGPT's release in 2022. When we met Nomic a few months later, we were seeing this phenomena of not necessarily subject matter experts, but software developers that really wanted to understand what's going into models, how are models being trained, and Nomic had this really powerful visualization that allowed someone that maybe doesn't day-to-day work in ML engineering or ML research to actually hover over different data points are going in to train the model, understand the relationship between those data points and curate that data. So we saw it from MongoDB perspective. Our persona has always been the software developer. We saw this as a really powerful tool to enable software developers to get dangerous with AI.

>> Right behind us, MongoDB has a little kiosk where you've got investment companies, your portfolio companies, and I walked by, I didn't see the branding of Nomic, but I saw this looked like a knowledge graph to me, this visualization. It was very cool. Explain what I was looking at.

Brandon Duderstadt

>> We call that the map, and that's part of our product, Nomic Atlas, which is distinct from MongoDB.Which is a fun little fun little game we always play. But in Nomic Atlas, every piece of your data set is a point on a scatter plot and two points are close if the content of that data is similar. So what you start to see are these emergent structures where we're able to group the data by its similarity, we're able to label those clusters with kind of semantics. You can think of them as landmarks, and we can apply a lot of different visualization techniques on top of it, letting you search it, filter that data, color it by different metadata. So suddenly you can detect these trends in your data, in these relationships that if you were just looking at it a table or maybe in something like a map plot lib visualization that's not interactive, it would be incredibly hard to discover. So one of my favorite quotes from one of our downstream customers SmarterX is they say, One of my favorite things about Nomic Atlas is I can find something new about my data within the first five minutes of using the product, and I can still discover new things about my data after using the product for hours.We think that really speaks to the power.

>> Then using that to build apps. One of the problems that we've had with our RAG was that ... somebody described this to me really resonated, is that these LLMs are like Swiss cheese. When the data's there, it's actually really good. When it's not, you've got all these holes and if the AI goes into those holes, it'll start grabbing from other data and that's when it really starts to hallucinate. I know others are working on that problem. So we've tried to figure out, Okay, how can we avoid either going into that hole or provide other data to fill those holes?Is that a problem that you're seeing and how do you deal with that?

Brandon Duderstadt

>> Yeah, so on the topic of RAG, there's a couple things here. One, that's certainly a phenomenon that we can visualize in the platform. If you take your RAG data set, you will put it into the map, and then you have your query stream streaming into the map. You'll see if there are queries that are appearing in locations where you don't have that context. If you're collecting ratings on that data, suddenly you can color by that and start to see, oh, the poorly rated things are in this area. There's no training data there, there's no RAG data there, We have to do something about that. But even going a step beyond that and getting into one of Nomic's other product offerings, which is Nomic Embed, one of the most important parts of RAG is retrieving the right data. One of the things that Nomic has done is we've released this model, Nomic Embed. It's the first truly open, that's open source, weights and data model that outperforms OpenAI's embedding models on both short and long context tasks. We've seen massive adoption of this thing. We passed a million downloads on Hugging Face and we're really excited about enabling the entire broader developer community to do RAG with their fully open source stack powered by the economics of that. Can you address that? You're getting better results. Are the economics also superior?

Brandon Duderstadt

>> One of the things that we think a lot about at Nomic, this goes into our third offering, GPT4ALL, is the unit economics of these models. I think what we're seeing right now are a lot of industries experimenting with the models so they're turning to something that's very general purpose, be GPT-4 or a Claude Opus. What I think you'll see is over time as organizations start to understand where in the flowchart of data through their organization these models fit, they're going to become more price sensitive to these things. So they're going to start turning to more specialized, more compressed, more edge deployable models. GPT4ALL comes into play is it's a platform that allows us to take these specialized models and compress them and run them either in the cloud on hardware that is more efficient or on the edge on a user device, and suddenly if you have a really well-specified operational domain, you can get the same level of performance as a GPT-4 or a Claude, but for a fraction of the cost.

>> Awesome. Thank you for that. Suraj, zooming out, can you just sort of describe MongoDB's sort of investment philosophy, and then specifically knowing what you've seen with Nomic over the last couple of years, how it applies there?

Suraj Patel

>> At a high level, MongoDB as a whole, our mission at the company is to empower innovators to use software and data to disrupt industries. So MongoDB Ventures, we look for founders that share in that mission. I think that when we met Brandon, as we mentioned, that was exactly what he was working on. He was thinking about the bottlenecks that, whether it's a software developer or ML researcher, as they're thinking about training a model, fine-tuning a model or putting a RAG system into production, what are the different ways that they approach this problem of what data sites should I use? How do I understand that data set? Then actually beyond that, they had also recently released a product called GPT4ALL, which had incredible open source adoption, allowed developers to run or actually anybody to run models locally on their machine. Again, it reflected on an incredible empathy for the developer andor the technical creator that really wants to bring AI into everyday applications. So that's why we invested in Nomic and super happy to do so.

>> I wonder if we could get your thoughts, both of you, on where you see LLMs going and AI going start with LLMs. There's a debate out there about whether these things become commoditized. You hear it on all talking head shows, and the premise is, well, because everything gets replaced, GPT-2 is who cares? GPT-4. So GPT-going to come out and nobody's going to care about GPT-4 anymore, so it's going to become one big commoditized bin. Others say, Well, no, as long as the innovation continues, it won't be commoditized.VC hat on, how do you think about it, and then ask the entrepreneur how he sees it.

Suraj Patel

>> I don't think the models are going to be commoditized completely. If we think about a world in which there's no post transformer architecture, if we're confined to transformer architectures, then GPU capacity is incredibly important and there's only a few players that will have the data sets and the GPU capacity to train a world-class frontier model. So there may be multiple players. Anthropics has done a great job with their latest cloud release, OpenAI always at the forefront. So there will be multiple players, but that list is not going to be 20 in our opinion, that list is going to be confined to half dozen or less.

>> Do you have a thought on that, Brandon?

Brandon Duderstadt

>> Yeah. The way I see this all playing out, I think of it as sort of concentric rings of intelligence. In the center most ring, you have models that run locally on your machine. These things are always going to be the least intelligent because they have access to the least number of FLOPS, but over time they'll become more intelligent. If we look at running Llama 3 on my MacBook now, it's a world better than running the original Llama release on my laptop just in terms of the quality of the outputs and so on and so forth. So you're always going to have something that you're going to want to be personalized running on your computer. Maybe there's sensitive data involved. Maybe you don't have the ability to communicate with a server, maybe it's just cheaper. That's almost always going to be cheaper. Then one ring out from that is going to be perhaps your organization's language model, which might be a fine tune or has RAG from your organization's context across maybe your company or your industry or what have you that you can call out to for certain kinds of problems. You're going to lose a little bit of that privacy and you're going to pay a little bit more for that, be able to bring in context that maybe you wouldn't have on the local machine. Then one ring out from that are going to be, I think, these big sort of AGI players, which are going to be the most expensive models where you have to send your data away and you have the least kind of data protections, but they are going to offer the most intelligence because they can utilize the most FLOPS. So the way I see this field evolving is we will always have these sort of concentric rings, and because of innovation, there will always be pressure that is taking the level of intelligence in the outermost ring and pushing it closer and closer to the device. So I think there's going to be room for all these players in the limit.

>> You mentioned FLOPS, and some people say, Well, FLOPS really isn't the issue. It's FLOPS per watt and things like thatwhich is, by the way, true. But I always say FLOPS matter because you look at the combinatorial effects of A6, now LPUs, and what you're able to do here, and as inference becomes the dominant workload ... I think last year at Mobile World Congress, MWC, Qualcomm I think showed a 7 billion parameter model running on a smartphone and I think they said to me in theCUBE that they'd be an order of magnitude more than that next year, which just blew me away. So that to me says there's going to be so much distributed power and the way in which applications are going to be written is going to completely change. First of all, do you agree and how do you see Mongo sort of playing in that world?

Brandon Duderstadt

>> I can almost guarantee you the model we're running is not a 7B model, but a quantized model. Those are very different things, and we can get into that if you want, but-

>> No, I get it, though. I get what you're saying. Right.

Brandon Duderstadt

>> I do think from the standpoint of if you look at the scaling loss for these models, it's obvious that if you can coordinate more compute towards a task, you are going to get a better end result. Even as the dominant paradigm moves the inference, we're seeing this idea of chaining multiple inferences together to achieve a better result. So if you look at, say, the idea of a validator prompt, which is not only asking a model for an answer, but then asking it to check its work, you're doing twice as many calls to the model, twice as many FLOPS, but suddenly the performance increases. So I don't see this idea of more FLOPS equals better outcome going away, and frankly, that's been the story of computing since the computer became a thing. As far as FLOPS per watt goes, I definitely see power becoming the next massive constraint here. It's going to be fascinating to watch the dynamics of the energy industry and the infrastructure for moving that energy over the next five years as organizations start to ramp up. The last kind of piece that I'll say here is the idea of synthetic data. One of the things that I think is going to really accelerate inference workloads is the fact that we see with things like DALL-E 3 for instance, the idea of recaptioning, that synthetic data can massively move the needle on the quality of these models and so now you have this massive inference problem before you can even train the next thing, and that's even going to further accelerate the demand for not only compute but for energy and so on and so forth.

>> Thank you. Thoughts on how MongoDB plays in that space?

Suraj Patel

>> I think people when they think MongoDB's AI play maybe default and think about our vector search, and certainly a lot of vector search workloads are related to AI. A lot of them are power and RAG architectures. But one of the things that Brandon touched on, which is this idea that when you put an AI application into production, there's multiple calls to an LLM or any type of AI model. There's responses and all of those calls and responses are recorded somewhere. I think the unanticipated things that we saw is that a lot of people are leveraging the fact that is Schema-less to record that type of data in MongoDB, and it's a new workload that's taking off quite quickly for us.

>> More real-time. Do you see a day ... because it seems like most applications today, the architecture is I'm calling one LLM at a time, and I don't think anybody's figured out how to do simultaneous calls to multiple LLMs. I don't even know what that world looks like. Do you see that happening? What's the technical challenge there? I wonder if could comment.

Brandon Duderstadt

>> I was talking to Harrison at Langchain and Jerry at Llama Index about this particular thing and so I was asking what kinds of agent or cognitive architectures are we seeing nowadays? Is it just a simple RAG with one LLM are we seeing more advanced things? Right now, I think we're in the very early days and they told me 90of the users are just LLM up to a vector database and that's kind of whole app. But those 10of users that are going beyond that and are today orchestrating many LLMs at different levels of sophistication, so maybe sometimes local, sometimes calling out to GPT-4 or a Claude and doing some of this more bespoke cognitive architecture modeling, they're seeing that start to pick up and they're seeing the real power behind those applications. So I think as developers get their hands on this technology and are enabled by some of the work that Mongo is doing now to actually have this stuff running on their machine in different dev environments and in the open source, they're going to be able to orchestrate increasingly complex apps that are going to be increasingly featureful and capable.

>> I think that's the thing that people sometimes miss when you listen to the All-In pod, they would go, It's so easy this stuffand you're going to be able to do more with far fewer developers, but to really get that big ROI, not so much the percentage, but the big numbers, the transformative ... Dev was talking about the waves and always happens, you hit the hype cycle or whatever, and you think about back in ... you were probably still in kindergarten, Brandon, but the dot com, you had Lycos, you had AOL was the granddaddy of them. So many of those are gone. Who ended up winning? It was Amazon, it was Google, it was obviously the social media folks that weren't even around in the early dot com days. I see something similar happening going to take some time. I know. Your investment horizon is oftentimes eight to 10 years to build a company. I still think it's going to take that long to solve some of these problems. You're just going to be able to do it maybe with a lot less resources or maybe you're going to need as many resources, but it's going to change the world in ways that we can't even imagine. Do you see the same dynamic playing out?

Suraj Patel

>> Yeah. We invest in infrastructure, so companies that are really enabling innovators to create the end user application. We love that area of the stack and we think that several winners will emerge from that area of the stack. If I think about the application layer, though, what's interesting is I think today we are seeing basically AI enable existing workflows. With every platform shift, you see this period of time where people are enabling existing workflows on the new platform and at some point we're going to get to a point where new workflows are created that are native to that platform. If you think about how long into the mobile cycle it took for an Uber to be created, which was really an experience that could only happen once there was mobile maturity, I don't think we're there yet in terms of AI applications and seeing truly AI native workflows yet. Certainly the other aspect that I think about a lot is the AI applications we are seeing today, they're making some tough math work. As Brandon was saying, if you're taking multiple calls to a model, now you have to figure out, Okay, do I host that model myself? Do I put it on the public cloud depending on how big the model is? Do I need a high performance GPU? Can I use the last gen GPU?All of these economic calculations, I don't think many of them have been figured out in a meaningful way yet.

>> I guess so many questions. First of all, Uber, that's a great example. We envision Uber for all that is real time people, places and things. Is all the action happening in the cloud today or is it actually creeping on prem? What are you seeing there?

Suraj Patel

>> Both. I think some of the model, LLMs that you talk to, they're finding interesting use cases in financial services, healthcare where people aren't comfortable sending calls outside of their four walls. Actually, Brandon and I were having a talk about this just yesterday. Maybe you want to touch.

Brandon Duderstadt

>> In terms of the industry, like VPC link?

Suraj Patel

>> GPT4ALL and the traction you're seeing in terms of local.

Brandon Duderstadt

>> Sure. One of the things that we see with GPT4ALL is the extreme of that, which is bringing the model to the end device. The entire sort of value prop of that tool chain is you can take something like a Llama 3 quantized to four bit or eight bit and run it at 20 tokens a second on a MacBook, and now suddenly you have the ability to use that if maybe you don't have wifi, maybe you have sensitive data, you don't want to pay for those GPU workloads, and so it comes back to this idea of the concentric circles of intelligence where I think in the near future, probably less than a couple of years, you're going to have language models burned into the actual chips of these devices. I certainly know Apple is investing in this idea with their M series processors and is super well positioned because of that, but I think that's going to become increasingly common.

>> Do you agree that you're going to need GPU power there to do that? Even in inference? It's simple matrix math, but there's a lot of it. Is that correct assumption that you are going to need GPUs at the edge?

Brandon Duderstadt

>> You're going to need something that looks somewhat like GPU. If you look at what Apple is doing, they have the integrated memory of things. It's not quite a GPU, but it's not quite not a GPU, and you're going to need something like that. But the ability to coordinate a massively parallel number of threads is really what you're looking for and I think we're going to see this whole spectrum of bespoke kind of end device hardware that does that.

>> Well, and you need a shared SRAM. That's one of the things we talk about all the time you need a big shared SRAM monolithic chip to do that? Or is this chiplet architecture with what I think a relatively slow asynchronous connection is going to do it? There's probably room for both. Topic for another day. Guys, thanks so much for coming to theCUBE. Really appreciate your time.

Brandon Duderstadt

>> Thanks for having us.

Suraj Patel

>> Thank you.

>> You're very welcome. All right, keep it right there. We're live at MongoDB Local from the Big Apple. You're watching theCUBE.