AWS Summit NYC 2024 | Jean-Simon Venne | BrainBox AI & Randall Hunt | Caylent

theCUBE.net

info Help
Sign in
Register

Clips
More from AWS Summit NYC 2024

Jean-Simon Venne

Chief Technology Officer & Co-Founder

BrainBox AI

Randall Hunt

CTO

Caylent

play_circle_outline Focus on enriching context with user data for generative AI differentiation

play_circle_outline Empowering AI Through Prompt Engineering and Client Team Integration: Enhancing Model Performance Evaluation

play_circle_outline Leveraging tools from AWS and Caylent for brainstorming and efficiency

play_circle_outline Impact of technology on reducing emissions and energy consumption in buildings

Info

Jean-Simon Venne | BrainBox AI & Randall Hunt | Caylent

Jean-Simon Venne

Chief Technology Officer & Co-Founder BrainBox AI

Randall Hunt

CTO Caylent

search

>> Welcome back, everyone, to theCube's coverage here in New York City for Amazon Webster. So I'm John Furrier, your host of theCube. Been covering all the new announcements. We've got Matt Wood coming up. We've got two great guests here. We've got a customer and partner, Randall Hunt, VP of cloud strategy, innovation at Caylent. And Jean-Simon Venne, Chief Technology Officer at BrainBox. You guys working together. Great example of AI. Welcome to theCube.

Randall Hunt

>> Thank you.

Jean-Simon Venne

>> Popup Cube, great to see you.

Randall Hunt

>> Good to see you again.

>> You've been Amazon for years as an Amazonian, now CTO, you've got customer partners together. AI has kind of been around for a while. You guys are working together. It's early days, right? We're seeing, this is clearly early days. Even now, we're just talking about, hey, models will work together. Oh hey, what, you think? Or hey, how do I work with a little small language model, large language model, and then what infrastructure do you run out? These are all the basic questions that are being discussed today. That's going to happen fast. You guys are doing work now with AI. What are some of the things you're seeing? What are you guys doing? Take a minute to explain what you're doing with AI and how you guys are involved.

Randall Hunt

>> So Caylent, we're an all-in on AWS, premier tier consulting partner. We build software for people. So a customer comes to us with an idea and we say we can build it, or sometimes we say we can't, because the models aren't there yet. And we've completed well over a hundred generative AI projects at this point. And what we've found is organizations like Jean-Simon's BrainBox AI, if they're willing to lean in and adopt a platform around generative AI, they're much more successful than the companies who are just trying to shove a model into something. The model is honestly the least durable part of any architecture, because they're improving so quickly. The platform that you build around the model, and the evals, and the testing, that's what we've seen with successful customers, and we've worked across every industry at this point. So we've done financial services, healthcare, life sciences, name them all, startup, enterprise, small business, DNB. And the key things that we've learned is, if you can get something into production within three months and you can get customers using it, getting feedback, you start that loop, those projects tend to be successful and they tend to move forward. But the ones that linger, and you spend a year and a half on R&D, those often don't make it.

>> They boil the ocean over or they miss the mark, the market's changed. Jean-Simon, talk about your side of the equation here, because obviously a couple of years ago there was the hype, people were experimenting with, and then the next year, this past year, a year ago you started to see real development, people couldn't get the low-hanging fruit, say I'm going to get these use cases. And then, now it's more like how do you scale it? So when you first started seeing opportunities, take us through the mindset. What did you look at? What was the view, what did you lock your eyes on, and how did you tackle it?

Jean-Simon Venne

>> When you look at what we've done in the last six years is we managed to connect over 15,000 building in 16 country. And we're having this, I would say the old AI with this deep learning model, which are giving us a prediction, and we're basically doing optimization in real time of these buildings. So we're kind of lowering usually the energy being spent by this building by 20 to 25%. What we start to realize a year ago is like, we have terabyte and terabyte and terabyte of data that we keep accumulating, and we keep generating more data. The prediction is an example, but we keep generating a lot more data for each of these building. And then, when you look at the dashboard that we had, it's like there's so many tabs. So the poor customer is looking at all that, going like, calling us to say, "On what tab do I find that graph again?" And it's kind of pretty obvious that it's overwhelming. And I think last summer there was this discussion with AWS and saying, you know what we should do? We should build Jarvis like an island man, using the LLM. And of course AWS went like, "Oh, that's not a small challenge, because you know what?"

>> The VPCs got to connect, they got to land.

Jean-Simon Venne

>> There's a latency problem, hallucination problem. How do you know it's the right answer? And I kind of forced the players, and said I want to build it. And he said, "Well, I mean honestly, let's build a team." And that's when basically Caylent came into the equation saying, "Okay, we need people that first of all had more experience than us, playing on the beta user on Bedrock." But then it was exactly what Randall was explaining. It's not just the LLM. You're really looking at building a ring around the LLM of tools agent, that will basically make sure that you're doing the right thing all the time. And that's what we've been building ever since last September, with a lot of wall being broke down. And it's funny, because when I look back where we were in December and where we are today, it's incredible the amount of work that was done, and then what we were successfully pushing forward.

>> They always say the people who break down those walls get bloody.

Jean-Simon Venne

>> Oh yeah.

>> And so, picked a good part with Randall. He's got Amazon wired, he's been there. He knows all a lot of the products. That's interesting is that, it's interesting in these new waves, it's like the people who have the problem and the people who solve the problem may be a mismatch. And this comes back down to like, hey, I've never really had a problem with a lot of data, but I have a solution for you, because I know you have a problem, so I'm going to engineer for it. So you have a lot of data, you're data full, you already had data, you had machine learning working, you had Amazon experience. So, good fit, I see that. But there's now some people don't have that problem. I just want to bolt on gen AI. So it seems to be having data, it seems to be the requirement. Do you see that same thing, where you got to know what you're doing with the data, or do they all have enough data to start? I mean, I guess that's my question. And in your case, you had a lot of data, so easy entry to get, you get a little bloody, but you had to get your arms around it. At least you knew what you could do.

Randall Hunt

>> Your data is definitely your differentiator when you're working with generative AI. So, anyone can invoke an LLM, do some prompt engineering, but it's the context that you're using to enrich the inference from the LLM. That's what differentiates your offering from some competitor's offering. And if you're able to find the right data, you can do a lot. And people focus too much, I believe, on finding data that is either in their existing OLTP store, or in their existing Redshift cluster, or analytics, or big data warehouse. They don't think about how they can enrich the context in real time with user interaction. One of the things that we built together was Aria, and maybe you want to talk about that, and-

>> Hold on, just want to back up before, because you said something I want to unpack real, we'll get to the Aria in second. Enriching that with either data or-

Randall Hunt

>> Where the user is....

>> or user data. So you've got user behavioral data from people, but also you can bring in other data sources, kind what Matt Wood was kind of getting to that today, or not.

Randall Hunt

>> And today they launched memory for Bedrock agents, which is able to store across 30 days the conversation histories, summaries, invocations, traces, all those good things. So you can enrich in real time. When I say enrich in real time with the context of the user, what I'm talking about is, where are they in the application? Where are they on the web page, what's the current state? What is their current intent? What are they trying to accomplish? And if you can get to that next best thought, next best action, that's when you have a generative AI solution that is going to wow your customers, because it's thinking ahead of what they're already doing.

>> I want to get into that and then we'll get into the solution, because I think this brings up the question of, if you believe that to be true, which by the way, I believe that, Randall, to be true, that's the use case, because that's generative, right? You're generating, but it requires data to be addressable in low latency real time milliseconds.

Randall Hunt

>> Yep.

>> Okay, a lot of people don't know... I got a data warehouse, it might not be able to be queried even in time. So context of having available data, it's got to be highly available or high availability has got to be in place. How do you guys look at that architecturally, what is the mindset? Because if you solve that problem, then you're at the next level. Are we there? Is the data architecture support that, do you guys see that? Are we seeing low latency generative queries or is that still on the roadmap?

Randall Hunt

>> I think every user is, every customer is on a different part of that spectrum. So accessing your OLTP store, or accessing Dynamo, or MemoryDB, those are trivial. And writing queries generatively is basically a solved use case at this point. There may be certain edge cases which you would evaluate through your evals and testing. Many customers don't have that data, to your point, in the right format. But I wouldn't say it's a blocker the way that it used to be.

>> Yeah, it's gettable. You can convert it.

Randall Hunt

>> I mean, here's the thing, if you're willing to spend a lot of money, you can just convert all of that unstructured data into structured data. The cost is just how many tokens are you going to put through? And not everyone's willing to spend that. The unit economics of it don't always make sense, but traditional data science and traditional big data workloads, ETL workloads, those are all still relevant, because you're trying to get the data into the format that's most accessible for the LMS.

>> And if you look at kind the projection of any new market, those economics will change as performance increases. So that self-optimizes over time and becomes a non-issue. And if they already have SQL, then that's comes some historical data too, right? I mean, you've got both.

Randall Hunt

>> And the unit economics right now, the way that it works is every single customer wants the highest quality tokens as quickly as possible at the lowest possible price. And that's what I mean when I say that the model is the least durable component of any architecture, because Haiku came out and immediately it's delivering high quality tokens quickly. Anthropic is innovating very quickly. So you always want to take advantage of that latest model.

>> Yeah, I think just like the comparisons to the web is a great one. Online speeds got faster as the online population increased, searching just got better. I think that's going to get better. I think really the application's key. So let's get into what you guys did next. I want to get into the, because you have a real powerful use case. I mean, you guys solve a problem for getting greenhouse gases. This is a real application. It's like, it's not fantasy. So take us through what you guys have at Aria. What's the application do? What specifically is the benefit?

Jean-Simon Venne

>> So when you look at the core tech itself, which has been in place like before last year, it's really modulating in real time without any human in the loop. All of the cooling, heating and ventilation of the buildings. And it's really reducing the footprint of the energy of that building by 20, 25% emissions. Sometime we push it to 40%, because the algorithm are clever enough to go work on the natural gas instead of the electricity, when the electricity is being produced by windmill, right? So that kind of modulation in real time is really getting and harvesting the value. So what we did as a next step is say, okay, this is great. We're delivering for the CFO of that organization. We're reducing that bill and for the system-

>> Yeah, and by the way, it could be a big bill too.

Jean-Simon Venne

>> Yeah, exactly. In this place-

>> I mean, empty building sitting there with no...

Jean-Simon Venne

>> You still maintain-

>> why, got to keep it low on maintenance.

Jean-Simon Venne

>> Absolutely. One of our customer approached us right before we started working with Caylent and said, "Okay, this is great. You're saving energy and emission, but we still have a lot of other lowing in food that we would like to resolve." And one of them is like, when you have a problem in the building, the building manager or the building engineer has to basically start to do the troubleshooting. So in this room it's too hot. So he's going to start to look in different data source and he's going to investigate, try to find what is the diagnostic, because it's probably a valve, or it's probably a malfunction of a motor somewhere. So that work takes one hour, two hour, and eventually you find that that's a problem with that valve, right? And then, then he knows, and then you're going to start to work and replace that valve, or find a way to open it manually. So when you look at Aria, it basically you ask the questions, can you investigate? And Aria goes connect to the existing data set and basically do that root cause analysis in a few minutes. So that 90 minutes is now becoming two minutes, and that person is now able to go to action and resolve that problem. So when you look at the productivity in a given day, this person usually leaves after the end of their shift with still a lot of problem unresolved, because eight hours and I'm done, right? And I'm fresh, I'm going to start work, keep calling, working on it tomorrow. So in this case, suddenly the productivity increase in our space of building management, and then building engineering, and there's a shortage of labor, which is incredible. And young generation don't go in that field. So we're coming with that tool, where it is giving the productivity gain, that the spirit-

>> It's like IT, a single pane of glass into the whole thing. But I think this is a use case that we're seeing across every vertical. IT, DevOps, security. Hey, I get the time it takes, I got to reduce the steps it takes to do the task. In this case-

Jean-Simon Venne

>> It's finding the root cause analysis.

>> Yeah, that's hard.

Jean-Simon Venne

>> Look at all the data-

>> Oh yeah, specialty and these knowledge is needed too. You're going to need to have specialism on that.

Jean-Simon Venne

>> Yeah, exactly. And what we're finding, it's interesting, we were testing Claude 3 when it came out, and clearly Claude 3 has some HVAC building engineering, building mechanic training. Because when it start to do the reasoning, you could tell that it did read a thermodynamic book. It understands it, which is kind of wow, this is-

>> Welcome to the digital twin world of talents. No, but this is a great use case. And I think this points to where we were just talking earlier on, the keynote announced with the industry analysis roundtable was, everyone's laughing about chat bots and then now agents. And we're like, well, agents are today, but one of the comments was that's going to be a primitive, like DLL was for Windows. You're going to see agents being under the covers doing things. So I can buy that. That's a good analogy, not quite accurate, but I get the analogy. As the agents happen, the next step is automation and reasoning. So this is reinforced learning comes in next. Are you guys looking at that direction? You got training and inference, so yeah, okay. They're trained on HVACs and buildings. You have data that, does it need to be trained? It's pretty much trained. Your data is trained. I mean, it's your data. I mean, does it be training or is it more of reinforced learning about if it's stale or not? But I'm sure it's like you don't really need to train your own data, do you?

Randall Hunt

>> Yeah, yep.

Jean-Simon Venne

>> Yeah, you have to review the data as a knowledge base basically. So you're tapping in that knowledge base and am basically benefiting that into the way you're going to answer that question. So really when you look at the role of the LLM, it's really use the LLM first to decode what is the human really asking? Prepare your plan of how I'm going to organize my strategy to answer that question. And then, really it's the worker that goes out, and go and get the knowledge, assemble all that knowledge together, so then you could return and start to prepare the answer with the LLM.

>> Okay, so let's get back to Jarvis, because I love that analogy. Again, why hallucinations are going to make him, Jarvis, kind of an idiot, or look smart on paper, but not. But you're now getting into the world where you have your own data. I'm sure it's in the neural network format that's vectorized, all embedded, up for retrieval and all that use case. All right, so now you're starting to build your neural connections. Is there a next level? How do you think about the next level of getting to that Jarvis, where there's a lot of content? Is there gaps in your system on data? Where do you see that going? Because neural network will get you right to the Jarvis answer.

Jean-Simon Venne

>> Well-

>> Or close enough. Well, you have-

Jean-Simon Venne

>> And I would love to get the feedback from Randall a little, because it's a challenge we're kind of carrying together. So using it as a BI or an analytic tool is one thing. Pushing it to be able to carry out mission, where you're going to be allowed to take decision and write back, to make executing the decision, is another level. And then, it's only then that you're really reaching the Jarvis, like an Iron Man level.

>> Well said, well said.

Jean-Simon Venne

>> And then, really you're reaching another level of productivity, because a human could say, "Well, you know what, I'm going for lunch. I'm going to give you this mission, please execute-"

>> Fix the valve while I'm gone. Long lunch that turned into.

Jean-Simon Venne

>> Let me know when it's done. So it's kind of a second steer, which is quite challenging, but at the same time, so exciting.

>> That's true next level, that would be legit next level. Randall, what's your reaction to that? Because this is really the conversation, it's hitting these levels and this is just trajectory. It's not so much, it's where we are now. So, I can see that next level coming, reinforced learning, reasoning, understanding more context.

Randall Hunt

>> I think it's important to imagine the LLM optimization pipeline as a graph, where your Y-axis is context optimization. That's where you're looking an answer up in an encyclopedia. So the context optimization is providing the resources, and references so that the LLM can infer a correct answer and cite that source. The LLM optimization along the X-axis is the umbrella of fine-tuning. And there's lots of things that we can do under that umbrella. There's continuous pre-training, where we can do unsupervised learning, where we're just feeding more tokens that the model may be unfamiliar with into the model, so that it can learn new vocabulary and learn new concepts. Then you have supervised fine-tuning, which is where you are going and picking out, I have a series of recorded inputs and recorded outputs. I would like for the model to fine-tune on this. And even under that umbrella you have things like parameter-efficient fine-tuning, using things like low-rank adaptations and all of these things, all of these techniques can be applied. And we do find that our most sophisticated customers are applying all of the above techniques. But you can often get started with just prompt engineering.

>> That's the entry level, right? You come in, understand how prompts work, that's an interactive, but prompts turn into automation.

Randall Hunt

>> Yeah.

>> Natural prompts happening on their own. And the machine learning spectrum is a function of data and people can code it. Or I mean, how do you manage the human labor on that? I mean, how would you advise clients? Because that's the path, right? I mean, okay, so how do I do that? Do I have to hire people? You guys do it? How do I run it? How do you see that playing out?

Randall Hunt

>> Well, one of the things that we enjoyed with working on BrainBox AI was, we work with, we don't work for, so it's not like we are some consultant, where we just go off into a hole and we chuck some code over the wall every week. We are deeply integrated with your team. And so, we work alongside the BrainBox AI team, so that we deeply understand their data and that we understand the formatting, the ways that we can access it. And we have a suite of evals, is what we call them, and it's a term of art, so everyone should know about it, where we go and we will run, I think we have 132 evals that we just run on every new model that comes out, and we can immediately tell if this model's going to perform better, worse for the specific use cases that we're testing. And so, we use multiple models within BrainBox AI and we built a lot of it on Bedrock. And I was saying earlier that with some of the changes in Bedrock agents, so we had built our own version of code interpreter before it existed in Bedrock. Now that it exists in Bedrock, we're going to take a look at how we could just leverage the code interpreter inside of Bedrock, as opposed to using the version that we wrote, which was a bit of a hack, if I'm honest.

>> Well, pre. Now they have leverage. You have leverage with the operating level of what they built, and then maybe you can add on.

Randall Hunt

>> And then, just to your point about the diffusion of knowledge within an organization, we found that once we get a project started with a customer, we're able to create that organizational muscle around leveraging generative AI. And so, once someone sees the initial success in it, they tend to double down on additional use cases.

>> And you guys get operating leverage off the charts. I mean, from a financial standpoint.

Jean-Simon Venne

>> But it's fascinating, because when you look at that kind of team, like you've got AWS supplying tools, and they keep producing new tools, giving us tools, and then we try to figure out how we're going to use the tools. And then you get a Caylent like brainstorming capability saying, okay, we could plug it like this, this. And then, we're coming with our team and BrainBox saying, okay, well we got this data, this data and we don't have that. And then, this is kind of becoming an incredible powerful team, which is moving the needle every week. And that's a secret, right? You keep moving and you don't know what you don't know, but when you hit the wall, you try to figure out how you move on.

>> Jean-Simon, congratulations on what you do. I really appreciate this case study. Great example of innovation. A builder's cloud, we always say Amazon is. Final minute we have left. Talk about some of the accolades you got on the efficiency in the reducing the emissions, because greenhouse gases, the sustainability is big. In fact, Amazon announced today that they're seven years ahead of their goal of a hundred percent neutral. Google not doing so well, but Amazon seven years early. You have a real impact in the world. What's some of the commentary and accolades you guys have gotten and how does that feel?

Jean-Simon Venne

>> I think we're still in this path to try to explain to everybody that technology could play an important role, right? Now, how do you apply it? So, the install base is already over 15,000 building. We're doing, example, some airport around the world, some building here in New York. And it's like, being able to shave without any CapEx investment. It's a pure software SaaS development, right? The deployment, you're shaving in between 20 and 25% energy, but the emission by 40%, and it's plug and play type of extraction data. It's quite fascinating.

>> And the bigger you get, the better it gets, right?

Jean-Simon Venne

>> You need to scale.

>> You can really have a huge, put a big dent in the universe so to speak, or in this case the climate.

Jean-Simon Venne

>> Yeah. And we have to remember that every month on this planet we're adding the equivalent of Manhattan in quantity of new building. So this kind of an infinite tab.

>> And also, you can have policy based stuff to manage buildings, and like you said, have policy based systems that are intelligent, and understand the environment and flex accordingly, and shut a floor down if it's not being used, and open to another floor up or direct people. Appreciate you guys. I know you-

Randall Hunt

>> A pleasure....

>> guys got to go. Appreciate it. Thanks for coming on. Appreciate it.

Jean-Simon Venne

>> Pleasure.

>> All right, more Cube coverage after this short break. We'll go right to the end for the interviews. They have to get pulled out and they go now. I'm John Furrier, host of theCUBE. We've got a great guest coming up next. Stay with us.

Jean-Simon Venne | BrainBox AI & Randall Hunt | Caylent

search

>> Welcome back, everyone, to theCube's coverage here in New York City for Amazon Webster. So I'm John Furrier, your host of theCube. Been covering all the new announcements. We've got Matt Wood coming up. We've got two great guests here. We've got a customer and partner, Randall Hunt, VP of cloud strategy, innovation at Caylent. And Jean-Simon Venne, Chief Technology Officer at BrainBox. You guys working together. Great example of AI. Welcome to theCube.

Randall Hunt

>> Thank you.

Jean-Simon Venne

>> Popup Cube, great to see you.

Randall Hunt

>> Good to see you again.

>> You've been Amazon for years as an Amazonian, now CTO, you've got customer partners together. AI has kind of been around for a while. You guys are working together. It's early days, right? We're seeing, this is clearly early days. Even now, we're just talking about, hey, models will work together. Oh hey, what, you think? Or hey, how do I work with a little small language model, large language model, and then what infrastructure do you run out? These are all the basic questions that are being discussed today. That's going to happen fast. You guys are doing work now with AI. What are some of the things you're seeing? What are you guys doing? Take a minute to explain what you're doing with AI and how you guys are involved.

Randall Hunt

>> So Caylent, we're an all-in on AWS, premier tier consulting partner. We build software for people. So a customer comes to us with an idea and we say we can build it, or sometimes we say we can't, because the models aren't there yet. And we've completed well over a hundred generative AI projects at this point. And what we've found is organizations like Jean-Simon's BrainBox AI, if they're willing to lean in and adopt a platform around generative AI, they're much more successful than the companies who are just trying to shove a model into something. The model is honestly the least durable part of any architecture, because they're improving so quickly. The platform that you build around the model, and the evals, and the testing, that's what we've seen with successful customers, and we've worked across every industry at this point. So we've done financial services, healthcare, life sciences, name them all, startup, enterprise, small business, DNB. And the key things that we've learned is, if you can get something into production within three months and you can get customers using it, getting feedback, you start that loop, those projects tend to be successful and they tend to move forward. But the ones that linger, and you spend a year and a half on R&D, those often don't make it.

>> They boil the ocean over or they miss the mark, the market's changed. Jean-Simon, talk about your side of the equation here, because obviously a couple of years ago there was the hype, people were experimenting with, and then the next year, this past year, a year ago you started to see real development, people couldn't get the low-hanging fruit, say I'm going to get these use cases. And then, now it's more like how do you scale it? So when you first started seeing opportunities, take us through the mindset. What did you look at? What was the view, what did you lock your eyes on, and how did you tackle it?

Jean-Simon Venne

>> When you look at what we've done in the last six years is we managed to connect over 15,000 building in 16 country. And we're having this, I would say the old AI with this deep learning model, which are giving us a prediction, and we're basically doing optimization in real time of these buildings. So we're kind of lowering usually the energy being spent by this building by 20 to 25%. What we start to realize a year ago is like, we have terabyte and terabyte and terabyte of data that we keep accumulating, and we keep generating more data. The prediction is an example, but we keep generating a lot more data for each of these building. And then, when you look at the dashboard that we had, it's like there's so many tabs. So the poor customer is looking at all that, going like, calling us to say, "On what tab do I find that graph again?" And it's kind of pretty obvious that it's overwhelming. And I think last summer there was this discussion with AWS and saying, you know what we should do? We should build Jarvis like an island man, using the LLM. And of course AWS went like, "Oh, that's not a small challenge, because you know what?"

>> The VPCs got to connect, they got to land.

Jean-Simon Venne

>> There's a latency problem, hallucination problem. How do you know it's the right answer? And I kind of forced the players, and said I want to build it. And he said, "Well, I mean honestly, let's build a team." And that's when basically Caylent came into the equation saying, "Okay, we need people that first of all had more experience than us, playing on the beta user on Bedrock." But then it was exactly what Randall was explaining. It's not just the LLM. You're really looking at building a ring around the LLM of tools agent, that will basically make sure that you're doing the right thing all the time. And that's what we've been building ever since last September, with a lot of wall being broke down. And it's funny, because when I look back where we were in December and where we are today, it's incredible the amount of work that was done, and then what we were successfully pushing forward.

>> They always say the people who break down those walls get bloody.

Jean-Simon Venne

>> Oh yeah.

>> And so, picked a good part with Randall. He's got Amazon wired, he's been there. He knows all a lot of the products. That's interesting is that, it's interesting in these new waves, it's like the people who have the problem and the people who solve the problem may be a mismatch. And this comes back down to like, hey, I've never really had a problem with a lot of data, but I have a solution for you, because I know you have a problem, so I'm going to engineer for it. So you have a lot of data, you're data full, you already had data, you had machine learning working, you had Amazon experience. So, good fit, I see that. But there's now some people don't have that problem. I just want to bolt on gen AI. So it seems to be having data, it seems to be the requirement. Do you see that same thing, where you got to know what you're doing with the data, or do they all have enough data to start? I mean, I guess that's my question. And in your case, you had a lot of data, so easy entry to get, you get a little bloody, but you had to get your arms around it. At least you knew what you could do.

Randall Hunt

>> Your data is definitely your differentiator when you're working with generative AI. So, anyone can invoke an LLM, do some prompt engineering, but it's the context that you're using to enrich the inference from the LLM. That's what differentiates your offering from some competitor's offering. And if you're able to find the right data, you can do a lot. And people focus too much, I believe, on finding data that is either in their existing OLTP store, or in their existing Redshift cluster, or analytics, or big data warehouse. They don't think about how they can enrich the context in real time with user interaction. One of the things that we built together was Aria, and maybe you want to talk about that, and-

>> Hold on, just want to back up before, because you said something I want to unpack real, we'll get to the Aria in second. Enriching that with either data or-

Randall Hunt

>> Where the user is....

>> or user data. So you've got user behavioral data from people, but also you can bring in other data sources, kind what Matt Wood was kind of getting to that today, or not.

Randall Hunt

>> And today they launched memory for Bedrock agents, which is able to store across 30 days the conversation histories, summaries, invocations, traces, all those good things. So you can enrich in real time. When I say enrich in real time with the context of the user, what I'm talking about is, where are they in the application? Where are they on the web page, what's the current state? What is their current intent? What are they trying to accomplish? And if you can get to that next best thought, next best action, that's when you have a generative AI solution that is going to wow your customers, because it's thinking ahead of what they're already doing.

>> I want to get into that and then we'll get into the solution, because I think this brings up the question of, if you believe that to be true, which by the way, I believe that, Randall, to be true, that's the use case, because that's generative, right? You're generating, but it requires data to be addressable in low latency real time milliseconds.

Randall Hunt

>> Yep.

>> Okay, a lot of people don't know... I got a data warehouse, it might not be able to be queried even in time. So context of having available data, it's got to be highly available or high availability has got to be in place. How do you guys look at that architecturally, what is the mindset? Because if you solve that problem, then you're at the next level. Are we there? Is the data architecture support that, do you guys see that? Are we seeing low latency generative queries or is that still on the roadmap?

Randall Hunt

>> I think every user is, every customer is on a different part of that spectrum. So accessing your OLTP store, or accessing Dynamo, or MemoryDB, those are trivial. And writing queries generatively is basically a solved use case at this point. There may be certain edge cases which you would evaluate through your evals and testing. Many customers don't have that data, to your point, in the right format. But I wouldn't say it's a blocker the way that it used to be.

>> Yeah, it's gettable. You can convert it.

Randall Hunt

>> I mean, here's the thing, if you're willing to spend a lot of money, you can just convert all of that unstructured data into structured data. The cost is just how many tokens are you going to put through? And not everyone's willing to spend that. The unit economics of it don't always make sense, but traditional data science and traditional big data workloads, ETL workloads, those are all still relevant, because you're trying to get the data into the format that's most accessible for the LMS.

>> And if you look at kind the projection of any new market, those economics will change as performance increases. So that self-optimizes over time and becomes a non-issue. And if they already have SQL, then that's comes some historical data too, right? I mean, you've got both.

Randall Hunt

>> And the unit economics right now, the way that it works is every single customer wants the highest quality tokens as quickly as possible at the lowest possible price. And that's what I mean when I say that the model is the least durable component of any architecture, because Haiku came out and immediately it's delivering high quality tokens quickly. Anthropic is innovating very quickly. So you always want to take advantage of that latest model.

>> Yeah, I think just like the comparisons to the web is a great one. Online speeds got faster as the online population increased, searching just got better. I think that's going to get better. I think really the application's key. So let's get into what you guys did next. I want to get into the, because you have a real powerful use case. I mean, you guys solve a problem for getting greenhouse gases. This is a real application. It's like, it's not fantasy. So take us through what you guys have at Aria. What's the application do? What specifically is the benefit?

Jean-Simon Venne

>> So when you look at the core tech itself, which has been in place like before last year, it's really modulating in real time without any human in the loop. All of the cooling, heating and ventilation of the buildings. And it's really reducing the footprint of the energy of that building by 20, 25% emissions. Sometime we push it to 40%, because the algorithm are clever enough to go work on the natural gas instead of the electricity, when the electricity is being produced by windmill, right? So that kind of modulation in real time is really getting and harvesting the value. So what we did as a next step is say, okay, this is great. We're delivering for the CFO of that organization. We're reducing that bill and for the system-

>> Yeah, and by the way, it could be a big bill too.

Jean-Simon Venne

>> Yeah, exactly. In this place-

>> I mean, empty building sitting there with no...

Jean-Simon Venne

>> You still maintain-

>> why, got to keep it low on maintenance.

Jean-Simon Venne

>> Absolutely. One of our customer approached us right before we started working with Caylent and said, "Okay, this is great. You're saving energy and emission, but we still have a lot of other lowing in food that we would like to resolve." And one of them is like, when you have a problem in the building, the building manager or the building engineer has to basically start to do the troubleshooting. So in this room it's too hot. So he's going to start to look in different data source and he's going to investigate, try to find what is the diagnostic, because it's probably a valve, or it's probably a malfunction of a motor somewhere. So that work takes one hour, two hour, and eventually you find that that's a problem with that valve, right? And then, then he knows, and then you're going to start to work and replace that valve, or find a way to open it manually. So when you look at Aria, it basically you ask the questions, can you investigate? And Aria goes connect to the existing data set and basically do that root cause analysis in a few minutes. So that 90 minutes is now becoming two minutes, and that person is now able to go to action and resolve that problem. So when you look at the productivity in a given day, this person usually leaves after the end of their shift with still a lot of problem unresolved, because eight hours and I'm done, right? And I'm fresh, I'm going to start work, keep calling, working on it tomorrow. So in this case, suddenly the productivity increase in our space of building management, and then building engineering, and there's a shortage of labor, which is incredible. And young generation don't go in that field. So we're coming with that tool, where it is giving the productivity gain, that the spirit-

>> It's like IT, a single pane of glass into the whole thing. But I think this is a use case that we're seeing across every vertical. IT, DevOps, security. Hey, I get the time it takes, I got to reduce the steps it takes to do the task. In this case-

Jean-Simon Venne

>> It's finding the root cause analysis.

>> Yeah, that's hard.

Jean-Simon Venne

>> Look at all the data-

>> Oh yeah, specialty and these knowledge is needed too. You're going to need to have specialism on that.

Jean-Simon Venne

>> Yeah, exactly. And what we're finding, it's interesting, we were testing Claude 3 when it came out, and clearly Claude 3 has some HVAC building engineering, building mechanic training. Because when it start to do the reasoning, you could tell that it did read a thermodynamic book. It understands it, which is kind of wow, this is-

>> Welcome to the digital twin world of talents. No, but this is a great use case. And I think this points to where we were just talking earlier on, the keynote announced with the industry analysis roundtable was, everyone's laughing about chat bots and then now agents. And we're like, well, agents are today, but one of the comments was that's going to be a primitive, like DLL was for Windows. You're going to see agents being under the covers doing things. So I can buy that. That's a good analogy, not quite accurate, but I get the analogy. As the agents happen, the next step is automation and reasoning. So this is reinforced learning comes in next. Are you guys looking at that direction? You got training and inference, so yeah, okay. They're trained on HVACs and buildings. You have data that, does it need to be trained? It's pretty much trained. Your data is trained. I mean, it's your data. I mean, does it be training or is it more of reinforced learning about if it's stale or not? But I'm sure it's like you don't really need to train your own data, do you?

Randall Hunt

>> Yeah, yep.

Jean-Simon Venne

>> Yeah, you have to review the data as a knowledge base basically. So you're tapping in that knowledge base and am basically benefiting that into the way you're going to answer that question. So really when you look at the role of the LLM, it's really use the LLM first to decode what is the human really asking? Prepare your plan of how I'm going to organize my strategy to answer that question. And then, really it's the worker that goes out, and go and get the knowledge, assemble all that knowledge together, so then you could return and start to prepare the answer with the LLM.

>> Okay, so let's get back to Jarvis, because I love that analogy. Again, why hallucinations are going to make him, Jarvis, kind of an idiot, or look smart on paper, but not. But you're now getting into the world where you have your own data. I'm sure it's in the neural network format that's vectorized, all embedded, up for retrieval and all that use case. All right, so now you're starting to build your neural connections. Is there a next level? How do you think about the next level of getting to that Jarvis, where there's a lot of content? Is there gaps in your system on data? Where do you see that going? Because neural network will get you right to the Jarvis answer.

Jean-Simon Venne

>> Well-

>> Or close enough. Well, you have-

Jean-Simon Venne

>> And I would love to get the feedback from Randall a little, because it's a challenge we're kind of carrying together. So using it as a BI or an analytic tool is one thing. Pushing it to be able to carry out mission, where you're going to be allowed to take decision and write back, to make executing the decision, is another level. And then, it's only then that you're really reaching the Jarvis, like an Iron Man level.

>> Well said, well said.

Jean-Simon Venne

>> And then, really you're reaching another level of productivity, because a human could say, "Well, you know what, I'm going for lunch. I'm going to give you this mission, please execute-"

>> Fix the valve while I'm gone. Long lunch that turned into.

Jean-Simon Venne

>> Let me know when it's done. So it's kind of a second steer, which is quite challenging, but at the same time, so exciting.

>> That's true next level, that would be legit next level. Randall, what's your reaction to that? Because this is really the conversation, it's hitting these levels and this is just trajectory. It's not so much, it's where we are now. So, I can see that next level coming, reinforced learning, reasoning, understanding more context.

Randall Hunt

>> I think it's important to imagine the LLM optimization pipeline as a graph, where your Y-axis is context optimization. That's where you're looking an answer up in an encyclopedia. So the context optimization is providing the resources, and references so that the LLM can infer a correct answer and cite that source. The LLM optimization along the X-axis is the umbrella of fine-tuning. And there's lots of things that we can do under that umbrella. There's continuous pre-training, where we can do unsupervised learning, where we're just feeding more tokens that the model may be unfamiliar with into the model, so that it can learn new vocabulary and learn new concepts. Then you have supervised fine-tuning, which is where you are going and picking out, I have a series of recorded inputs and recorded outputs. I would like for the model to fine-tune on this. And even under that umbrella you have things like parameter-efficient fine-tuning, using things like low-rank adaptations and all of these things, all of these techniques can be applied. And we do find that our most sophisticated customers are applying all of the above techniques. But you can often get started with just prompt engineering.

>> That's the entry level, right? You come in, understand how prompts work, that's an interactive, but prompts turn into automation.

Randall Hunt

>> Yeah.

>> Natural prompts happening on their own. And the machine learning spectrum is a function of data and people can code it. Or I mean, how do you manage the human labor on that? I mean, how would you advise clients? Because that's the path, right? I mean, okay, so how do I do that? Do I have to hire people? You guys do it? How do I run it? How do you see that playing out?

Randall Hunt

>> Well, one of the things that we enjoyed with working on BrainBox AI was, we work with, we don't work for, so it's not like we are some consultant, where we just go off into a hole and we chuck some code over the wall every week. We are deeply integrated with your team. And so, we work alongside the BrainBox AI team, so that we deeply understand their data and that we understand the formatting, the ways that we can access it. And we have a suite of evals, is what we call them, and it's a term of art, so everyone should know about it, where we go and we will run, I think we have 132 evals that we just run on every new model that comes out, and we can immediately tell if this model's going to perform better, worse for the specific use cases that we're testing. And so, we use multiple models within BrainBox AI and we built a lot of it on Bedrock. And I was saying earlier that with some of the changes in Bedrock agents, so we had built our own version of code interpreter before it existed in Bedrock. Now that it exists in Bedrock, we're going to take a look at how we could just leverage the code interpreter inside of Bedrock, as opposed to using the version that we wrote, which was a bit of a hack, if I'm honest.

>> Well, pre. Now they have leverage. You have leverage with the operating level of what they built, and then maybe you can add on.

Randall Hunt

>> And then, just to your point about the diffusion of knowledge within an organization, we found that once we get a project started with a customer, we're able to create that organizational muscle around leveraging generative AI. And so, once someone sees the initial success in it, they tend to double down on additional use cases.

>> And you guys get operating leverage off the charts. I mean, from a financial standpoint.

Jean-Simon Venne

>> But it's fascinating, because when you look at that kind of team, like you've got AWS supplying tools, and they keep producing new tools, giving us tools, and then we try to figure out how we're going to use the tools. And then you get a Caylent like brainstorming capability saying, okay, we could plug it like this, this. And then, we're coming with our team and BrainBox saying, okay, well we got this data, this data and we don't have that. And then, this is kind of becoming an incredible powerful team, which is moving the needle every week. And that's a secret, right? You keep moving and you don't know what you don't know, but when you hit the wall, you try to figure out how you move on.

>> Jean-Simon, congratulations on what you do. I really appreciate this case study. Great example of innovation. A builder's cloud, we always say Amazon is. Final minute we have left. Talk about some of the accolades you got on the efficiency in the reducing the emissions, because greenhouse gases, the sustainability is big. In fact, Amazon announced today that they're seven years ahead of their goal of a hundred percent neutral. Google not doing so well, but Amazon seven years early. You have a real impact in the world. What's some of the commentary and accolades you guys have gotten and how does that feel?

Jean-Simon Venne

>> I think we're still in this path to try to explain to everybody that technology could play an important role, right? Now, how do you apply it? So, the install base is already over 15,000 building. We're doing, example, some airport around the world, some building here in New York. And it's like, being able to shave without any CapEx investment. It's a pure software SaaS development, right? The deployment, you're shaving in between 20 and 25% energy, but the emission by 40%, and it's plug and play type of extraction data. It's quite fascinating.

>> And the bigger you get, the better it gets, right?

Jean-Simon Venne

>> You need to scale.

>> You can really have a huge, put a big dent in the universe so to speak, or in this case the climate.

Jean-Simon Venne

>> Yeah. And we have to remember that every month on this planet we're adding the equivalent of Manhattan in quantity of new building. So this kind of an infinite tab.

>> And also, you can have policy based stuff to manage buildings, and like you said, have policy based systems that are intelligent, and understand the environment and flex accordingly, and shut a floor down if it's not being used, and open to another floor up or direct people. Appreciate you guys. I know you-

Randall Hunt

>> A pleasure....

>> guys got to go. Appreciate it. Thanks for coming on. Appreciate it.

Jean-Simon Venne

>> Pleasure.

>> All right, more Cube coverage after this short break. We'll go right to the end for the interviews. They have to get pulled out and they go now. I'm John Furrier, host of theCUBE. We've got a great guest coming up next. Stay with us.