theCUBE + NYSE Wired present the East Coast AI Leaders Executive Series | Andrew Hoh, Lastmile.ai

Clips
More from theCUBE + NYSE Wired present the East Coast AI Leaders Executive Series

Andrew Hoh

Co-Founder

LastMile AI

play_circle_outline LastMile AI seed state company, focusing on democratizing AI technology

play_circle_outline Leveraging knowledge and experience from Facebook Meta in the AI space

play_circle_outline Risks and benefits of using open source vs. closed source AI models

play_circle_outline Building a Strong Team: Tackling Complex AI Challenges with Passion and Expertise

Info
Transcript

Andrew Hoh, Lastmile.ai

Andrew Hoh

Co-Founder LastMile AI

At the NYSE for theCUBE Studio East, Andrew discusses LastMile.AI's focus on democratizing AI technology. The company specializes in evaluating and deploying large language models for Fortune 500 customers. They stress the importance of robust review processes and testing for production-grade systems. LastMile.AI highlights challenges with using foundation models from others and the importance of thorough inspection and testing before integration. Based in New York City, they are seeking individuals passionate about machine learning and infrastructure. Excite... Read more

explore Keep Exploring

What is the focus and mission of the company that raised their seed funding from Gradient Ventures and was started by a team from Facebook? add

What lessons can be learned from the machine learning efforts and infrastructure at Facebook/Meta for other companies looking to implement similar technologies? add

What has been the impact of the open source models on the market and developers? add

What is the core DNA of the company and what type of problems are they passionate about solving? add

bolt Powered by CUBE AI

Andrew Hoh, Lastmile.ai

search

>> Hello. Welcome back to theCUBE. Are here at the NYSE for theCUBE Studio East, part of our NYSE Wired Community partnership with theCUBE, linking Silicon Valley and Wall Street. I'm John Furrier, your host of theCUBE. Just heard the closing bell. We're wrapping up day one of three days of wall-to-wall coverage on Wall Street, and of course, all the top experts on the East Coast are gathering the AI. We're here with Andrew, the co-founder and president of LastMile.AI. Andrew, great to see you. Thanks for coming on theCUBE here in the NYSE.

>> Yeah. Thank you for having me. Pleasure to be here. Pleasure to talk to you about AI. We were riffing a little bit before, so I'm excited for the conversation.

>> Well, let's set the table. Take a minute to explain what you guys do, where you are at your journey, funding, product, stats, give us the numbers.

>> Yeah. So we are a seed state company. We raised our seed from Gradient Ventures, which is Google's AI Fund. We raised 10 million from them, and the company started from a team from Facebook. We all left to start this company and what we're really focused on is how do we democratize this AI, this new technology for the rest of the world. Essentially, we've been building it out at a company like Facebook and we thought with generative AI taking off, how do we really build that out for everyone else and bring the power to everyone else as well? Specifically, we focus a lot on evaluation and testing. We call ourselves LastMile AI because there is a last mile of work that is so incredibly difficult with testing productionizing, and so that's what we help out with. We help with evaluating testing and bringing people along for that last mile.

>> So the last leg of the journey, so to speak, in what context, app infrastructure, what specifically LastMile, can you just share that piece of it or is it all LastMile?

>> It's a combination of infrastructure and machine learning. So when you're working with these large language models, you find yourself building applications and you want to either fine tune the model, you want to make a RAG system, you want to add your own data. Oftentimes, it's really difficult to get the performance you need. Oftentimes it's even difficult to figure out whether it's working properly or not. So what we do is we help you on that last bit of the journey, especially with testing and evaluation where we want to make sure that you feel comfortable with what you're releasing to your customers.

>> Yeah. I've been living in Palo Alto for the past 25, 26 years, it's fun to watch the Facebook rise in 2007, when they opened up the platform, and knowing a lot of friends that worked there. I know for a fact it's been a huge machine learning effort. Vertex Ventures guys got Jonathan over there and a lot of heavy duty techies coming out of Facebook. There was a ton of work on data. Obviously, you guys also where Facebook was known for open sourcing, the open compute spec, which came out of Facebook and Microsoft, but there was a ton of machine learning at Facebook, now Meta and recent last, the AI stuff's been phenomenal. So you had large scale there. So when you look at the last mile, what did you learn the most from the Facebook Meta infrastructure that jumped out at you and said, "Okay. Facebook is a unicorn in itself, is in the stone requirements." Not everyone is Facebook from a scale perspective, but when you go to other people, they have those problems now. So that seems to be the value. What were those problems? How do they translate to the broader market?

>> Yeah, really a good question. So what was really exciting for us was working in the ML space. Me personally, I've been in it for probably seven to eight years, always difficult for companies outside of these tech companies to train their own models. So oftentimes as we were working on this infrastructure in ML, there's basically tech companies who can find the ML talent to build these models and clean the data. But oftentimes it's a luxury to have it. Nowadays with foundation models and large language models, a large bulk of it is there, Facebook, open source Llama, a bunch of great open source models. So now there's this last mile worth of work to get that to work for you and to really empower your business. And that's what we saw, which is we are building all these things inside of Facebook and we found that we're building it for truly from zero to one now that they've open sourced a lot of these. Now all of these problems, interesting problems that happen at the end are still unsolved, and so let's go make it for the world.

>> Yeah. What's great about is the open source tsunami of goodness that has hit the market. It's been quite the developer frenzy. I mean, put it in perspective. For the folks that are watching that, may not understand the scope and magnitude of what's happened in the past 3 years, even the past 24 months. The explosion of enthusiasm and the frothiness and the intoxication of the developer community has just been off the charts. Give us a taste. It's the younger generation, it's the old and new, it's the systems guys and the young guns, I mean it's in everyone in.

>> Everyone is so excited about it. I think because it has that magical first moment. The first time you write a query or a prompt to ChatGPT and it shows you a response, it opens up a light bulb. We've seen it with executives, engineers, data scientists, ML. Everybody sees the potential and everyone gets excited for it. So when you look at the open source community, there's a lot of popular open source libraries and frameworks on how to use these models, chaining a lot of these models together, more automation, now agentic workflows. You've probably talked a lot of agentic folks who are big into agententify everything.

>> We just wrote three killer pieces on the economist about digital twins and how that's going to come from manufacturing to every process. I mean, it's just been an amazing run of just creativity. I think there's going to be some real breakthroughs that we haven't even seen yet. I think you mentioned string chaining things together. You're going to start to see people start doing more of that, less of this monolithic coding base. It's going to be a lot of like re-usability, collaboration, connected, intentional relationships forming.

>> That's the cool thing. I know we talked a little bit about machine-level code. These models are extremely good at creating code and code generation, and so oftentimes when people start thinking about how computers are made and the operating system, you think about all of the machine-level operations. And then you start thinking about what if they can be dynamically generated by a large language model? And this is why there's such a big frenzy on what is HCI in the future where human-computer interaction, it feels like it's ripe for change. When you go lower and lower and you start using an LN.

>> It's funny you said HCI and like hyperconvergence infrastructure. Well, they're both going on. You have HCI and the hyperconvergence, which basically made Facebook huge is now being democratized and open source. I mean, Jensen Wong at Nvidia says GenAI is bringing supercomputing to the masses. They are. And that's a total democratization wave, but human-computer interaction, the other half of HCI I, that interface is going to be audio. Hey, Cube. Give me my videos. Hey, Facebook. Hey, Meta. Hey, my glasses, my Ray-Bans. This is all now augmenting the human in the loop.

>> Exactly. The best way I view it is it feels like mouse and keyboard is still a little bit outdated in terms of all of the other information. We talk about how audio, video site, all of it has so much data that can be used to help convey your intentions and really transcribe that into making a device such as a computer work for you. So it's exciting times. It is really exciting.

>> Andrew, you mentioned earlier about the machine, we were saying on cameras as connected dots with the folks watching that people are coding down to the chip level, kernel level. Getting as close down to the hardware is possible because there's efficiencies there that are game-changing to the app, which translates into revenue or the value proposition of HCI on both sides. This is a huge trend. You agree with that? You're seeing that happen?

>> Yeah.

>> Okay, cool. Okay. Check. No debate. That's it.

>> It's expensive.

>> Okay. No debate. So I've been saying this for two years, last year hardcore. Okay. But there's money there. Perplexity is new. Everyone's doing because there's benefits. That's where the money is. And smart money goes to the way action is. Okay. What does that enable? I mean, that brings back the the small ball, I call the baseball analogy. Play some small ball, smaller code dealing with memory. Back in the eighties, what I was coding, you had to do memory management tricks, swap out the disk. Now you have different constraints. You have design around how many clusters you're dealing with. So the scale and the new AI infrastructure, you get down to the low level. There is efficiencies in this new abstraction layer on top that will enable data layers, data pipelines, hydration of data, so a whole other level of coolness that goes on. If you get it right down here, your reaction to that?

>> The big with AI and ML always is that eventually it'll get a lot cheaper. Everybody's betting that it'll get a lot cheaper, which means once it does get cheaper, what are all of the tools, constant and VC talk picks and shovels that people need to really scale up? You can imagine all the applications, all the use cases that will use it because it's gotten so much cheaper right now when you look at OpenAI's o1 model and you see how expensive it's on inference, you start thinking, "Is this really possible to use that frequently?" But everybody's thinking about how this is going to get that much cheaper. So one step further is thinking if it's really, really cheap, where can I use it? How can I use it and how can I improve my business?

>> All right. Andrew, you guys just formed out of Facebook, you and your friends. Talk about the journey. What was the moment when you came together? What was the vision? What's your vision for the team? Where are you with the company? How many people are on the team? I'll say 10 million is a nice seed round. Not too shabby. Okay. All good. So I mean, you're on a great wave right now, so you're looking good. What's the vision and where are you?

>> Yeah. So we're currently a team of 12. A lot of us come from big tech and a lot of us been working either in the infrastructure space, the ML infrastructure space or the machine learning space. And right now we're building that platform out for testing and evaluating, and we're bringing Fortune 500 customers to production. Constantly, when we listen to all of our customers and potential customers, their biggest feedback is, "We've started a POC. It's really, really hard to get us into production. What can you do to help?" And that's where we come in and we really shine. The more that we go through this, the more we're finding that this new flavor of ML infrastructure for January needs to exist. So a lot of things that we're building applies quite well across many use cases. And funny enough, you've been in this industry for so long, it's very similar technology to old school ones, like whether it's a model server for general BI or a model server... It runs the same.

>> It's funny. I taught this old school security guy at the Google Mandiant conference, hardcore threat intelligence, and they're skeptical, but they're definitely leveraging, they're leaning in and they said, just candidly, "Look, it's just another application," and he's not wrong, but pragmatically, that's what it is. Just abstract review. I got to onboard it, going to go into production. It's the same game, but different mechanisms now because of the requirements.

>> Yeah, that's exactly right.

>> What are some of those requirements around the onboarding to production? I mean, obviously Meta, you do one little thing wrong in production, that thing blows up, roll back, try again.

>> Yeah. The really wild thing is first, I think Meta has a big philosophy of online experimentation. You're testing things out. AV tests are running all the time to figure out what's best for users. For Fortune 500 customers, especially FinTech, finance firms as well as insurance, they don't really have that AV test. We cannot have anything happen where it's misinformation, any hallucination. It needs to be rock solid. So what we spend a lot of time is how do we build the right rubric and criteria, and then how do we clean the data and make sure that your system's better so it can be launched into production? So we find that we have the strictest review possible, and oftentimes companies struggle to really figure out exactly why it's not right. They have that eyeball test. You give it to someone on your team, they look at it, they're like, "Looks good." Somebody else says it doesn't work. So what we do is we decode that. We actually train models. We train models to evaluate it, and then we seed that information back into your system so you can refine it and get something production grade. So it's been a really fruitful experience getting more customers happy.

>> Yeah. Must be a lot of fun.

>> Yeah.

>> So talk about the funness of having this experience where you're now going to be onboarding all these apps you had experienced. The one question that I've been asking folks, and I'd love to see what your thoughts on this is that the word resilience has been defined from mostly storage/cyber recovery, like ransomware. Like, "Okay. Shoot, we've been bricked," or, "Okay, pay the Bitcoin roll back." Recovery is a concept we all know, but GenAI, how do you recover? Is there resilience? Is there observability? Are there the tools fully in place if someone... We saw this in the news, I think we covered it, where someone had an open S-free bucket with passwords unencrypted, it got trained. Okay. You got to unwind that. I mean, this is scary stuff. When you say, "Okay. But there are people working on this." This is resilience. This is going to be table stakes. This is where people say, "Okay. Show me the resilience, and then you're in production." What's your thoughts on this?

>> Yeah. So a lot of the best practices on resiliency come from ML into the GenAI space. And I'll touch upon what's different, but you typically want to know what's the data lineage, what's the data that's going in? Audit logs, I want to know what went in and what came out, and I want to verify whether it's right or wrong, and what tests actually ran. What are the suite of tests? And then ideally, you also incorporate some tests on biases, the ethics around it. And some companies have an ML committee, they'll look and review as well, the use case. And then the GenAI part, which makes it so difficult, is sometimes you don't own the foundation model, and so you're using somebody else's trained model and you're using it for your system. So where does the liability lie and how do I make sure that we are risk-free? And part of that is really understanding which model are you depending on? And so there's open source models where people can at least inspect a bit more, run more tests on it before they bring it into their organization. Closed source models, you're hoping that the providers have-

>> You're at the whim of the closed source.

>> Maybe model cards or white papers on how they've done it, but it is a little bit rolling dice of it because you can never really inspect.

>> So your only real solution is on the close side, is have the good grounding and evaluating either in real time the zillions of responses.

>> That's right. But it's expensive, so they get to you the other way where if you're validating it, then it is very expensive.

>> Well, the good news with the Llama models, look at the cost side of the tokens and the hosting, it's worth building your own infrastructure to host a Llama model than to go with the close on that side.

>> Yeah.

>> Okay. Great. How's the journey for you guys? Where's your head? You guys having fun?

>> Yeah. I mean, it's a great journey. I think now's the time to work on our business because AI's really taking off and we've been working in this space heads down for a while. Now that AI's really taking off, it's fun to contribute back to the community. Bring in some thoughts about how reviewing the space and making impact with the product. A lot of exciting news. I think the last big one was o1 that came out, which was super exciting. A model that does reasoning really opens the door for a lot of new use cases. And we've talked about multimodality. That's also a really cool one that's coming. So it's exciting. I think this is the time to really use these models and learn them because the space is moving so fast and the technology is so impactful, and so people ahead of it will yield the gain.

>> Where are you based out of?

>> New York City. We're in the Empire State Building.

>> So your team's here?

>> Yeah, that's right.

>> All right. Well, put a plug in for what you guys are looking for. Are you hiring? What kind of people? What's the DNA of the company? What do you guys wake up in the morning and say, "Hey, we just love to eat machine learning for breakfast?" What's the vibe? What's the DNA of the company? Give a plug.

>> Yeah. The DNA of the company is if you really like solving hard problems, especially with machine learning and infrastructure, that is our core DNA. We've been doing it for so long across many different companies. We're really passionate about it. So when we see challenging problems, such as the how do you trust large language models, we get excited and we start thinking about the research side and technology side and how to apply it. So if you're really interested in that and you're interested in machine learning or infrastructure, reach out to us in New York.

>> And if you're reading papers on the weekend for fun, then you're definitely a good candidate. I mean, the paper tsunami that's coming out, the research that's coming out, really amazing work. I mean, I read a paper on LLM routing last week, just serendipity. Someone passed, that's pretty cool. They're talking about how to audition versus predictive LLM routing around. To your point about what language model should I use at any given time? And that's a policy network problem. That's a networking problem applied to a data layer. That's fascinating.

>> That's the beautiful thing that oftentimes I see a lot of computer science principles be brought back in. So deciding on which models to use is a networking problem. And we see also quorum selection in terms of how do I select the best answer when I run three, that's a distributed data problem. It's been there for so long. It's researching it, but as cooler names now. I think AI has given a lot cooler names.

>> And the scale and the magnitude of the data is phenomenal. Well, we're in New York. We're going to be doing a lot more. We'll certainly definitely see at our events where we put on with NYSE and theCUBE partnering together. Great stuff. Andrew, thanks for coming on.

>> Fantastic.

>> Okay. You're watching theCUBE here. We wrap up day one of Media Week. And starting next year, you'll see us every day here, programming here at the NYSE. I'm John Furrier, your host at theCUBE. Thanks for watching.