SC25 | Josh Grossman, Supermicro & Christopher Sullivan, Oregon State University

Clips
More from SC25

Josh Grossman

Principal Product Manager

Supermicro

Christopher Sullivan

Director of Research and Academic Computing for the College of Earth, Ocean and Atmospheric Sciences

Oregon State University

play_circle_outline Introduction of guests Josh and Chris discussing their project on spotted owls.

play_circle_outline Assessing the Impact of Barred Owls and Lumber Industry on Endangered Species' Habitats: Safeguarding Spotted Owls

play_circle_outline Advancing Forest Biodiversity Research: Autonomous Recording Units and Algorithms for Sound-Based Species Identification

play_circle_outline Future potential applications of technology in monitoring various endangered species.

play_circle_outline Positive impact of their work on owl populations and related ecological industries.

Info
Transcript

Josh Grossman, Supermicro & Christopher Sullivan, Oregon State University

Josh Grossman

Principal Product Manager Supermicro

Christopher Sullivan

Director of Research and Academic Computing for the College of Earth, Ocean and Atmospheric Sciences Oregon State University

In this interview from SC25, Josh Grossman from Supermicro and Chris Sullivan from Oregon State University join theCUBE’s Dave Vellante and Savannah Peterson to discuss a groundbreaking application of edge AI: saving the endangered spotted owl in the Pacific Northwest. Sullivan details how his team utilizes bioacoustics and autonomous recording units to monitor forest biodiversity, managing a massive data pipeline that generates hundreds of terabytes. The conversation highlights the critical shift from centralized model training to inference at the edge, illu... Read more

explore Keep Exploring

What project is being discussed that involves the monitoring of an endangered species? add

What strategies are being implemented to save the spotted owl and maintain its habitat? add

What measures are being taken to monitor the habitat of an endangered species in the Pacific Northwest? add

What should scientists do if they want to monitor specific species, such as abalone or whales, and how can they collaborate with companies like Supermicro? add

What impact has the initiative had on the owl population, and is it effective? add

bolt Powered by CUBE AI

Josh Grossman, Supermicro & Christopher Sullivan, Oregon State University

search

>> Good afternoon, nerd fam, and welcome back to St. Louis, Missouri. We're here midway through day one of our three days of coverage here on theCUBE. My name's Savannah Peterson. I'm really pumped to be bringing you a super special segment next with Dave Vellante. Dave, what's your favorite bird?

>> I think the owl might be my totem.

>> Is it?

>> Might be. It's either that or the cardinal.

>> I love it.

>> I haven't quite figured it out yet.

>> Those are very different birds.

>> It took time but it's ticking.

>> Yeah. Well, I mean I'm obviously into birds. There's a bird on my forearm, my mother's a birder. I'm all about the birds. And our next guests are going to tell us about how they are saving one of those breeds. Please welcome Josh and Chris. Thank you guys for being here.

>> Thank you for having us.

>> Thank you, guys.

>> It is not every day in enterprise tech land that I get to talk about spotted owls, so I got to say thank you for that.

>> Well, I'm always excited about spotted owls.

>> So we just got to dive right into that and then we will break down the tech that's powering this, but tell us about the project. Tell us how you're saving them.

>> So I work with the United States Forest Service to monitor spotted owl, which is an endangered species to help us what we consider maintain habitat. So some owls like the barred owl actually have crossed states and come into our habitat and they're taking away the resources. And that coupled with the lumber industry who farms trees in Oregon, they actually want to be able to take down trees without affecting habitat as well. And so our goal was, we've been working on this since 1996 and our goal is to really enable the lumber industry to do things in a positive way and to feel good. No one wants to-

>> That's not something that most people associate with logging, for example.

>> Yeah. Right.

>> It's not always the most sensitive group of people in terms of the public perception.

>> Correct.

>> So I think that's actually quite lovely to hear.

>> When we talk to them, no one wants to take down and hurt animals and everybody wants to maintain the forest. And we've even run simulations, which we'll talk about using the tech, that shows that hyperdiversity in forests is needed to actually maintain species. So if I don't have the animals, we can't disperse the seeds properly. And you actually go to mono dominance with the trees and you'll see that where you have just a pine tree forest. And when you actually go down to the Amazon, you have a tremendous amount of diversity and that's because of hyperdiversity maintaining all the species. And so this is an important aspect. And as we lose keystone species, the forest will collapse. And if that happens, we will not have the same planet that we lived in.

>> Oh my gosh. I mean that's the truth. I'm curious, so obviously endangered, how did you choose the spot at owl for this particular project?

>> It actually landed on the endangered species list, and so the United States Forest Service was asked to start monitoring the habitat for that. And so the Pacific Northwest is the main habitat. And so we actually installed... We right now have about 5,000 autonomous recording units between San Francisco and Canada that are actively sampling the sound. And what I did is I wrote an algorithm where we can find all the animals in the forest using sound. And so I can just record the forest and I can tell you what the species are. I can tell you when a tree falls in the forest and no one was there.

>> Does it make a sound?

>> Yeah, no, it does. It really does. I could tell you what .

>> Mystery solved, y'all. Mystery solved.

>> And so we actually got that up to about 130 species right now.

>> Wow.

>> Yeah. And so we try and couple that with camera trap data as well. And we can actually then get down to even male and female with the sound. I can do Doppler shift and tell you which direction they're heading. We can even get down to the individual bird sometimes and know and track that individual bird. Wow.

>> So, Josh, the edge has gone from this sort of data-rich, but information and insight-poor environment and now you're bringing AI to the edge. How should we be rethinking about AI at the edge?

>> Well, there's a whole kind of shift if you will. So we were in the previous kind of... And everything's so compressed time-wise, so let's say in the previous three to four years because that's when we sort of had this big bang moment with ChatGPT and so forth. As we move past the training of the models, now we're getting to the deployment of the inference models and that's I think what we really would focus on now. So there's still going to be a lot of training happening, but inference is going to really move into the fore and that's a lot of what Chris is doing.

>> Yeah. I mean, in the end we leveraging different technologies for training that we are at inference. And by pushing inference out towards the edge, we reduce the impact of managing the data and we also create a better temporal aspect to that data. If I can't get through the data quickly enough, it's meaningless for the forest and the lumber industry to actually make active decisions. And so we really need a more real time or semi-real time solution, which is why I need to push the compute out to the edge as far as possible.

>> And there are different workloads too. So when you ask a really complicated question to one of the AI bots about, "What should I do with my retirement?" And it's thinking, that's not the kind of question that you're going to have a server at the edge answering. But for something like this, it's absolutely perfect and those models are already trained. So there's a whole variety of different use cases and this is one that works perfectly at the edge for inference.

>> How much of the data do you persist? How do you decide what to persist?

>> So let's talk about the data for a minute because AI is a data problem, it's not a processing problem. The processing comes from data. And so right now between those autonomous recording units, we're collecting about 250 terabytes of data every month and a half. And I then need to process that, it turns into about a half a petabyte to a petabyte for one experiment. And we have to keep all the raw data. I have all the raw data going back to 1996. And that's because as I changed the models and I retrain the models on larger beautiful machines that we've been getting, we were able to go back to the data and actually pull more information back out. As I increase the number of species-

>> I was wondering about that because you've got quite the historic data there.

>> Yeah. We're trying to look at how things have changed over decades, not just small pictures of time. We do this not just in the forest but also the ocean. I have the Ocean Observatory Initiative as part of our grant and it's a $1 billion project where I have minute-level data coming from the ocean and it's lasting for 25 years. And so we have to have a both local temporal aspect, but a large temporal aspect so we can see changes over time because they don't happen rapidly, they happen slowly and gradually, and we'd be able to react in such a way that we can change it back.

>> So you store it all?

>> I store it all, and then I have petabytes and petabytes and petabytes. It's scary.

>> And that varies by use case, right? I mean-

>> And one of the interesting things is the model that Chris is talking about in terms of the workflow, that's a variety of different verticals. So it's interesting that that continuous pre-training and training of models with new data and then there's things like RAG which allow you to get even more context. That model for the way we're using AI works in a variety of different scenarios. So it's really interesting to see Chris has got a really interesting use case because it's saving the planet and it's awesome, but you can use that in a lot of different areas as well.

>> Well, that's what's great about the hardware is that it's able to be used across different experiments and I can move different experiments on and off the same piece of hardware. It's not like certain other types of lab equipment. We treat these like they're a piece of research equipment and we treat them research equipment. And in that sense, it's one of the greatest pieces of research equipment because it's-

>> Unbiased....

>> non-fungible, I can do anything with it. It's what I decide and how creative I can be as a scientist.

>> So let's dig in there because it's so cool that you have such a long history both of working together as well as with this data set. I mean, what a true snapshot of an environment and now all these other species. So I'm very curious about the journey as to when new things have been unlocked along the way and what you think might get unlocked as we continue to move forward.

>> So I just published another version of our network that we do and we are very excited about that. And that's the point is that what we're trying to do is really get down to individual birds and stuff like this and be able to separate pieces out a little bit more. And so as we build and the data is being held, we have to get the processing to match that. The other piece about this is, I mean, spending a tremendous amount of labor to go collect the data. So right now all of those devices have cards and so we can't just put Starlink on all the devices because the cost would go through the roof. And so I literally have to send people out to collect data right now. And that aspect of it is the actual part that limits us.

>> I bet.

>> The truck roll.

>> Yeah. And they're bringing me boxes of data and then I have to transfer it around and do all of this. And that delays the rate at which I can do things. And so working with Supermicro, they were able to build us a workstation that put liquid-cooled GPUs onto a workstation. Putting GPUs out in the forest is not easy. They take a lot of power and generate a lot of heat, and I don't have a data center out there.

>> Yeah. And you don't have a team monitoring that in a tree house somewhere to try and make that work. Yeah.

>> Right. And so what we worked with Supermicro to learn was that we could train on Grace Hoppers and then push out to this workstation that had liquid cooling built into it. And so I didn't need to have a data center, I didn't need to have cooling. I could put it at a ranger station and then try and do a wireless mesh from those devices back and that's how we should be working more. And so that way I can get semi-real-time or real-time data coming back so that way I can always know what's going on. And that's where the data needs to reside. Our limit has always been either processing or disk space. I could either take processing out there or I could take disk space out there, but I was never able to take both.

>> And now you can. And so we heard from Solidigm earlier that by next November they're going to have a 256 terabyte drive.

>> Yeah, smaller than our freaking cell phone.

>> You guys are going to double, triple, quadruple the GPU power that you can fit into the system.

>> Yeah. And the shared memory footprint is making a real game-changing.

>> So explain that, explain what that does to the economics.

>> So basic... Go ahead.

>> This is me. Okay, so I have so much data. So we're taking the sound data and we're talking about petabytes of it. And so when I start to try and train a model on this, the GPUs that we had in place were not going to do the job. And then I have to start moving to super pods and connecting multiple DGXs together. And then along came this thing called the Grace Hopper. And the Grace Hopper allowed us to take advantage of unified memory. That means that the system memory, even though it may not have the same level of performance, it changed the scope of work that we could do with the data sets that we had.

>> At a much better cost point too. That's one of the critical things.

>> It's the cost point. It's the fact that I'm not putting out a quarter million dollars to half a million dollars to do the same work.

>> Exactly.

>> I can get one of these machines for $30,000 to $60,000.

>> And the TDP and energy draw as well.

>> That's actually pretty low.

>> Because of the GPU costs are so much less expensive for an N minus one or?

>> Because before, we were taking multiple GPUs which have each a cost to them and trying to stitch them together to get memory. We're not doing that anymore. We take the system memory that I've already paid for on the box and I'm leveraging that. This was done a long time ago with Nvidia and IBM through a coherent accelerated processor interconnect called CAPI.

>> Yeah, right.

>> Yeah, .

>> In the end that was abandoned for what we have now. And so really unified memory is the end-all goal. And we've even seen that, in certain cases, I've gone to my CUDA code where I run the simulations to understand hyperdiversity and in one line I could change it one line of code and I took advantage of unified memory. How easy of a change is that to change your scope of work?

>> Wow.

>> And I went from landscape sizes that were very small to the size of the Amazon rainforest.

>> But I just want to put a finer point on it, make sure I'm clear, but when the next-generation GPU comes out and it's cost-effective enough, you'll want it or not necessarily?

>> Well, I do want it.

>> Okay. So what I'm trying to get to, because everybody's talking about the depreciation cycle, you saw the hyperscalers, they extended their depreciation cycle from Florida now six years, people are saying the depreciation cycle on these GPUs should be two years. But here's my question, will today's training GPUs become tomorrow's inference?

>> Yeah, so I actually just connect multiple Grace Hoppers together now.

>> And you can do both. So I think that's a yes, right?

>> Yes.

>> It depends on the workload, right?

>> Yeah, that's right.

>> So the reason we're getting these benefits of kind of the economies here is that where it used to be eight GPUs, you can have now one GPU and one Grace Hopper processor with 576 gigs of shared memory where you used to have to have eight GPUs. And so that kind of shared memory economics, and although as he mentioned it's tiered, so it's not all apples for apples, it's definitely good enough to do these sorts of training drills. So there's definitely going to be an opportunity to use those GPUs and inference I think as well.

>> So that's good news for the industry, all this-

>> The power's lower too.

>> All this bubble talk, and that was my next question, what does that do for the power envelope?

>> Right. So the power envelope for the large training data sets is going up, but there's also core efficiencies associated with that because of things like liquid cooling. And we've got the liquid cooling on the disk that Chris mentioned. But in addition to that, we've got everything from, you can do forced air cooling, you can do dry cooling, you can do side-cars, which are basically retrofitting an existing air-cooled data center with liquid. So there's lots of different opportunities and it's all about the workload. There's a variety of different workloads and they each have their specific sweet spot and we can help to tailor that so you make the most of your dollars.

>> And that hybrid configuration that's here to stay or is that like a stock cap?

>> Definitely. So what we're seeing is for the most part, Greenfield, which would be new data centers, are going to be fully liquid-cooled. But because of the power requirements, a lot of folks need to stay with their existing air-cooled data centers. And that's where this hybridization really plays an important role. And as you get to the edge, that's even more the case. And then there's a whole 'nother discussion we can have about containerized data centers where you've basically got the whole data center in a box.

>> Shipping a pod?

>> Yeah, exactly. So there's lots of different options and there's basically if you work with someone like us that can give you that expertise, we can help to walk you through that whole-

>> Let's talk about the water cooling and the performance. We've all been buying these GPUs and the air-cooling, you're actually giving up half the performance. We showed this with Supermicro and Nvidia. They allowed us to publish some work that shows that these A100s that were older were actually faster than the modern Grace Hopper. And it's because the liquid cooling gives you back the performance you paid for, but we're losing. And so we can stick A100s out into the field that are literally faster than the H100s that you're buying today.

>> Okay, that's wild. And this are-

>> This is great news.

>> It is. And this is really outer edge. I mean, this is the edge of the .

>> Yeah, that's right.

>> But in a really lovely way. Josh, I got a question for you. When Chris came to you and was curious to go do this, first of all, how did you go about even determining what was going to be the best to survive in these environments for an extended period of time?

>> Well, fortunately with somebody like Chris, Chris knows, as you probably have gathered listening to him, he has a very clear opinion on what he wants to do.

>> That design brief was pretty tight. Yeah.

>> I know the technology, guys.

>> So I was just very happy to work with him on it. But I want say I'm a former abalone diver, I want to make this stuff. They closed the abalone fishery in California. The utility of this in a lot of different areas is going to be huge, so I just want to help to spread the good news of this.

>> The other piece, I get a weekly call with Supermicro for the last I don't know how many years, and I get to talk to an engineer . She's an amazing engineer. She's the one who helped design the liquid cool GPU workstation. She had one under her desk. Talking with the engineer who designed the technology and having that level of access is not normal, but Supermicro rolls out the red carpet to me, and so they help me change the way we do science using technology.

>> Which is gorgeous. So you mentioned that this solution can work across a lot of different things, not just for the owls and not just for other animals in the forest. What are some of those other applications where it's best?

>> Particularly with the GH200 that Chris was talking about, we've just released a world-class benchmark and stack, which is a financial benchmark. So that would be in the FSI space, financial space. And then in additional, there's something called Gurobi which is, this is really interesting stuff, this is kind of manufacturing automation. So it allows you in regression. So in a nutshell, it's basically when do I turn on the power plant? When do I turn it off? When do I drill for oil? When do I stop drilling? Even if I'm building a new superstore, where should I put the displays based on past data? So there's lots of different areas where this makes sense and that's just with the GH200, there's all the other-

>> So I think it's important to understand that people were reluctant at first to go towards the GH200 because they saw it as a separate architecture and thought the software stacks weren't going to be there. And that's not true. We've been playing in ARM for a long time. We've had the Nvidia Jetsons, we've had a whole bunch of different ARM technologies, and all of them brought all that software forward. And so bringing these GH200s online was so easy. It was just they rolled out, we had them up in three hours, people were using them. And so I think I want to remove that concept that moving your workloads towards the ARM processor is going to be difficult. That's not true. We found just tremendous return from moving to that and it cost us nothing.

>> So how close are you now? I just want to go back, circle back, how close are you now to real-time monitoring? And what's the bottleneck? Is it compute, is it data, is it location?

>> It's humans bring the boxes?

>> It becomes the wireless mesh. I have students in the engineering department right now working on the wireless mesh using LoRaWAN and how we can increase because as I increase the bandwidth, I increase the power. And getting power in the forest is difficult. I mean, solar panels, there's a canopy, I don't get much sun. And so that's what we're working on right now and we are submitting more grants to get the money to buy those devices. Right now we have edge devices doing Starlink, but they're very few and far between because of the cost and the forest service asked us to move to a different model. And so if we can get the LoRaWAN and the wireless mesh to work, which is our last piece, we should be off to the races by the next year.

>> Okay. So, compute, you got covered. The data, you've got covered. It's really that wireless capability.

>> It's the interconnect between the two.

>> Yeah, it's the edge device to the-

>> It's the forest, the problem

>> To the ranger station.

>> Yeah. .

>> And the fact we're going to put these in ranger stations.

>> Yeah. Okay. So this is a fun game. So for the other environmentalists who might be listening, the scientists out there in the world right now who are watching this thinking, "Oh my goodness, we should monitor insert X species here."

>> Abalone, condor.

>> Yeah, abalone. I love that. I've been to the abalone farm in Cayucos when it was still open, when the fishery was still open.

>> Just think the whales, for example, shipping cargo vessels off the coast will hit whales and kill them. And they don't want to do that. How do we leverage this edge AI to put buoys out there that tell them where the whales are and so we don't lose them and stuff like this? We really need to implement-

>> So what would you tell that person who's thinking about that? How do they get started? How does the animal-loving scientist get started with somebody like Supermicro?

>> Well, actually you contact Supermicro and say, "Give me an engineer." And that's what they do. They just roll out engineers.

>> We will roll out the red carpet for you.

>> You really do make it sound easy.

>> It was though. That's why I'm saying. So that's why I have a 20-year relationship with them because I can always go to them and I can count on them to basically put the engineer behind it and say, "Go talk to the engineer. Here they are." And it doesn't matter how much time it takes, the engineer is always there and will continue the conversation and find the solution for me.

>> And the other thing is the product of a public university, this kind of fundamental science that's being done at our public universities that we support with the government's absolutely critical, so let's keep doing that. And it's creating huge results like this that in the private sector, people aren't really incentivized to do. So what Chris is doing is really... As you can probably imagine, you could have a different job in the private sector, so we really need to support and validate what Chris and folks are .

>> We need industry partners, especially right now as funding models are changing with the government. And we need to recognize that those changes affect your ability to get cool things to come out of science. I mean, the fact that GPS works and the fact that you have your phones and all these things that we have are driven from science first. And so we really need industry partners like Supermicro to help us get that message across and change the way we're doing stuff.

>> And we'll work with you to ring out the best possible value for your investment. And I'm personally, as you can tell, I'm excited about it because I support public universities and the work that he's doing.

>> For a good reason. Yeah.

>> I look at it like this. If I'm not here every year talking about how technology changed the science, you guys failed. All of this room failed, because that's in my opinion, what you're here to do. And every year there should be something new for me to apply and it should be exciting. That's the way it should be.

>> Yeah, it should be. All right, last question for you, how has this positively or otherwise impacted the owl population? Is it working?

>> So, it is and that's the benefit. That's why we've been doing it so long and that's why we keep pushing out publications around this is because it does work and the forest industry counts on us to do that. We farm trees, we've even put signs up, what year this was planted and they really count on us to communicate to them where those habitats are. We would like to be able to give them information on a daily basis, not monthly and stuff like this.

>> So hopefully you can say that soon .

>> And he didn't even talk about the dodging of crabs. You're saving that .

>> Yeah, .

>> And it's a great because these are public-private partnerships that really yield fruit.

>> So I use the same machine to go out and put on the ships and stuff like this as well. And so the crab industry's having us talk about putting them on their fishing boats because we put sensors and cameras into the thing so they don't have to bring them up. They're even trying to get us to put buoys out there so they don't have to take the boat out. They can just wake up, check the crab pots, and do something else all day. And I'm behind that. I don't want them using fuel.

>> I was just going to say .

>> This is a brilliant idea and that these types of devices are what I can put on a ship because it has the liquid cooling, I have a generator with enough power, it plugs into a 550 .

>> The architecture enables that.

>> It does.

>> So if you want to change the world for good, work with Supermicro and Chris and we'll work to make it happen.

>> You've convinced us.

>> Good.

>> Very much convinced us. Josh and Chris, thank you so much.

>> Thank you. Thanks, guys.

>> This has been not only fascinating and educational but also very heartwarming. Thanks for saving the owls.

>> Yeah, right. I'm going to share it with my kids.

>> Yeah.

>> AI's got some good things that come out of it as well.

>> Dad is going to work today.

>> Right.

>> We can confirm. Do you want to say hi to the kids? Go ahead.

>> Hi, guys.

>> Yeah, there you go.

>> Emily.

>> Right there. Yeah.

>> Steven, Emily. Hi.

>> All right, gentlemen, thank you so much and thank you, Dave.

>> Thank you, Savannah. Thank you, Dave.

>> Yes.

>> Thank you, guys.

>> Bye.

>> And thank all of you, wherever you might be or whatever tree you might be roosting in. We're here in St. Louis, Missouri. It's Supercomputing 2025. My name's Savannah Peterson. You're watching theCUBE, the leading source for enterprise tech news.