KubeCon + CloudNativeCon NA 2024 | Morgan McClean, Splunk

Clips
News
More from KubeCon + CloudNativeCon NA 2024

Morgan McLean

OpenTelemetry Co-founder, Senior Director of PM

Splunk

Profiling and workload management: MultiKueue and OpenTelemetry’s role in AI and observability

KubeCon is a simultaneous celebration and solemn assessment of open-source advancement over time. A major theme at this week’s event is MultiKueue, an artificial intelligence-driven project that helps manage distributed workloads across multiple clusters.MultiKueue and OpenTelemetry are two such initiatives enabling AI models to queue tasks efficiently, particularly across multicloud environments.Splunk’s Morgan McLean discusses observability innovations with theCUBE.“There’s lots of things happening in OpenTelemetry,” said Morgan McLean (pictured), OpenTelemetry co-founder and senior director of product management at Splunk

play_circle_outline OpenTelemetry adoption in major banks, airlines, businesses

play_circle_outline Importance of observability tools in analyzing complexity

play_circle_outline State of observability report showing 78% IT leaders value OpenTelemetry

play_circle_outline Latest Updates on OpenTelemetry Adoption and Exciting Insights from Splunk and AppDynamics Revealed

Info
Transcript

Morgan McClean, Splunk

Morgan McLean

OpenTelemetry Co-founder, Senior Director of PM Splunk

Morgan, an OpenTelemetry expert, discussed the project's rapid adoption and value in the market. He highlighted the benefits of OpenTelemetry in providing insights and capturing data from various sources across organizations. He also mentioned the upcoming integration of AppDynamics into Splunk, emphasizing the added capabilities for high-scale web services and three-tier applications.

The State of Observability report revealed that 78% of IT leaders are either using or planning to use OpenTelemetry and seeing value from it. This adoption is attribut... Read more

explore Keep Exploring

What has been the adoption rate of OpenTelemetry compared to other CNCF projects, such as Kubernetes? add

What is the real power that OpenTelemetry brings in terms of analyzing complexity in cloud services and infrastructure? add

What percentage of IT leaders are either using or planning to use OpenTelemetry and seeing value from it? add

What are some exciting developments and future plans related to projects like OpenTelemetry and related capabilities like Splunk Observability and AppDynamics? add

bolt Powered by CUBE AI

Morgan McClean, Splunk

search

Savannah Peterson

>> Good evening, nerd fam, and welcome back to Salt Lake City, Utah. We are here at the very last segment of our first day of coverage here at KubeCon North America. Two more days of coverage still to come. My name's Savannah Peterson, here with the one and only Rob Strechay. Rob, what is the coolest thing you saw today that didn't say, "Kubernetes," on it?

Rob Strechay

>> Oh, that didn't say Kubernetes on it?

Savannah Peterson

>> I'm really throwing you a wildcard here on the last thing.

Rob Strechay

>> Yeah, well, I like the new project, MultiKueue

Savannah Peterson

>> Oh, that counts.

Rob Strechay

>> And it's K-U-E-U-E, so it took me a little time to figure out-

Morgan McLean

>> It's going to take a lot of people a bit of time.

Rob Strechay

>> Spelling's fundamental and not for me.

Savannah Peterson

>> With you and I being dyslexic, right now my brain is already like, "I don't know."

Rob Strechay

>> It hurt to do that. But that was really cool about distributing AI out with multiple queues across different clusters and multiple clusters. And the demo that was done by CERN was fantastic.

Savannah Peterson

>> I'm going to have to check that out. I haven't had a chance.

Rob Strechay

>> It was not technically all Kubernetes.

Savannah Peterson

>> Yeah. No, it's all good. We're here to celebrate Kubernetes.

Morgan McLean

>> Every time I see CERN demo something I'm always amazed, and I remember. I'm like, "Oh yeah, the first web server was at CERN." They have a long history-

Rob Strechay

>> And the colliders and all the amount of data that they're doing. But why don't we welcome him in?

Savannah Peterson

>> I was going to welcome him. I love that our conversation is already so natural. Morgan's like, "What's up? I'm here anyway." Morgan, thank you so much for taking the time to come hang out with us.

Morgan McLean

>> Thank you for having me. Yeah, I'm happy to be here.

Savannah Peterson

>> Yeah. So speaking of projects, a lot of OpenTelemetry fans in the house.

Morgan McLean

>> Yes. Yep.

Savannah Peterson

>> You are one of the origin minds of that project.

Morgan McLean

>> Yes. Yep.

Savannah Peterson

>> What's going on in OpenTelemetry land right now?

Morgan McLean

>> Lots. At the conference, lots going on because we have our own fancy project booth. I don't know if you've seen it, but-

Savannah Peterson

>> I actually went by. I was very impressed.

Morgan McLean

>> It's the third time we've done it. It's a really, really nice space for the community. But no, there's lots of things happening in OpenTelemetry. It's been now, for several years, the second largest or second most active CNCF project, Kubernetes of course being the biggest one. But I think in the last year or two the adoption of the project has really taken off. I've said this in interviews like this before, but when we started OpenTelemetry, if you told me it would be as widely used across whole swaths of the market as it is today, I would've been a little skeptical that it would be so rapidly adopted. And yet we see major banks, airlines, businesses that are only relatively early in their Kubernetes adoption journey already diving into OTel and adopting it widely, which is very, very exciting for us.

Rob Strechay

>> Yeah. I think when you look at it, and obviously being from Splunk, and you see a lot in the observability market in general.

Morgan McLean

>> Yep.

Rob Strechay

>> Before that you saw it from the cloud hyperscaler side.

Morgan McLean

>> Yes.

Rob Strechay

>> Same with me.

Morgan McLean

>> And that's where I was working previously. Yeah.

Rob Strechay

>> And it was a different one.

Morgan McLean

>> Yep.

Rob Strechay

>> But when you start to look at how complicated things are, to me, OTel made a whole ton of sense for like you're saying because it was easy.

Morgan McLean

>> Yeah.

Rob Strechay

>> And has that always been one of the design points, where you looked at-

Morgan McLean

>> It has been.

Rob Strechay

>> Yeah.

Morgan McLean

>> And I mean even broader than OTel is observability tools. Cloud services, cloud infrastructure, systems like Kubernetes have made it very, very easy for random firms of any size to scale up and build really exciting web services that, 15 years ago you would've had to go buy a data center and staff it in order to scale up like that. That's really great. But the challenge is it also induces a ton of complexity, and you need observability tools to analyze that. And to use observability tools, you need to extract the right information out of your infrastructure, out of your services, which seems like it should be easy and yet, prior to OpenTelemetry, it wasn't. And so that's the real power that OpenTelemetry brings, is you can get the data you need out so you can understand what's going on and how to fix things.

Savannah Peterson

>> Yeah, absolutely. You mentioned two things I want to tie together there a little bit. You mentioned that you've seen really rapid adoption in the last year or two years.

Morgan McLean

>> Two years in particular, yeah.

Savannah Peterson

>> In particular. That happens to parallel the adoption of a few other things, an acronym a lot of people are talking about. Starts with A and then ends with I. Do you think that those things are in parallel? Do you think that the things that people are building are more complex right now?

Morgan McLean

>> Those are more complex because Kubernetes itself is so well-adopted, because all of these other sort of systems now exist where you can build things at high scale if you choose to and if you'd need to. And so OpenTelemetry just filled that void of, well, if you build it, you now need to be able to operate it. You now need to be able to extend it. You now need to be able to understand what it is you built so when your new junior engineers join the team, they don't have to all bother the same person on a whiteboard to go tell them how the 100 services work. And as the person who had to draw the whiteboard drawing before any of this stuff existed, that's-

Savannah Peterson

>> Sensitive to that.

Morgan McLean

>> Yeah.

Savannah Peterson

>> I feel that as you're saying it out loud.

Morgan McLean

>> Yeah, it resonates with me. And so it's critical. And so it fills that void that was totally empty and that was, in some ways, blocking adoption of systems like Kubernetes and others because you could build something that would become unmanageable very, very quickly.

Rob Strechay

>> I think one of the things, and this is near and dear to my heart, is the profiling feature that is coming to OTel.

Morgan McLean

>> Yeah.

Rob Strechay

>> And fill people into that that aren't as deep into OTel, and what that is.

Morgan McLean

>> It's the perfect time to.

Rob Strechay

>> Yeah.

Morgan McLean

>> If you wind the clock way, way back before Kubernetes, before cloud services, before any of this, you'd be running some sort of web service or online service that people could access and you would use logs to analyze it. And probably by the mid-2000s, even the most relatively green engineer graduating from a CS program knew that if you walked into a company and you're trying to run things at some kind of scale, and if they're not even looking at logs, you should probably walk right out the door and find a new job. And that became true with monitoring later over time. And I think distributed tracing we saw with OpenTelemetry, that really opened that up. And you've seen a of observability in APM products as a result. That's great. People understand those now. They didn't understand tracing when I started doing this sort of gig, but we've changed that. Profiling is one that's very, I think, extremely powerful and yet is not that well-known. And it resonates with me. I work at Splunk now. I used to work at Google. Google does system-wide profiling of their entire fleet and they save a lot of money every year by doing this, by finding slow code. When I was at Google, we launched what I think is the world's first distributed profiling product. And amongst our biggest customers it was very, very successful. They loved it. They could save a ton of money with it. It's great. And since then, various observability vendors, Splunk and various others have brought out similar features. This is great, but with most of them, all of them, it remains somewhat niche. Everyone knows what logging is, everyone knows what monitoring is, everyone knows what tracing is. And you talk about profiling, and the only people who sort of know what it is are ones where their bill is on the order of like $50 million a month for compute. And that always kind of irked me. I love working on it. It's really exciting to me, being able to look at your code and actually understand which functions within your code are costing you a ton of money is powerful and exciting.

Savannah Peterson

>> It's really powerful.

Morgan McLean

>> And you can't do it at development time. You can profile stuff at development time but for a large production system, you can never emulate its load properly on a single box, or even run it. And so it gives you insights that you have no other way of capturing, really, for real workloads, and you can save tons of money with it. Because as an engineer, maybe you wrote something that's slow. You have no way of knowing it.

Savannah Peterson

>> Right.

Morgan McLean

>> As soon as you get profiling going, you can see it.

Savannah Peterson

>> It's like process mining. This is process mining on the-

Morgan McLean

>> Sort of, yeah.

Savannah Peterson

>> To a degree. It's very... yeah-

Morgan McLean

>> It's even deeper. You're actually looking into the actual code and understanding-

Savannah Peterson

>> Which is amazing.

Morgan McLean

>> I'm exaggerating. It's not line by line but it's function by function and see.

Savannah Peterson

>> Yeah.

Morgan McLean

>> And so you see those calls and those call stacks, and you look at a flame graph and you can understand, like, "Wait a second. This thing that says it's using half my CPU, that's a random regular expression I wrote to concatenate a few things."

Savannah Peterson

>> I can imagine this illuminates a lot of easy wins, to a degree, where you wouldn't have-

Morgan McLean

>> I think they're NDA, but we have some customers at Google, some of the biggest ones, who saved obscene amounts of money within like an hour or two of looking at a profiler and being like, "Hey, your profiler's broken. It says that I'm burning like 1/3 of my compute on this random function." We're like, "It's not broken." And they're like, "Oh boy." But you have no way of knowing otherwise.

Rob Strechay

>> Funny enough, they don't want to be known for writing bad code.

Morgan McLean

>> Well, it's not bad because you look at it and anyone could've done it.

Rob Strechay

>> Yeah.

Morgan McLean

>> You could be the best engineer in the world, but if you have no way of knowing, there's literally no signal coming back to you, how would you know?

Savannah Peterson

>> Right.

Morgan McLean

>> And with OTel, with this coming in, there's numerous benefits but, frankly, I think the biggest one is it'll just go more mainstream. More people will be exposed to it, more people will use it. And then I think you'll see profiling analytics get integrated into more and more things. Whether it's lower down in the stack, or maybe you're in your development environment. Maybe you're searching through your source code and it's actually highlighting things, saying, "This is kind of slow. You should look at this," or other places. But I think you're going to see it proliferate, and it's very, very exciting to me. It's very personal. Yeah.

Savannah Peterson

>> We couldn't tell it all, sitting there.

Morgan McLean

>> To be clear, there's lots of people working on profiling in OTel. I'm sure many of them have similar stories. This is not like-

Rob Strechay

>> I'm sure they all take it to heart, too. Absolutely.

Morgan McLean

>> I've written zero lines of code on this. I'm just attached to the project. But it's something I've wanted to see become well-adopted and mainstream for a very long time, and I'm extremely excited about it.

Savannah Peterson

>> Yeah. Is adoption happening fast?

Morgan McLean

>> Not yet. It's not finished yet.

Savannah Peterson

>> Okay.

Morgan McLean

>> We passed some major milestones in the project this year. But if you look at the journey of OTel, we started with tracing. We delivered metrics two years ago. We delivered logs last year. This is the big year with the profiling. Probably launch more early into next year, but it's the big year where we've been working on it. But it's now part of the spec. Getting things in a large open source community like this to become part of a core specification takes a lot of time. There's a lot of debate, a lot of opinions. That part is now effectively complete. And so now we just need to go finish implementing things. That got a boost. Our friends at Elastic donated their big profiling agent. That was fantastic. At Splunk, we're donating a lot of the work behind .NET profiling. There's work going on in Java for the other languages. In a few months you're going to see that actually be usable. I expect adoption to be relatively quick.

Savannah Peterson

>> Yeah.

Morgan McLean

>> Every time we've added a new signal to OpenTelemetry, adoption has gone quickly. And that is in some ways surprising. OTel adds a lot of value, and the fact that you get multiple signals coming through the same interface, there's correlations and things that open up even more value. But the challenge with metrics or with logs is that many people are capturing those already through different means. That can slow adoption. And yet it's been quick. With profiling, most people aren't capturing this data. It's relatively new for most organizations. They're not going to have the friction of, like, "Oh, but we use this other thing and now we need to go migrate." It's new. It's easy to use. It's easy to pull in. There's lots of solutions for it. It's going to be pretty exciting.

Rob Strechay

>> I'm excited now. You've got me pumped.

Morgan McLean

>> Yeah.

Rob Strechay

>> That's for sure.

Morgan McLean

>> As a person who, I haven't written a lot of code in OTel, but writes a lot of code in my spare time. It's really nice to have. Yeah.

Rob Strechay

>> But it actually leads back to... and again, like we were talking beforehand. We talked back in February when I was sitting in an airport-

Morgan McLean

>> Yeah. Minneapolis, yeah.

Rob Strechay

>> And it took us so long to nail down the talk from the analyst side and get the briefing done post-acquisition and what everything is going on. Now Splunk and AppDynamics are coming together.

Morgan McLean

>> Yeah.

Rob Strechay

>> And there's a lot of new Kubernetes innovation and integrations going on.

Morgan McLean

>> That's correct, yeah.

Rob Strechay

>> Give us an update on what's going on there, because it would seem like AppD has been around in the APM market for-

Morgan McLean

>> For a long time.

Rob Strechay

>> A long time.

Morgan McLean

>> Yeah, and it's a very successful business. Yeah.

Rob Strechay

>> And you guys have been in there on the security side and observability side.

Morgan McLean

>> Yep.

Rob Strechay

>> So how are things going with that? And what new stuff is going-

Morgan McLean

>> It's going well. It's exciting. As you pointed out, that's probably the biggest news for Cisco right now or for Splunk right now, is integration of AppD into Splunk. And so it's part of the Splunk business. It's Splunk AppDynamics. I think there's a few areas of excitement. I think the big one is that Splunk Observability was really aimed and remains aimed at people running high-scale web services, often on Kubernetes. And so the type of analytics you need to be successful there are usually things where you are looking at the broader view and understanding how a request propagates across your dozens, maybe hundreds of services across Kubernetes clusters and other technologies. AppDynamics is more aimed at more typical three-tier applications where you've got a client that goes to the edge, that goes to the actual app processing, that goes to a database, and back. And as such, AppDynamics has typically needed features where it needs to dive more into the code. I realize we were just talking about profiling. A little different because this is actually tracking individual requests, things like that. So features like business transactions, snapshots, things like that. There's a lot of power in bringing those capabilities together, both that broader view as well as the ability to dive into function by function, like, "Morgan clicked this button. Here's the exact path of execution that happened." And so there's a lot of work going on right now to bring those AppDynamics capabilities to our existing observability customers while still using OpenTelemetry and everything else, and also continuing to invest in AppDynamics using, whether it's OpenTelemetry or other investments where we can expand not just as feature set but also expand the breadth of languages and frameworks and things that it can capture data from. And certainly integrating that with the rest of the Splunk portfolio.

Savannah Peterson

>> Speaking of data, y'all just released a state of observability report a few weeks ago.

Morgan McLean

>> We did, yeah.

Savannah Peterson

>> Can you give us some nuggets from that?

Morgan McLean

>> Yeah. I always struggle to remember numbers, but I think it was like 78% of IT leaders are either using or planning to use OpenTelemetry and seeing value from it.

Savannah Peterson

>> That's a lot.

Morgan McLean

>> That is a lot, yeah.

Savannah Peterson

>> Talk about market saturation.

Morgan McLean

>> Yeah, certainly in terms of mind share,

Savannah Peterson

>> Right. "Mind share saturation," I guess I should say, but yeah.

Morgan McLean

>> And as I mentioned earlier, we're seeing that roll out very, very aggressively across a lot of these organizations. No, it's huge. It goes back to what I was saying earlier of, if you told me when we started this project it would be quite so well-adopted, I would've been a bit skeptical. And yet the value that it provides I think is reflected in that exact number there, where these people gain huge value from OTel. And there were some other nice nuggets in there about the power of observability. I think it was 2.5x return on investment for investing in observability tools that an organization. No, I think this speaks to the value that people are really getting from these tools. KubeCon's a pretty serious conference. Most things that are being pitched are actually quite valuable, but in the broader SaaS software space, there's things where, if you're a CISO or CTO, you wonder, this is quite an expensive check to write. Am I actually going to get anything for this? I think of observability solutions, Splunk being I think one of the premier ones, the choice is quite clear. If you don't have this you are going to struggle as an organization. You will ship code more slowly, you will have more outages, the outages will be more severe, and so on. And the value is very strong.

Rob Strechay

>> Yeah.

Savannah Peterson

>> Yeah.

Morgan McLean

>> You see that there.

Rob Strechay

>> Yeah, and it would seem... because again, knowing that study because I've actually read through it multiple times over multiple years, a lot of people were looking at it and saying, "Hey, I'm looking to consolidate down the number of vendors as well."

Morgan McLean

>> We see a lot of that, yeah.

Rob Strechay

>> That would make a lot of sense. Is that also helped by the fact that OTel is out there and the fact that, just some standardization on that and things of that nature?

Morgan McLean

>> Yeah. Well, with standardization, when you think of tool consolidation, there's two huge benefits to OpenTelemetry. The first is you can do it.

Rob Strechay

>> Right.

Morgan McLean

>> Swapping observability tools, or not even swapping. Having multiple and going to a smaller set, historically it was very challenging. They all had their own agents. And ripping out and replacing those agents was often very expensive, very challenging. Your engineering team had to sit down and deal with it. If you're a decision maker, that's a risky bet to make, especially if it doesn't end up panning out. With OpenTelemetry, you send your data into OTel capturing with OTel. You can send it out to all of those tools, if you'd like. You could use all of them to their full breadth. It's going to be a little expensive, but you can also slowly migrate between them. It makes that transition a lot easier. Secondly, you actually get more value out of your data. When you're using two different tools like a tracing tool over here and a metrics one over here, and maybe different parts of your org are using different tracing tools, you're not getting the full picture. If you're using Open Telemetry, whether you're using one or more tools, you get the full breadth and, secondly, you get all of the correlation between the data. If you have traces that are tied to a service that's slow, you look at the spans that are slow. Then you can immediately hop to the system metrics and go, "Oh, okay. I see the CPU consumption's high. That's why this slowed down, and so on." If you're fragmented between tools across the organization, a lot of those things that seem easy, those connections are very difficult, if not impossible to make, and it takes you hours when something goes wrong. So no, I do think it's a good point. OpenTelemetry, there's always pressure for tool consolidation.

Rob Strechay

>> It's the changing factor.

Morgan McLean

>> Yes.

Rob Strechay

>> It's not as big a radical change.

Morgan McLean

>> It's not why they're consolidating, but it means they can consolidate when they want to, which means they want to even more.

Rob Strechay

>> Yeah. Absolutely.

Savannah Peterson

>> It's all about choice out there for the community.

Morgan McLean

>> Yeah.

Savannah Peterson

>> Last question for you, Morgan, because this has been a blast. And we look forward to having you on again..

Morgan McLean

>> I'm happy to be here.

Rob Strechay

>> Yeah.

Savannah Peterson

>> What do you hope to be able to say next time we have you on the show? Let's call it KubeCon London or KubeCon Atlanta, that you can't yet say today?

Morgan McLean

>> We talked a lot about profiling, so there's profiling adoption.

Savannah Peterson

>> I was almost going to say you can't say, "Profiling adoption," because I feel like you've said that and I know that's a deep core principle of yours.

Morgan McLean

>> There's some minor things in OpenTelemetry. The project is due for graduation. I expect at one of those two KubeCons we'll be able to announce that. The OTel collector is hitting stability, so all of its internal APIs... not internal. The APIs that people in the community use for more add-ons will be locked down so that those won't be changing. That'll, I think, add even more explosive growth to the usage of the collector. But those are relatively small. In terms of the big things I'd want to talk about or hope to talk about, outside of OpenTelemetry it'd probably be the things we were talking about, like Splunk and AppDynamics and the other capabilities we're bringing to the field with like Splunk Observability and AppDynamics. There's a lot of exciting things happening there that we can't talk about externally a whole lot yet, but hopefully we'll be able to show off in the coming 12 months.

Savannah Peterson

>> Well, we look forward to hearing all about it and helping you show off. Morgan, thank you so much for being here with us.

Morgan McLean

>> Yes. Thank you for having me.

Savannah Peterson

>> And Rob, what a great first day.

Rob Strechay

>> Yeah, it's been fantastic. I think the collection of people that we've had on that bring the expertise that we've been able to dig into has been fantastic.

Savannah Peterson

>> Yeah, me too. I just love this show, and I hope you all love it as much as we do wherever you might be watching. We're here in Salt Lake City, Utah. That's a wrap on day one of KubeCon North America. My name's Savannah Peterson. You're watching theCUBE, the leading source for enterprise tech news.