Black Hat USA 2025 | Kat Cosgrove, Minimus

Clips
More from Black Hat USA 2025

Kat Cosgrove

Head of Developer Advocacy

Minimus

play_circle_outline Introduction of Kat Cosgrove, a Kubernetes expert and head of developer advocacy at Minimus.

play_circle_outline Open source projects like Kubernetes have a robust community but require trust among contributors.

play_circle_outline Open source practices enable quicker issue detection, benefiting project security through transparency.

play_circle_outline From Embedded Linux to Kubernetes: Kat's Journey in Container Security and Minimal Secure Images with Minimus

Info
Transcript

Kat Cosgrove, Minimus

Kat Cosgrove

Head of Developer Advocacy Minimus

At Black Hat USA 2025, theCUBE’s Jackie McGuire sits down with Kat Cosgrove, head of developer advocacy at Minimus and Kubernetes release team lead, for a deep dive into securing one of the world’s most critical open-source projects. Cosgrove shares how Kubernetes maintains resilience at scale, balancing openness with rigorous security practices across 1,300+ monthly contributors. She explains why transparency, robust governance and “security hygiene” make Kubernetes inherently more secure than many proprietary systems, even in the face of sophisticated threa... Read more

explore Keep Exploring

What is the role of the person being interviewed and what responsibilities do they have regarding Kubernetes? add

What measures are in place to ensure that individuals with elevated permissions in a project like Kubernetes do not misuse their access? add

What are the differences in security incident disclosure between open source projects like Kubernetes and private companies? add

What factors influenced your decision to join Minimus? add

bolt Powered by CUBE AI

Kat Cosgrove, Minimus

search

Jackie McGuire

>> Welcome back to Black Hat 2025 at the Mandalay Bay in Las Vegas, Nevada. I'm Jackie McGuire, your practice lead, principal analyst and favorite media host for security. I am here with Kat Cosgrove of Minimus. Welcome, Kat.

Kat Cosgrove

>> Howdy, happy to be here.

Jackie McGuire

>> Awesome, you are the head of developer advocacy.

Kat Cosgrove

>> I sure am.

Jackie McGuire

>> And we had you on theCUBE, I think at Open Source Summit, right?

Kat Cosgrove

>> Yeah, yeah.

Jackie McGuire

>> And you are a Kubernetes expert.

Kat Cosgrove

>> Yeah, I own the Kubernetes release team sub-project, so three times a year it's my job to make sure that Kubernetes gets outside the door safely and relatively on time. We consider release dates to be kind of a moving target.

Jackie McGuire

>> Yeah, yeah. I'd rather ship right than on time, to be honest with you.

Kat Cosgrove

>> Oh, yeah. Dude, same. I also am a tech lead for SIGDocs, so I own the Kubernetes website and the Kubernetes documentation.

Jackie McGuire

>> Wow.

Kat Cosgrove

>> Pretty small job.

Jackie McGuire

>> Yeah, I'm sure they can replace you with AI.

Kat Cosgrove

>> Oh, so quickly, so quickly. They definitely could. We've definitely gotten some really sketchy drive by PRs before, but we did establish a policy-wide project of not allowing that.

Jackie McGuire

>> For the three people watching who don't know what Kubernetes is, can you explain what Kubernetes is?

Kat Cosgrove

>> Yeah, Kubernetes is a tool for orchestrating large clusters of containerized applications or containerized code. You could do all of this manually by hand without something like Kubernetes, but it would be a pain to scale your application up and down by hand. Kubernetes handles all of that for you, makes sure that your containers and your applications stay up. If one of them goes down, it'll, depending on how you set it up, spin it back up again for you. So it makes sure that you can handle an adequate amount of load from users without much interaction from you.

Jackie McGuire

>> So it's kind of like what we're trying to call AI agents right now, right? It just does things automatically for you.

Kat Cosgrove

>> It just does things automatically for you. It's really big, fancy, super complicated automation. That's all it is, and that's what a lot of AI is too, honestly.

Jackie McGuire

>> But it's incredibly helpful. I threw a digital music festival in Minecraft where we had to have a whole bunch of different servers, and anytime one server hit more than 100 users, we had to automatically replicate. And so doing that stuff manually would make most projects unfeasible economically and from a manpower perspective, so it's really important.

Kat Cosgrove

>> It powers a frightening amount of today's infrastructure. It is the second largest open source project in the world behind Linux itself.

Jackie McGuire

>> Yeah, so no big deal.

Kat Cosgrove

>> No big deal. We're managing, I think 1,300 currently contributors a month, so it's a lot of cats to herd.

Jackie McGuire

>> I'm a security person, so let's talk about the security of that with an open source project, because you brought up Linux. We had xe-utils year. Which year was it, 2023? I don't know, I'm in a time warp. They're obviously with open source. It is one of the most critical methods of creating technology, I would say, in our society.

Kat Cosgrove

>> For sure.

Jackie McGuire

>> And I think it's what the utopia of technology is supposed to look like, right? But unfortunately, it also creates opportunities for people who do contribute meaningfully for a while, to then do bad things. So how do you think about balancing the benefits of having this robust community of experts with the need to babysit them and make sure they're not bad actors?

Kat Cosgrove

>> There's not actually a ton of babysitting involved in a project like Kubernetes. We are a very large, very mature project, so we have a lot of automated controls in place to make sure that the wrong people don't have dangerous permissions. However, somebody like me with significantly elevated permissions, I could in theory, if I wanted to, and I could get one or two other people with similar permissions to work with me, do some really nasty stuff to the Kubernetes code. I could. So there is just some trust that has to exist, assuming that I and two of my friends are not going to go sneak something scary into Kubernetes. But a lot of it I feel is there's just more publicity on breaches and incidents with open source today than there was 10 or 15 years ago. Open source software is inherently more secure than proprietary software because we can move faster. If I did something to screw with Kubernetes, it would get noticed and rolled back so fast, so fast it would make your head spin. It would be unfathomably difficult for me to hide it. And if I did manage to hide it long enough for it to make it through a release cycle and actually get pushed-

Jackie McGuire

>> Which is why we've really only seen a couple of these in decades, right?

Kat Cosgrove

>> Yeah.

Jackie McGuire

>> Where everywhere else, you're seeing vulnerabilities left, right, middle.

Kat Cosgrove

>> It just doesn't happen.

Jackie McGuire

>> Well, and I think that's another thing that's really important is not just you have experts contributing, but to your point, you also have the experts monitoring. And we've seen a number of vendors in the last couple years who had these critical security incidents or CVEs, and they don't disclose them for days, weeks, sometimes months. And because you don't have independent third-party eyeballs, you're subject to the whims of a private company. So I think that's something to highlight as being extremely important as well.

Kat Cosgrove

>> Yeah, the radical openness of open source means that issues get noticed pretty quickly, and a project as large and mature as Kubernetes functions a lot like a business. We have an org structure, there are people I report to, there are people that report to me. We have a security response counsel as well. They're relatively independent, they all sign NDAs. I have to sign an NDA a as the owner of the release team sub-project because sometimes I have to know in advance, when there's something coming down from Go or whatever. But-

Jackie McGuire

>> Well, I guess arguably, there's enough North Korean hackers working at private companies that you're no more vulnerable than those private maybe. And then because of the extra eyeballs, probably less, right?

Kat Cosgrove

>> We function a lot like a company, but it's a company you can see straight through. Our whole org chart is public. All of us are easy to get a hold of and all of the code is in public. And because it's an open source project, if you notice something, PR is welcome, file an issue.

Jackie McGuire

>> Yeah, so it's almost like you guys are just practicing basic security hygiene.

Kat Cosgrove

>> Yeah.

Jackie McGuire

>> Like privileged access management and things like that.

Kat Cosgrove

>> Yeah. And we're doing it all in public and other people do learn from us as a project. A lot of people give talks about the way Kubernetes is structured from an organizational perspective, or about how security works within the Kubernetes project. So other smaller projects get to learn and figure out how to behave from us with our cumulative decades, probably centuries of experience, something like 200-ish maintainers, all of them with quite a lot of experience.

Jackie McGuire

>> I guess that was my next question, was what do you feel like the community could be doing to better help this effort since it is so key to almost every technology we use?

Kat Cosgrove

>> Yeah, we have a lot of contributors, but we also have a lot of work to do. Some SIGs are better staffed than others. SIG release is very, very well-staffed. That's where I spend most of my time because we are very public facing, we're very flashy. People want to get involved in the release team because they want to put it on their resumes. But there are other parts of the project that are understaffed in spite of the fact that we have 1300-ish monthly contributors and they need help, too. And more eyes, more hands, more different kinds of experience make for a better and more secure project. It doesn't matter how much experience you have, somebody who's new, like a relatively junior engineer is going to see things and have ideas that we won't with years of experience. We'll miss it.

Jackie McGuire

>> Are there specific areas of expertise you want people to reach out if they have?

Kat Cosgrove

>> We do, in fact, always need more people helping with docs. We don't allow features into Kubernetes that have user-facing changes without documentation, but that standard is pretty minimal.

Jackie McGuire

>> every tech company in the world could use more docs.

Kat Cosgrove

>> They could, yeah. So there's nothing in Kubernetes that you can touch that isn't documented, but that documentation can always use improvement and it can always use localization into other languages. So if you speak a language that is not English, please do come help. I think the second largest localization project is Hindi.

Jackie McGuire

>> If you're a Hindi speaker and you want to contribute to the community who has helped you build the foundation for most of the technology you use, please reach out because documentation isn't sexy, but it's something I think you can do one thing that helps thousands and thousands and thousands of people.

Kat Cosgrove

>> For sure, making the project more easily usable and easier to learn the intricacies of how it works does make it easier for us to keep it secure. Because somebody could dig in and find, "Oh, hey, that actually doesn't behave exactly the way it's documented," and I think that's exploitable. That's a problem we need to fix. We do also have a security SIG. All other SIGs have to answer to some degree, to horizontal SIGs like security and docs. So if you are specifically interested in Kubernetes security, SIG security is a really, really great place to go. And the SIG security chairs are very sweet and very responsive people.

Jackie McGuire

>> That's amazing. What drew you to Minimus?

Kat Cosgrove

>> Well, I work in containers right now.

Jackie McGuire

>> You couldn't contain yourself?

Kat Cosgrove

>> I couldn't contain myself. Yeah, I spent a long time working just generally in DevOps and I was getting a little bored working generally in DevOps while in my spare time, I was doing Kubernetes maintainer stuff. So everything I do for Kubernetes was happening in my spare time after work, because I was working for some DevOps tooling company or some infrastructures code company. And it wasn't directly relevant to my job to still be a maintainer. But Minimus is explicitly a container security company building minimal secured images, and that is directly relevant to my job as a Kubernetes maintainer. So first of all, they let me do Kubernetes Maintainership on the clock, which is a great way for any company to support open source. If you can't throw money at the project directly, you can throw people at them.

Jackie McGuire

>> Yeah, and if we think people are always working 40 hours a week remotely, that's probably not true.

Kat Cosgrove

>> It's not true.

Jackie McGuire

>> And if you let your employees have two or three hours, four hours a week to work on something that's meaningfully fulfilling to them, I don't think you're going to find that it's at the sake of their other work. It will most likely be in addition to because we've all got bandwidth for things that spark joy.

Kat Cosgrove

>> For sure.

Jackie McGuire

>> But the next Zoom meeting isn't usually that for me.

Kat Cosgrove

>> It's not, yeah. They let me do something that does spark joy, that I genuinely enjoy doing on company time because it's beneficial for them, too. But also, I think it's a cool solution to a problem that is going to increasingly rear its head. People are using a lot of just super bloated container images with God only knows what.

Jackie McGuire

>> I'm the poster child.

Kat Cosgrove

>> Yeah.

Jackie McGuire

>> I am the poster child. I had John Morello on yesterday from Minimus, and I was telling him as was a data scientist, the rule was you open your Jupyter Notebook, you PIP install and import everything you ever thought you wanted in the world, even if you don't need it. And then you just push it. And I think obviously, having fewer dependencies and packages and all of that thing, it makes the containers more secure, but it makes them way cheaper to run. And at the end of the day, if you are a CISO and you are trying to sell, "Our containers will be more secure," you are dead in the water every day. If you are selling your CEO by making our containers more secure, we're going to save 50%, 75%, 80% of the costs, that's a really easy way to get somebody to write a check.

Kat Cosgrove

>> It sure is. But before I was a developer advocate, I was a software engineer and I was specifically an embedded Linux engineer, and that's how I learned Kubernetes. I was trying to run K3S on really, really, really small devices. And having smaller containers makes it easier to do more on a smaller device with fewer resources.

Jackie McGuire

>> They also crash less.

Kat Cosgrove

>> Yeah, they crash less because there's less to debug when something weird goes wrong. It feels better in every possible way. So from a technology standpoint, I am a minimalist, even though in real life I'm very much a maximalist. My house is full of weird knickknacks and crap.

Jackie McGuire

>> I am the epitome of conservative minimalism, obviously.

Kat Cosgrove

>> Oh, I can tell. Yeah, you for sure are.

Jackie McGuire

>> I try to shrink into the background as much as possible.

Kat Cosgrove

>> You're so normal.

Jackie McGuire

>> What's up next for you? What's next on your agenda, what are you excited about this year that you're working on?

Kat Cosgrove

>> Whoo, I've got about a million and one conferences this year, about a million and one conferences. But I get to go to some new places that I've never been to before. I've never been to Australia and I'm getting shipped to Australia to keynote DevOpsDays Wollongong.

Jackie McGuire

>> That's amazing. I haven't been either and that would be-

Kat Cosgrove

>> I'm really jazzed for that one so that'll be fun. I'm going to take a few extra days and go hiking in the Blue Mountains, I think, do some rock climbing.

Jackie McGuire

>> I recently started rock climbing indoors, and it's-

Kat Cosgrove

>> Oh my God, welcome to the hobby.

Jackie McGuire

>> And I have three kids and I bring them with me, and it's so cool to watch them challenge themselves. And I had done it before, but I was always too scared to boulder, and I just started bouldering this time around-

Kat Cosgrove

>> Bouldering is my preferred rock climb as well.

Jackie McGuire

>> Yeah, I had my biggest fall from 12 or 13 feet up a couple days ago.

Kat Cosgrove

>> Did you scream?

Jackie McGuire

>> No, no.

Kat Cosgrove

>> Good job.

Jackie McGuire

>> No. And I landed, I crumpled, I did the whole thing. And it was one of those things that I was like, "This is a milestone in my life of being terrified and doing it anyway." And that's always been my thing in life, but it felt like an extra accomplishment to be like, "No, I really did go for that last rock and I fell off and I got back up and I did it."

Kat Cosgrove

>> Good for you. Yeah, that's really key, not getting freaked out or discouraged by failing. It's a sport that really relies on you being okay with failing repeatedly, which is-

Jackie McGuire

>> And only competing against yourself. You have to be very self-motivated and I think it's a lot of things you can carry into work in terms of holding yourself accountable to being an increasingly better version of yourself.

Kat Cosgrove

>> Yeah. And I'm a developer advocate. I give a lot of conference talks, I do a lot of demos on stage, and I have to be okay with that demo failing on stage and not fall apart when that happens. Because it is a when, it's not an if. It is going to happen to you someday, so having a sport that also forces you to be okay with falling over in public-

Jackie McGuire

>> Then you figure out none of your friends and family stop loving you because you failed, so you just keep rolling.

Kat Cosgrove

>> Yeah. Usually the audience, when your demo barfs on stage, they feel a little bad for you but they're not thinking you're, "Oh my God, this loser can't even get through a demo," because we've all seen it happen before.

Jackie McGuire

>> It makes you more relatable. I always say that me being a little bit of a hot mess, I think makes me at least a little bit relatable and charming.

Kat Cosgrove

>> For sure. Yeah, it does. I've had a laptop fully crash during a demo on stage before. Fortunately, I had a second laptop because somebody who mentored me early in my career was like, "This happened to me once. Always travel with two laptops, have your demo ready to go on two different machines and-"

Jackie McGuire

>> Redundancy, resiliency.

Kat Cosgrove

>> Have some redundancy.

Jackie McGuire

>> Almost like cyber hygiene.

Kat Cosgrove

>> Almost, yeah. And I didn't believe her and then it happened to me at SCALE, at the SoCal Linux Expo, so not a great feeling. That's a pretty good conference. It's a difficult conference to get a speaking slot at, and so didn't feel good crashing there but I was so glad I had the backup.

Jackie McGuire

>> Yeah. Well, Kat, thank you so much for joining us. Everybody should go find you, .at Cosgrove, you're amazing, brilliant. If you're a K8s user, you definitely need to be following Kat. Appreciate having you, thank you so much.

Kat Cosgrove

>> So happy to be here, thank you.

Jackie McGuire

>> For theCUBE, I'm Jackie McGuire. We are in Las Vegas, Nevada for Black Hat 2025. We will be right back with more amazing guests and great content. Stay tuned.