D2DO315: Running Agents Securely at Scale

Ned Belavance (0:07)

Today's podcast is sponsored by Gartner. The Gartner IT Infrastructure Operations and Cloud Strategies Conference, December 8th through the 10th in Las Vegas, brings together heads of infrastructure, cloud ops pros, platform engineers, and security leaders. Learn more at gartner.com/io.

Michael Levan (0:26)

Efficiency is top of mind for everybody. Better results, better quality, it's all there, but it's just a matter of...

The industry needs to... You can't put best practices around something that changes every 18 hours.

Ned Belavance (0:42)

Welcome to Day 2 DevOps, where the dev-oops is in the details. I'm Ned Belavance, and I'm joined by my protocol co-host, Kyler Middleton. Hey, Kyler.

Kyler Middleton (0:51)

Hey, Ned. Today, we're discussing all the things AI.

How do you observe it? Where does the data come from, and how do you store it? Will we have enterprise AI on our laptops? Will we end up buried in paper clips? Should I keep asking questions to myself? Let's talk about all the things.

Ned Belavance (1:09)

Guiding us through all of this AI is our guest, Michael Levan, a protocol content creator in his own right, and an AI architect at solo.io. In fact, he is hands-on doing this stuff all the time. So this is not just drawing stuff on the whiteboard, he's down in the trenches doing it. So let's talk to him about it. Michael Levan, welcome back to Day 2 DevOps. We're very happy that you're here. Really enjoyed our conversation last year.

Talking about, what did we talk about? We talked about Wasm last year.

Michael Levan (1:41)

Yes. Yeah.

Ned Belavance (1:43)

Something else has come up since then.

Michael Levan (1:45)

Something else has become important in this space. Just a little bit.

Ned Belavance (1:51)

Why don't you tell the good folks out there what you've been up to since we last talked to you?

Michael Levan (1:56)

Yeah. Last year. Okay. So one of the big things was I went from being self-employed to joining a company called Solo IO., which has been awesome. And I've done everything and anything. Right now, I'm an AI architect, so focusing on all things agentic across the entire stack, from agents to your gateways, to your registries, to your evaluations, and everything in between. And since then, yeah, I've definitely moved from the standard, I don't know what you would call it. I don't even know what you would call it now, but the standard cloud native pieces of the puzzle to definitely more AI focused, more agentic.

But again, it's like so much of this still runs on the environments that we were running everything else on already. Like the majority of all things agentic are still running on Kubernetes, right? Like people still need a way to orchestrate and scale out agents, right? And if you're not using a managed service, like a foundry or a bedrock or something, you're probably running it in Kubernetes because that's the place to scale nowadays. So there's a lot of overlap. And another thing that I would say too is, like if you have the background in monitoring, observability, performance optimization, cost optimization, programming, architecture, like you're not just throwing away that stuff, you're bringing it with you a thousand percent. You know, like a hundred percent of what I've done throughout my career, I am now using in the agentic space.

Ned Belavance (3:29)

How did you... So I guess I'm curious, was it a natural progression for you to end up in this AI architect space from where you were before? Was it a series of consulting engagements that got you into it, or you were just tinkering on your own personal curiosity? Like what drove you to dig deeper into the world of AI?

Michael Levan (3:49)

Yeah, so I think it was three things. Number one, it was just tinkering, right? Like the reason why I've always stayed up to date with my career has been because I just enjoy this stuff, right? Like when people ask me what I do on the weekends, I'm like, I'm programming for fun, and then during the week, I'm programming at work. So like I'm just always interested in this stuff. So originally, I was like building out agents from scratch. I was like looking into like fine-tuning models, seeing what went into building models. Obviously, I didn't do that because I didn't have millions of dollars for any data centers. But I was able to dive into like light fine-tuning with Laura and stuff like that. L-O-R-A. It's like a memory-optimized way to fine-tune models. So I was diving into that. And then I started diving into the frameworks, Crew AI, Lang chain, all of them.

And then just kind of like one thing led to another, right? It was like it was popping up more and more.