Reliability Without the Guesswork: Foresight AI with Kolton Andrus, CEO of Gremlin

Kolton Andrus (0:00)

The other part of the equation is, people is the hard part of technology, and getting people to do the right things, more than just the SRE team or the platform team, but the engineers across the company, really are required having the right incentives and the right measurement in place. I'm told not to say this, but we want boring systems. We want systems that just do the right thing without us having to worry about them. Plans have dates, otherwise, you're really, it's just wishful thinking. And one of my favorite one is one time, people were talking about an issue they encountered and Jeff said to them, this sounds like it's happening to you.

Joel (0:43)

Can you explain a little bit to start like what Gremlin does and then we can jump into that shift?

Kolton Andrus (0:49)

Yeah. So Gremlin, we're about, the analogy I typically use is that of like crash testing a car or a vaccine.

You know, we want to go out, we want to inject a little bit of harm. We want to reproduce real world scenarios to understand what happens when they behave. And we want to go fix them so they don't impact customers, so they don't actually end up resulting in harm in production.

Joel (1:16)

Okay. And so you guys were on the show years ago. I talked to Matt.

Tell me a little bit about like where you've gotten, that's where you got your start, but where are you now?

Kolton Andrus (1:28)

Yeah. So I think one of the things we found is what we built was an expert level tool, and you needed to do a bunch of homework essentially in order to get value out of it. You need to really understand the system. You need to design these experiments. Our platform makes it safe and secure, but you had to really understand how everything fit together. So that was the technical piece. There were some aspects that we have adjusted over the years to make the technical bar lower. How do we give people more out of the box? How do we integrate with their alerting, their monitoring, their logging, so we can tell them if their system behave correctly? That was part of the equation. The other part of the equation is people is the hard part of technology and getting people to do the right things more than just the SRE team or the platform team, but the engineers across the company really are required having the right incentives and the right measurement in place.

One of the ways I like to phrase this is, hey, if I prevent a big outage, am I going to get promoted?

Because you can get fired if you create a big enough outage, but will you get promoted if you prevent one? And oftentimes, if things just run smoothly, no one notices. So how do you convey this to your leadership or to your boss that you've done a bunch of work? The reason the system went smoothly was not because you got lucky, it was because you went in and put forth a lot of effort to help ensure that things went that way. And that's really where you have to start measuring the types of issues you found, the issues you corrected, the improvements that were made over time. And that's where we came up with a reliability score that really gives leadership a clear idea why did things get better and help them to track it. The other side of that equation is accountability. If leadership doesn't have any visibility into what's happening, what's improving or what isn't approving, then they can't go and ask teams to go prioritize work or lean on them a little bit to make sure that work gets done. So that was the social side of things.

I don't know if you want me to get into the AI side of things as well.

Yeah. So then right now, the thing we've been working on the last six to nine months is around, how do we just do all of that for people? That was really the vision. One of my favorite, I'm a Simpsons nerd, and one of my favorite Simpsons episode is when Homer becomes the garbage commissioner.

He's got this great song, just like, can somebody else do it? And sometimes that's how we feel about some of our systems. Wow, can somebody else just do that for me? So that's really been our long-term vision at Gremlin. How much can we do for people? Can we analyze their systems? Can we tell them what would go wrong? Can we tell them what risks they have? Can we design the experiments that they should run? Can we run them for them? When we run them, can we analyze the results and tell them what didn't go well or what went well? And can we recommend the fixes and the remediations out of those? And so I think that's what's exciting about today's era is now much more than five years ago, we can do many, many more of those things. We can do that whole life cycle that I described, start to finish with a little bit of human oversight, a little bit of human nudging and guiding here or there. But in a pretty much autonomous fashion.