The Organization Behind Chaos Engineering with Matt Fornaciari of Gremlin

**Dan** (0:02)
Nobody likes being the engineer on call, especially if you're the one being woken up at 3 a.m. due to an outage. It feels chaotic. It feels like there should be a better way.
This is exactly what Matt Forni was thinking when he co-founded Gremlin, the world's first chaos engineering platform. In this episode of Dev Interrupted, we get into exactly what chaos engineering is, how it came to be, and how you can implement it at your organization.
Also, be sure to join our upcoming Dev Interrupted watch party on May 20th. I'm bringing back some of our favorite guests from Netflix, GitHub, and Honeycomb to discuss how they're creating continuous improvement at their organizations. Sign up using the link below.

**SPEAKER_2** (0:53)
This episode is sponsored by LinearB.
Give your dev team the power to improve with team-based metrics, high-risk code alerts, and the world's first project board based on real-time get activity. Sign up free at linearb.io.

**Dan** (1:08)
Matt, thanks for joining us on DI today.

**Matt Fornaciari** (1:11)
Yeah, absolutely, Dan. Thanks for having me. I'm looking forward to having a good chat.

**Dan** (1:15)
Yeah, yeah. Really cool to have you on here.
And so we just kind of dive right in. Can you start by giving us a definition of what is chaos engineering and maybe a quick background on how it all got started?

**Matt Fornaciari** (1:30)
Yeah, absolutely. I mean, you're not the first to ask. I've definitely had to explain this several times, which is part of the occupational hazard of starting a category, right? Chaos engineering, I think it's originated way back in the day. I think it really goes back to disaster recovery and trying to get ahead of that, being proactive.
I think there's some folks that think that this all started at Netflix with Chaos Monkey, which I think is where it first became really popularized. But we were doing this back in the Amazon, my Amazon days back in 2010 And I think even before that, there was a guy named Jesse Robbins, who used to run through data centers and just pull cables out of the wall, think of the nickname, the master of disaster.
And so the practice that's been evolving over time, but really what chaos engineering is, it's the practice, the art, if you want to call it that, of introducing controlled chaos. So not just going in and randomly shutting stuff down or breaking stuff without really a purpose, but really thinking very analytically and very scientifically around, great, what are a bunch of hypotheses that I have about how my system works? And how can I actually go exercise those hypotheses to see if they hold true, if I can validate or reject those hypotheses? So you use that chaos the same way you would very apt these days. You inject a little bit of the failure to see if you can build up a tolerance similar to a vaccine or something along those lines.

**Dan** (3:00)
Yeah. Is it only something that is for kind of production, like the health of your production or the stability of production, or are you also doing it in the QA practices? Where does it live?

**Matt Fornaciari** (3:13)
Theoretically, you should be running chaos just about everywhere. What we're doing under the covers with chaos engineering is building that mentality, that practice, that muscle, and that thought process about reliability. Chaos engineering is really one of the tools in the toolkit of the SRE, of the reliability engineer, or just in building reliability. What you want to do is sort of build not only that practice from a engineering, put it into your CI, CD pipeline, that kind of thing, but also the culture around, cool, let's start to think about how can this fail up front when you're developing code, all the way through production.
I've heard the term shift right a lot more these days, which that one drives me a little bit crazy, but the idea that I don't think it's really shift left or shift right. It should live in every step of the pipeline. It should go all the way from dev to staging, production, if you've got QA in the middle, you know, all those kind of things.

**Dan** (4:11)
Yeah, I'm usually hearing like shifting left, getting, you know, empower developers, find things earlier, which will save time and money. What do you think are the major benefits of having a chaos engineering practice in place for an engineering organization?

**Matt Fornaciari** (4:32)
I mean, for me, back in the Amazon days, the best benefit I can think of is my pager didn't go off at 3 a.m. anymore. It went off at, you know, 3 p.m. when we're running these controlled experiments and when we're actually like validating our assumptions with like eyes on glass. And I'm not waking up and rolling out of bed trying to like wipe the sleep from my eyes and figure out what the hell is going on with my, am I allowed to swear? Sorry, I forgot to ask earlier on, but yeah, it's.