The Problem with MTTR: Learning from Incident Reports | Courtney Nash

**Courtney Nash** (0:00)
We just have this really seemingly clean, tidy number. Our MTTR went up and it went down. And then the question I always ask people is, well, if it went up, what are you gonna do?
What are you gonna do? Oh, well, I guess we gotta go figure out why. How are you gonna figure out why? Well, I guess we gotta go look at all those incidents. And then I ask people, well, what if it went down? Wouldn't you wanna know why?
Like why you're getting better or why things are getting better?

**Dan Lines** (0:19)
I only wanna research it if things are bad, otherwise, no.

**SPEAKER_3** (0:23)
Every engineering team could use an easy win right now, including yours. That's why Linear B collaborated with dozens of engineering teams to crowdsource the real, smart ways companies have improved their productivity quickly and easily without wasting more hours at work, spending a lot of money or burning out already stretched teams. On February 15th, Linear B will be presenting our free scaling developer efficiency workshop. Designed to help you gain up to 30% increased productivity, this free online workshop will empower your teams with strategies and solutions that let them focus on what they do best.
Sign up to the workshop for free at linearb.io/efficiency or head on over to the show notes to sign up today.

**Dan Lines** (1:06)
Hey everyone, welcome to Dev Interrupted. My name is Dan Lines and I'm here with Courtney Nash, who has one of the coolest, possibly made up titles, but possibly real, internet incident librarian.

**Courtney Nash** (1:23)
Yep, that's right.

**SPEAKER_3** (1:23)
Is that right?

**Courtney Nash** (1:24)
Yep, you've got it.

**Dan Lines** (1:25)
Welcome to the show.

**Courtney Nash** (1:26)
Thank you for having me on.

**Dan Lines** (1:27)
I love that title.

**Courtney Nash** (1:29)
Still possibly made up, possibly real.

**Dan Lines** (1:30)
Still possibly made up.

**Courtney Nash** (1:31)
We'll just leave that one out there for the listeners to decide.

**Dan Lines** (1:34)
Let everyone decide. What could that possibly mean? We have a, I think, maybe a spicy show. A spicy topic.

**Courtney Nash** (1:42)
It's a hot topic show.

**Dan Lines** (1:43)
Hot topic, especially since we're at DevOps Enterprise Summit, where we hear a lot about the Dora Metrics, one of them being MTTR.
And you might have a hot take on that. The end of MTTR, or how would you describe it?

**Courtney Nash** (2:01)
Yeah, I feel a little like the fox in the hen house here, but Gene accepted the talk, so, you know, there's that.

**Dan Lines** (2:09)
So it's on him.

**Courtney Nash** (2:09)
Yeah, it's all Gene's fault. So I have been interested in complex systems for a long time. I used to study the brain, and I got sucked down an internet rabbit hole quite a while ago.
And I've had beliefs for a long time that I haven't had data to back up necessarily. And we see these sort of perverted behaviors, not that kind of perverted, but where we take metrics in the industry and then Goddard's Law, pick whatever you pick, or people incentivize them, and then weird things happen. But I think we spent too little time looking at the humans in the system, and a lot of time focusing on the technical aspects and the data that come out of the technical side of systems. So I started a project about a year ago called The Void. It's the Verica Open Incident Database, actually real, not made up name. And it's the largest now collection of public incident reports. So if you all have an outage, and you hopefully go and figure out and talk about what happened, and then you write that up, that's out in the world. So I'm not writing these, I'm curating them, I'm collecting them.
So I have about 10,000 of them now, and a bunch of metadata associated with all these incident reports.

**Dan Lines** (3:24)
So these are all different companies or sizes?

**Courtney Nash** (3:28)
Every size you can imagine. From probably there's one or two person startups, or a project that someone's maintaining even sometimes, all the way up to massive, multinational conglomerate corporations. About over 600 organizations in The Void.

**Dan Lines** (3:43)
And usually when I hear about a publicly available incident, it's usually like a security breach. I guess I would ask like, where are you getting the, like how do you get these?
Yeah.

**Courtney Nash** (3:55)
Yes. I would love for people to send them to me more.
And there's a very inefficient way of doing that right now that I'm hoping to improve upon. But most of this was, there were some initial databases, if you will, or small efforts at this. Dan Liu had a GitHub repo called Kubernetes AF that had a bunch of, that's where I started because I was doing Kubernetes. Okay. Research for Berica. And then some other folks I knew had actually kept all of the SRA weekly incidents and like collecting those and hanging onto them. So some people just sent those to me. And then a lot of it is just going out and collecting them online, like literally, because web scraping, because that's fun. Everybody likes doing web scraping.