The AI Productivity Paradox

Darin Pope (0:00)

This episode is sponsored by Allstacks.

Jeff Keyes (0:02)

Are there more deployments going through? Yes, are there more tickets being closed? Yes, are there more confidence in the ability to take an idea and vibe it through to come up with something? Absolutely.

Are we truly more efficient? Do we have more throughput? Or is the Dora Report talked about creating more instability?

Also, yes, there's problems that are happening. We're not really seeing the throughput. We're not really seeing the increase in quality. In fact, while individuals report better individual productivity as a team, we can't really answer the question of, are we shipping more value?

Darin Pope (0:43)

This is DevOps Paradox, episode number 368, the AI Productivity Paradox.

Welcome to DevOps Paradox. This is a podcast about random stuff in which we, Darren and Victor, pretend we know what we're talking about. Most of the time, we mask our ignorance by putting the word DevOps everywhere we can and mix it with random buzzwords like Kubernetes, serverless, CI-CD, team productivity, islands of happiness, and other fancy expressions that make us sound like we know what we're doing. Occasionally, we invite guests who do know something, but we do not do that often since they might make us look incompetent. The truth is out there and there is no way we are going to find it.

Yes, it's Darren reading this text and feeling embarrassed that Victor made me do it. Here are your hosts, Darren Pope and Victor Farsi. Victor, you use AI for daily development. I use AI for daily development. A lot of developers are using it, 90 percent are using it. Most of them are saying it's helping them be productive. Is it making you more productive, Victor?

Viktor Farcic (1:55)

Let's put it this way. If you tell me no more agents for you, I'm growing tomatoes.

I'm not going back to software industry. That's exactly the decision from my part.

Darin Pope (2:05)

One study found that developers who felt 20 percent faster were actually measurably slower. So here's the question for today. What if a whole team is confidently shipping nothing faster?

On today's show, we have Jeff Keyes from Allstacks. Jeff, how are you doing?

Jeff Keyes (2:26)

I'm doing fantastic.

Darin Pope (2:28)

Did I say your last name right?

Jeff Keyes (2:30)

No, Keyes. We have to make it messy. If you speak more Latin languages, people say Keyes because it looks like Reyes, and in English, most people say Keys, but it's Keyes.

Darin Pope (2:40)

And I even have it phonetically in front of me, and I did it with a dash, so it's Keyes. Did our story sound even familiar there?

Of what's happening, in fact, let me ask it a little bit differently. We're not debating whether or not AI is helping type code, right? We can see it typing code really fast. What we're really trying to figure out is it helping us shi- Any faster. Breathe in, Jeff.

Jeff Keyes (3:04)

For sure. Are there more deployments going through? Yes. Are there more tickets being closed? Yes. Are there more confidence in the ability to take an idea and vibe it through to come up with something? Absolutely.

Are we truly more efficient? Do we have more throughput? Or is the Dora report talked about creating more instability?

Also, yes, there's problems that are happening. We're not really seeing the throughput. We're not really seeing the increase in quality. In fact, while individuals report better individual productivity as a team, we can't really answer the question of, are we shipping more value? There are some reasons why that I believe are part of that story. But is AI helping develop a productivity? 100 percent.

Viktor Farcic (3:54)

Everybody says that on individual level, AI is helping, whatever that means.

The logical conclusion must be that then there is something structurally problematic on a company level, a process level, right? Because if I'm more productive and you're more productive and Darren is more productive, but when we combine our productivities, that does not turn up to be more productive as a whole, then my only explanation can be, okay, your process does not work for this. Yes.

Jeff Keyes (4:29)

I like the explanation that AI is an amplifier, and it will amplify the good and the bad. And I think the challenge of the bad is so much that it's off-setting some of the good.

If I remember right, looking back at some of AI's impact according to the research that was done last year, individual effectiveness went way up, but software delivery instability also went up. Amazing thing that happens to pipelines that were mostly right. When they're mostly right and you put more through them, you got more failures and more issues to go solve, which means higher incident rates, higher deploy failures, and the rest of it. The most notable example from my point of view was a recent session we ran to exemplify a new feature. We ran a proof of concept, and part of that proof of concept included this text editor as part of it. And not thinking it through, we just built it all, and the agents went to work and did great things, including building a text editor. It's happy to do whatever we feed it, and whatever it sees, and wherever it goes. Is that exemplifying what problems we see in the industry? I think so. I think there's a lot of, do platform components get used well? Are existing APIs being duplicated? Are bad patterns being replicated because that's what we did in the past? Maybe. How much are we learning from agentic development from what we've already done?