Publishers Block Just 22% of AI Bots. Here’s What They’re Missing

Rob Kelly (0:01)

In terms of AI bot traffic, have you met someone more knowledgeable about it than you?

Gavin King (0:07)

I'm sure they're out there, but I've got to say, I'm probably in the top five. I stare at this every day. My background is in network performance, at big tech companies.

I spend more than a normal workday every day on it. So this is what I live and breathe right now.

Rob Kelly (0:40)

Welcome to Media and the Machine, I'm Rob Kelly. This show is about AI and how to profit from it. I believe this is the biggest technology shift of our lifetime. I'm a three-time founder and CEO of Content Driven Businesses, and each week, I talk with the people on the inside of AI and Content. The ones building this new world and learning what works firsthand. I'm an optimist about AI, but I'm also a realist, so every show ends with me asking our guests what AI means for our jobs, our families, and our future.

My guest today is Gavin King, founder and CEO of Known Agents. He was a software engineering manager for Meta's Instagram for seven years before that. And as you just heard, he lives and breathes AI bot traffic.

Gavin and I dig into a remarkable shift happening on the web. One third of website traffic is now coming from bots. We break down what those bots are actually doing. And bots often get a bad rap. But according to Gavin's data, 96 percent of the bots actually follow the rules when publishers tell them to stay away. Meanwhile, publishers block only 22 percent of bots. We go over live data about what percentage of bots are blocked by such companies as Gannette, CNN, and AdWeek. We also get into the bad news. Just 0.1 percent of human traffic is currently coming from AI chat referrals and what publishers can do about it, from paywalls and subscriptions to MCPs and building businesses that AI agents can transact with directly.

At the end, Gavin shares his advice for young people entering a job market being reshaped by AI, whether his own future kids will even go to college. Please enjoy my conversation with Gavin King.

Can you just share kind of the history of bots and how important it is to both technology and how it touches pretty much any human using the web?

Gavin King (2:28)

Yeah, so I mean, bots really aren't a new thing. You know, they're as old as the Internet itself. Like Googlebot's been around for a while. But I think the big change that's happened recently is that there's just like this like Cambrian explosion of bots and like what they're actually doing.

Rob Kelly (2:42)

So pre-AI, Google would make in effect a photocopy of the web. The bots would go out, grab the content that it wanted to help serve to people doing Google searches.

What's changed now with AI?

Gavin King (2:55)

Yeah, so there's, so instead of just kind of one category of bots, these search engine crawlers, now there's, you know, dozens, probably like about 15 to 20, at least the way that we categorize different categories of bots.

Rob Kelly (3:06)

Okay, so we're looking at your knownagents.com/insightspage.

One headline here is 33% of traffic now to websites is coming from bots. These are good bots, you know, average, medium bots, bad bots, all the bots, about 33%. And I double checked because I have three sites up now that Known Agents is running on and I range from 32 to 38%. So I'm sort of in the normal kind of range there. But there's kind of a surprising stat here on the robots.txt effectiveness. Can you first off just define what that means? And what's kind of the surprising part of the data here?

Gavin King (3:50)

Yeah, totally. So what a robots.txt file is, is it's a file that you put on your website. Before a bot starts looking at any of your pages, it should read this file, see if it's allowed to visit particular pages. And if not, then it should back off. It is sort of an honor system.

And yeah, there's sort of a current narrative right now that many bots, if not all bots, just ignore this completely. And that's just not true, right? Like this data set, this is thousands of websites that we're looking at the traffic of. Many of the big bots, especially of the bigger companies, they actually do follow the rules. So if you put something in for OpenAI or Anthropic, they actually will back off and not visit your pages. So yeah, I think a lot of people, like I said, immediately jump to firewalls or bot blocking enterprise software, when they can get 96% of the problem solved just by having a full coverage robots.txt.