Common Crawl Changes Everything for Harmonic Centrality

James Dooley (0:00)

Common Crawl is showing in LLM training data.

Wow, now this just means that harmonic centrality has become way more important than ever. Some people might be listening to this and saying it's always been important, but many people in the SEO industry are obsessed with DR or PageRank Distribution. But now with regards to being cited in the LLMs, we now have information, factual information to say that the LLMs are using it as part of their training data. I'll give you one or two little examples. I'm not going to go too deep into it, but we've had certain websites blocking all bots apart from CCBot, or we've had certain websites that when all the bots come and visit the page, it shows page A, but it cloaks and shows page B when CCBot comes along, and it's got unique information to CCBot. This information, which only CCBot can see, is now being cited in nearly all of the large language models. This is new information of what we have. I also went through quite a little bit when I was in Sofia last week with Alan Klatz, and he confirmed the exact same thing.

Now knowing Kasra, that Common Crawl is being used. Now obviously, we don't know how much it's being used and stuff like that. Now knowing that Common Crawl is being downloaded, and being used in ChatGPT, and in many other large language models, like what are you thinking now with regards to optimizing for Common Crawl or trying to get a higher CC rank using like Harmonic Centralita?

Kasra Dash (1:41)

I think it puts a big spanner in the works.

Like what you were saying, so many people are used to looking at DR or domain authority scores and stuff. I think this is just another metric. Equally, I think it also gives links builder, or link builders another place to look at, especially if their clients as link builders are trying to get sighted in AI more.

Again, so many people just care about DR or spam scores and stuff. This is completely different, and not only that, this isn't like a made up metric. This is a metric that we all know that AI is looking at, whether it's ChatGBT, Claude, et cetera. One thing I would say, though, is we don't know exactly how much of a direct ranking factor it is. Like a few tests that we have been running is, it seems like it's almost like Ahrefs. Ahrefs buys data every few months. ChatGBT also updates that set data piece every few months as well. So it's not something like you can manipulate and instantly you get an impact. There's probably better ways to get instant impacts. But if you want to be part of the core training data, this seems like more of a path in.

James Dooley (3:15)

Yeah, for sure.

Just to repeat and echo what you've just said there for anyone who's listening. So what they'll do is they'll download it each month, and then it's only when they physically update. So let's say it goes from Fable 5.1 to Fable 5.2 or Opus 4.8 to the next model and stuff like that. It's only when they've updated the model and they've had it as part of the training data, and that they've downloaded from the Common Crawl latest month. So Common Crawl only runs once a month. It goes and crawls the whole of the Internet or as much as it is what it can see. It starts with a seed set of sites, and the closer you are to those seed set of sites, the better the CC rank can be. So you want to be as close to those trusted seed set of sites as possible. The higher the CC rank, the more you're being crawled by Common Crawl, and knowing that Common Crawl now is being fed with regards to the LLMs, now this should be part of your AEO or GEO strategy.

Kasra Dash (4:16)

Yeah, so there's different parts to Common Crawl. I'm just sharing my screen now so everyone can see.

Like they've basically got like the web graph, they've got the URL index, the CDXJ index, they've got all of these different indexes. What we're specifically looking at is the web graph. So as you can see, the last update was June, July, August. You've obviously got May, June, July and stuff like that. So every month is adding on an additional month. So this tells us that A, it obviously crawls the web once a month, but this doesn't necessarily mean that ChatGVT updates their actual information once a month. So just bear that in mind.

I think James, weren't you saying that it takes potentially three to four months after every single time it grabs the information? Yeah.