Subquadratic SSA: Shattering the Quadratic Bottleneck | 22nd June 2026

**DailyNews** (0:00)
Welcome to Colaberry AI Podcast, brought to you by Colaberry AI Research Labs and Carl Foundation.

**SPEAKER_2** (0:05)
Yeah, glad to be here.

**DailyNews** (0:06)
So imagine throwing your entire monolithic codebase, dozens of sprawling financial filings, and just massive contradictory legal contracts into an AI model all at once.

**SPEAKER_2** (0:19)
Just dumping it all in there.

**DailyNews** (0:20)
Right, dumping it in, and actually getting accurate nuanced reasoning back, completely bypassing the need for complex retrieval pipelines.

**SPEAKER_2** (0:28)
Which is kind of the holy grail right now.

**DailyNews** (0:30)
Exactly. Today, our mission in this Deep Dive is to dissect a highly technical and frankly audacious set of claims coming from Subquadratic's new SubQ 1.1 Small model.

**SPEAKER_2** (0:41)
Yeah, they're making some massive waves with this one.

**DailyNews** (0:44)
They really are. They're claiming they've achieved a 12 million token context window using roughly 1,000 times less compute than standard models. Okay, let's unpack this.

**SPEAKER_2** (0:53)
It's a huge claim.

**DailyNews** (0:54)
It is. So if you're an engineer or a technical leader listening to this, our goal today is to rigorously analyze the underlying mathematical methods they used, tear apart these benchmark results, and most importantly, identify the specific things to try if you get access to this model.

**SPEAKER_2** (1:11)
Because the premise they are presenting, it's a fundamental paradigm shift.

**DailyNews** (1:16)
Right.

**SPEAKER_2** (1:17)
And to truly understand the weight of what Subquadratic is proposing, we have to start by grounding ourselves in the brutal mathematical reality that makes their claims so controversial to begin with.

**DailyNews** (1:29)
Quadratic bottleneck.

**SPEAKER_2** (1:30)
Exactly. The quadratic bottleneck of dense attention. I mean, for years, the foundation of how transformer models understand language has been the self-attention mechanism.

**DailyNews** (1:39)
Right.

**SPEAKER_2** (1:39)
And in a standard dense attention architecture, every single token in a sequence has to mathematically compare itself against every other token to calculate a relevant score.

**DailyNews** (1:49)
Because it's essentially calculating a massive matrix of probabilities, right? So if you double the input length, the computational cost doesn't just double, it quadruples.

**SPEAKER_2** (1:58)
It explodes.

**DailyNews** (1:59)
Yeah. If you scale your context window from, say, 10,000 tokens up to 1 million tokens, your compute requirement doesn't increase by a factor of 100 It increases by roughly 10,000 times.

**SPEAKER_2** (2:12)
And that exponential explosion in compute is the exact reason why the entire AI engineering industry has spent the last few years building massive, fragile architectures of workarounds.

**DailyNews** (2:24)
We're talking vector databases, right?

**SPEAKER_2** (2:26)
Right. Vector databases, retrieval, augmented generation, or arrayed frameworks, aggressive chinking pipelines, all of it.

**DailyNews** (2:33)
Complex agent orchestration layers.

**SPEAKER_2** (2:35)
Exactly. All this heavy infrastructure exists for one simple reason.
Dense attention models just cannot affordably hold an entire massive document in their working memory and reason over the whole thing at once.

**DailyNews** (2:47)
Which means every retrieval system we build is essentially a force gamble. We are making an implicit algorithmic bet about what information actually matters before the model even gets to read the whole text.

**SPEAKER_2** (2:57)
Yeah, you're hoping your pipeline grabs the right pieces.

**DailyNews** (2:59)
I always think of our current RRAG pipelines like trying to understand a highly complex 200-page corporate merger contract by only being allowed to read isolated sentences on pushcards.

**SPEAKER_2** (3:13)
That's a great way to put it.

**DailyNews** (3:14)
You might retrieve a highly relevant clause on page 2, but you completely miss the semantic context where that exact term is severely limited on page 46 and then quietly altered again in an appendix.

**SPEAKER_2** (3:26)
Right, you get the fragments, but you completely lose the architecture of the document.

**DailyNews** (3:31)
Exactly.

**SPEAKER_2** (3:33)
What's fascinating here is how the industry has previously tried to solve this without relying on those fragmented RRAG pipelines and why those attempts haven't quite cracked the enterprise problem.

**DailyNews** (3:43)
You mean the compression models.

**SPEAKER_2** (3:44)
Yeah, we've seen compression alternatives. Architectures like Mamba, Retinet and RWKV. They attempt to bypass the quadratic scaling problem by compressing past context into a small fixed size memory state as they read.

**DailyNews** (3:57)
Almost like taking a massive high definition image and compressing it down to a low resolution JPEG.

**SPEAKER_2** (4:02)
That's exactly what it's like.

**DailyNews** (4:03)
I mean, it works reasonably well if your goal is generating a high level summary or doing basic sentiment classification. You get the gist of the picture.

**SPEAKER_2** (4:10)
But when your model needs extreme specific recall, meaning it needs to pull one highly specific obscure variable from millions of tokens ago, that compression algorithm often just loses the detail entirely. The resolution isn't there.

**DailyNews** (4:27)