Emergency Pod: Mamba, Memory, and the SSM Moment

**Erik Torenberg** (0:00)
Turpentine is a network of podcasts, newsletters, and more covering tech, business, and culture, all from the perspective of industry insiders and experts. We're the network behind the show you're listening to right now.
At Turpentine, we're building the first media outlet for tech people by tech people. We have a slate of hit shows across a range of topics and industries, from AI with Cognitive Revolution to Econ 102 with Noah Smith. Our other shows drive the conversation in tech with the most interesting thinkers, founders, and investors, like Moment of Zen and my show Upstream. We're looking for industry leading hosts and shows along with sponsors. If you think that might be you or your company, email me at erik.turpentine.co.

**Nathan Labenz** (0:46)
My sense is that neither the human brain nor the transformer are the end of history. The purpose of this episode today is to really sound an alarm and say that I think we now have that new architecture. We're going to see more effective agents, more compelling long-term assistance, more compelling long-term AI friends and companions. All of this, if I had to guess, I would say it probably happens several times as fast as the transformer era. Hello and welcome to The Cognitive Revolution, where we interview visionary researchers, entrepreneurs, and builders working on the frontier of artificial intelligence. Each week we'll explore their revolutionary ideas, and together we'll build a picture of how AI technology will transform work, life, and society in the coming years. I'm Nathan Labenz, joined by my co-host, Erik Torenberg. One of the most important big questions that I get asked is what are the chances that somebody invents something better than the transformer?
Since 2017, with the introduction of the transformer, attention is all you need. Transformers have dominated the field. One of the big realizations that I had as a relatively recent entrant to the field a couple years ago from my multimodal perspective at Waymark was, oh my God, it's a transformer solving all of these different problems. As I'm watching art creation come online, as I'm watching language models get dramatically better, as I'm watching all sorts of niche tasks advanced rapidly, image captioning, image matching, video captioning, all these different things. So many things, everything, everywhere, all at once, but all the transformer.
So I think I've been pretty clear in recent months on a bunch of different episodes that this is hard to predict. I sometimes call it the hundred trillion dollar question, which is the size of the global economy today. But my sense is that neither the human brain nor the transformer are the end of history.
If you watch my AI scouting report, I go through the history of expectations and predictions about how far AI might make it given raw compute guesses. These date decades back, most notably in the 90s with Kurzweil publishing his Singularity is Near, drawing these exponential curves and saying, hey, right around 2020-ish, 2020, 2025, that's when you're going to have enough compute to match the power of one human. And then later you're matching all of humanity and becoming truly superhuman. Well, the human level AI has shown up roughly on schedule. It is, as we've covered, you know, in many different ways, human level, but not human-like. In many ways, it is very alien. And yet this power, which now is in most domains ahead of the average human and in many domains really closing in on expert performance, this has basically all been driven by the transformer. The attention mechanism is doing everything for us.
So I've been really interested in this question. How likely is it that somebody will invent something better than the transformer? And I've been watching out for it, and there have been a few plausible candidates this year, which I've mentioned a number of times. Those include most notably Retnet, which is a publication out of Microsoft collaborating with Tsinghua University in China. We also had Lily Yu from Meta on to talk about the Megabyte architecture, which was still fundamentally transformer, but the hierarchical approach that was different and in meaningful ways. And of course, there have been a ton of improvements, but there have been a few candidates for things that, hey, this might start to look like something that could be even better than a transformer.
The purpose of this episode today is to really sound an alarm and say that I think we now have that new architecture.
And it is called The Selective State Space Model, aka Mamba, published in just the last couple of weeks. I saw this paper pretty much as soon as it came out, saw some of the claims and have gone really deep into not just trying to understand it. That was about the first week since it came out, but then also really begin to project into the future. What is this going to mean? I think if my AI scouting paradigm is good for anything, it should be good for identifying new research that really matters and at least giving a pretty good guess about what that information, what that new research, what that new capability is likely to unlock in practical terms. So that's what I'm going to try to take you through here today. It is going to be hopefully accessible throughout. I always try to use vocabulary words and then the most plain spoken terminology that I can. So I'll try to make it as accessible as possible throughout. It will also definitely be technical. I will be getting into the weeds for sure deeply. But I actually want to start off with something a little bit different today.