Titans: Neural Long-Term Memory for LLMs, with author Ali Behrouz

**Erik Torenberg** (0:00)
Hello, and welcome back to The Cognitive Revolution. Today, I am thrilled to share my conversation with Ali Behrouz, PhD student at Cornell and lead author of the fascinating paper on integrated large-language model memory, Titans, Learning to Memorize at Test Time. This paper represents another significant step forward in addressing what I've often called the missing middle in memory for large-language models. We've got world knowledge baked deeply into the model weights, and we've got attention-based working memory in the context window. But that missing middle layer, the integrated, persistent, and ever-evolving longer-term memory that humans use to maintain coherence and identity over time, still seems like a necessary piece for success in high-context environments and has been a frontier to watch in AI for some time now. It was once famously said of computers that you could see them everywhere but the productivity statistics, and to a lesser degree, that's been true of AI as well, at least so far. In part, this is because the technology itself is really only recently useful, and people take time to adjust. Tyler Cowen recently went reverse-opera, pointing out to an influential audience and saying, You are a bottleneck. You are a bottleneck. But I think often the bigger barrier is best summarized by another famous Tyler quote, Context is that which is scarce.
To be honest, I've never been entirely sure what Tyler meant by that in the context of humans, but in the context of AIs, it makes perfect sense. Today's AIs know a literally superhuman amount about the world at large, but out of the box, they know little to nothing about the individuals and businesses that they're meant to serve. Meanwhile, assembling and maintaining relevant context for them, especially since it's often spread out across Slack, email, documents, GitHub, meeting transcripts, task management systems, you name it, that is tedious work, and the AIs have really only recently, I would say truly with Gemini 2.5 Pro, started to properly reward it. But for the sake of argument, imagine a world in which context is not scarce for AIs. A world in which an LLM trained specifically for a major company, say GE or 3M for example, knows as much about that company, including its products, its history, its team, its internal processes and debates, its finances. It knows as much about the company as it does about the world at large. Obviously such a model could be created. No company has more than 1% of the data on which the AIs are already trained. And if it were created, it would immediately know more about the company than any single person at the company. It might still be trained to search official records to ground its analysis and work, but unlike today's model, it would know, in a way very similar to humans do, when it's actually found what it's looking for. And it would almost certainly pick up many of the subtle patterns that constitute what I call how we do things around here, which could make it relatively easy to manage, especially as compared to the overall process of hiring, onboarding, and retaining human knowledge workers. Startup costs for a model like this could reach into the millions or perhaps even tens of millions of dollars, which is notably roughly where OpenAI has publicly priced their custom models offering. But for a hundred-year-old enterprise, that would still be a bargain for an AI that you can drop in and have do a significant portion of the work at the company, particularly since you'll be able to amortize those costs across as many copies as you need. For smaller businesses, meanwhile, which have much less data, using today's fine-tuning prices as an anchor, I would expect costs to be more like tens of thousands, maybe into the low hundreds of thousands of dollars. Obviously, still affordable. We're starting to get a glimpse of this future as individual ChatGPT users with their new memory features these days, but I'm not aware of a productized version of this that works well at scale. And I think that's ultimately because there's still the gap in the foundation models themselves, which scaffolding isn't quite enough to fix. The bottom line, then, is that it seems plausible to me that the main thing between where we are today and a future full of drop-in knowledge workers that begin to very quickly and dramatically disrupt the labor market is simply a breakthrough in long-term memory. And that's why I think today's topic, Titans, is such a big deal. Unlike rag-type systems, which store data explicitly and then make it searchable via a mix of traditional or embedding similarity or graph searches, or even more integrated strategies like Mamba and other state space models, which encode memory as a matrix of numbers that gets updated as part of each forward pass, the Titans architecture that Ali and his co-authors propose uses a neural network, which itself is updated via gradient descent at runtime as the LLM's memory module. This is a qualitatively different approach, and in my humble opinion, represents a significant conceptual advance. So with all that in mind as motivation, in this conversation, Ali and I explore the technical details of Titans, including how he conceptualizes and takes inspiration from human memory systems, how the associative memory loss function works, the role of surprise and momentum in updating the neural memory module, and the various approaches they experimented with for integrating this long-term memory module with the standard attention mechanism. Beyond the technical details, this conversation also offers a fascinating glimpse into how a highly original and obviously quite brilliant researcher thinks about pushing forward in a relatively new architectural direction. I was really struck by how many times in this conversation I wanted to dig in and understand the reasons behind the decisions that Ali and co-authors had made. But his response was basically that they see all of this as very early foundational work, and so they just did the simplest thing possible for now, trusting and expecting that others will come along to improve and refine it later. Overall I really enjoyed this conversation, and while it is pretty technical at times, I think Titans is one of those relatively few papers that is worth taking some time to grok. The intuitions behind it are elegant, and this line of work might just produce the last major technical unlock needed for AI to hit an inflection point in economic value and impact. As always, if you're finding value in the show, I'd appreciate it if you'd take a moment to share it with friends, write a review on Apple Podcasts or Spotify, or just leave us a comment on YouTube. Of course, we welcome your feedback either via our website, cognitiverevolution.ai, or by DMing me on your favorite social network. And finally, a quick reminder, I'll be speaking at Imagine AI Live, May 28th through 30th in Las Vegas, the Adapta Summit, August 12th and 13th in São Paulo, Brazil, and the Enterprise Tech Leadership Summit, September 23rd through 25th, again, in Las Vegas. If you'll be at any of those events, please send me a message and let's meet up in person. For now, I hope you enjoy this technical deep dive into the frontier of large-language model memory with Ali Behrouz, PhD student at Cornell and author of Titans.