Why We Need Continual Learning

**Malika Aubakirova** (0:00)
The model is basically frozen, but the new experiences, new knowledge still persists. Humans are not AGI, but we still learn on the job, we learn from experience, and that's what makes kind of humans kind of unique. And so that's kind of like the ultimate test, like how do we define that we got to continual learning? It's like, well, is there a system that is able to learn on the job and get better through use just like humans? In all of the labs that we talk to, even the labs don't just tackle one approach, they actually have multiple teams that tackle continual learning through the different kind of paradigms.
Any honest argument about continual learning pretty much has to start with in-context learning, because it genuinely works.

**SPEAKER_2** (0:47)
What if today's AI models can't actually learn? Right now, most systems are trained once, deployed, and then frozen in time. They can reason, retrieve, and generate, but they don't truly update from experience. To compensate, we've built layers around them. Context windows, retrieval systems, agent scaffolding, these approaches work, but they also raise a deeper question. Are we just working around a limitation, or have we reached the ceiling of what this paradigm can do?
There's another path, one where models don't just respond, but improve, where they learn continuously, adapt to new information, and evolve over time, more like humans do.
In this episode, Elena Burger speaks with Malika Aubakirova, partner on the AI infrastructure team at a16z, about why continual learning matters, what's missing today, and what it would take to build systems that actually learn from experience.

**Elena Burger** (1:44)
Good afternoon, everyone, who is currently monitoring the situation at 2:33 p.m. Pacific time.
I'm Elena. I work on the new media team, and I'm here with Malika. Malika, do you want to introduce yourself?

**Malika Aubakirova** (2:04)
Yeah. Thank you so much for having me here. I'm a partner on the AI infrastructure team, excited to chat more about continual learning.

**Elena Burger** (2:12)
Yes. Today, Malika published a piece called Why We Need Continual Learning. I think even before we go into this piece, I think first, Malika, you should just talk about your process writing it, because it seems like you spoke to every single AI researcher under the sun, so we'd love to just hear what your process was, and then we can kind of get into the meat of the piece itself.

**Malika Aubakirova** (2:41)
Totally, absolutely. Like, in fact, that's actually the reason why we didn't name all of the individuals involved, because we had the opportunity and luxury to talk to a number of just incredible top researchers, founders, PhD students, we organized continual learning dinners. And so honestly, this piece was shaped largely by their insights and learnings and made this piece much more sharper and grounded than anything else we could have written on our own. So definitely, thank you all for just incredible insights.

**Elena Burger** (3:17)
Yeah. And so the piece opens, so first, everybody can now see, you know, this reference to the great Christopher Nolan movie Memento. So the piece opens, sorry?

**Malika Aubakirova** (3:31)
With a twist on machines.

**Elena Burger** (3:33)
With a twist on machines.
So why did we open the piece with this metaphor? You know, what is the kind of like Memento-like experience of AI models today and kind of like what is the frame that we used here?

**Malika Aubakirova** (3:53)
Yeah, absolutely. So I guess I'm not sure if everyone in the audience has seen Memento movie. If you haven't, here is kind of like a quick blurb, but basically the main protagonist, Leonhard Shelby, has a form of this amnesia where he cannot form new memories. So he goes about his life with kind of like this cut-off date after which point he has kind of these long-term memories but really cannot retain anything new that he experiences. And so what he does is he uses the sticky notes where he writes some of the notes to himself. He pulls out his Polaroid camera to capture moments as he goes on about his life. And I mean, he even tattoos some of the memories that he wants to imprint in his memory basically throughout kind of the movie. And so why does this matter, right? Like this is just kind of like an explanation, but why does this matter? And like AI models, like honestly, it kind of maps one-to-one to how AI models work today. So we have the training phase where we basically encompass all of the world knowledge and that part is what we call pre-training. And after the training phase, we basically have the cut update after which point we deploy the models into the world and we call that inference.