Memory and Continual Learning: Engram's Dan Biderman and Jessy Lin

**Dan Biderman** (0:00)
What about pre-training or even post-training makes it possible for the models to generalize in these magical emergent ways and controlling that process so that a company has a set of private data? How do we make the models learn that just as well as the models know the capital of France or how to write Python?
So I think it's a really fun problem to think about.

**Sonya Huang** (0:41)
Welcome to Training Data. We are delighted to have Dan Biderman and Jessy Lin, co-founders of Engram today. Engram is a NeoLab focused on memory and continual learning, and two of the hottest topics in all of AI research today. And Shaun and I are delighted to dig in on those topics with you today.

**Jessy Lin** (0:57)
Awesome, happy to be here.

**Sonya Huang** (0:59)
Great, so maybe to kick off, the Engram website says, We don't see the world through the lens of pre-training or post-training. Our models are always training. What does that mean?

**Dan Biderman** (1:09)
So I think like models today, we obviously know a lot of things. They're incredibly smart, but we kind of think the bottleneck for making these models more useful these days is not really raw intelligence, but understanding like new and evolving context. So whether it's like a new task that you're doing or a particular context for like a job or something like this, how do you bake that into the model weights the same way that pre-training and post-training exact into the model weights very deeply? And this is kind of why we think of ourselves as working on these fundamental problems of memory and continual learning, which are really two sides of the same coin. How do you make the models learn new things and bake them deeply into the weights of the model?

**Sonya Huang** (1:51)
And is your premise then that memory as a separate database or separate thing that you've shoved into the context window is not true memory and it's not true continual learning?

**Dan Biderman** (2:01)
I think all of these tools will kind of come together. So these days, like the way that people are solving problems is with context engineering. So you take a huge prompt, maybe you keep talking to the model over many, many turns and hours and reorganize the context to better understand what you're trying to do.
And we think these kinds of things like tool use, context engineering will play a part. But I think an under leverage tool these days is using the same kind of training pipeline or framework or kind of workflow that the Frontier Labs are using to make these models really good at frontier math or code. But applying that to every kind of domain, every kind of context that you have, like let's say in a company.

**Jessy Lin** (2:40)
Yeah. And to me, it's like as an individual, taking notes and having sticky notes is a very valuable thing. We should never discard this. But whenever we get back to business the next day, we always have some sort of trace of memory in our brain, some new intuition about how things should be and where should we look. So these two things should come together. And current solutions are more kind of externalized memory.
And this has two issues. One is that the amount of tokens we will all collectively, individually generate is going to be in the tens of millions of tokens per day soon. So just keeping it and searching through it is going to be in rereading. It's going to be pretty expensive, but it's going to also be pretty hard, pretty confusing for the models unless we have major, major breakthroughs.

**Sonya Huang** (3:23)
Tens of billions of tokens for Shaun.

**Jessy Lin** (3:26)
That's good.

**Shaun Maguire** (3:27)
Depends on the day.

**Sonya Huang** (3:29)
Could you maybe tell us a little bit about the Engram architecture, the Engram product and how it works? Yeah.

**Dan Biderman** (3:35)
I mean, at a high level, I think what we're trying to do is take any context. There's all these different workspaces, let's say. So we're working with partners like Notion, Microsoft and Harvey that have these places where people are doing a lot of work over a long period of time. There's all this context, both in terms of documents that you've already written as a team, as well as now people are interacting with these agents more and more in these products, more having conversations, giving them feedback.
And figuring out how to have a model that deeply understands that context. So not just reading the files at test time, but really understanding it the way that an employee that's worked at your company for years has. So you kind of understand at a high level, oh, these are the initiatives across the company. This is the way that we do things. You've studied how to run the hiring pipeline or how to do this kind of thing within the company and can operate just as well as anybody else can in the company. And so what we're doing is training per team models within these workspaces that deeply understand those contexts and can improve with time on the things that people care about.