Why Your AI Agent Gets Brain Damage, and How to Fix It - Tyler Barnes, Mastra

Shane Thomas (0:00)

You're working for a long time, a long session, and all of a sudden, you hit this imaginary window that then now causes a massive compaction event and you lose context and you feel like you have to just restart the session.

Tyler Barnes (0:10)

A lot of people have hated compaction just because it's like your agent, it's like they got brain damage as soon as it happens.

Shane Thomas (0:19)

I'm with Tyler, founding engineer at Mastra, and we're gonna be talking about memory, specifically observational memory. We're gonna talk about agent signals, but first, a lot of people have probably seen you before, but it's been a while since you've been on the show. What were you doing before Mastra, and then what are some of the things you've worked on since you've joined Mastra, which was very early. I mean, you joined when we were in YC.

Tyler Barnes (0:40)

I joined Mastra, like you said, early on. I had been working on my own agents and stuff, and I think you and I would get on calls just to hang out sometimes, and we kind of demo stuff to each other, and I was getting pretty excited by the stuff you guys were working on.

Shane Thomas (0:54)

I remember you telling me, it was very early, but you had basically wired up a camera where if your cat came, it would detect that this is a cat, so feed it, and if your dog came, it would detect that it wasn't a cat, don't feed it, or something like that.

Tyler Barnes (1:10)

It would play like an alarm to scare the dog away, so it was always eating the cat food.

Shane Thomas (1:14)

So you're using some AI there to detect which...

Tyler Barnes (1:17)

I think it was like GPT-4R or something, you know?

Shane Thomas (1:19)

I think it was before that, actually, but it was somewhere in that range of models.

Tyler Barnes (1:24)

It was, yeah.

Shane Thomas (1:25)

We always talked just models and what we were building. But then you joined Mastra, and you've obviously worked on all different areas, for sure, but you've specialized in a couple.

Tyler Barnes (1:37)

Yeah, I guess memory was the first thing.

Shane Thomas (1:39)

So tell me a little bit about the first version of Mastra memory, because we've gone through some iterations now. There's still a lot of people that used the initial memory system that we built.

Tyler Barnes (1:49)

Yeah, there's quite a few people. I think it started with three types.

Maybe we just started with the message history, very simple. The last X number of messages, like 10 or 20 or however many you want. Then we added working memory, which is like the agent can use a tool to update a chunk of context. So over time, it can keep track of something.

Shane Thomas (2:08)

Did that context basically just sit in the message history, or sit in the system prompt, or how did the context actually, what was the underlying mechanism to make that happen?

Tyler Barnes (2:17)

It did sit in the system prompt, which is not great for prompt caching. We've actually fixed that since, with the newer version of memory.

Shane Thomas (2:23)

When we built that, no one cared about prompt caching.

Tyler Barnes (2:25)

That's true. It wasn't a thing people were talking about.

Yeah. I think it existed, but not all the providers even had automatic prompt caching at that time. It was like, a lot of the time, you're just paying uncached prices all the time.

Shane Thomas (2:36)

And so you had message history, working memory, what else?

Tyler Barnes (2:41)

And then the other one was semantic recall, which is just RAG. So every new user message and a system message, you would basically do a RAG query and then insert into the system prompt again, some relevant context. At the time we ran LongMemEval on it and got like a really high score. And it was somewhat controversial because it was like, oh, you just use RAG and you can score very highly on this benchmark. We've since gotten a much higher score, but it was quite interesting that such a simple system could work so well.

Shane Thomas (3:08)

Yeah. And I mean, the interesting thing about it was you could basically take a huge message history and only insert the parts that mattered. So you'd actually have less context, which at the time, for folks, that was great. You didn't send as much context, it was cheaper. You had the extra lookup, of course, so maybe you had some extra latency, but you probably made up that latency because you're sending less tokens to the LLMs. But over time, we've realized that maybe there's other approaches that could even be better, especially as prompt caching became more prominent.