Context Engineering for Agents - Lance Martin, LangChain

**Alessio** (0:04)
Hey, everyone. Welcome to the Latent Space Podcast. This is Alessio, founder of Kernel Labs, and I'm joined by Swicks, founder of Small AI.

**Swicks** (0:11)
Hello, hello. We are so happy to be in the remote studio with Lance Martin from LangChain, LangGraph and everything else he does. Welcome.

**Lance Martin** (0:19)
It's great to be here. I'm a long time listener of the pod, and it's finally great to be on.

**Swicks** (0:25)
You've been part of our orbit for a while. You spoke at one of the AIEs, and also obviously we're pretty close with LangChain. Recently though, you've also been doing a lot of tutorials. I remember you did R1 Deep Researcher, which is a pretty popular project, and Async Ambient Agents. But the thing that really prompted me to reach out and say, okay, it's finally time for the Lance Martin pod, is your recent work on context engineering, which is all the rage. How did you get into it?

**Lance Martin** (0:54)
Well, it's funny.
Buzzwords emerge oftentimes when people have a shared experience. And I think lots of people started building agents, kind of early this year, mid this year, quote unquote, the year of agents. And I think what happened is when you kind of put together an agent, it's just tool clanging a loop, it's relatively simple to lay out. But it's actually quite tricky to get it to work well. In a particular, managing context with agents is a hard problem. Karpathy put out that tweet, canonizing the term context engineering. And he kind of mentioned this nice definition, which is context engineering is the challenge of feeding an LM just the right context for the next step, which is highly applicable to agents. And I think that really resonated with a lot of people. I, in particular, had that experience over the past year working on agents. And I wrote about that a little bit in my piece talking about building OpenDeep research over the past year. So I think it was kind of an interesting point that the term captured a common experience that many people were having and it took hold because of that.

**Alessio** (1:53)
How do you define the lines between prompt engineering and context engineering? So is the prompt optimization, like context engineering, in your mind? I think people are confused. Are we replacing the term? What is it?

**Lance Martin** (2:08)
Well, I think that prompt engineering is kind of a subset of context engineering. I think when we kind of move from chat models and chat interactions to agents, there is a big shift that occurred. So with chat models, working on ChatGPT, the human message is really the primary input. And of course, a lot of time and effort is spent in crafting the right message that's passed to the model. With agents, the game is a bit trickier, though, because the agents get in context not just from the human, but now context is flowing in from tool calls during the agent trajectory. And so I think this was really the key challenge that I observed, and many people observed, is like, oof, when you put together an agent, you're not only managing, of course, the system instructions, system prompt, and of course user instructions, you also have to manage all this context that's flowing in at each step over the course of a large number of tool calls. And I think there's been a number of good pieces on this. Madness put out a great piece, talking about context engineering with Madness. And they made the point that the typical Madness task is like 50 tool calls. Anthropics multi-agent research is another nice example of this. They mentioned that the typical production agent, and this is probably referring to Cloud Code, could be other agents that they've produced, is like hundreds of tool calls. When I had my first experience with this, and I think many people have this experience, you put together an agent, you're sold the story that's just tool calling in a loop. That's pretty simple. You put it together. I was building deep research. These research tool calls are pretty token heavy. And suddenly you're finding that my deep researcher, for example, with a naive tool calling loop was using 500,000 tokens. It was like a dollar to two dollars per run. I think this is an experience that many people had. And I think it's kind of that the challenge is realizing that building agents is actually a little bit tricky because if you just naively plumb in the context from each of those tool calls, naively you just hit the context window of the LLM. That's kind of the obvious problem. But also, Jeff from Chrome out spoke about this on the recent pod. There's all these weird and idiosyncratic failure modes as context gets longer. So Jeff has a nice report on context rot. So you have both these problems happening. If you build a naïve agent, context is flowing in from all these tool calls. It could be dozens to hundreds. There's degradation in performance with respect to context length, and also the trivial problem of hitting the context window itself. So this was kind of, I think, the motivation for this new idea of, actually it's very important to engineer the context that you're feeding to an agent.