The Hidden Layer Every AI Agent Runs On | Tony Holdstock-Brown, Inngest

**Turner Novak** (0:02)
Tony, welcome to the show.

**Tony Holdstock-Brown** (0:04)
Thanks for having me. Good to see you again.

**Turner Novak** (0:05)
Good to see you too. I think probably the best place to start, what is Inngest for people who don't know?

**Tony Holdstock-Brown** (0:10)
Yeah. So Inngest is a small SDK that gives you durable execution on any platform. The TLDR is you build step functions, and they automatically retry. They will succeed every single time your functions run, and we abstract all of the infrastructure there. So no queues, no state, no events. It just kind of works.

**Turner Novak** (0:26)
So somebody who has never heard any of those words before, they just said, like, why is that all important?

**Tony Holdstock-Brown** (0:33)
Yeah, totally. So the TLDR of this is, when you're running something like agents, so you're running some sort of process like order shipments, non-AI workloads, you might be interfacing with API providers that fail. You might have systems that fail because, for example, your database is down. Typically, the way you get around this is building this crazy system of queues and events and spending a lot of time. You get months on infrastructure.
When you use something like Inngest, you write some really basic code that says, in this step, we are going to call Anthropic. In this other step, we are going to call a tool that Anthropic's LLM wanted us to call.
If Anthropic fails because of rate limits, we will automatically retry that step exactly where it is in the function without losing any of the previous context because we store all of its state. You basically do zero work to get durability out of the box. That's really cool because then any engineer who wants to create these step functions, whether that's an agent harness or if that's something like e-commerce orders or health insurance, you get all of this out of the box without spending any time on infrastructure.

**Turner Novak** (1:42)
Do people spend a lot of time on it if they're not using Ingest? Yeah.

**Tony Holdstock-Brown** (1:46)
This is actually where Ingest comes from. I spent months rebuilding the same infrastructure on Kafka and then on Qs, and it was terrible.

**Turner Novak** (1:55)
This was in 2019-ish.

**Tony Holdstock-Brown** (1:58)
Yeah, this is a while back. Yeah, it was a while back. Qs still exist. Everyone uses classic example is SQS.
Really complex to work with. Even harder with AI because you're going to be building really complex chains of logic for your agent harnesses. If you're not doing that, then when your agent harness fails, you're essentially trying to retry in memory, and or your entire agent is going to just fail, which sucks.

**Turner Novak** (2:28)
How do people get around this today? If I'm not using something like Ingest, and I'm just straight up shooting this all with my own code, building this all myself, how do you do it?

**Tony Holdstock-Brown** (2:39)
You either don't, and you live with the fact that an agent is going to get halfway through, die, and you're going to retry the entire thing, which is a ton of wasted effort time, or you're trying to string together this architecture yourself, building queues and so on. And that is going to take you a ton of time and be relatively inflexible. And that's kind of the antithesis of AI. With AI, you need to be able to update your code, your agents, your harnesses, your prompts, your models really quickly because models are going to change relatively frequently. You're going to learn a bunch as you run things in production, and you're going to take that learning and put it back in your application code to continue improving your harness.
So in general, it's sort of got to the point in which you need to develop using some sort of harness, and your harness is almost always going to be using Dribble execution so that it can generate the traces, retain state, retry correctly.

**Turner Novak** (3:31)
Is this not something that's just built into the AI products already? It just kind of sounds like it's so table stakes. As somebody who's not in the weeds every day like you, I'm just like, oh, that should have been a solved problem because it's so obvious.

**Tony Holdstock-Brown** (3:42)
It should have been a solved problem and it's so obvious. I think there's been some attempts at trying to fix this. For example, agent SDK is a thing, but it doesn't go as far as you need. It allows you to create some sort of harness yourself, but it doesn't go as far as you need in order to give you the durability, the reliability, concurrency controls, constraints for each of your users. This is also relatively new in that we've been going for four or so years. These are all relatively new things in the scope of infrastructure. So I think a lot of people have also been trying to experiment with what an agent SDK could look like, like an agent framework itself, and some of those learnings are that each particular LLM call is in a step, and that LLM will return some data, which will maybe ask you to do some tool calls, and you run this in a loop up until you hit some goal.