⚡️How Claude 3.7 Plays Pokémon

**Alessio** (0:02)
Hey everyone, welcome back to another Latent Space lightning pod. This is Alessio, partner and CTO at Decibel. There's no swicks today. We got a special co-host, Veeboo, which if you're a part of the Latent Space community, on Discord you definitely seen. Welcome Veeboo as a co-host, first time.

**Veeboo** (0:19)
What's up guys?

**Alessio** (0:21)
In that we had David Hershey from Anthropic today, who's the person behind Claude Plays Pokémon. It's funny, I saw we were first DM'ed about playing Magic the Gathering together in NSF. I really...

**David Hershey** (0:32)
On all of the different nerd angles you can get me.

**Alessio** (0:35)
I'm glad to be here. David is the person doing this and I was like, okay, I'll DM him and then yeah, it was cool. We already had a touch point. So welcome to the show. This is our second Anthropic episode. We had Eric Schlantz from the SWE agent before. So welcome.

**David Hershey** (0:53)
Thank you. Glad to be here. Excited to talk Pokémon.

**Alessio** (0:55)
Yeah. So let's give a little background on this. So Sonnet 3.7 came out a couple of weeks ago. I don't know. Time goes by so quickly. This week? I don't know, man. It feels like two weeks ago. And then you had this Claude Plays Pokémon thing that kind of went viral where if people remember, there used to be this thing called Twitch Plays Pokémon where people could go on Twitch and kind of type in the chat and then busy like figure out what the next action that the emulator would take is. What you've done instead is given it to Claude and basically have Claude figure out how to walk through it. I'm looking at it right now. So far it's been stuck in Mt Moon for 52 hours. Poor guy. It probably met 15,000 ZooBets. So yeah, let's talk about what gave you the idea for it, kind of the origin story that we can go through the implementation.

**David Hershey** (1:41)
Totally. Yes, I actually started working on it in like June of last year for the first time. And for me, so I work with customers at Anthropic, and I just like really wanted to have some way for myself to be able to like experiment with agents, like in a real way, some framework, some harness where I could actually just like go to town and try some different things and see, see what actually worked to get Claude to do like pretty long running tasks in general. And so I like had that in one hand, and then I was like, OK, what is the thing that will make me the most addicted to making this work? Like how will I grind to the hardest actually trying this? And Pokémon was like a pretty clear answer. Someone else that Anthropica actually like tried once to hook it up. So I had a little bit of like the shell of what I needed to actually put it together and to like kick off what became an obsession a little bit in the coming months. So yeah, like I played with it in June in the Switch trying things out. This was like Sonnet 3.5 came out in June of last year was just when I started kicking around. It was very good, but you could see like kind of signs of life, but like not much really happened. And then ever since then, as we released new models, it's sort of been like the way that I get to know one of our new models a little bit, right? So we released the new version of Sonnet 3.5 in October and like used this to like really kind of see like what's it better at and it got better. Like you could see it start to like, it could get out of the house somewhat reliably, which was not always true. And it got a starter and I like even named it sometimes. Like it was like doing stuff, not great, but like it could move. Along the way to like, I'm just like, we have a Claude Plays Pokémon Slack channel. Like I'm sort of just like giving people updates. So over time, as I'm like posting gifs and up-par eyes updates, like I'm just like slightly growing in popularity of a cult following internally of people who are somewhat interested. But then like a couple of weeks ago, I was bashing an early version of Sonnet 3.7 and it just like, you can just tell it had like, it was a little different. It's clearly not still good. As you said at the top, like it's, it's in Mt Moon for its 50 something hour. This is a little bit worse than average from what I've seen so far by now. But like, this is like, you know, about on brand. It doesn't really have a great sense of direction. It's pretty bad at seeing the screen, stuff like that. But like, it plays the game, you know? It gets Pokémon, it catches Pokémon, like it caught its first Pokémon, it got out of the variety the first time, like a whole bunch of stuff happened for the first time or like could squint and see a thing play in the game. And yeah, like posting updates obviously internally, it was very fun. Like people were just like kind of going wild at the fact that this was actually happening finally. And it was like entertaining enough that I could kind of see it. And the other side is like we kind of just like got finally a sense that this was like an actually useful way to measure what was going on with this model. You know what I mean? Like there's one thing that it's like fun and fun follow along, but like internally, like I think we got more of a sense that like you could actually use this as a bit of a measuring stick for what's going on in the model. I've spent, you don't want to know how many hours I spent staring at Claude play Pokémon, I have to have seen and read like millions of words that Claude has generated in the course of playing Pokémon over the last eight months. So like you can kind of get a feel for like what's actually going better, what's it getting better at and that kind of thing. And with this particular release, like I think the fact that it got this much better at this kind of reflects a lot of things that we wanted to be true about the model to begin with.