Ep 66: Member of Technical Staff at Anthropic Sholto Douglas on Claude 4, Next Phase for AI Coding, and the Path to AI Coworkers

Jacob Effron (0:00)

Sholto Douglas was a key part of the Anthropics Claude 4 models. It was really fun to sit down with him on the day these models got released. We talked about a bunch of things, including how developers and builders should think about this next generation of Anthropic models. We talked about what the trendline mean for where these models will be in 6-12 months, 2-3 years from now. We hit on what's required for reliable agents, and when these models will get better in domains like medicine and law, kind of mirror the advances they've already made in coding. And then we hit on his views on alignment research. Where we are today, what's working, what still needs to be done, and his reaction to the AI 2027 work. This was just a fascinating conversation with a brilliant mind in LLM research. I think people will really enjoy it. Without further ado, here's Sholto. Well, thanks so much for coming on the podcast.

Sholto Douglas (0:42)

No, it's a joy. It's a really cool little group.

Jacob Effron (0:43)

Yeah, no, I appreciate you getting in this little cave with us.

Sholto Douglas (0:47)

Yeah.

Jacob Effron (0:47)

It's always fun. By the time this podcast comes out, well, the world will have Claude 4 I'm sure people will play around with it, but I'm curious. You're one of the first people to get to play around with these models. What gets you most excited about them?

Sholto Douglas (0:58)

There are another step up in software engineering, that's for sure, and Opus is really an incredible software engineering model. More and more, I have these moments where I go and ask it to do something incredibly ill-specified in our large monorepo, and it's able to go and do it in quite autonomous and independent way. They're going and discovering the information and figuring this out, running tests, and that blows me away every time.

Jacob Effron (1:19)

Every time we get a new set of models, we have to re-characterize a mental model of what works, what doesn't. How has your model now changed when you're coding? Like what you use these models for and don't with this lead.

Sholto Douglas (1:30)

So I think the biggest one is time horizon expands a little. So I think you can characterize model capability improvements along two axes. One of those is like the absolute intellectual complexity of the task, and the other one is the amount of context or the amount of successive actions that they're able to meaningfully reason over and include. And these models feel substantially better along the second axis. Like they're really able to take multiple actions and like figure out what information they need to pull in from their environments and then act on those. So giving it like, it's the time horizon. Also the support that we like the like Claude code and this kind of stuff. The fact that it now has access to all of the tools to be able to do this in a useful way and you aren't sitting there like copy pasting from a chat box is like a pretty meaningful improvement in that regard too. There is a wide range of tasks where I'm looking at an hour plus or like many hours of work where I would have done and it's just there churning away in front of me doing these like in terms of human equivalent time.

Jacob Effron (2:28)

People are going to get these models when this podcast comes out for the first time. What is your like advice on the first thing they should try?

Sholto Douglas (2:33)

First thing they should try? I mean I think honestly try and plug them into your work. That's the biggest one is like sit down and ask it to do the same thing. The thing that you were about to do first off in your code base that day and watch as it figures out what information needs to pull in and figures out what to do. I think you'll be pretty impressed.

Jacob Effron (2:54)

Yeah. I mean now that you have these new capabilities, obviously you have tons of people that build on top of these models. What are you hoping is newly enabled for builders that take these models and build applications?

Sholto Douglas (3:04)

So I think this is a concept of a product exponential in some respects. We have to be constantly building just ahead of the models capabilities. I like to think about this in terms of like say like Cursor, and Windsurf, and Devon, and these products. If you look at Cursor, they had a vision for coding would be that was substantially ahead of where the model capabilities were for a while. Cursor didn't hit PMF until the underlying models like Claude 3.5 Sonnet took off such that the assistance that they wanted to give people was able to be realized. Then Windsurf went, I would say like substantially more agentic, and that enabled them to get a reasonable slice of market share by really pressing harder on that product exponential. What we're starting to see now with Claude code, but also with the new Claude Github integration and with OpenAI codecs, and also Google's coding agents, so everyone's really using coding agents.