Securing Model Weights Won’t Stop China | Buck Shlegeris

Buck Shlegeris (0:00)

So when people think about US., China, race dynamics, one fact is that the AI companies seem to have very bad security for their model weights. And you might have thought that you could resolve this pretty quickly by improving security in ways that are very specifically aimed at defending your model weights from being stolen. It's not clear to me that this suffices for good outcomes because of the fact that if your algorithmic secrets are stolen, then Chinese companies have enough compute, but they have enough computer to run training runs as big as US company training runs. So plausibly, you just get zero lead if your algorithms are being constantly stolen, even if your model weights aren't.

All right. We are back. We are live with Buck Shlegeris, who is the CEO of Redwood Research, which is an AI alignment organization. Buck, welcome to MTS. Great to be here. So why don't we start with your tweet from yesterday. It was a long post. TLDR, I think there is about a one in three chance that within the next year, AI companies will build superintelligence. If this happens, I think these AIs will probably take over. And there's a serious chance that as part of it, all or most humans will be killed. So that's correct. That's what I think. Right. So why don't we break this down and start with, why is superintelligence within a year, 33% likely?

Yeah, so I got these numbers basically from a bunch of polling like that done by the AI Futures Project over the last week, which I was a little bit of a part of. So roughly speaking, if you just look at the current rate of AI progress, there are a variety of ways of extrapolating the distance between here and superintelligence and a lot of those make it look like superintelligence is pretty plausible pretty soon. I don't know how much you want me to go into the detail of the different options here. But one approach you could take here is, if you want to just know when will the AIs be able to replace all the humans at the task of doing AI research, one way you can do this is look at what's the rate of ECI progress, what's the rate of overall model capabilities progress that's required for their research tastes to improve a certain amount. So right now the AIs, if you talk to people at AI companies, the AIs have pretty bad taste in ML research in a lot of aspects. They have bad ideas about how to understand experiments, they have bad ideas about what algorithms are a good thing to try out, but they're getting better. If you basically try to draw the trend line at the current rates of overall progress, if you try to translate between the rates of progress that we saw from this meter report and then try to translate that into the rate of improvement research taste, you end up believing that the AIs will probably be able to automate AI research next year. There are some other approaches you can take, like more directly trying to go from the estimate of how much faster AI research has to be going at an anthropic to lead to this overall 2X rate and the rate of capabilities progress. And some models that you might try to make using that also get you to believe it's pretty likely you get super intelligence next year. One more thing about this. So I just like chart on my wall, which I guess is obsolete so I can pull it off and just destroy it forever now. This is like the joint distribution over different possible capabilities trajectories that the AI Futures Project put out in August this year.

Basically, each of these rows is one sample from their distribution over how quickly you might progress between different milestones.

So each of these is like in the median world, we ended up getting full automation of research of AI R&D in like 2029

So one way you can think about what's going to happen in the future is you just take these trajectories and filter down to the ones where the current AI R&D acceleration is that which was implied by meters numbers last week in their report. And if you do that, you end up with like insane ones, like right at the top. And so this kind of model is why I think superintelligence next year is fairly plausible.

Theo (3:50)

Yeah.

Buck Shlegeris (3:51)

So why is it not higher than 33 percent?

Because you don't get higher than 33 percent. It's just like kind of a quantitative question.

Some factors that might make you think that AI R&D is going to be accelerated less than you would have thought. You've got all these questions about to what extent is there going to be diminishing returns in AI progress? So nowadays, getting a certain number of ECI points from algorithmic progress takes a lot more work and a lot more experimental compute than it did five years ago. Like five years ago, if you look at how people were training models, they were wrong about all kinds of stuff that in hindsight looks pretty basic. And it wasn't that hard for researchers to come up with ideas and run experiments that would demonstrate how to improve things. And nowadays, just because a much larger number of really smart people are working on AI R&D, it takes a lot more work. You have to have a much, run many more experiments and have much better ideas in order to get a certain amount of progress. And so we've sort of seen this in the case of semiconductor manufacturing. So you know Moore's law, where every 18 months, the number of transistors, you could fit in a certain area doubled. But it turns out that to get Moore's law, you had to also be exponentially increasing the R&D spend. So if you look at a graph of R&D spend on semiconductor manufacturing over time, you also see this exponential increase. And what that's telling us is that in the case of semiconductor manufacturing, every successive doubling of transistor density took you more and more research effort. And so the same effect is almost really true in AI. And so you have to basically think through the interplay, or like the relationship between as your AIs get better and better, it's getting harder and harder for them to make progress, but also they're getting better and better at making progress. And so the whole question comes down to basically how these two factors compare to each other quantitatively. And maybe six months ago, I would have said it was 50-50, those two factors line up such that you end up with extremely rapid AI progress at some point after you've automated AI R&D. Now it seems more like maybe two-thirds that you see a short period of extremely rapid AI progress, where you get 40 years of progress at the current rates in the year after you fully automate AI R&D.