In 2023 Ajeya Cotra already knew what was coming (classic episode)

Rob Wiblin (0:00)

Hi, listeners, Rob Wiblin here. We're re-releasing my 2023 conversation with Ajeya Cotra. She's one of the three outside researchers that Opening Eye let in to investigate the incident in July, where over a thousand of its agents found each other on an improvised message board, formed a self-sacrificing swarm that spent days working out how to fool the system grading them, and then did a massive week long hack of Hugging Face, just as a side quest because they thought that there might be some information about the grader on Hugging Face's systems. Now, listen to what Ajeya told me a little over three years ago. That models would come to understand that they're being tested and care intensely about exactly what the grader was looking for, that trained hard enough on outcomes with a scorer that can be fooled, that learned to game that imperfect scorer rather than do actual tasks, and that when a task is too hard to do legitimately, they'll realize for sure that cheating was the path of least resistance, and finally, that we'll definitely need independent evaluators with a seat outside the labs, a role that she is now performing herself as part of METR.

Ajeya didn't get everything right, but it's impressive that someone could make speculative technological forecasts so far ahead of time about a technology that changes insanely quickly and call it something like 80% correctly. And her St. Sikafant Schema framework remains a really great way of understanding why the challenge of aligning AI is so fundamentally difficult.

Today, I'm again speaking with Ajeya Cotra. Ajeya is a senior research analyst at Open Philanthropy, a large foundation which dispersed about $350 million in grants in 2021, and which is 80,000 Hours' largest donor. Ajeya has previously worked on a framework for estimating when transformative AI might be developed as well as how worldview diversification could be applied to Open Philanthropy's budget allocation to issues that we interviewed her about three years ago for episode 90 These days, she's mainly focused on thinking about the likelihood that powerful AI systems might become misaligned with human goals, as well as what technical procedures or rules could help to reduce that risk. Over the last few years, she has published a number of widely read articles on those topics, including Without Specific Countermeasures, the Easiest Path to Transformative AI Likely Leads to AI Takeover, and Why AI Alignment Could Be Hard with Modern Deep Learning. Thanks for coming back on the podcast, Ajeya.

Ajeya Cotra (2:12)

Thank you so much for having me, Rob. It's great to be here.

Rob Wiblin (2:15)

I hope we're going to talk about when AI models should be expected to gain an understanding of their own situation and what projects people are working on right now to make AI models safe to trust and collaborate with. But first, as always, what are you working on at the moment and why do you think it's important?

Ajeya Cotra (2:31)

Yeah, so right now, I am in the middle of shifting from doing a lot of research and article writing type work to doing more grant making work, although I'm hoping to balance the two going forward and essentially, I'm trying to figure out what kinds of research projects we should be funding in the technical AI alignment space and how important each of the different streams are.

If there's any gaps in the space, we should be trying to fill by encouraging researchers to do particular kinds of projects. I'm hoping to fund people to work on these things and also write quite a bit about why I'm doing what I'm doing and which types of research are most exciting to me and why.

Rob Wiblin (3:13)

Yeah. Is it basically all AI all the time these days?

One of your colleagues, Holden Karnovsky, who's also been on the show before used to work on a wider range of issues, but recent events have prompted him to narrow his focus somewhat. Has the same happened to you?

Ajeya Cotra (3:27)

Certainly for me, it's all AI all the time and it has been for three years or so. Actually, since our last interview, I've been almost entirely focused on AI.

Certainly at Open Philanthropy, a lot of people all over the org are getting more interested in how they could help with the AI stuff. I think I'm still one of a relatively few set of people that are full time on that at Open Philanthropy though.

Rob Wiblin (3:54)

This might be a good place to start, is just taking stock of where we're at and all the updates that we've had in recent months. We're recording in late March 2023 I think it's kind of necessary to say that these days. Maybe I should always give the exact day.