Surveying 2,700+ AI Researchers on the Industry's Future with Katja Grace of AI Impacts

**Katja Grace** (0:00)
Forecasting the future is quite hard. We are having to make decisions about these things.
The best guesses that we can get are valuable, and so it's important to hear what AI researchers think about this. Amidst quite a lot of uncertainty here, the chance of things just staying pretty similar and nothing coming of this to drastically change people's lives seems quite low. This seems like a pretty important thing to be keeping an eye on and trying to have accurate opinions about what the public thinks about these things. Things will affect what policies happen and they think that could change the rest of the future forever.

**Nathan Labenz** (0:36)
Hello and welcome back to Turpentine AI. Before today's show, another quick reminder to take a second to sign up for our new Cognitive Revolution feed. Our latest episode on that feed is an outstanding conversation with the great Linus Lee of Notion AI. So find that and be sure not to miss any of the new episodes that will be exclusive to that feed by visiting our website CognitiveRevolution.ai to subscribe.
Today I'm excited to share my conversation with Katja Grace, founder of AI Impacts, who recently published what I believe to be the most comprehensive survey of elite machine learning researchers' expectations that has ever been conducted.
As we all try to make sense of this AI moment and understand where things are going, Katja's survey provides an incomparable resource. With over 2,700 respondents, all of whom have published in one of the top six AI conferences in just the last couple of years, it offers a nuanced and detailed picture of what the people closest to the technology really believe. And the results are truly striking. Arguably the biggest takeaway is that there is no consensus expectation within the field. Most individuals express high uncertainty on the most important high-level questions, and only relatively small minorities confidently predict a very positive or very negative future. Researchers definitely do not dismiss existential risk, with median estimates of a 5-10% chance that AI could lead to human extinction. And meanwhile timelines for key capabilities have shortened since the survey was last conducted just a year ago.
There is strong agreement that AI safety should be prioritized more than it is today, perhaps in part because no major mechanistic interpretability breakthroughs are expected in the near term.
Overall, this conversation offers a uniquely rigorous field-wide take on critical questions that are all too often poorly asked and sloppily debated. After first digging into the methodology to better understand how Katja and team went about defining their terms, framing their questions, and summarizing the responses into aggregate views, we go on to consider implications for policy makers, the public, and the field itself. While there's of course nothing to say that ML researchers will ultimately be correct about AI outcomes, I would call this a must-listen for anyone who wants to ground their own personal AI world models in the considered opinions of the people who are collectively inventing the technology.
As always, if you find value in the show, please share it with others who you think would appreciate it. I genuinely believe that both the Doomers and the EACs should update their P-Dooms toward these community estimates. A 5-10% chance of extinction is obviously a very critical issue. But the shape of the other 90% of outcomes is also well worth worrying about.
Of course, I'm always eager to hear your feedback, so please feel free to reach out via your favorite social media platform. And now, please enjoy this exploration of the expectations and the substantial uncertainties of the AI research community with Katja Grace of AI Impacts. Katja Grace, founder of AI Impacts, welcome to The Cognitive Revolution.

**Katja Grace** (3:55)
Thank you.

**Nathan Labenz** (3:56)
I'm excited for this conversation. You have just recently completed one of the most comprehensive surveys of the worldviews and expectations of elite machine learning researchers.
And as we try to grapple with this AI moment and try to make sense of what's happening and where things are going, I think it's a really incomparable resource. So I'm excited to just spend the full hour kind of digging into that and understanding both how you went about collecting this opinion and also what the aggregate opinions are. For starters, you want to just introduce maybe yourself and AI Impacts a little bit and give a little bit of the inspiration or motivation behind the survey?

**Katja Grace** (4:39)
I am broadly interested in trying to make the world better. For the last 10 years or so, I've been involved in trying to figure out what's up with the future of AI, since it seems like potentially a pretty important thing to try and cause to go well. I've been variously sceptical about whether it's likely to cause human extinction or not, but interested in figuring out these kinds of things. So AI Impact is an effort to try and answer questions like this and just various high level questions about what will happen with the future of AI that are relevant to decision making for lots of people.