Welcome to AI in the AM: RL for EE, Oversight w/out Nationalization, & the first AI-Run Retail Store

**Nathan Labenz** (0:00)
Hello, and welcome back to The Cognitive Revolution. Or in this case, I should say, welcome to AI in the AM. This is the third time that my friend Prakash Narayanan and I have done a live stream together. And this time, we figured we should give it a name, and he also took the initiative to create a new look for the show, with real-time AI transcription and AI-powered comment moderation. Check out the video, and I think you'll agree that he's done a really nice job with the look and feel. As you'll see, in some ways, we are still very much figuring out both what we want the show to be and how best to organize and produce it. One thing we're going to look at after this episode is creating a mechanism where we can easily signal to one another when we'd like to ask a follow-up question or move on to another topic. Nevertheless, when it comes to the quality of guests and conversations, I think this episode is right where we want to be. Our guests for this episode were Sergiy Nesterenko, CEO of Quilter, which is using reinforcement programming to train AI systems to perform circuit board design, a problem with an insanely high-dimensional search space, complicated physical constraints, and relatively low volume of available training data. After that, we spoke to Andy Hall, Professor of Political Economy at Stanford, who's doing a bunch of interesting work to characterize models' behavior in political contexts and who's also working to design independent AI governing bodies that he hopes will allow the public to exercise some oversight over AI companies without requiring nationalization. Then finally, we welcome Lukas Peterson and Axel Backlund from Andon Labs. You may know them from their autonomous vending machine work, but today we'll be talking about the new AI-operated retail store that they've recently opened on Union Street in San Francisco. The store, which is managed entirely, including the hiring of human staff, by an AI agent, currently has a 2.6-star rating. But I personally still can't wait to visit.
For me, the big takeaway from this series of conversations is once again, that the future is coming at us much faster than we can process it. Assumptions that seem safe from one perspective become very questionable in the face of increasingly powerful and autonomous AI systems. With that in mind, I want to add just a bit to my answer to the very first question that Prakash asked me in our opening discussion. Namely, why are we now suddenly seeing violent outbursts directed at AI lab leaders?
First of all, while it's certainly possible, and I would very much hope that the recent attacks on Sam Altman's home will ultimately prove to be a random blip signifying nothing, my honest assessment is that by default we should expect to see more of this kind of thing. And not because of the super high P doom numbers coming from the AI opposition camp, but simply due to the fact that more and more people are now becoming aware of the extreme reality of the AI situation. It wasn't that long ago that Sam, Dario, Demis, and Ilya all signed, alongside many other luminaries, a statement saying that mitigating the risk of extinction from AI should be a global priority alongside other societal scale risks such as pandemics and nuclear war. And though the record does show that each of leading AI companies was founded with a certain awareness of and intention to address the hard problems of AI safety. And of course the upside, it goes without saying, is unquestionably immense. In practice today, they are developing what they recognize to be destabilizing and likely dangerous technology pretty much as fast as they possibly can, while repeatedly failing to live up to their own prior safety and social commitments. In significant part because, and here I am quoting Sam Altman's immediate reflections after the Molotov cocktail incident, quote, being the one to control AGI as a quote, ring of power dynamic to it.
All while by their own accounts, we still face anywhere from a 5 to 20% chance of something like, as Sam himself again famously put it, lights out for all of us. And the US government's main concern seems to be making sure that nobody can constrain their ability to use the technology for autonomous weapons, domestic surveillance, or anything else for that matter.
I can't emphasize enough, objectively, this really is a crazy situation. Those of us who stumbled onto the idea that all this might happen years ago have had a lot of time to get accustomed to it and to position ourselves to do what we think we can about it. And many of us have reconciled ourselves to the idea that some version of it is inevitable. But that doesn't mean that we should try to tell people who are only now learning about this that they are wrong for freaking out about it. A 1 in 20 chance of human extinction is not low, and absolutely is worth freaking out about. I'm reminded of two movies that memorably illustrate how I think many people will respond to learning the facts about AI. In the 1998 movie Armageddon, when an asteroid is found to be on course to destroy the Earth, it's just understood that it's heroic for individuals to risk and in the end even to sacrifice their own lives to save the world. And that doesn't hinge on whether there's a 5% or 20% or 99.9% chance that the asteroid really will hit the Earth. The heroes would be heroes in any case.