Governing Frontier AI, with CA Senator Scott Wiener, Author of SB 1047

**SPEAKER_1** (0:00)
Turpentine is excited to announce our new show, The AI Daily Brief, hosted by Nathaniel Whittemore.
The AI Daily Brief is a daily show that covers all things AI, from legislation to new technologies in the market, to the philosophical and ethical debates around generalized intelligence. If you're looking for an edge to stay up to date on everything AI, subscribe to The AI Daily Brief at the link in the description.

**Nathan Labenz** (0:23)
Hello, and welcome to The Cognitive Revolution, where we interview visionary researchers, entrepreneurs, and builders working on the frontier of artificial intelligence.
Each week, we'll explore their revolutionary ideas and together we'll build a picture of how AI technology will transform work, life, and society in the coming years. I'm Nathan Labenz, joined by my co-host, Erik Torenberg. Hello, and welcome back to The Cognitive Revolution. Today, I'm speaking with California State Senator Scott Wiener about SB 1047, his proposed legislation to establish safety testing and risk mitigation requirements for advanced AI models, which the bill defines, as does the Biden executive order, as models trained with 10 to the 26th flops of compute in 2024 and other models with similar capabilities, even if they require less training compute to get there in the future.
It's an admittedly complicated definition, the ambiguity of which reflects the general uncertainty about how this technology will evolve going forward. In our conversation today, Senator Wiener begins by sharing his enthusiasm for the potential of AI, as well as the story and motivation behind SB 1047 He explains how he's attempting to take a light-touch approach, not to eliminate but at least to minimize risk, without harming prospects for continued AI innovation.
He addresses some of the misconceptions he's seen in the online discussions surrounding the bill, and he responds to various good faith objections that have been raised. For a more thorough analysis, I definitely encourage you to listen back to our previous episode, where Nathan Calvin from the Center for AI Safety Action Fund and Dean W. Ball from the Mercatus Center joined Steve Newman and me to discuss the potential benefits, as well as the drawbacks of this bill, from a generally pro-technology point of view. I think that episode holds up very well in light of today's conversation. So, where do I come out on SB 1047?
Personally, I often describe myself as an adoption, accelerationist, hyperscaling pauser. Which means that, like Senator Wiener, I really want to see us get the incredible value of things like primary care AI doctors. And by the way, I'll soon have an episode on Google's new Med Gemini model, which shows just how close we are to that exciting reality. And yet, at the same time, I do believe that we are playing with a new kind of fire in AI, and that it's critical that we exercise caution on our way to superhuman AI strategists and scientists. SB 1047 is very much in that spirit, and for that reason, I am inclined to support it. While it's true that the small number of companies pushing the frontiers today seem to be proceeding responsibly without being forced to, and while I, as a longtime libertarian, absolutely recognize that a new government agency could become a bureaucratic nightmare, it seems to me that the stakes associated with frontier AI development are high enough and the competition is fierce enough that this bill constitutes a prudent step. Of course, there's more that government could and perhaps should do with respect to AI, and I did ask a number of questions about other possible rules meant to mitigate, for example, arms race dynamics between leading developers. But in the end, I came away from this conversation quite sympathetic to Senator Wiener's point that you can't do everything in one bill, and that the focus of this bill is on ensuring that serious testing is done and safety mitigations are applied before models are released. As you'll hear, one change that I would still love to see, and which would turn me from a somewhat cautious to a very enthusiastic supporter of the bill, is stronger support for independent third-party testing. I believe it's critical that independent experts who are motivated solely to find the truth have the opportunity to thoroughly test frontier models during the training process, and to be able to report their results directly to the government, without fear of loss of access to the models or other retaliation from the developers. This is a tricky balance to strike, with open questions including who should be trusted to test, what sort of access should they have, and what remedy should the labs have if testers begin to behave in bad faith. But as challenging as those questions are, evidence is mounting that a government requirement for third-party testing is needed if we are to achieve Senator Wiener's stated goal for this bill. For one thing, my experience on the GPT-4 Red Team, and if you haven't already heard it, I definitely recommend my Did I Get Sam Altman Fired episode for the full story of that experience, shows just how precarious an independent tester's position can be vis-à-vis the frontier developers. And that's assuming one can get access in the first place.