AI Discourse Deranged: Assessing LLM Generalization Takes and Polarizing Regulatory Debate

**Erik Torenberg** (0:00)
Turpentine is a network of podcasts, newsletters and more, covering tech, business and culture, all from the perspective of industry insiders and experts. We're the network behind the show you're listening to right now.
At Turpentine, we're building the first media outlet for tech people by tech people. We have a slate of hit shows across a range of topics and industries, from AI with Cognitive Revolution to Econ 102 with Noah Smith. Our other shows drive the conversation in tech with the most interesting thinkers, founders, and investors, like Moment of Zen and my show Upstream. We're looking for industry leading hosts and shows along with sponsors. If you think that might be you or your company, email me at erik.turpentine.co, that's E-R-I-K at turpentine.co.

**Nathan Labenz** (0:45)
Oh, look at this. It proves that language models cannot generalize.
And this is basically insane. Instead of Helen of Troy, I was thinking this tweet is like the Helen of Transformers. If we start to mislead or embrace pretty obviously wrong headed conclusions about what is, it cannot be good for our downstream discourse of what should be done about it.

**Erik Torenberg** (1:13)
The concern here is that this is a Trojan horse or a wedge into sort of a governing body that has the reputational credibility and then the legal ability to regulate who or who not can innovate.

**Nathan Labenz** (1:27)
The alternative is we're going to shit on the people that are trying to establish the best practices, then that's what's going to bring down the heavy handed regulation. If you want to prevent that regulation, show me that there's no problem. Show me that you have it under control. This may be the time to build, but it's definitely not the time for ideology. Hello and welcome to The Cognitive Revolution, where we interview visionary researchers, entrepreneurs and builders working on the frontier of artificial intelligence.
Each week we'll explore their revolutionary ideas and together we'll build a picture of how AI technology will transform work, life and society in the coming years. I'm Nathan Labenz joined by my co-host Erik Torenberg.

**Erik Torenberg** (2:07)
All good on your end?

**Nathan Labenz** (2:09)
Yeah, I got a few bones to pick today, but aside from that, you know, everything's going well. It's been a little bit of a quiet period the last, you know, 10 days as I've really been digging into all the open AI releases and, you know, trying to feel out what they're good for and what they're not good for. I think that'll be subject of another episode because I want to do at least a couple more experiments before I give a summary of my findings. One spoiler is the vision component, which I was expecting to be a huge unlock. I think it really is going to be a huge unlock. And in part, that's also because it's quite cheap.
You can pass in 12 images for one cent and then you do pay also for what it generates in response to that. But 12 images for a cent gives you, you know, a lot of ability to kind of take slices out of videos or just take periodic screenshots of stuff, you know, all sorts of monitoring solutions. A lot of passive stuff I think can happen with the vision because it's just so easy to like collect that sort of information. And since it's so effective and cheap at processing it, I think it's going to be a really big deal. But that's not what we're here to talk about primarily today.
Basically, I have just had a burn my saddle, you might say, over the last week or so with a couple of aspects of the AI discourse online, where I'm just like, guys, let's all be better than this. So I want to take these topics one by one, take them apart, analyze a bunch of the different contributions that people made to the ongoing discussion and give my message to all these people. And again, you can hold me accountable as we go. Before doing that, I wanted to take a moment, and this might become a bit of a ritual, to give a strong kind of nod and pay respects to the value of accelerating the adoption of existing AI technology. And I had kind of two findings that were just relevant in the last few days that I wanted to highlight, if only as a way to kind of establish some, hopefully, credibility and common ground for the critiques that are to come. But not only that, because I think these are also just like, you know, meaningful results. So the first one comes out of Waymo.
And they did this study with their insurance company, which is Swiss Re, which is a giant insurance company. So here, I'm just going to read the whole abstract. It's a kind of a long paragraph, but read the whole abstract of this paper and just, you know, reinforce because it's kind of a follow up to some previous discussions, especially the one with Flo about like, you know, let's get these self drivers on the road. So here's some stats to back that up. This study compares the safety of autonomous and human drivers. It finds that the Waymo One autonomous service is significantly safer towards other road users than human drivers are, as measured via collision causation.