The AI control gap - who gets to say 'it's safe?'

Dave Vellante (0:00)

Welcome to Breaking Analysis No. 329 I'm Dave Vellante. We see a control gap between what AI can do and what the evidence supports trusting it to do.

You know, an awesome demo or an MVP, it doesn't prove that a system will stay within the limits of an intended deployment's goals. The question is, who gets to decide when the evidence is sufficient to declare that AI is safe? We believe that decision needs independent oversight and enforceable responsibilities.

This week, we draw on an extensive conversation with Appian CEO Matt Calkins, who helped us connect public policy to the agents acting inside a business and to the opportunity to make those systems dependable enough to receive more trusted authority. In this week's episode, we explore the AI control gap, the starting point of which is addressing AI safety through alignment. We're going to question some of the prevailing narratives out there and propose a public policy regime that has more teeth than what has been put forth thus far.

And we bring in some data from our partner at Qualitate to capture the customer sentiment on some of these issues.

Let's start with who can actually say no. So this tongue-in-cheek illustration comes from Amit Govran, and we're using it, of course, with his permission. He wrote a post last week in the Q Research called Grading Others' Homework. And for those listening and not watching, Amit conceived a gathering at the White House. You have Elon, and Zuck, and Dario, and David Sacks, and the POTUS, you got Jensen, all these other tech leaders. And they're smiling at each other, smugly around a table, and they're showing off each other's, their quote unquote teachers' grades that they've given to each other. Great job. Golf clap. So the humorous depiction here underscores a much more serious question, which is how much authority should companies have to self-police? And when they do that, what's the next step? So the September 29th White House Accord includes external assessments and oversight by an independent committee of each company's board. Now, those commitments, they're a good first step. But the accord itself does not create a public authority with compulsory access, associated binding conditions, and penalties for noncompliance. So an assessment or a self attestation, that can identify a problem, but public policy has to establish what happens next, in our opinion. So we asked Matt Calkins, what public regulation should actually look like? And his answer starts with a duty to prevent damage. Please play the clip.

Matt Calkins (3:04)

This is the kind of time that we really need the government to step in. And in similar instances where an industry has the capability to cause a lot of damage, we always ask that industry to behave responsibly. We ask banks to have sufficient funds and not too much leverage so that they don't cause people to lose their savings. For example, we expect power plants to have adequate safety procedures, and we don't just rely on a lawsuit after a nuclear meltdown or something. We always ask industries capable of doing damage to be responsible now and not pay the price after damage is done in the form of civil penalties.

Dave Vellante (3:45)

Now, another key point that Matt made in our conversations is the cost of doing this today is actually low because when you think about it, not a lot of damage has been done. AI has really been not that capable in terms of the harm that it can do. But the longer we wait and the more capable models become, the more expensive the fix becomes. Now, we agree with this principle. The standard should be set in the public interest. Independent AI experts should be able to examine the evidence that's produced by the LLM vendors and their self-policing, and an authority that has teeth should be able to enforce the results.

So, when the evidence doesn't, when it falls short for the high-risk activity, that independent authority needs the power to require some kind of remediation or restrict the activity or stop it completely.

And the vendors, for their part, they have to have clarity on what they have to demonstrate before they can proceed. So, there's got to be a clear test that's transparent. So, this creates a basis for public confidence that a voluntary promise alone cannot provide. As of now, this authority is pretty fuzzy. The tech leaders, according to the president, have a moral authority to do the right thing. Now, Chamath, last week on All In, had this to say, Alex, please play the clip.

Matt Calkins (5:19)

They read it, and everyone's like, yeah.

SPEAKER_3 (5:20)

Which part do you think has the most teeth, will be the most effective at making things more secure? Which if you...