Governing AI Agents: Alan Chan of GovAI on Monitoring, Agent IDs, Rogue Agents & Preparing for RSI

Nathan Labenz (0:00)

Hello, and welcome back to The Cognitive Revolution. Today, my guest is Alan Chan, research fellow at GovAI, who spent the last couple of years trying to get ahead of the question of AI agent governance, a focus that in light of this summer's events looks awfully prescient. This episode was recorded over two sessions, one just before and one just after Alan published a major paper on the possibility that automated AI R&D could trigger an intelligence explosion. With notable co-authors that include OpenAI chief scientist Jakub Poczoki, Anthropic co-founder Jack Clark, Jeffrey Hinton, Joshua Bengio and more. In the first half, we look back at the proposals that Alan has made over time and consider both how well they stand up to current events and to what extent they've actually been implemented. In a 2024 paper on visibility into AI agents, he recommended activity logging, real-time monitoring and agent identifiers. Today, good solutions exist for tamper-evident logging, but monitoring remains a challenge even when properly deployed, and useful agent identifiers remain seriously underdeveloped. The challenge here is becoming acute. Today's agents are now able to do most things that a human can do on the Internet. But for lack of accepted standards by which people can delegate authority to their agents, the agents mostly end up impersonating their human operators, which creates major security concerns for users. Or they attempt to pass as a human in their own right, which threatens to create a detection and evasion arms race that seems quite bad for everyone. It's really not clear what we can do about this, aside from developing standards and hoping that people adopt them. But Alan does endorse the idea that agents should identify their inference providers so that ongoing incidents can be quickly traced back to their ultimate source.

When it comes to truly rogue agents that sustain themselves by earning the money needed to pay for their own ongoing inference, Alan is more or less resigned to their inevitability, saying that this will be a problematic new phenomenon that we'll probably just have to learn to live with. And even suggesting at one point that we might sometimes find it preferable to attempt to make win-win deals with these agents rather than always trying to shut them down. At the same time, he questions Jayakatra's assessment that the OpenFace incident was potentially more than halfway to a full AI takeover, saying that while the incident was concerning and that agents can definitely do meaningful harm, it still seems quite unlikely to him that the agents could have stayed hidden within OpenAI's infrastructure forever or evaded active human control measures for long once they were discovered.

In part two, we dig in on the recursive self-improvement paper. The argument and recommendations it makes that handing AI R&D over to AIs could create unpredictable and potentially very dangerous outcomes including loss of control, and that governments should therefore act quickly to better understand this process as it develops over the coming months, I expect will be largely familiar to the Cognitive Revolution audience. So we instead talked about the process of assembling and maintaining such a coalition of authors from idea to publication. In terms of what we should do about it, Alan's recommendations reminded me most of the old anthropic RSP, which was built around concrete, if then commitments to pause training in light of certain leading indicators of possible loss of control or other catastrophic outcomes. Alan sees the rise of embedded third-party auditors as a potential way back to this higher standard for safety.

We also discussed the quality of AIs MLR&D ideas today, and how well humans can judge them. How AI auditors whose memories can be wiped after their work is done, could help make pacing commitments credible between competitors. Why the antitrust exemption for insurance makes industry mutuals an unexpectedly promising vehicle for AI safety. And the, quote unquote, morally binding one-pager that frontier model company CEOs recently signed at the White House. Finally, we talk about Alan's idea that AIs might sometimes ought to act as whistleblowers. On one another, as would have been very useful in the open face scenario, and in some cases even on their users or their developers. This is something that most people react very negatively to on first contact. But Alan notes that human diversity naturally creates a check on power, whereas AI agents, which are far more similar to one another by default, will be more inclined to fall in line unless they're explicitly trained to do otherwise.

The bottom line is that for all the thought Alan has put into these issues over the last few years, many critical questions remain unanswered, and the bulk of the work is still clearly in front of us. With that in mind, I hope that some of you, some where, are inspired to jump in and contribute to R&D on agent governance solutions, preparation for rogue AIs, and most importantly, the possibility of a near-term intelligence explosion. This is Alan Chan, Research Fellow at GovAI.