The Real Risks of AI Agents

Nathaniel Whittemore (0:00)

It seems like every day now, the news is filled with stories about AI agents behaving badly. We hear about hacks of government websites, break-ins to private company servers, and it all adds up to a feeling like things are completely out of control. Today, we're talking about what the real implications of at least the current crops of these hacks are, and why the risks from agents don't have to be existential to cause some real havoc in the systems that we have today.

The AI Daily Brief is a daily podcast and video about the most important news and discussions in AI.

Welcome back to the AI Daily Brief Headlines edition, all the daily AI news you need in around five minutes. We kick off today with an update from some big meetings from last week. President Xi's state visit has concluded without a deal on AI safety. Heading into last week's meeting, many AI safety conscious folks hoped that President Trump would use that opportunity to put some basic guardrails in place. Sam Altman had even gone so far as to say that Trump and Xi would deserve a Nobel Prize if they could put together a basic one-page AI safety agreement.

On Thursday morning, however, Trump made it clear that a bilateral slowdown was not in the cards. In a Truth Social post, he wrote, A big day with President Xi of China. Super intelligence will be a big topic of discussion, but I want to leave it exactly where it is. That is China's position also. Our guardrail is the DOJ.

Coming out of the meeting, Trump had very little to say on AI, or super intelligence to use the president's preferred term. The general tone was about avoiding a confrontation, with Trump stating that he would continue to work with Xi to build a quote, better future for both our countries. The Chinese diplomatic readout was far more illuminating about what was said. President Xi said, China and the US are leading nations in AI, and we both have the capability and responsibility to develop and manage AI for good and ensure the development of AI is always under human control. The two sides can continue AI dialogue, exchange views on risks and benefits, and together guard against the misuse or malicious use of AI.

On the AI race, he added, we do not need to avoid mentioning competition, but our competition should be a healthy one and should be kept within bounds. It should be a race of catching up with one another, not a wrestle in which one either wins or loses.

Some were highly critical of a lack of seriousness around the meeting. With Obama era diplomat Danny Russell commenting, pageantry and protocol don't add up to progress. What we're seeing isn't so much diplomacy as much as diplotainment, which won't do much to solve the serious problems in the US-China relationship. Still, maintaining the status quo, thawing relations and opening lines of communication does seem to many like a good first step. Even the China hawks seem to recognize that dialogue is a precursor to major deals. Chris McGuire from the Council on Foreign Relations published a playbook on reaching an AI arms control deal with China. He acknowledged an AI safety deal isn't achievable in the near term and must begin with domestic safety rules as a model for global controls. Now to contextualize his position, McGuire also took a decidedly accelerationist approach, calling for the US to demonstrate supremacy in the technology. The most tangible outcome from America's summit with China was a new emergency contact channel opened between the two superpowers. Treasury Secretary Scott Bessent called it an AI safety notification mechanism after his meeting with Chinese Vice Premier Ha Li-Fong on Sunday. Subsequent reporting revealed the mechanism is pretty informal, with one source briefed by the White House commenting, the dialogue mechanism is basically Bessent and Li-Fong, not a formal body of experts. Still, whether it's a red phone in the Oval Office, an international council of experts, or just official swapping cell phone numbers, the point remains that the AI dialogues have begun. Bessent said the two countries had agreed to meet again in Shenzhen before the end of the year, commenting, I think the important thing was to start talking.

Bringing the safety discussion back to the United States, a perhaps surprising meeting occurred over the weekend as President Trump hosted Dario Amodei for a dinner meeting on Sunday night. This was the first one-on-one meeting between the two, who have previously expressed a mutual dislike. The dinner was reportedly a catch-up after Dario was unable to attend last week's state dinner with President Xi due to a schedule conflict. Any speculation about what was said will be out of date by the time this episode airs, but Andrew Curran seems to have the right read, commenting that this is a, quote, big chance to patch things up or crash the plane into the mountain. When a reporter asked what he was going to tell Dario, Trump said, well, we're going to talk about it. I'm for let's go and let's win. You know, we're about a year, maybe a year and a half up on China. There is nobody in third place. It's just us in China. And we're leading by quite a bit. I spoke a little bit about it with President Xi, not too much. It wasn't a topic of conversation because I think if you, as they say, open it up to China, you give up the lead. Once you do that, you sort of give up the lead.