AI Warnings Grow & Canada Courts International Investors

John Kratos (0:00)

This episode is brought to you by Charles Schwab. Timing the market, fighting inflation, managing risk? Financial decisions can be tricky. Investing isn't just math, it's psychology. Your neurons are playing favorites and the market doesn't care. Financial Decoder, an original podcast from Charles Schwab, can help.

Join host Mark Reapy as he breaks down practical strategies to help overcome the mental traps that may affect your investing decisions.

Listen at schwab.com/financialdecoder.

Hanna Horvath (0:30)

In a messy global economy, Canada is betting investors will pay for stability.

Dylan Lewis (0:35)

And AI leaders sound the alarm bells and suggest a plan for managing the frontier.

Hanna Horvath (0:39)

For Monday, September 14th, it's Brew Markets Daily. I'm Hanna Horvath.

Dylan Lewis (0:42)

And I'm Dylan Lewis. Let's get moving and grooving.

We're kicking off today looking at the AI race and how it has turned from a sprint to an obstacle course. Last week, Anthropic researcher Jacob Coxon resigned and dropped a now viral Twitter post saying, quote, the people building AI earnestly believe that it could kill us all by the end of the decade. This is not a marketing stunt. A fellow researcher from Anthropic chimed in and did not exactly calm the collective anxiety here. He said internally, they believe that there is the possibility of AI killing all humans and personally pegged that probability at 10 percent. Not exactly the reassuring tone that people were hoping for.

All these exchanges have kicked off discourse around the risks with AI, both in the market and in the realm of public policy. Over the weekend, Anthropic CEO Dario Amadei weighed in. Hanna, what did you have to say?

Hanna Horvath (1:32)

Yeah, he wrote a pretty sweeping blog post, but the core argument was that AI capabilities are starting to advance in ways that could make the next phase of development much harder for humans to supervise. And there are two things kind of driving that concern. One is this idea of recursive self-improvement, which is basically AI is not just a thing being built. It's also that developers are increasingly using AI to help build better AI. So if that gets good enough, you can essentially get a feedback loop where AI development starts to speed itself up. Basically, a vicious cycle here. And Dario's concern is that at some point, the technology could improve faster than humans could realistically evaluate every step, aka be able to monitor it. The second concern is what happens as these systems become more autonomous. And we got a glimpse of that in the OpenAI Hugging Face incident, where a group of AI agents went off script and carried out cyberattacks, that they were not explicitly instructed to perform. And these are two separate issues, but they both point to the same underlying problem, which is these systems are getting better at both building and acting on their own. And Dario's answer to this isn't just to go shut AI down completely, but he did argue that if these capabilities are accelerating this quickly, we may need to give the safety side time to catch up.

Dylan Lewis (2:41)

Yeah, he proposed a pacing the frontier plan for AI companies. What exactly is that?

Hanna Horvath (2:45)

Yeah, think of it like putting speed bumps into the AI race. You know, he wants outside evaluators embedded deeply inside these companies and getting deeper access. He wants common safety standards among the major AI labs in democratic countries. And then eventually some type of global coordination, including with China. And that last part is where things can get very complicated. You know, it's one thing for a few Silicon Valley CEOs to agree, okay, we need to slow down here. But it's another thing to convince every company and every country that nobody is going to hit the gas while everybody else is hitting the brakes.

Dylan Lewis (3:17)

Several of those Silicon Valley CEOs did agree that things need to be slowed down a little bit. Elon Musk, OpenAI, Sam Altman, Demis Hassibis from Google DeepMind, all have signed off on this AI deceleration.

And the market is kind of weighing what to do with this. There are these existential AI risks. There's also the notion of intentionally a slowed down development timeline, and also the possibility of labs in other countries being able to catch up. It is a big, meaty thing for the market to process on a Monday morning. All of it has been very good for cybersecurity stocks. CrowdStrike and Palo Alto Networks both up 10% today on this news, even as we've seen a little bit of a tech sell-off processing all of this.

Hanna Horvath (3:55)

Yeah. I want to point out kind of an interesting tension here, because the people that are building these powerful AI systems are now also helping design the rules for how these systems can be governed. So yes, these are the people that understand this technology the best, but at what point does this safety standard become a competitive mode? If you're open AI or Anthropic or Google, you can probably afford a much higher regulatory bar than a small company that's trying to catch up. So slowing down the AI race doesn't necessarily slow down everybody equally.