AI:AM: A Level We Shouldn't Pass? Notes from The Curve + Tokens vs. Salaries & Is SaaS Cooked?

Nathan Labenz (0:00)

This week, my notes from The Curve, the conference where Frontier Lab Insiders and their critics share a room. Plus, the memory wall, and what agents are doing to software. This is AI in the AM.

There likely is, let's say, a level of intelligence that we just shouldn't go past.

Thomas Sohmers (0:21)

What?

Nathan Labenz (0:22)

Yeah, first time I'd ever heard that from a Frontier Lab leader, then I just asked to follow up, like, okay, well, how do we operationalize that? Would you be prepared to sign on to something like a limit on the number of flops to go into the next pre-training run? But what actually happened was he basically said, yeah, I think that could be reasonable. Prakash Narayanan, my co-host, on what a ceiling would actually require.

Prakash Narayanan (0:49)

When we say that we shouldn't go exceed a point of intelligence, that automatically means that you have to look at the inputs going in.

Nathan Labenz (0:57)

Thomas Sohmers, co-founder of Positron, which builds AI inference chips around memory.

Thomas Sohmers (1:02)

I was looking at a quote that we had a year and a week ago for memory, and it's gone up four and a half times since then. I really hope that this isn't the case, but I wouldn't be surprised if this is going to end up being another doubling over the next year.

Nathan Labenz (1:17)

Sean Wang, known as Swix, who runs Latent Space and the AI Engineer conferences.

Sean Wang (1:24)

The phrase I've recently taken to is, I don't want to pay for someone else to go through LM psychosis.

I can pay for my own psychosis, that's fine. But when you work for me and I'm paying your tokens, you better be actually producing thoughtful stuff. I have two or three employees right now who are basically under performance review because they're just giving me Claude's law. Yeah.

Nathan Labenz (1:46)

I pity our Claude that's going to have to try to cut this down into a highlights episode, Evan. I think this is a banger content from start to finish today.

Welcome to the AI in the AM weekly highlights. The narration is my cloned voice. Please tell us what worked and what didn't, so we can make the next one better.

Part 1, notes from the curve. Last weekend, I was at the curve at Light Haven in Berkeley.

Prakash asked me about the divides I saw.

One of the big observations in general in AI is that events have come pretty fast. You would think if you were looking ahead a couple of years ago, that with two years of additional information and revelations, and actually seeing how strong the AIs get and what they do and who's been right or wrong, you would think that people would start to agree on the state of things, and that has happened a lot less in the broader world than I would have expected. But I did feel that there was some of that felt at the curve. But I do think the big thing that people have come together on, at least to a significant degree, is it does look like the AIs are going to get really powerful, and this is something we're really going to have to reckon with and can't just hope goes away or feel like maybe if we just wait, the bubble will burst or the fever will break or what have you. This time around, people were definitely like, okay, it's getting pretty serious.

There were very few perspectives that, as Helen Toner once famously put it, long timelines have got super short. I would say another corollary to that was like, what is called normal in terms of expectations is getting increasingly pretty incredible. Even I would say the people there who had the most modest expectations for what AIs will be able to do were like, yeah, it's going to do an awful lot. The question now on the capabilities front has gone to, is RSI going to fom? Is it going to be fast for a while and then level off? Is that more than it's meant to be? Yeah, we've got these super powerful things, but is it really going to run away from us? Those are still the capabilities questions that we're getting asked. But there has been, I think, some healthy updating across the community just based on what we've seen.

Prakash asked whether anyone from the Frontier Labs said limits should be enforced on the players in this space, themselves included?

Yeah, that was a huge topic. I mean, probably the biggest divide in terms of perspective is really just inside the Frontier companies, outside the Frontier companies. Everybody knows that they're living in the future and they have visibility that the rest of us don't have. And so, I actually had a chance to talk to and hear from, in some cases, in sessions, in some cases, just in, you know, passing conversations, all Chatham House rules, all abstract way. But we're talking to people who are like founders, executives, top researchers at Frontier companies and a small set of them. I asked a couple of them, why come here? You know, what's in it for you? You know, and before doing a couple of sessions where I was like interviewing or moderating, I asked the same question, what makes this next hour a good use of your time? And they repeatedly expressed that it's just important to them to share as much as they can with the broader community. Definitely not sharing their trade secrets, of course, still, but really trying to give us a candid take on what are they seeing internally? What do they believe internally? You know, what's their outlook? What's the plan? How can we come together and, you know, potentially do some of these constraints? And there were a number of things that I actually thought were, like, kind of newsworthy.