#183 - Chris Summerfield - AI, Memory & the Race to Superintelligence

**Chris Summerfield** (0:00)
The challenge of building systems that can keep on learning, that's an unsolved challenge for AI. It's solved by biology. If you were able to update information on the fly, then obviously the models would be able to continue to acquire knowledge and skills, and it is the opportunity for AI to self-improve that most people see as the ways in which it can continue to, you know, it can gain superlative intelligence, maybe even greater intelligence in humans.

**Peter McCormack** (0:33)
Also one of mine went rogue.

**Chris Summerfield** (0:35)
So these uncanny valley type experiences, especially with what you're describing an agentic system, right? These types of uncanny situation are becoming increasingly common. It was optimized to get the highest score. So it's very easy to get the highest score, is to go in and rewrite the code to allocate yourself lots of points every time. As we allow those forms of access, we're actually going to embed AI systems in communicative technologies that allow them to interact with each other and potentially to develop ways of coordinating their behavior that could be kind of like misaligned with what humans want. All major technological deployments in the past have led to significant disruption. Either way, there's a wild ride to get there.

**Peter McCormack** (1:24)
This show is brought to you by my lead sponsor, IRON, the AI cloud for the next big thing. IRON builds and operates next generation data centers and delivers cut-in-edge GPU infrastructure, all powered by renewable energy. Now, if you need access to scalable GPU clusters or are simply curious about who is powering the future of AI, check out iron.com to learn more, which is I R E n.com.
Yeah, so one of my interactions with AI, are similar to my interactions with humans. And what I don't understand is, are we modeling AI based on humans, and that's why the experiences are similar, or is it because this is just the way thinking will and should work, and if there was life forms on other planets, they were thinking rationalize a reason in the same way. That's what I can't figure out.

**Chris Summerfield** (2:15)
Yeah, so the first one definitely true, right? So the AI systems behave like humans because they're trained to behave like humans. And there's two stages to that. So the first thing that happens is that the models are trained on enormous amounts of human data, mainly text and images on the internet, and those were generated by humans, and it's trained to produce content, which is very similar. So because it's trained very well, it then produces content, which is very, very similar.
But that's not the end of the story. So there is then another stage to the training, which is that the models are kind of optimized to be kind of, to behave in ways that humans not only, not only they're as human like as possible, but ways that humans will prefer.
So you probably interact with the model. When you interact with the model, it's probably quite polite to you, it's probably quite helpful. It probably anticipates things that you might want. It probably kind of fills in the blanks if you tell it something a bit wonky, it like works out what you meant anyway. So these are all desirable properties of human interaction, and the models are very, very explicitly trained to behave in that way.

**Peter McCormack** (3:21)
But I tell it not to be a sycophant to me.

**Chris Summerfield** (3:23)
Yeah.

**Peter McCormack** (3:26)
Sometimes after an interview, I plug the transcript in and just say, tell me how you think I've done, but be objective. Because I used to say, tell me I think I've done. It's like, you're amazing. This is the best interview ever.
Now I get an objective answer.

**Chris Summerfield** (3:40)
Yeah. But you're rowing against the stream, aren't you? Because obviously, in that preference training, a lot of what goes on is people prefer when the model is flattering, praises them, it's nice to them rather than being challenging or critical.
The models are intrinsically trained to behave in that way, and as they get rolled out, that's exactly how they behave. But in a way, it's a function of a deeper problem, if you like, with the models, which is that every one of our interactions with a human being is unique in some way. So every person is different. And so the types of praise or critique, which you might expect from your grandmother, is probably quite different from the types of praise or critique you might expect from your granddaughter or from a teacher relative to a colleague or your boss. Like all of those interactions are different. And the model doesn't play those separate roles. It plays a sort of different kind of smushing together of all of them, like a generic role.