Designing How AI Grows — Tom McGrath

Tom McGrath (0:00)

Which is like Neil Nanda says, SAEs are dead.

There you go, you can use that for the intro. I think interpretability is, I think of it as a natural science, like physics, biology, chemistry, but it's a natural science that you do completely on the computer. And so this means that we should be able to, again, speed run science once we have agents that can do experimental work for us, and the ability to do that experimental work as fast as they need it to happen. So we have real scientific work to do, no barrier to research, scientific work to be done, and it's sort of gated on both empirical data collection and theory building.

Tim Scarfe (0:37)

Dario Amodi, so he had a blog post called The Urgency of Interpretability, right? And he gave this wonderful analogy of a bus. We're all on the bus, and we're hurtling down the road, and we can't stop the bus, but we can potentially steer it. But the window is foggy at the front, so we can only really look in the rear-view mirror. And also, the steering wheel doesn't work very well, so you can steer it a little bit once every few hours or something. So interpretability is a bit like defogging the front window. And what you're proposing is the ability for us to steer the bus, essentially.

Tom McGrath (1:11)

So I feel like if anything is going to go, if any science is going to get revolutionized by intelligence, we should make sure that it's interpretability. And I think that it's possible that it just goes an order of magnitude faster.

In the next couple of years than it has in the last decade. That's when I think, why am I optimistic about interpretability? It's partly because I think we're starting to have really good traction, but also because I can imagine this incredible speed up.

Tim Scarfe (1:38)

Right, which is almost like there's a little man inside our brain, and it's a form of convergent evolution, because we interact with the world using our physical affordances and so on. And could it also be the case that there's a little world inside neural networks?

Tom McGrath (1:54)

Oh, very good.

Tim Scarfe (2:04)

You had quite a nice piece actually in your intentional design blog, where you were saying that there's almost a spectrum of possibilities, right? There's, you know, we can write a program to do something, or we could admit a lot of ambiguity. And where on that spectrum do we want the foundation models to sit? And when we build applications, where do we want those to sit?

Tom McGrath (2:22)

Yeah, and that's sort of part of the point of the intentional design idea is, at the moment, you can have like one or the other.

You either write a program, like it's the Stone Age, or you get a model to do it. And that model will have been trained, like it just gets whatever it gets from its training process. We can't select, like when you write a program, that stuff only goes in if you intend it to go in and some bugs.

But like, we want to be able to have this sort of spectrum where you have more like engineering ability in the model creation process. So you can say like, I want to learn this, but not that. And I think that that's like, I think that's quite a hard thing to do. We're sort of trying to imagine a new way of doing machine learning which brings intelligence into it. But I think we could really change the way we do machine learning if we can figure that out.

Tim Scarfe (3:17)

I know. I mean, when I interviewed the Apollo research guys, I mean, they were kind of talking about these conflicting objectives. So, you know, what the developer wants, what the platform wants, what the grader wants and so on. And I suppose this is kind of talking about this, you know, when you're in the intelligence regime, it's really, really difficult to specify exactly what you want. And it's very, you know, possible in a novel situation for the calculus to change, right? And the model, all of a sudden, it will decide to do this and instead of that, which makes me think that engineers are going to have to increasingly take more responsibility.

Because, do you think it's possible in principle just to kind of train models that could robustly deal with all of these novel situations? Or do you think it's more of a, you know, engineers have to take some responsibility?

Tom McGrath (4:01)

Probably some of both. Like currently, I think it's not possible to, our current training methodologies don't seem sufficient to give this kind of control over training. And so it's all on, you know, human engineers with their AI assistance to secure, secure these systems sort of in a different way.