Prof. Jakob Foerster - ImageNet Moment for Reinforcement Learning?

Jakob Foerster (0:00)

The ARC challenge as a target for the community is a terrible idea. That's not what it's supposed to be. It's not supposed to be something where we then design methods to solve the ARC challenge. Fundamentally, AI is trained on the collective outputs of humanity. This technology belongs to everyone, including people we don't like. It's centralized resources in the interest of the common good, in the interest of the public, and not in the interest of maximum profit. Because to me, the biggest alignment challenge is not between AI and humans. The biggest alignment challenge is between those people who hold the keys to power, who control these systems, and the rest of the population.

Tim (0:37)

Jakob, welcome to MLST.

Jakob Foerster (0:40)

Thank you for having me. Great to be here.

Tim (0:42)

It's amazing to have you here. Tell us a little bit about your background.

Jakob Foerster (0:45)

I run the first lab for AI research at the University of Oxford, which is nowadays about 30 people working on anything that is cutting-edge, interesting, not supervised learning, thinking beyond current state of the art. I do this 50% of my time and the other half I spend at the fundamental research group at Meta AI.

Tim (1:05)

Oh, amazing. Well, I watched your talk at ICML earlier and it was really good because you were sketching out a potential ImageNet moment, you know, like an Alex Kruchewski moment for reinforcement learning. What was the elevator pitch?

Jakob Foerster (1:17)

So I think reinforcement learning in the last decade or so really hasn't lived up to its potential, which raises a question. Why is it that deep learning has been this revolutionary success story, while deep reinforcement learning, which held that great promise, really hasn't quite delivered in terms of real world impact? And we have a hypothesis at Flair, which is that deep reinforcement learning had lost the hardware lottery, because deep learning is perfectly suited for the GPU. We can really keep all the course busy, churning through data efficiently. In contrast, the way that deep reinforcement learning has been done in the field is running reinforcement learning environments on the CPU, but running the agents on the GPU.

And this has meant all sorts of complications and complexity, both in algorithm design and in terms of what hardware we need and how we are going to develop algorithms for this, which has really slowed down the field. And this meant experimentation very slow and difficult. And we are currently in this revolution really, which is for the first time allowing us to run environments and agents together jointly on the GPU, making deep reinforcement learning finally a winner in the HAT lottery. And then hopefully, this will be the step to really make reinforcement learning work in the real world.

Tim (2:46)

And I suppose it's not just the hardware lottery, it's also the bitter lesson, which is this idea that when we scale up compute, we get dramatic performance improvements. But the thing holding back deep reinforcement learning to date has been this bottleneck that so much of it is running on the CPU. It does remind me, though, of, you know, there was that article by Alex Irpan in about 2018, and he was saying deep learning just doesn't work. Reinforcement learning doesn't work yet. And he was also pointing out some of the other problems, which is that, you know, if you change anything, or you do anything wrong, the whole thing's broken. So why is reinforcement learning so sensitive to the architecture and the parameters?

Jakob Foerster (3:21)

I think what has happened is because experimentation has been so slow, we've only been able to train our algorithms on very specific environments, and we've only been able to hill climb on that very small set of environments, which means the things are fundamentally brittle. And now for the first time, the experimental loop has been sped up by orders of magnitudes, which means that now we can really start to develop methods that are robust. Remember, the scientists, we ourselves, are doing meta-learning. We're trying to discover methods that can generalise.

Tim (4:00)

MLST is sponsored by CentML, which is the compute platform specifically optimised for AI workloads. They support all of the latest open-source language models out of the box, like Llama, for example. You can pay on consumption essentially, or you can have a model which is always working, or it can be freeze-dried when you're not using it. All of the models that they deploy support the OpenAI API specification out of the box, which means it's just a one-line change in your application to switch over to CentML and start saving money and make your application go faster.