Why Scaling Prediction Cannot Create Intelligence - Alexander Mattick

Alexander Mattick (0:00)

Effectively, information can be very expensive. And in fact, I would say most information is very, very expensive. If you completely say, well, constraints don't matter, I don't need any prior information, I don't want to encode anything in there, we still have the problem of now needing to actually relearn everything from scratch. He believes that you can figure out a suitable reward function for every task. That is enough. I think that energy-based models are not really useful. At least not useful anymore. I think at world models and so both JEPA and World Models are ultimately branding.

There were JEPA like models before JEPA, and there were world models before world models. Right now, all of these methods are not set up like this. I think they are set up to look very nice in demos, and to actually do some really impressive stuff.

How many times did it fail to do something?

SPEAKER_2 (0:51)

Amazing. Well, Alexander Mattic, it's amazing to have you back on MLST. It's been many years since you were on the show, and we were so grateful for you when you last came on the show because you helped us do some research for our interview with Yann LeCun and Randall Balestriero about the Spline Theory of Neural Networks, and we did that intro all about interpolation and extrapolation. But just for folks at home, so your nickname is Zickzack on Yannic Kilcher's Discord server, and this gentleman is by far the smartest person on Yannic's Discord server, and that is saying something, believe you me. So, Alexander, it's wonderful to have you on the show. Can you give us a bit of a background? This episode is supported by Cyber Fund.

If you're building at the frontier of AI, they want to hear from you. Cyber Fund believes the future belongs to AI natives who wants to achieve the impossible, and that is why they're introducing the monastery for AI native founders. It's an environment of pure focus and rapid execution for founders operating at AI native speed, and they're offering teams $2 million each to participate. Apply now at cyber.fund.

Alexander Mattick (2:08)

Yeah. My name is Alexander Mattick. I first did my bachelor's at FAU in Erlangen in Germany. Then I also did my master's there. My bachelor's was in computer science. My master's is in artificial intelligence.

And then I went over to Fraunhofer, which is kind of one of the biggest applied research institutes.

And now I'm also dual affiliated with the Technical University of Nuremberg. It was kind of still under construction, kind of in the very much in the build up of becoming a real thing. So it's really exciting to actually be one of the first people who actually does its PhD there.

SPEAKER_2 (2:50)

Well, folks, today we're going to get into savant levels of technical detail. We're going to talk about inference. So things like energy based models and diffusion of flow matching. We're going to talk about theories of deep learning, constrained reinforcement learning and world models and maybe some other stuff as well. So why don't we start with inference?

So can you just sketch out the history of this? So obviously back in the olden days, we might have parameterized a basic normal function. And then we had things like energy based models and flow matching and normalized flows, all this kind of stuff. Sketch out the whole history.

Alexander Mattick (3:24)

In the very beginning, so one of the first ideas is of course, if we have these like nice parameterized densities, well we actually know regardless of any parameter we put in, we will end up with a normalized density. So imagine again, the normal function, but there are plenty of others like a categorical distribution, it's the same idea. But the core idea was, well, the difficult thing when we try to do inference is that we always have this, like the density has to be normalized at some point. The first idea was of course to say, and you go and say, well, let's just pick a group of functions which is always going to be normalized. This gives us the very traditional variation inference problem. Effectively, we have a normal distribution with unknown parameters, and now we try to fit them onto data.

The issue with this is, and people notice this very quickly, is that now you are very constrained by the actual density. Let's say your data is not normally distributed. Let's say it's bimodal. Now, your actual density no longer fits. Basically, what would happen is if you imagine you have a mode here, and you have a mode here, then just fitting a normal would put all the mass basically in the middle and try to split the difference.