Why Models Are AI’s Next Training Dataset with Damian Borth

**Sam Charrington** (0:00)
One of the biggest questions facing AI today is how foundation models keep improving as high-quality training data becomes harder to find. Some researchers are betting on synthetic data, others on inference time reasoning. Today's guest has his chips on something very different.
Every trained model represents thousands or even millions of GPU hours spent discovering what works. Instead of treating those weights just as the end of the training process, what if they're also the beginning of the next one?
Damian Borth, Professor of AI and Machine Learning at the University of St. Gallen, sees trained models themselves as data. Data that can be learned from, analyzed, and even used to generate entirely new models. When I asked him to explain the idea behind weights-based learning, here's where he started.

**Damian Borth** (0:47)
So we basically thought about this very simple idea.
What happens actually if we take the weights of trained neural networks as the input to train a neural network to understand these weights that we have out there, much, much better. Thinking about that, that you can treat the weights as an input modality, gives you suddenly this opportunity of, can we be much, much faster in creating new weights for particular tasks? Or can we be much more precise in analyzing weights when somebody gives me a neural network that I'm not knowledgeable about and I never saw before?

**Sam Charrington** (1:23)
I'm Sam Charrington, and this is The TWIML AI Podcast. For over a decade, I've been exploring the ideas and innovations shaping the future of AI through conversations like this one that help you understand what's real, what's next, and what matters. Let's jump in.

**Damian Borth** (1:46)
We started, like in 2000, 2021, the work on what we call weight space learning. And weight space learning is a quite interesting way at looking at machine learning in general. That's currently the major topic. We also do a little bit of work in remote sensing and then representation learning on tabular data. We're now walking and combining everything together to focus more on weight space learning, which I think is a really interesting way forward to solve a couple of problems that the community currently encounters.
And started from a very esoteric idea to something that works surprisingly well.

**Sam Charrington** (2:30)
You know, we think about weights as the product of training a model. And, you know, we get some utility out of them. We may use them for things like explainability or manipulate them when we're quantizing or something like that. But the idea seems to be that, you know, there's so much more that we can learn from these weights.

**Damian Borth** (2:51)
Exactly, so if you think about machine learning, machine learning has this idea of you have data and some output in classic supervised machine learning data and some predictions. And you train neural network in between to mimic the data set, mimic the distribution of the data set. And the outcome during this very expensive training procedure is a set of weights, a configuration of parameters that define the neural network, like the DNA of the neural network.
This is classical machine learning, supervised, unsupervised, self-supervised, that fuels a lot of innovation over the last 10 years and with GENAI moved to the next stage. If you look at what happened over the last couple of years, more and more of those models have been published. Publicly, are online accessible at repositories like Huggingface or GitHub. So we basically thought about this very simple idea. What happens actually if we take the weights of trained neural networks as the input to train a neural network to understand these weights that we have out there much, much better. So to take another analogy, language models, you take a big model, you train this on every single centers on the internet. At the end, you have a language model able to analyze language and to generate language.
You can take the same idea for pixels. You take a big model, you train on all the pixels on the internet and you can analyze pixels and you can generate pixels. We do the same idea on all the weights of trained neural networks, so we can analyze weights of neural networks and we can generate weights of neural networks.
As straightforward as it is, obviously, there's a little more into the details. But thinking about that, that you can treat the weights as an input modality, gives you suddenly this opportunity of thinking about, okay, what would be language translation in neural network models? What would be generation of words and tokens that are worse and generation of tokens that are weights? And can we be much, much faster in creating new weights for particular tasks? Or can we be much more precise in analyzing weights when somebody gives me a new network that I'm not knowledgeable about and I never saw before? And then, you have this new entire world, this empty space of things you can do with weights, that you kind of carry into the community and hope that there's somebody listening and continuing and building up a community which happened over the last two, three years, which is very exciting because there are more people about that. And yeah, weights are exciting, not only as the output of learning, but as the input for learning.