How Deep Learning Finally Cracked Messy Tables - Frank Hutter

SPEAKER_1 (0:00)

There is so much NFL on ESPN right now.

Monday Night Football, plus pregame and postgame. NFL Live, NFL Primetime. Everything you need to stay on top of every game, every week.

Frank Hutter (0:14)

Unbelievable.

SPEAKER_1 (0:16)

And now, NFL Network is on the ESPN app. More football, more coverage, more ways to watch. Football's on, all season long, on ESPN and streaming on the ESPN app.

SPEAKER_3 (0:31)

This episode is brought to you by TicTac. Lace up and get your game on because the TicTac lineup just got a new teammate. Dr. Pepper TicTac. This new double threat puts two of your favorite flavors into one box of mints. And when two iconic flavors come together, the result can only be iconic.

Score big. Try Dr. Pepper TicTac today.

SPEAKER_4 (0:54)

Deep learning did not work for tabular data, and now it works dramatically better than CatBoost and XGBoost.

Frank Hutter (1:01)

TabPFN is really this natural progression of AutoML, where we learn this entire algorithm that is executed in a forward pass, a million rows times a thousand columns. So if you were to throw this into an LLM, you would have a billion elements, and you would need to tokenize each of these numbers, so that would maybe be three or four tokens each. So you have three or four billion tokens in your context, and LLMs wouldn't be very happy there. You're getting some observational data, and you want to predict if I change this variable here, what will happen? So if I give this patient this medicine, what will happen?

SPEAKER_4 (1:38)

Frank, it's amazing to have you here. Welcome to MLST.

Frank Hutter (1:41)

Thank you for having me. It's great to be on the show. Actually two years ago, people asked me, hey, is there a possibility that you might be on MLST at some point? And I was like, maybe at some point. And yeah, super exciting to be here now. Yeah, hey, my name is Frank. I have been a machine learning professor for the last 12 years before starting Prior Labs as a CEO, now co-CEO research. I'm very excited about that. I focused a lot on AutoML, automated machine learning, or yeah, started the workshop series on AutoML, ran that for eight years. We transitioned that to a conference on AutoML, co-wrote the first book on AutoML, so really did a lot in that space, but then also moved over to deep learning, worked a lot on optimization of deep learning, so Adam W and Kosan Liling is from Ilya Loshchulov and myself. That's being used for training any transformer in the world. Then we put the two together and did deep learning for tabular data.

That is something that didn't use to work, and with TabPFN, we actually made it work and are scaling this up, building foundation models and are really excited to revolutionize this world.

SPEAKER_4 (3:00)

I think one thing that people might not appreciate is that tabular data is absolutely everywhere and speaking from bitter experience, it's a nightmare to work with tabular data. Anyone who's built a machine learning model with tabular data, obviously like pandas and scikit-learners got some stuff in there, but you have to deal with missing values. What do you do with the categorical features? How do you do some feature engineering? How do you do transformations that reflect the semantics of the problem? You need to really know what you're doing and it feels kind of kludgy. Even though you can make an ML model, you always feel dirty afterwards because you feel like you've done all of these shortcuts and corrupted it in some way.

Tell me about that.

Frank Hutter (3:42)

Yeah, I fully agree. Tabular data is very dirty and I think that's one of the reasons that deep learning took so long to actually do well for it.

There's been countless attempts for deep learning for tabular data. 2019, TabNet by Google was really hyped. Thousands of citations, yeah, the new thing for tabular data and it just doesn't work. It doesn't generalize to new datasets.

It just, you know, you have heterogeneous data, you have outliers, you have missing values, you have categorical values, you have binary values, you have ordinal values, you have missing data, missing at random, missing not at random, all kinds of these data complexities. And yeah, you need to actually take that into account and just the standard deep learning approach, so to like get treated everything as the same, like any image is the same in terms of, you know, like there's pixels and you typically have very, very similar spatial relationships between the pixels and so on. And you can take images, like you can have an image net that is just all, basically cover all kinds of different images in the world and learn on that one data set. But what you need for tabular data is actually, yeah, a whole lot of different tables, because if you have one table from medicine and then you have another table from insurance, there's just nothing you can learn in terms of the individual rows from one, can't tell you anything about the other one. But what you can do and where you can do the transfer is on the levels of these patterns, detecting the patterns, how the different features interact, etc., detecting causality potentially. And on that level, using in context learning, you can actually transfer across different data sets. And that's why within context learning, we finally saw this breakthrough for Turbulent Data.