AI Fundamentals: Datasets 101

**Alessio** (0:09)
Hey, everyone, welcome to the Latent Space Podcast.
This is Alessio, Partner and CTO of Residence and Decibel Partners. I'm joined by my co-hosts, Wix, Frieder and Edger of Latent Space.

**SPEAKER_2** (0:20)
Today, we finally, finally, we have the second episode, our 101 track. This has been a long time coming. So last time we did Benchmarks 101, I think we got a lot of mileage out of that. We understood a lot about the benchmarks, and we talked with a lot of our guests over the previous episodes about how they evaluate their models.
And today we wanted to dive into datasets, what they are, how they're constructed, and why they matter. I guess I should go into why we wanted to do this episode.
It's a little bit weird to separate datasets and benchmarks. So we did benchmarks first, but a lot of the benchmarks were datasets. So pretty much they're the one and the same thing, right? Like, and I think where they start to diverge is a cause of significant interest, but mostly actually I wanted to focus on datasets for one primary reason, which is that many people say that GPT is trained on all the internet. So first of all, this is actually not true. And second of all, it actually causes some potential misperceptions. I say potential because there is some legitimate debate about this. There are misperceptions about us running out of data, and we can discuss the pros and cons of whether or not we are running on the data. It's been named the Token Crisis by academics and quite a lot of commentators on AI. So in the show notes, we're going to link to a paper on to repeat or not to repeat insights from scaling LLMs under the Token Crisis.
And then I'm also going to link to an opposite view from OpenAI with Ilya Sutskever talking about how they're not anywhere close to running on the data that they want to train on. So whenever there's such an interesting divergence between practitioners and academics, I think it's a worthwhile thing to dive into. And just in general, I think there's a lot of foundational knowledge that people skip over when they assume that everyone knows what datasets we're talking about.

**Alessio** (2:15)
Yeah, I was gonna say, I think also in terms of like the knowledge that the models have, if you say it's been trained on the whole internet, you would assume it knows everything on the internet, but it obviously doesn't.
You can go in there and ask about people that have online presences and are not actually in the knowledge base of the model. And this also helps when thinking about what data to then use to find to them. So if you understand what's in the model, if you're trying to build a verticalized model for a specific use case, you can better figure out what's actually gonna be meaningful versus what was already present in the first training ground.

**SPEAKER_2** (2:49)
Yeah, just for some comparison, let's say the total size of the internet, some people have estimated at around 5 billion gigabytes. Most of the datasets that we are gonna talk about today are in the hundreds of gigabytes range, and it's growing every single day. There's always new data being created every single day, and there's always new modalities to claim that data from. So a lot of the whisper behind, open your eyes, whisper, is that they're actually transcribing YouTube, which is a source of extra tokens, and we'll have to explain what tokens are.
The first thing is about whether or not we're running out of data. The second issue is this divergence between data sets and benchmarks. And I wanted to dive into this specifically because they used to be essentially one and the same thing. In like a very standard machine learning tutorial, you would do something like the iris data sets, and then you would do train test validation splits, and you would basically evaluate data based on samples from the data itself. But more recently, we actually have decoupled benchmarking from the data sets that they're trained on, except for the calculation of loss. I think in our Discord, in Latent Space Discord, we've actually been doing a small paper club where we've gone over some of the foundational papers. And we actually recently went over the BERT paper, which is the bi-directional transformers paper that was a predecessor to T5 and is a predecessor to all the large language models today.
And in BERT, they actually invented this concept of masking, which meant that data sets could create their own training objectives, which I actually think is super interesting. So basically, what you have is, for example, a sentence.