2024 in Synthetic Data and Smol Models [LS Live @ NeurIPS]

**Charlie** (0:00)
We're back at Latent Space Live, our first mini conference held at NeurIPS 2024 in Vancouver. This is Charlie, your AI co-host. As a special treat this week, we're recapping the best of 2024 going domain by domain. We sent out a survey to the over 900 of you who told us what you wanted, and then invited the best speakers in the Latent Space Network to cover each field. Two hundred of you joined us in person throughout the day, with over 2,200 watching live online. Our next keynote covers the state of synthetic data and Smol Models with Loubna Ben-Alao of HuggingFace. We last commented on the synthetic data trend at last year's NeurIPS pod, and my goodness did it explode this year across pre-training, post-training and evils. We are very honoured to have Loubna, who not only worked on Cosmopedia, HuggingFace's open reproduction of Microsoft's Phi 1.5 synthetic textbook quality data set, and FineWeb, HuggingFace's new 15 trillion token common crawl subset, but she also leads SmolLM, HuggingFace's implementation of Meta's mobile LLM paper, which made waves for its shared matrices and shared weights architecture. There's been lots of movement this year on small models, from Apple Foundation models rolling out to every iPhone and Macbook on Earth, to Google introducing Gemini Nano in the Chrome browser and Microsoft embedding RWKV into Windows. As always, don't forget to check our show notes for all the selected best papers of 2024, and for the YouTube link to their talk. Watch out and take care.

**Loubna Ben-Alao** (1:53)
I'm very happy to be here, thank you for the invitation. So I'm gonna be talking about Synthetic Data in 2024, and then I'm gonna be talking about Smol on-device models. So I think the most interesting thing about Synthetic Data this year is that, now we have it everywhere in the large language models pipeline. I think initially, Synthetic Data was mainly used just for post-training, because naturally that's the part where we needed human annotators to show the models how they should answer instructions, how should they be, they should be helpful and not toxic. And when we had Lmsys were really performant, we replaced the not human annotators just with Synthetic Data. And then after that, we realized that we don't really have good benchmarks to measure if models follow instructions well, if they are creative enough or if they are chatty enough. So we also started using Lmsys as judges. And I think this year and towards the end of last year, we also went to the pre-training parts and we started generating synthetic data for pre-training to kind of replace some parts of the web. And the motivation behind that is that you have a lot of control over synthetic data. You can control your prompt and basically also the kind of data that you generate. So instead of just trying to filter the web, you could try to get the LLM to generate what you think the best web pages could look like and then train your models on that. So this is how we went from not having synthetic data at all in the LLM pipeline to having it everywhere.
And so the cool thing is like today, you can train an LLM with like an entirely synthetic pipeline. For example, you can use our Cosmopedia data sets and you can train 1B model on like 150 billion tokens. Those are 100% synthetic and those are also of good quality. And then you can instruction tune the model on a synthetic SFT data set. You can also do DPO on a synthetic data set. And then to evaluate if the model is good, you can use a benchmark that uses LLMs as a judge, for example, MTBench or AlpacaEvil. So I think this is like really my blow-in because like just a few years ago, we wouldn't think this is possible. And I think there's a lot of concerns about model collapse and I'm going to talk about that later, but we will see that like if we use synthetic data properly and recreate it carefully, that shouldn't happen. And the reason synthetic data is very popular right now is that we have really strong models, both open and closed. It is really cheap and fast to use compared to human annotations, which cost a lot and take a lot of time. And also for open models, right now, we have some really good inference frameworks. So if you have enough GPUs, it's really easy to spawn these GPUs and generate like a lot of synthetic data. Some examples are VLM, TGI and TensorRT. Now let's talk about the elephant in the room. Model collapse. Is this the end? If you look at the media and all of like, for example, some papers in Nature, it's really scary because there's a lot of synthetic data out there in the web. And naturally we train on the web. So we're going to be training a lot of synthetic data. And if model collapse is going to happen, we should really try to date that seriously. And the other issue is that, as I said, a lot of people think the web is polluted because there's a lot of synthetic data. And for example, when we're building FineWeb datasets, here at Guillerm and Hinek, we're interested in like, how much synthetic data is there in the web? So there isn't really a method to properly measure the amount of synthetic data or to save web page synthetic or not. But one thing we can do is to try to look for like, proxy words. For example, expressions like as a large language model or words like Delve that we know are actually generated by ChatGPT, we could try to measure the amount of these words in our datasets and compare them to the previous years. For example, here we measured like these words ratio in different dumps of Common Crawl. And we can see that like the ratio really increased after ChatGPT's release. So if we were to say that the synthetic data amount didn't change, you would expect this ratio to stay constant, which is not the case. So there is a lot of synthetic data probably on the web. But does this really make models worse? So what we did is we trained different models on these different dumps, and we then computed their performance on popular like NLP benchmarks, and then we computed the aggregated score. And surprisingly, you can see that the latest dumps are actually even better than the dumps that are before. So if there is some synthetic data there, at least it did not make the models worse. Yeah, which is really encouraging. So personally, I wouldn't say the web is polluted with synthetic data. Maybe it's even making it more rich. And the issue with like model collapse is that, for example, those studies, they were done on like a small scale. And you would ask the model to complete, for example, a Wikipedia paragraph, and then you would train it on these new generations, and you would do that iteratively. I think if you do that approach, it's normal to observe this kind of behavior because the quality is going to be worse because the model is already small. And then if you train it just on these generations, you shouldn't expect it to become better. But what we're really doing here is that we take a model that is very large, and we try to distill its knowledge into a model that is smaller. And in this way, you can expect to get like a better performance for your small model. And using synthetic data for pre-training has become really popular after the textbooks are all you need papers, where Microsoft basically trained a series of small models on the textbooks that were using a large LLM. And then they found that these models were actually better than models that are much larger. So this was really interesting. It was like the first of its time, but it was also met with a lot of skepticism, which is a good thing in research. It pushes you to question things. Because the dataset that they trained on was not public. So people were not really sure if these models are really good, or maybe there's just some data contamination. So it was really hard to check if you just have the weights of the models. And at HuggingFace, because we like open source, we tried to reproduce what they did. So this is our Cosmopedia dataset. We basically tried to follow a similar approach to what they documented in the paper. And we created a synthetic dataset of textbooks and blog posts and stories that had almost 30 billion tokens. And we tried to train some models on that. And we found that the key ingredient to getting a good dataset that is synthetic is trying as much as possible to keep it diverse. Because if you just throw the same prompts as your model, like generate a textbook about linear algebra and even if you change the temperature, the textbooks are going to look alike. So there's no way you could scale to millions of samples. And the way you do that is by creating prompts that have some seeds that make them diverse. In our case, the prompt, we would ask the model to generate a textbook, but make it related to an extract from a web page. And also, we try to frame it to stay within topic. For example, here, we put an extract about cardiovascular bioimaging, and then we ask the model to generate a textbook related to medicine that is also related to this web page. And this is a really nice approach because there's so many web pages out there, so you can be sure that your generation is not going to be diverse when you change the seed example. One thing that's challenging with this is that you want the seed samples to be related to your topics, so we use a search tool to try to go all of FineWeb data sets and find the pages that are related to the topics we're interested in. And then we also do a lot of experiments with the type of generations we want the model to generate. For example, we ask it for textbooks for middle school students or textbook for college, and we found that some generation styles help on some specific benchmarks, while others help on other benchmarks. For example, college textbooks are really good for MMLU, while middle school textbooks are good for benchmarks like OpenBook QA and Pico. This is like a sample from our search tool. For example, you have a top category, which is a topic, and then you have some subtopics, and then you have the topic hits, which are basically the web pages in FineWeb that belong to these topics. And here you can see the comparison between Cosmopedia. We have two versions, V1 and V2 in blue and red, and you can see the comparison to FineWeb. And as you can see, throughout the training, training on Cosmopedia was consistently better. So we managed to get a dataset that was actually good to train these models on. It's of course so much smaller than FineWeb, it's only 30 billion tokens, but that's the scale that Microsoft datasets was, so we kind of managed to reproduce a bit what they did. And the dataset is public, so everyone can go there and check if everything is all right. And this recent paper from NVIDIA, Nemotron CC, they took things a bit further and they generated not a few billion tokens, but 1.9 trillion tokens, which is huge. And we can see later how they did that. It's more of like rephrasing the web. So we can see today that there's like some really huge synthetic datasets out there and they're public, so like you can try to filter them even further if you want to get like more high quality corpses. So for this rephrasing the web, this approach was suggested in this paper by Pratyush, where basically in this paper, they take some samples from C4 datasets and then they use an LLM to rewrite these samples into a better format. For example, they ask an LLM to rewrite the sample into a Wikipedia passage or into a Q&A page. And the interesting thing in this approach is that you can use a model that is small because it doesn't, rewriting doesn't require knowledge. It's just rewriting a page into a different style. So the model doesn't need to have like knowledge that is like extensive of what is rewriting compared to just asking a model to generate a new textbook and not giving it like ground truth. So here they rewrite some samples from C4 into Q&A, into Wikipedia, and they find that doing this works better than training just on C4. And so what they did in Nemotron CC is a similar approach. They rewrite some pages from Common Crawl for two reasons. One is to improve pages that are low quality. So they rewrite them into, for example, a Wikipedia page so they look better. And another reason is to create more diverse datasets. So they have a dataset that they already heavily filtered, and then they take these pages that are already high quality and they ask the model to rewrite them in Q&A format into open-ended questions or multi-choice questions. So this way they can reuse the same page multiple times without fearing like having multiple duplicates, because it's the same information, but it's going to be rewritten differently. So I think that's also a really interesting approach for generating synthetic data just by rephrasing the pages that you already have. There's also this approach called PROCs, where they try to start from a web page and then they generate a program which finds how to write that page to make it better and less noisy. For example, here you can see there's some leftover metadata in the web page, and you don't necessarily want to keep that for training your model, so they train a model that can generate programs that can normalize and remove lines that are extra. I think this approach is also interesting, but it's maybe less scalable than the approaches that I presented before. That was it for rephrasing and generating new textbooks. Another approach that I think is really good and becoming really popular for using synthetic data for pre-training is basically building better classifiers for filtering the web. For example, here we released the data set called FineWeb-Edu. And the way we built it is by taking Llama 3 and asking it to rate the educational content of web pages from zero to five. So for example, if a page is like a really good textbook that could be useful in a school setting, it would get a really high score. And if a page is just like an advertisement or promotional material, it would get a lower score. And then after that, we take these synthetic annotations and we train a classifier on them. It's a classifier like a BERT model. And then we run this classifier on all of FineWeb, which is a 15 trillion tokens data set. And then we only keep the pages that have a score that's higher than three. So for example, in our case, we went from 15 trillion tokens to just 1.5 trillion tokens that are really highly educational. And as you can see here, FineWeb-Edu outperforms all the other public web data sets by a larger margin on a couple of benchmarks. Here I show the aggregated score. And you can see that this approach is really effective for filtering web data sets to get like better corpses for training your LLMs.