Better Data is All You Need — Ari Morcos, Datology

**Alessio** (0:04)
Hey, everyone. Welcome to the Latent Space Podcast. This is Alessio, partner and CTO at Decibel, and I'm joined by Swix, founder of Small AI.

**Swix** (0:11)
Hello, hello. And we're so excited to be in the studio with Ari Morcos, CEO, co-founder of Datology.

**Alessio** (0:16)
Welcome.

**Ari Morcos** (0:17)
Thank you so much for having me.

**Swix** (0:18)
Ari, so you first came across my radar. I mean, I guess Datology is like a relatively, I guess, exciting, well-hyped startup, at least with the fundraising and the higher profile of the people that you hire. I reached out to book this interview after you worked on the RC. I don't even know how to pronounce it. R-K?

**Ari Morcos** (0:35)
RC, yeah. RC. It's inspired by a real transformer that was called RC.

**Swix** (0:40)
Yeah, the RC Foundation models. You guys have been doing a lot of data work. How would you describe Datology today?

**Ari Morcos** (0:46)
Yeah, so our mission at Datology is to take everything around the data side of machine learning. So going from you have a bunch of data sitting in storage to you're going to feed it into a model via a data loader. There are a ton of choices you would make in that process, ranging from how you're going to filter the data, how you're going to sequence the data, what synthetic data you're going to generate, if any, how you're going to batch the data, all of those things. Those will have a tremendous impact on the performance of the model that you train on the data. One of my favorite catchphrases is models are what they eat. If you show them great data, they're going to be really high quality. If you show them low quality data, they're going to be low quality. But this is a frontier research problem. How do you actually do this effectively? How do you do this automatically at scale? It has to be automatic to be able to process trillions of tokens, billions of images, things like that. That's our mission at Datology is to take that whole process, make it really easy so that anybody can get access to state-of-the-art data curation without needing to be an expert themselves. In doing so, help the folks we work with to train models much faster to much better performance and to also help them train much smaller models to the same or better performance, which I actually think is some of the most exciting stuff going forward. But fundamentally, that's what we do at Datology, is help people curate their data so they can train models faster, better, smaller.

**Swix** (1:59)
So the key words for that, data curation is a service, data efficiency, all those terms. In the pre-chat before we started recording, you mentioned that there's a cool story around how you got into data in the first place, right? You were at GDM, you were at Meta as a research scientist. Describe how that became an interest.

**Ari Morcos** (2:15)
My PhD is actually in neuroscience. So I come much more from an empirical science background. I actually spent time trying to teach mice how to count, and then analyze the activity of thousands of neurons in the brain while mice did count, and try to understand how did that actually happen? What were the neural dynamics that enabled that? That's actually initially how I got into machine learning, was as a means to analyze my neural data sets. I also started my PhD 2011 So AlexNet came right after that, Tarii Dekuyan right after that. Lots of evidence that AI was going to be very, very exciting, which led to me transitioning. But as a result, because I had this somewhat different background of being trained as an empirical scientist rather than as a computer scientist, my real first mission when I joined AI was to try to build more of a science of deep learning. Something that I think is still true today in many cases is that deep learning is an empirical science, but most people that have computer science backgrounds were trained more in the context of a branch of theory. Everything was very provable. That was the initial pushback to deep learning, actually was that you couldn't prove anything in it. But deep learning is at its core an empirical science. We have to run large experiments. We understand the rules for how we design these systems, but the properties that come out of them when we actually train them on a ton of data are emergent and unexpected. So I always really wanted to write these papers where they had two halves, where the first half of the paper was trying to understand why is this representation desirable or undesirable? Why does the model good or bad? And then understand that and then use that understanding to then improve the model. And that was always my goal. That was kind of the perfect paper. Rather than just throwing spaghetti against the wall and seeing what stuck, we were able to really understand why something didn't work and then use that understanding to improve it. Unfortunately, it turns out that it's not so difficult to do the first half of that, try to understand the system, but really, really difficult to actually use that understanding to improve the system. A lot of times what would happen is you go to optimize for this variable, you find, hey, here's this property of representations that makes models good. You go and you optimize for that and then it turns out that wasn't a causal variable. That was a correlate and it doesn't actually work. So I maybe wrote 30 papers where we did that first half and maybe only three or four where we did that second half. That was always frustrating and dissatisfying to me. Then around 2020, I had several papers that all slapped me in the face at the same time with the same insight which is that all that really matters is the data. I had come into all three of these papers very much focused on inductive biases. How do we put better inductive biases into models, either through changing the objective or through changing the architecture, which is where most of the field was and still where you see a lot of the papers at the big conferences are about architectures and various tweaks to architectures. But I had these multiple papers, all of which made this clear takeaway that the data is the only thing that matter. I'll give you one example. There's a paper we had called CONVIT, where the idea was to take a vision transformer and initialize it as if it was a convolutional neural network.