Alex Ratner, Snorkel AI CEO on Data-Centric AI and AI Development

Grace Gong (0:01)

We're on. Hi, Alex. Welcome to Venture with Grace.

Alex Ratner (0:05)

Thanks for having me on. I'm excited.

Grace Gong (0:07)

I'm so excited since I was just telling you about my researcher night last night. So I have all the benchmarks ready, so we'll yap up everything. Before we start, I want to say a quick shout out to our amazing sponsor. This episode is brought to you by Nebius, the ultimate club for AI innovators. Nebius provides AI infrastructure. You can count on combining reliability, as feed with flexibility and engineering support, unmatched by hyperscalers. AI leaders like Meta, Shopify, and Higgsfield are dependent with Nebius to run the AI workloads. Plus, Venture back startup can save up to 150K on compute custom and apply for access. Visit nebius.com or nebius.com/startup to learn more.

So I want to start with your journey.

Should we start with the journey or different benchmarks? Maybe the journey.

Alex Ratner (0:51)

Benchmarks is a juicy topic that could think of the entire session. I think I'm so happy to start with the journey a little bit, although I'm eager to get into the benchmark discussion as well.

Grace Gong (1:01)

Amazing. So you are at Stanford and you and your professor, Christopher Ré, invented a new way to basically attack different data. Maybe we could start from what were the original research and how did it evolve into a Snorkel?

Alex Ratner (1:16)

Yeah. Well, taking a step back, when we started this is maybe 12, 13 years ago when I was first working in the lab and eventually then the core Snorkel team that started in Chris Ray's lab at Stanford, we were working on pretty much what all AI people worked on, which was the models and the algorithms. We were trying to get AI to work for more specialized settings. We were working with folks in the biology department.

Hope everything is okay.

Grace Gong (1:51)

My chair's arm fall off this morning.

Alex Ratner (1:55)

Oh, no.

Grace Gong (1:56)

I'm sorry. I put it back and it just fall again. I'm sorry. I apologize for the mess.

Alex Ratner (2:02)

No. You might hear similar clatters on mindset. I usually like to keep my desk very clean, but my kids just pile up all these random offerings, and so that every once in a while, something crashes off when I'm noticing. That's my excuse if you hear a similar clatter. But yeah, so we were talking to all of these experts that are trying to apply AI or what we called machine learning back in those ancient days to biology, healthcare, with Stanford Healthcare and the VA Healthcare system. We're working with journalists and a lot of the big tech companies that funded the lab, like Google and others.

We were trying to push new fancy algorithms for joint inference and X, Y, Z, and you can get extra five points of accuracy and sharper theoretical sample complexity, convergence bounds, and blah, blah, blah. And they kept saying, okay, that all sounds great, but can you help me with all this label that I need to get these models to work?

And the first couple of times to tell a story, honestly, we just kind of ignored that.

We kind of took the stance that most of the AI world, certainly AI academic world took back then and still sometimes takes today, which is data, data labeling and generation creation. That's not our problem.

That's janitorial work. That's something that you use a crowdsourcing or staffing for or something. It's boring. And the more we kept hearing this refrain, and also the more we looked and saw this at the time it was deep learning, some of the first NVIDIA GPUs were starting to hum down the hallway.

And then many LLMs were also soon then after starting to become a thing.

We saw that not only was data and data labeling and preparation and generation a problem that was bottlenecking real-world application, especially in complex specialized settings.

But it was going to become arguably the main way that you kind of programmed AI as the models and algorithms got kind of more standardized and black box. So we came up with this term data centric AI and we decided that actually data was going to be the center of AI development if these trends continued. Or it was an academic project. So we thought maybe that would happen. It seemed like an interesting counterculture thesis. And the time, like what happens in a lot of the market today still, again, data generation labeling now what we call environment development for reinforcement learning was really just being approached as a staffing and logistics problem. And we thought if it was really going to become one of the key ways that you actually measured and train and develop AI, as again, the models and algorithms became more black box and less admissible to direct development and intervention, then you had to kind of treat it like a technology problem. And so, you know, our research agenda really started saying, okay, how do we label and then generate data and environments for these complex expert settings using a synthesis of human experts? We can talk about this, but, you know, human experts are going to be at the center of the loop for valuable data. And in our strong opinion, it's kind of simple logic, the data that's valuable for the frontier is the data that models don't already know. And that has to come from somewhere. But we thought as data got more complex, it wouldn't be scalable or even feasible someday for humans alone, just to kind of develop all this really complex data at scale. You needed to accelerate them with AI. And so that's kind of where the research started of how do you, you know, automate some of the data development. And if you want to get a little bit more into what we do today that we're really excited about using today's parlance, although it's just a kind of rebrand of what we've been working on for years, how do you create a recursive self-improvement loop for data where specialized AI models help humans create and curate and build data sets and environments, but then their feedback further specializes and improves those agents. That's what we build in our platform today. So we started off on that research at Stanford. I started a lab at the University of Washington. I'm not active now, but was there for a bit. And then eventually it became clear that this data problem indeed was becoming a major bottleneck and a major part of how AI was developed. We spun out the company and then, yeah, it's been about six plus years. We started off just licensing the technology platform because if you got kind of the incumbent approach that is still a big part of the ecosystem today, which is kind of staffing, crowdsourcing, how do you get as many human hours as possible? And then we were taking the kind of, we were purely just building a technology platform to automate first data labeling, and generation, and curation, and synthesis of environments, et cetera, with humans, but to accelerate them for really complex, expert settings. And then about a year and a half ago, we launched more of a verticalized offering on our platform called Data as a Service, where we could kind of match what the frontier labs generally want, which is not a self-serve SAS license, but rather just lots of data, lots of RL environments yesterday. And then that's been a big uptick in growth and lots of excitement. The same core idea, same core technology platform, same core IP of merge humans with AI in kind of compounding ways to match the rising complexity. Basically, as AI gets smarter and becomes superhuman in certain areas, and we can talk about where and how we measure that if we get into benchmarking, data to train and teach it becomes more and more sophisticated and harder for humans alone to do. We call this shift data 1 to 2.0. Data 1 is when a model doesn't know much, and any new bit of information is additive, so you just want to throw as many data points, as many human hours at it. Data 2 is when you're trying to teach like an advanced student. You need to craft a curriculum. You need to come up with really advanced, high-quality complex data points targeting exactly where they don't know something. That's more of a research and technology problem, and that's where we're playing and winning today.