Training the AIs' Eyes: How Roboflow is Making the Real World Programmable, with CEO Joseph Nelson

**Nathan Labenz** (0:00)
Hello, and welcome back to The Cognitive Revolution. Today, my guest is Joseph Nelson, CEO of Roboflow, a computer vision platform that supports more than one million engineers and more than half of the Fortune 100 as they seek to turn proprietary image and video data into a competitive advantage. We begin with an overview of computer vision capabilities today. Joseph notes that while language is fundamentally a human construct and inherently optimized to be understood, the real world contains a fat tail of chaotic scenes which are not at all optimized for understanding. And thus, just as the vision transformer came about three years after the original transformer, computer vision today is roughly where language capabilities were three years ago with the introduction of Chatch GPT and GPT-4. Which is to say that while frontier models can do amazing things, and most problems can be solved if you're willing to put in the work to fine-tune and pay any inference cost, we have a long ways to go before foundation models will really be able to do it all. To make this concrete, Roboflow maintains a site called visioncheckup.com, which highlights the spatial reasoning, precision measurement, and grounding failures that still plague even the best multimodal models today. And importantly, even when frontier models can solve a particular task, you can't wait 40 seconds for a reply when you're powering instant replay at Wimbledon or monitoring for defects on a high-throughput manufacturing line. And so there's often still a lot of work left to do to get vision models running efficiently enough to meet production latency and edge deployment requirements. This is where Roboflow comes in, and I was super interested to hear Joseph describe what it looks like to go from an open-source vision model to deploying your own task-specific model today. He emphasizes first the importance of establishing clear requirements upfront, because the performance thresholds the different customers need to hit on their respective use cases can vary really quite dramatically. From there, the process often involves distilling frontier model capabilities into much smaller models, like Roboflow's own RF Deader model, which they derived from Meta's Dino V2 backbone using a really interesting training technique called Neural Architecture Search, which in turn uses a weight-sharing technique to train thousands of network configurations at once, all within a single training run. This process ultimately produces a set of models of varying sizes that collectively map out a performance Pareto frontier. And today, Roboflow has productized this approach so that anyone can now run it on their own data set and come out the other end with an end-of-one model that is optimized specifically for their problem. From there, we cover a number of additional topics as well. Joseph explains that Chinese companies have consistently led in computer vision, and how much the American open-source ecosystem currently depends on Meta, but also why he's optimistic that Nvidia will fill the gap if Meta's new AI leadership changes priorities. He also describes how coding agents are expanding the market for Roboflow's tools, how skills are emerging as a new go-to-market vector, and how Roboflow plans to use a first-party agent to guide users through the process of building computer vision pipelines. We also discuss the state of AI's aesthetic taste and why the inherent subjectivity of aesthetic preferences makes this such a hard problem. We hear about the emerging S-curves that Joseph is watching, including world models, vision language action models being developed for robotics, inference time scaling for vision, and wearables which are now selling millions of units per year. We get his vision for how computer vision contributes to a good life as AI matures, which includes everything from precision agriculture and food safety, to self-driving commutes and real-time sports analytics.
And finally, he explains why he worries that overly opinionated regulation could accidentally siphil all sorts of surprising but valuable use cases, and why he recommends that policymakers focus on outcomes instead of trying to regulate the tools that people are using. When it comes to computer vision, Joseph has quite literally seen it all. So whether you're looking to catch up on the field like I was, or looking for a practical framework with which to approach a specific challenge, I think you will find a lot of value in, and I hope you enjoy, my conversation with Joseph Nelson, CEO of Roboflow. Joseph Nelson, CEO at Roboflow. Welcome to The Cognitive Revolution. I'm excited for this. So regular listeners will know, I really got into AI in a full-time obsessive way, in my role as founder of Waymark. And it was such an exciting time four years ago, when things were just starting to work. I ended up going really deep on what was available in computer vision at the time with clip and blip and blip2 and clip embeddings and trying to figure out how the problem that we had at the time was, we have all these small business users. We had developed a pretty good technique for scraping their websites and their kind of online presence and creating an image library for them. But then what to do with that image library, right? It was just initially just a total jumble of photos. We couldn't make any sense of it. We made very blind guesses as to what we would actually put into content for them. And that obviously had a long way to go before it really started to work. So I had a ton of fun in like 2022 into 2023 time frame, getting deep into the weeds on that stuff. And obviously a lot has happened since, so I'm really excited to catch up on a few years of computer vision progress in 90 minutes or so. Maybe let's start by just kind of setting the stage. Like, where are we today in terms of computer vision? You can come at that from a lot of different angles. Maybe start with like use cases. You know, what are the use cases out there that are really well established, they're driving the most volume, that are driving the most value. Give us kind of a survey lay of the land.