Chelsea Finn: Building Robots That Can Do Anything

**Chelsea Finn** (0:00)
Hi, everyone. I'm really excited to talk about developing general purpose robots and how we might actually truly develop and bring intelligence into the physical world. So to start off, I'd like to talk about this problem, which is that if you want to truly solve a robotics application, you essentially need to build an entire company around that application. You need to build a different company for logistics, for wet lab automation, for robots and kitchens, for surgical robots and so on. And this is really, really hard to do because that company needs to make new hardware, develop custom software, design unique movement primitives for that application, handle edge cases and so on. You have to do all of that from scratch if you want to solve a robotics problem. And as a result, a lot of robotics companies haven't been very successful in actually bringing robots into the physical world successfully in our daily lives. I co-founded a company called Physical Intelligence that's trying to solve this problem. And in particular, we're trying to develop a general purpose model that can enable any robot to do any task in any environment. And we think that this sort of generalist model may work better and be easier to use than purpose-built models, just like we've seen in the development of foundation models for language and other applications. For example, if you want to build a coding assistant, you don't nowadays develop something specifically for coding, but you develop and you build on models that were trained on large amounts of data, not just on code. And essentially, this is the problem of trying to develop these sorts of foundation models and bring this sort of intelligence into the physical world, rather than the digital world where they largely are today. So how do we do this? In this talk, I'd like to talk about how we go about doing this. And if we were to take a lesson from language models, we know that language models have taught us the importance of scale. And so one possible conclusion would be that perhaps scale is the most important ingredient for developing these models. And if you were to say this conclusion is true, then you might look to certain data sources for large scale data. So for example, we might look at data from industrial automation. And you get tons and tons of data of robots doing tasks over and over again like this. But this sort of data isn't going to allow robots to go into disaster zones or to make a sandwich or to bag groceries. And so this massive scale doesn't have the diversity of behaviors that we need in order to solve this general problem. Alternatively, maybe we look at data from YouTube, which has also a massive data source and many videos of humans doing tasks that can be useful for training robots. But at the same time, we don't learn how to write by watching other people write, and we don't become expert tennis players by watching Wimbledon. And so even though there's a massive scale of data here, it's very challenging to use, and there's also a gap between the embodiment of robots and humans. And lastly, we might look at data from simulation. You can also get a massive scale of data here, but this data lacks realism and also has a gap from reality. And so I think the lesson here is that scale is necessary for developing these models that can generalize in open-world conditions, but they're subordinate to actually solving the problem. So you need scale, but it's not sufficient for the entire problem. And so at Physical Intelligence, this is an example of a data episode that we've collected. This is in honor of our first anniversary, which was a few months ago, where here you can see a teleoperator in person who's operating some leader arms to control the robot, to light a match and light a candle with the match. And with this sort of data, we can train robots to do a variety of different tasks. And so what I'd like to talk about is some of our recent results at trying to develop sort of Physical Intelligence with large scale real robot data. I should mention this is large scale by today's robot standards, and arguably a minuscule amount of data compared to the sorts of robot data that we should have in the years to come. And so in particular, we'll be looking at whether robots can do a variety of dexterous long-horizon tasks, whether robots can succeed in places they've never been, whether robots can respond to open-ended prompts and interjections. And even if you're not excited about robotics, I think that the lessons that we've learned from trying to address these problems are applicable outside of the physical world. So can we develop robots that can have complete dexterous long-horizon tasks? And in particular, in this first part, I'd like to talk about how we trained a Pi Zero Foundation model to do this task, which is to unload a dryer and fold laundry. And to date, I think this is the most impressive thing that I've seen a robot do in the physical world. It's really hard.