Fully autonomous robots are much closer than you think – Sergey Levine

**Dwarkesh Patel** (0:00)
Today, I'm chatting with Sergey Levine, who is a co-founder of Physical Intelligence, which is a Robotics Foundations Model company, and also a professor at UC Berkeley, and just generally one of the world's leading researchers in robotics, RL, and AI. Sergey, thank you for coming on the podcast.

**Sergey Levine** (0:16)
Thank you, and thank you for the kind introduction.

**Dwarkesh Patel** (0:18)
Let's talk about robotics. So before I pepper you with questions, I'm wondering if you can give the audience a summary of where Physical Intelligence says that right now. You guys started a year ago, and what does the progress look like? What are you guys working on?

**Sergey Levine** (0:31)
Yeah, so Physical Intelligence aims to build robotic foundation models, and that basically means general purpose models that could, in principle, control any robot to perform any task. We care about this because we see this as a very fundamental aspect of the AI problem. The robot is essentially encompassing all AI technology, so if you can get a robot that's truly general, then you can do hopefully a large chunk of what people can do.
And where we're at right now is, I think we've kind of gotten to the point where we've built out a lot of the basics. And I think those basics actually are pretty cool. They work pretty well. We can get a robot that will fold laundry and that will go into a new home and try to clean up the kitchen. But in my mind, what we're doing at Physical Intelligence right now is really the very, very early beginning. It's just like putting in place the basic building blocks on top of which we can then tackle all these really tough problems.

**Dwarkesh Patel** (1:25)
And what's the year by year vision? So, one year in, now I got a chance to watch some of the robots. And they can do pretty dexterous tasks, like folding a box using grippers. And it's like, I don't know, it's like pretty hard to fold the box, even with like my hands. If you had to go year by year until we get to the full like robotics explosion, what is happening every single year? What is the thing that needs to be unlocked, etc.?

**Sergey Levine** (1:47)
So, there are a few things that we need to get right. I mean, dexterity obviously is one of them. And in the beginning, we really want to make sure that we understand whether the methods that we're developing have the ability to tackle like the kind of intricate tasks that people can do. As you mentioned, like folding a box, folding different articles of laundry, cleaning up a table, making a coffee, that sort of thing. And that's good, like that works. I think that the results we've been able to show are pretty cool. But again, like the end goal of this is not to fold a nice t-shirt. The end goal is to just like confirm our initial hypothesis that like the basics are kind of solid.
But from there, there are a number of really major challenges. And I think that, you know, sometimes when results get abstracted to the level of like a three-minute video, someone can look at this video is like, oh, that's cool. Like that's what they're doing. But it's not. Like it's a very simple and basic version of what I think is to come. Like what you really want from a robot is not to tell it like, hey, please fold my t-shirt. What you want from a robot is to tell it like, hey robot, like you're now doing all sorts of home tasks for me. I like to have dinner made at 6 p.m. I wake up and go to work at 7 a.m. I like to do my laundry on Saturday, so make sure that's ready, this and this and this. And by the way, check in with me like every Monday to see what I want you to do to pick up when you do the shopping. Right. Like that's the prompt. And then the robot should go and do this for like six months, a year. Like that's the duration of the task. So it's ultimately, if this stuff is successful, it should be a lot bigger. And it should have that ability to learn continuously. It should have the understanding of the physical world, the common sense, the ability to go in and pull in more information if it needs it. Like if I ask you like, hey, tonight, like, you know, can you make me this type of salad? It's okay. You should like figure out what that entails, like look it up, go and buy the ingredients. So there's a lot that goes into this. It requires common sense. It requires understanding that there are certain edge cases that you need to handle intelligently, cases where you need to think harder. It requires the ability to improve continuously. It requires understanding safety, being reliable at the right time, being able to fix your mistakes when you do make those mistakes. So there's a lot more that goes into this. But the principles there are you need to leverage prior knowledge and you need to have the right representations.