Robot-Use Agents: Why General-Purpose Models May Win in Robotics

Francois (0:00)

One of the big surprises the last few years has been the generalizability of coding agents across different domains. And now Frontier researchers are showing that this includes controlling robots. This led MIT professor Philip Isola to suggest in a recent viral essay that we may be entering the era of robot use agents, where general purpose models could make different robots more capable. So today Francois and I invited the founders of Waddle Labs and RoboCurve, two groups of startups that are working at the very frontier of making robots more capable with LLMs.

Maybe you guys want to briefly introduce yourselves, and just say a little bit about what each of your companies focuses on.

Hanmei (0:40)

I'm Hanmei. I'm from Waddle Labs together with Vincent. We work on building LLMs that control robots, and we do this by doing two things. Building a harness that allows the LLMs to do this very effectively, collecting data and then using that data to train better LLMs.

Jay (0:54)

Hi, I'm Jay, co-founder of RoboCurve.

We are an evalus company for physical AI. We measure everything, any robot, any model, including LLMs, and also classical approaches such as VLAs, Vision Language Action Models, and Word Action Models, and we evaluate all kinds of embodiments like hands, grippers, arms, humanoids, quadrupeds, all kinds of stuff.

Francois (1:20)

In the last few weeks, videos from both of you guys went quite viral on Twitter, and I think both of them are actually quoted in that Philip Isola essay talking about robot use agents.

The videos showed things like LLMs being able to unscrew caps, being able to communicate between multiple robots, being able to do tasks like uncapping a pen, for example. And so I thought maybe what would be fun is for Francois and I to dig in with you guys on some of the research that led to this moment to begin. And then maybe we can also show some demonstrations of what this can actually do. To start, why don't we talk about some of the early research around transfer between language models and robot control. So I want to talk about some of the early work on pre-training language models for decision making, and then also the RT2 paper, which established the VLA. So maybe Jay, do you want to tell us a little bit about some of this early work involving being able to use language models to do any kind of robot control?

Jay (2:12)

One of the earliest successful approaches of using AI on robots is the RT2 paper, where they use a pre-trained language model on web text and images and use that to control robots. The interesting thing about that is that it's a fine-tuned version of a language model, and instead of outputting English, for example, they just output what we call an effective pose, which is the coordinates that you can then translate into joint commands that can control the robots.

In some cases, in some sense, it's very similar to what we see now with large language models, just that instead of fine-tuning, the models are good enough to just do it out of the box.

Vincent (2:58)

And this is actually quite similar to the mapping that I have in my head is kind of like the COT moment. And so when we did GSM 8K and you said, Susie had $8, she spent five, how much does she have now? It had to output four hash marks and then the answer and an EOS. There was no ability for it to chain of thought. And so then we allowed it to actually have like, okay, let me go eight minus five is three.

And so like, hash, hash, hash, hash, three. And she allowed it to do this kind of thinking before it actually gave an action or an answer.

And similarly now, like VLAs were basically like, an RT2 is basically like, it has to output an action. There's no like, I can't allocate more compute for a more complex task. And then now I have this code chain of thought thing that I can do. And I can say, okay, even if you were using the example where you're not actually outputting the code, you're just giving it to Astra and let it think, think, think, think, think, and then output an action. Similarly now, like, and then with code, it's a more Kolmogorov complexity or lower Kolmogorov complexity code length to just use code and really compact code to just like, here's the code, just go.

Jay (4:07)

I totally agree with you on that. I feel like at the end of the day, it's a lot about a bit of less, right? If you give the agents or if you give the AI model more autonomy and if you unshackle it a bit more and give it more resources, it can actually do a lot of the things that we find you in it to do.