How Physical AI Learns Across Language, Video and Action — Ming-Yu Liu

SPEAKER_1 (0:00)

Propel Fitness Water. With Gatorade Electrolytes, Zero Sugar, and Vitamins. Propel hydrates better than water, to help you get the most out of your workout and get back to your best self. What propels you?

Propel with Gatorade Electrolytes.

SPEAKER_2 (0:15)

When you need to build up your team to handle the growing chaos at work, use Indeed Sponsored Jobs. It gives your job post the boost it needs to be seen, and helps reach people with the right skills, certifications and more. Spend less time searching, and more time actually interviewing candidates who check all your boxes. Listeners of this show will get a $75 sponsored job credit at indeed.com/podcast. That's indeed.com/podcast. Terms and conditions apply. Need a hiring hero? This is a job for Indeed Sponsored Jobs.

Ming-Yu Liu (0:45)

This car is about making a left turn. Can you envision the trajectory the car is taking and the video generated?

Tim (0:54)

Guess what?

That road was never filmed. It was made by NVIDIA's Cosmos3. It reads video, it simulates roads, and it's learning how to handle objects.

Ming-Yu Liu leads the research, and this is a paid partnership with NVIDIA. Now, Cosmos, it takes text, video, audio, and actions as input. Let's start with the simplest case.

Ming-Yu Liu (1:22)

As a model, what happened in the videos?

How many cars are in the video? Whether the robot succeeded in complete and action in the video? Okay, there are three cars, and the robot succeeded in pick up the apple and put in the basket. When you have video and test as input, and generate test, it becomes a visual language model.

Tim (1:44)

So why don't we flip this thing on its head? The model stops describing the scene and starts generating it.

Ming-Yu Liu (1:52)

55 Cosmos Dreams is a set of skills to build a closed-loop simulator, not just for self-driving. I can employ 100 drivers, drive 100 cars, go to different intersections and figure out whether it works better. Or I can use this simulator to verify the accuracy. You are not limited by the size of your fleet.

Any time you need to do a lot of testing, if you have a computer, just launch it.

Tim (2:25)

OK, so what if we took the same idea, but on an even harder problem, like robots, for example?

Ming-Yu Liu (2:32)

You know, navigation, you don't want your physical device to touch any other scenes. But manipulation tasks require all kinds of interactions. When there's an interaction, there's accretion. There's a potential deformation, you know, when scenes are manipulated. So it's more challenging. We will reach a state where we will be able to use neural simulation to handle this complex manipulation well. Once we reach the status, you can test your robot policy with the war simulator.

Tim (3:05)

OK, so one model, Cosmos3, handling three jobs. Handling objects in robotics is the future. It's very exciting.

Ming-Yu Liu (3:13)

So you can see that you are looking at a very diverse, versatile model architecture.

Tim (3:20)

And we now have action as a first-class citizen, which means the model might actually take an action, and then its next observation could be caused by the thing that it did before.

Ming-Yu Liu (3:31)

We start with a language model.

We either train it from scratch ourselves, or take it in an open model, right? Start with the language model. It has a basic understanding of the language structure, you know, understand language. Then we have a vision encoder that can tag a video or a set of images' inputs. Then we connect this vision encoder to the large language model. Connecting this vision encoder to the language model, we kind of build a vision language model, right? And sometimes you can also start with a pre-trained vision language model. Okay, now this language model, you know, is operating in an autoregressive manner.

We did a token at the time, right? Okay, it learns to connect, you know, visual output to the language space. It thinks in language. It explains what it sees, right? So you have a vision language model. Okay, now we take this vision language model, you know, the pre-trained weights, and then initialize a generator, right? So one key thing is that this vision language model is auto-regressive, and our generator is actually diffusion, is bidirectional. Now this bidirectional generate tower learn to generate video, action, audio.

Okay, in this tower, every token attend to every other token. So when you generate the video or the action, there's a coherence among the chunk it generates. Among each generation, the signal within this chunk is coherence. And it also diverges the token from the vision language model path, the recent tower, which we call is the recent tower. And so it also understands the intraction, what you want to generate. And in that generator training, there are two stages. Free training, learn the basic, and mid-training, we add action in.