NVIDIA World Models Explained: What Developers Can Build

Nate B. Jones (0:00)

So, we're saying that it's going to be faster to have a robot in my house that will do the laundry and fold it correctly than one that will do perfect scrambled eggs.

Ming-Yu Liu (0:07)

I think so. It's hard to say that it knows the physics.

You know, we actually don't know it knows the physics, but instead of creating a thousand different kitchen, you can quickly create a thousand different kitchen in the world model. As I enter the physical world, you will see more and more policy models.

Nate B. Jones (0:27)

Today's conversation is about world models. I'm here at Nvidia. People know Nvidia because of chips, but they actually do a ton of model development as well. I'll be chatting with Ming-Yu, who is the head of the Cosmos world model family. For those of you that don't know, a world model is a model that produces actions the way LLMs produce text. So for example, if I have a robot arm, I could tell the robot arm to take an action, like picking up the water bottle.

And that is a direct action that's produced. It's not like it's writing text that says, pick up the water bottle. It actually is issuing guidance to a robot to do that. World models are already around us. I've got a little robot lawn mower named McAllister and it mows my lawn. It has a tiny world model inside it. And it just outputs where that little robot lawn mower goes next on my yard. That is just a little bit of what I dove into today. I'm really excited to chat. Maybe you start by introducing yourself. Tell us about who you are, what you're doing at Nvidia, all of that.

Ming-Yu Liu (1:30)

Yeah, so my name is Ming-Yu Liu. I'm the vice president of Cosmos Lab. I lead Cosmos Lab for Cosmo platform development.

And I start my career as a GenAI researcher. I was the first group of people who believe that the future visual content will be generated by deep learning. But 10 years ago, when I did a job interview here, I told our chief scientist that I know today we are using computer graphics to generate visual content, but in the future, it's going to be using AI.

And it was quite controversial at the time, because the image could be generated by AI model at the time was so limited. The resolution is small and often the human look like a zombie. I just feel like this is such a nice paradigm, put all the problems into a learning problem, and you don't need to have all kinds of hand-designed rules to create a 3D world, just data-driven and create as long as this paradigm scale. The mutual content will be created through AI. And in the past 10 years, we have seen tremendous progress in the field, gradually, one by one. And now, I think most of the, I would say, it's growing, maybe, exponentially, the amount of content in the, we sure is generated by AI nowadays.

Nate B. Jones (3:25)

Super exponentially, it's growing.

Ming-Yu Liu (3:27)

Yeah, exactly.

Nate B. Jones (3:28)

So you were early. So 10 years ago, you saw this vision that AI was going to be a big part of video creation.

Ming-Yu Liu (3:34)

Yeah.

Nate B. Jones (3:34)

Now, you're in charge of the Cosmos family and where that's going. I think everyone assumes that Nvidia is about GPUs because that's all we hear, right?

But I learned that we have more software engineers at Nvidia than you have hardware engineers. And so it's a software company too. There was a hugging face acquisition recently. Tell me a little bit about sort of within the Nvidia family. What's it like to be focused specifically on, I guess we could call it a world model. Is that a fair way to describe it?

Ming-Yu Liu (4:05)

Yes, world model. So, Nvidia is a developer company. So we build our computers to help developers build amazing products to serve their customers.

And developers often need more than just the GPUs. You also need CUDA so that you can better program our GPU to extract all the muscle powers from our GPU. And there are also libraries we build on top, and also like an AI model we build on top. So we are kind of building layers of software on top of our computers. Now our computer including GPU, CPU and networking devices, connect them together so that we can better serve our developers.

We provide them tools so they can kind of save some energies and move faster. We imagine they may figure out a way to build better tools than what we provide, right? But also through building these tools, we better understand what kind of computer architecture we need to have for the future challenge. So we are building vertical to better knowing how to build our computers to serve the whole ecosystem.