World Models, Robotics, and the Future of 3D AI

Justin Johnson (0:00)

Language models are these general horizontal engines for processing streams of discreet text, discreet tokens. Those have had tons of applications from everything that we know and love today. And our thesis is that there exists another category of model called world models that should be based in visual understanding, should be based in physical understanding, that can be used to generate, simulate, reconstruct worlds. And if we can build these with the right generality, they should be applicable to tons of different industries, from entertainment to VR to construction to robotics. We live our lives in this physical built space around us and we need models to help us with these things as well.

SPEAKER_2 (0:31)

Language models gave AI a way to work with words. What happens when models can understand and simulate the physical world?

World Labs co-founder, Justin Johnson, joins Theo Jaffee and Sofia Puccini on MTS, to discuss Atlas and the broader idea behind world models. They explore how models could generate and reconstruct environments. What separates them from video generation and applications across gaming, VFX and robotics, including turning a few photos of a real space into a simulation for training a robot?

Theo Jaffee (1:05)

We're live with Justin Johnson, who is a co-founder of World Labs, which is a startup that builds spatial intelligence products. And just today, they announced Atlas, the world's first multimodal world model that generates image and video frames with pixel perfect camera control and reconstructs them in 3D. Model the world, move the camera and simulate space and time. So congrats on the launch. The launch video looks really cool.

Welcome to MTS.

Justin Johnson (1:29)

Yeah, thanks for having me.

Theo Jaffee (1:30)

Absolutely. So tell us a little bit more about what this product is, specifically, what you can do with it, what people use it for.

Justin Johnson (1:37)

Yeah, so this is not a product launch. This is a model announcement. So the products are coming later. This is the base model. It's going to be used to power our future products from World Labs. But at its heart, it's a world model. It does basically three different kinds of things. Generation, reconstruction and simulation.

Generation is you want to generate new worlds that don't exist. I can start from a text prompt, I can start from an image prompt, then I can direct the camera and fly through that world and whatever, have the model generate a world that never existed before. So that obviously has tons of applications across many things in creativity. From VFX to gaming to all the kinds of use cases that people are using there. The second major capability of Atlas is reconstruction. There, sometimes I don't want to generate a new world. Sometimes I have an existing space in the real world that I like to model and bring into virtual reality or the virtual world in some way. So there what Atlas can do is sparse reconstruction. I can take just a few photos of a space, as few as one, up to a hundred or more. And the more photos you give it, the more you can reconstruct that existing real world space accurately.

And the third thing we can do is simulation. And this is going after more robotics.

Well, both VFX and robotics. But there we want to be able to take an existing space, re-create that space in 3D, then simulate how a robot might behave in that space. And we show some examples of all these different use cases on the blog.

Sofia Puccini (2:56)

Yeah.

Theo Jaffee (2:57)

Is the Ultimate Vision for World Labs something more like an environment for teaching robotics how to work, or is it more like a metaverse that people will spend time in?

Sofia Puccini (3:07)

All of the above, probably.

Theo Jaffee (3:07)

Or is it just kind of all of the above?

Justin Johnson (3:08)

Yeah, I think it's all of the above. The central thesis here is that there's a new, there's a different kind of horizontal model that we can build called a world model. Like language models are these general horizontal engines processing streams of discrete tax, discrete tokens. Those have had tons of applications from everything that we know and love today. And our thesis is that there exists another category of model called world models that should be based in visual understanding, should be based in physical understanding, that can be used to generate, simulate, reconstruct worlds. And if we can build these with the right generality, they should be applicable to tons of different industries, from entertainment to VR to construction to robotics.

We live our lives in this physical built space all around us and we need models that help us with these things as well.