Runway’s WorldPrompt and the Engineering of Real-Time Worlds

Swyx (0:03)

Okay, we're here with Anastassios from Runway with me and Vibhu in the studio. Welcome.

Anastasis Germanidis (0:08)

Good to be here.

Swyx (0:09)

Congrats on all your success and progress with Runway, your opening offices all over the world. Did you envision this when you first started out?

Anastasis Germanidis (0:16)

Not quite. I think even when we started, we had this idea that, you know, it was more a matter of when, not if, we were seeing the early generative models of 2016, 2017, and just extrapolating. Assuming, you know, we, resolution, quality increases predictably over time, there's going to be a point where most of content will be generated. And that was maybe the initial, the initial thesis of Runway was we will need, as a result of those generative models, rethink how creative tools are made.

And as we built out the research behind our generative models, it then became clear that they were useful far beyond that as well.

Swyx (0:57)

And it is more obvious now with like the real world stuff and the world models that we'll talk about later. I'm just kind of curious how you go from a background in like ZocDoc and computer vision into Runway.

Take us back to that early conversation with Chris and whoever else is on your founding team.

Anastasis Germanidis (1:14)

I was always split into those two worlds. One was the, you know, I had my own art practice. I was making a lot of interactive art, I think for a long time.

And then on the other side, I was working in startups and I was working as a male engineer, as a backend engineer at different companies. I've always been interested in coding and computation and especially interesting simulation and brought it back into my early artwork as well. And at the same time, I was interested in...

Swyx (1:43)

The personal site has a few, right?

Is there one that we should pull up? Just in case there's something that's like... I just like to go down memory lane. Okay, what is this?

Anastasis Germanidis (1:51)

So this was a project that I made, I think, back in 2015 where I built this software that would give voice instructions to people in a gallery space. So it would basically coordinate interactions between people. And so it will first give you an identity, like you're an architect, you're 30 years old, and you like sports.

And then it would match you with another person, and you have this completely generated interaction. Obviously, language models were not quite there at the time. And so it was a mix of some templates and some Markov chain generated text. And it would just completely simulate these small talk conversations between everyone in the gallery space.

So I was always very fascinated on the one end with the generative models and the early machine learning work that was being out at the time. But at the same time, there was a separate thread of simulation and what it means, like what can we learn about humans by creating those very simple models of their interactions and their behavior.

Vibhu (2:56)

Did you generate the props or the 30-year-old, whatever? Was it you generating them or how did you?

Anastasis Germanidis (3:03)

Exactly. So the program would just generate those from...

A lot of it would be kind of ModLib style of just, you have lists of different professions, lists of different personality types, lists of different ages, things like that. And then it would just combine those things together. And then maybe the next project we go is Uncanny Valley, Uncanny Road, which was one of the first projects that we built with one of my two co-founders, Chris. This was taking Pix2PixHZ, which was one of the early image-to-image models that NVIDIA released back in 2016 or 2017

And it was a model that would take a semantic map of a scene and then generate photorealistic, let's call it, output. Obviously, very early days, so it was not very high-fidelity outputs. But I think it was the first image generation model that we generate at 1K resolution. And it was all trained on self-driving datasets. So the semantic categories it would support were only things you would encounter on the road. So it would be pedestrians, traffic signs, bikes, stoplights. And so that was actually one of our first indications that we built this and people making all this very surreal imagery of a million pedestrians or a million traffic signs or gigantic humans. And it was an indication that you could take a model that was trained on this very boring dataset, essentially, of not that many interesting things happen when you're on the road. And then you can repurpose it and go very out of distribution and make something that was artistically compelling. And that was the summary of the thesis of Runway in some ways. That you can take the same generative models and if you look at them from another direction, if you build interesting tools around them and you give them to artists, they're going to do things that you don't expect.