The Researcher to Founder Journey, and the Power of Open Models

**Andreas Blattmann** (0:00)
If you have an open model, there will be jail breaks, but there will be ways to mitigate those jail breaks. This is what we see in many other research fields. If you think about, I don't know, cryptography or something, there it's basically similar. You just improve your algorithms, then you have some people who jailbreak it, and then you improve further. No one certainly doubts that cryptography is really important for everything like we have on the web, and whenever exchanging information. And no one debates about whether open research is good or not.

**Derek Harris** (0:31)
Welcome to the A16z AI Podcast. I'm Derek Harris. This week, we have a very interesting discussion between A16z general partner Anjney Midha and the co-founders of a new generative AI model startup called Black Forest Labs, which they recorded live and in person in, as the company's name might suggest, Germany. The founding team, Robin Rombach, Patrick Esser, and Andreas Blattmann, drove the research behind the stable diffusion models, and recently started Black Forest Labs to push the envelope of image and video models, and to help keep the open research torch lit. In addition to discussing their new family of models, called Flux, Robin, Andreas, Patrick and Anjney also went into the transition from research to product, and then from building products to starting a company. In addition, they address the benefits of open research in AI and why it's important to learn from the greater community, rather than develop behind closed doors. But before we get started, here are some brief introductions from each of them to help you associate their voices with their names. First, Robin.

**Robin Rombach** (1:33)
I'm Robin, co-founder of Black Forest Labs. We're focusing on making image and video models as widely available as possible.

**Derek Harris** (1:41)
Then Patrick.

**Patrick Esser** (1:42)
Patrick Esser. I'm one of the co-founders of Black Forest Labs. I've been working in this area for a while, started at the university. I got excited when I saw the possibility that we can actually teach computers to create the images.

**Derek Harris** (1:56)
Finally, Andreas.

**Patrick Esser** (1:58)
Hi, my name is Andreas.

**Andreas Blattmann** (1:59)
I'm amongst the co-founders of Black Forest Labs. A couple of years ago, I started with those two guys working on image and then later video generation.

**Derek Harris** (2:09)
As a reminder, please note that the content here is for informational purposes only, should not be taken as legal, business, tax, or investment advice, or be used to evaluate any investment or security, and is not directed at any investors or potential investors in any A16z fund. For more details, please see a16z.com/disclosures.

**Anjney Midha** (2:32)
So you guys along with Dominik were four of the co-authors on Stable Diffusion. Why don't we go back all the way to the origin story? Where did you guys all meet?

**Robin Rombach** (2:41)
Yeah, we met at the University of Heidelberg, where we all did our PhDs or tried to get our PhDs.
And I actually met Andreas there who's from the Black Forest, just as me, village next door, basically. We didn't know each other before, but then we met in Heidelberg during the PhD. And it was a really nice time. We did a bunch of, I would say, pretty impactful works together. Starting with, we started with normalizing flows, actually. Tried to make them as good as possible, which was hard, which would probably still be hard. Then switched to autoregressive models, did this work that's called VQGAN. And then later on, after this DDPM paper, which really showed that diffusion models can generate nice images, we also looked into that and applied the same mechanism, the same formalism that we were working on before. We just latent generative modeling technology, where basic assumption is that when you want to generate media like images or videos, there's a lot of redundancy in the data. That's something that you can basically compress away, map the data into a lower dimensional latent space, and then actually train the generative model, which can be a normalizing flow, an autoregressive model or diffusion model on that latent space, which is computationally much more efficient. We did that then basically with latent diffusion. We did a bunch of tweaks to the architecture, introduced a text conditional unit, and we're amongst the first to do texts to image generation with diffusion models.

**Anjney Midha** (4:03)
If you guys think back to that moment in time, when it wasn't obvious maybe that diffusion models would be so good at various kinds of modalities, image generation, video generation, audio generation. That's more clear today, but was it as clear back then? What were the biggest debates you guys were having as a group back then?