🔬ESMFold2: The Bitter Lesson is Coming for Proteins - Alex Rives, BioHub

**Alex Rives** (0:00)
So ESM-C is also approaching programmable biology, but I would say in a very different way. It's approaching it from this kind of world modeling perspective, where the idea is basically you have a predictive model, and you're going to search the world model to find protein molecules that satisfy kind of whatever design criteria that you have. So we've been able to use this to actually now go and design many protein binders. But I think sort of most excitingly, we've been able to use this to actually design antibodies, SCFVs.

**RJ Honicky** (0:33)
Hello. Welcome to the Latent Space AI for Science Podcast. I'm RJ. Haneki, CTO of Miraomics.

**Brandon** (0:40)
Yeah. And I'm Brandon. Today, it's a pleasure to have Alex Rives, Head of Science at BioHub. Yeah. Would you like to interest yourself real quick?

**Alex Rives** (0:48)
Yeah. Yeah. Thank you for having me here. It's great to be here.
I'm Head of Science at BioHub. I'm a computer scientist, and I work on AI for biology, and a lot of my work has been on language models for biology.

**Brandon** (1:01)
By the time this podcast is released, you will have put out several new, exciting, interesting models.
Going over them, I couldn't help but have the kind of thought that you might be the most bitter lesson-pilled person in protein biology right now. Can you give a little context about what that means for biology and why you're so committed and excited to this route?

**Alex Rives** (1:23)
Well, I'll take that. I believe in scaling laws. I guess I've been working on this since the summer of 2018
My team, when we were at MetaFair, trained really the first transformer language model for protein biology. I've always thought that there would be kind of emergence of biological information as you train the model to predict the next token that evolution creates. So, our team has really explored that idea over a number of different years, and we've really kind of, I think, seen the scaling curve and really seen as we have increased models by an order of magnitude kind of in each generation that there's this emergence of new capabilities.

**Brandon** (2:08)
Yeah. So, you say emergence of capabilities, scaling over generations. You've been working at this, as you said, for I guess it would be eight years now, something like that.
It didn't always work that way, right? Like, there was signs that scaling might work. You know, we'll be getting to some new results where I think really you've kind of clearly demonstrated this hypothesis in a way that hasn't happened before. But you seem to have like a strong commitment to this in a way that I'm not necessarily sure I would have been so convicted that it would work in the same way. I mean, proteins are not this, protein language is not the same thing as natural language. There are similarities. But if you start sampling a transformer, a normal language transformer at a temperature, you're going to get gibberish. You sample a protein language model at infinite temperature, you're going to get something which is a valid protein, if not a not interesting protein. Despite the fact that it is a different domain for a different reason, I'm not necessarily sure that I would a priori assume the natural language model insight would transfer over. So what is specifically about the proteins that you thought was special or would make this also valid?

**Alex Rives** (3:17)
Yeah. I mean, it's a really interesting question, I think, kind of a deep question across AI right now more broadly. And I think what's so interesting is AI right now is such an empirical science. And so we don't have theory that can always guide us in these things, but we have this really strong empirical evidence of scaling. The thing that I was motivated by is, you know, if you think about evolution, and, you know, you think about the data that we have around proteins, we have databases that have billions of protein sequences. And, you know, those sequences contain patterns. And, you know, it had long been known. So, you know, this is going back, you know, decades kind of before, you know, we started working on this with language models, but that there are patterns, the sequences of protein families that come there because of the constraints that evolution is operating under. So, you can think about, you know, like a protein sequence that folds into a three-dimensional structure in space. And you can, you know, imagine that there are two residues or amino acids that are in this sequence that might be in contact in that folded structure. And so, evolution isn't free to choose those independently from each other. If it makes a choice at one position, it kind of has to make another choice that's going to be compatible at the next position.