Why AlphaFold Didn't Solve Protein Folding — Pushmeet Kohli, Google DeepMind & Sal Candido, Biohub

Brandon Anderson (0:05)

So, great to be here. What an exciting morning. So many cool announcements. I think the future of Bioscience is being announced right here.

So, this is the modeling session.

We are all modelers. So, of course, it's natural for us to talk about data. So, to start out with, I think one of the things that I like to think about when it comes to data is like, is there a scaling, like how do you scale data properly? And this has brought me to this question about the bitter lesson, but recast in the frame of data. So, the bitter lesson for those of you who are not AI people is the statement that methods that scale win eventually. If you can just scale enough, it wins. And so, my question, I'll start with Sal here, is, is there a bitter lesson for data?

Sal Candido (1:00)

Yeah, I mean, for sure. I think that you obviously need the right data, right, like in order for it to work. So, I think like one misconception of scaling laws is that, you know, scaling laws are everywhere and they always exist. I think a lot of the work is actually finding that scaling law, right? Like, so a lot of what we really do is trying to figure out, what's the situation where if you put more compute into it, if you put more data into it, you actually get a better result out. And the reason that's a great situation is then, once that happens, you can kind of just turn the crank, right? Like becomes an engineering problem, which is something I like. But that has to do with architecture, but it also very much has to do with data. So, if you don't have the data with the right information statistics to solve the problem that you want, which is, I think, probably pretty obvious, like if you think about it for a second. Like you're not going to get a model that has the capabilities and the understanding you want. So, you can only really pull the information from the data that you have and then use that to generate like to generalize beyond, right?

Brandon Anderson (2:09)

Yeah. So, when it comes to data collection, how do you think about what types of data modalities are best? And maybe a kind of related question is, do you think that modelers have a tendency to work on problems which are the data is available, rather than maybe the data that, or the problem that solves your goal most, like maybe has the most impact for translational medicine?

Sal Candido (2:31)

Oh, for sure. So, I think that like, I wouldn't speak for all modelers, but like I'm lazy, so I'm going to work with what's available. And I think there is actually like a good side and a bad side to this. And like the good side is that I think like when you're doing like machine learning or like conventional machine learning, you're really looking for like the most pristine, high-quality like data examples that you can find. But when you actually, if you're like me, you're actually someone who goes and you're kind of like rooting around in the back room looking through people's junk, like what's there? So like a concrete example of this is when we train, like when we train a protein language model, we train on metagenomic sequences which are not the highest quality data. In fact, much of that data, I can guarantee you, isn't even like a real whole protein. And yet that makes the performance of the model go up for designing real proteins that work, that make for understanding proteins that like we know are things.

And so that's like the positive side of it. But that can lead you to a negative place where you say, well, let's just scale up the data that we can generate easily. And I think that's not necessarily the way to do it. I think that's one of the things that's really exciting to me about what we're talking about here today with the BBI.

It's really going out and saying, like, what is the data that we need to solve the problem? It's the right resources, but also the right community. Like, I think that one thing that we do at, or we try and do at Biohub anyway, is to work in the open, work with the community and move the whole community forward. And I think that's a really critical piece for this, because if you just have people who are building models, we're going to lean towards the data that exists. If you just have people who generate the data, they're going to lean towards the things that can be generated. But if you can work together as a community, every step of the way in an open fashion, then you can actually figure out what is the data that we need, and then you can find that scaling law.