Model Quality, Fine Tuning & Meta Sponsoring Open Source Ecosystem

**Sarah** (0:05)
Hi, No Priors listeners, time for a host-only episode. This week, Elad and I talk about the path to better model quality from here.
The potential of fine-tuning, RLHF, RLAIF, RAG and retrieval systems generally, Meta sponsorship of the open-source model ecosystem, and finally, the beginning of a new set of consumer applications and social networks. Thanks for tuning in. So one thing everybody is thinking about is what it takes to get to 10x or 100x better AI systems. I think it'd be useful to just enumerate the elements to step function better. Elad, what do you think?

**Elad** (0:42)
It's interesting because there's a few different aspects of that that people always talk about. There's scalability of data sets and compute and parameters and all these things. But the reality is, I think a lot of people believe that in order to 10x or even 100x use cases and usages for AI, outside of that, there's things that could just be done on existing models today. So you don't need to wait for GPT-7 or whatever. You could start with GPT-4 or GPT-35 and add these things. And I think they are kind of bucketed into five or six areas. Number one is multimodality. So that means being able to use text or voice or images or video is both input and output. So you should be able to talk to a model, type to it, upload an image and ask about the image. And then it could output anything from code to a short video for you.
Second is long context windows. So basically, when you prompt a model, you basically are feeding it data or commands or other things. And everybody realizes that you need longer and longer and longer context windows. So magic, for example, is doing that for code. You should be able to dump an entire code repo into a coding model instead of having to do a piecemeal.
Third, which we're going to talk about today is model customization. So that's things like fine tuning, something known as RAG. There's data cleaning, there's labeling, there's a bunch of stuff that just makes models work better for you.
Fourth is some form of memory. So the AI actually remembers what it's doing.
Fifth is some form of recursion. So looping back and reusing models. And then six, which is related is potentially a bunch of small models that are very specialized being orchestrated by a central model or sort of AI router that says, well, for this specific task or use case, I'm going to route the prompt or the data or the output into this other model that's doing this other thing, which is basically how the human brain works, right? You process visual information through your visual cortex, but then you use other parts of your brain to make decisions, right? And so it's very similar to what evolutions were decided was an optimal approach. But I think it's really interesting because I think many people in the field know that these five or six things are absolutely coming and they can dramatically improve the performance on existing systems. Again, 10X, 100X better for certain things. And so it's more just a matter of when, right? It's not really an if anymore. A bunch of people are working on different aspects of this.
And I think it's all coming really fast. And so there's sort of two things that came out in the last week or two that are really relevant to this. It'd be great to get your thoughts on. One is OpenAI announcing that they're not gonna allow people to fine tune models. And then second is Google, where they looked at human generated feedback versus AI generated feedback for models and sort of fine tuning models that way. And if you wanna tell people a bit more about what happened with OpenAI and why that's important.

**Sarah** (3:16)
Yeah, so fine tuning as a capability has been offered by OpenAI for several years, right? But they've made like a specific investment in allowing people to do that with more sophisticated models in particular, like three, five, and then also making it possible for more enterprise use cases, right? And if you think about sort of like why that matters at all, as you said, like, you know, you have a bunch of these labs who are working on general capability and working on the sort of direction of scaling laws, like transformers predictably improve with scale data and compute.
But I think what's really interesting is like the way every, the way these models end up being used in many business or even consumer application contexts is against a specific task, right? And so we've talked a lot about like where research effort is being put or compute is being spent in the industry right now. And there's a really, I think there's a really interesting question of, we don't even know how good models can be at certain scale, right, at 70 or 30 or 100 billion parameters or more, but not at GPT-4 scale based on really high quality data and curation of that data because it hasn't been explored. And so I think we should talk about some of the different ways you get these models to actually operate against a specific task with either fine-tuning with RLHF against the reward for your task or with RAG, as you said, in terms of retrieving from a data set that you've specified, right? And there's reasons you would do all three of these, but I think it's actually a pretty big step for OpenAI to enable this because I think there was, at certain points in the research world, there's been a narrative that fine-tuning doesn't really matter, right? The general model matters.