#1934 - Lex Fridman

Lex Fridman (0:12)

How are you?

Joe Rogan (0:13)

Good to see you, my friend.

Lex Fridman (0:14)

Good to see you.

Joe Rogan (0:15)

Hey, what have your people done? You're AI people with this fucking chat GPT shit. This scares the fuck out of me.

Lex Fridman (0:21)

What do you mean?

Joe Rogan (0:22)

You're AI people.

You're wacky coders. What have you done?

Lex Fridman (0:26)

Yeah, it's super interesting.

Joe Rogan (0:28)

Fascinating.

Lex Fridman (0:28)

Language models, I don't know if you know what those are, but that's the general systems that underlie chat GPT and GPT. They've been progressing over the past maybe four years aggressively. There's been a lot of development. GPT-1, GPT-2, GPT-3, GPT-35.

And chat GPT, there's a lot of interesting technical stuff that maybe we don't want to get into.

Joe Rogan (0:52)

Sure, let's get into it. I'm fascinated by it.

Lex Fridman (0:55)

So chat GPT is based on, fundamentally on a 175 billion parameter neural network that is GPT-3.

And the rest is what data is it trained on and how is it trained. So you already have like a brain, a giant neural network, and it's just trained in different ways. So chat GPT-3 came out about two years ago, and it was like impressive but dumb in a lot of ways. It was like you would expect as a human being for it to generate certain kinds of text, and it was like same kind of dumb things that were off. You know, like, all right, this is really impressive, but it's not quite there. You can tell it's not intelligent. And what they did with GPT-35 is they started adding more and different kinds of data sets there. One of them, probably the smartest neural network currently, is Codex, which is fine-tuned for programming. Like, it was trained on code, on programming code.

And when you train on programming code, which ChatGPT is also, you're teaching it something like reasoning, because it's no longer information and knowledge from the internet. It's also reasoning. You can, like, logic. Even though you're looking at code, programming code is you're looking at me like, what the fuck is he talking about?

Joe Rogan (2:15)

No, no, no, that's not what I'm looking at. I'm looking at you like, oh my god.

Lex Fridman (2:18)

But reasoning is a... In order to be able to stitch together sentences that make sense, you not only need to know the facts that underlie those sentences, you also have to be able to reason.

And we take it for granted as human beings that we can do some common sense reasoning.

Like, this war started at this date and ended at this date, therefore it means that the start and the end has a meaning. There's a temporal consistency. There's a cause and effect. All of those things are inside programming code. By the way, a lot of stuff I'm saying, we still don't understand. We're, like, intuiting why this works so well. Really? These are the intuitions. Yeah, there's a lot of stuff that's not clear. So, GPT 3.5, which chat GPT is likely based on, there's no paper yet, so we don't know exactly the details, but it was just trained on code and more data that's able to give it some reasoning. Then, this is really important, it was fine-tuned in a supervised way by human labeling. Small data set by human labeling of, here's what we would like this network to generate. Here's the stuff that makes sense. Here's the kind of dialogue that makes sense. Here's the kind of answers to questions that make sense.

It's basically pointing this giant Titanic of a neural network into the right direction that aligns with the way human beings think and talk. So it's not just using the giant wisdom of Wikipedia. I can talk about what data sets it's trained on, but just basically the Internet. It was pointed in the wrong direction. So this supervised labeling allows it to point in the right direction to when it says shit, you're like, holy shit, that's pretty smart. So that's the alignment. And then they did something really interesting, is using reinforcement learning based on labeling data from humans. That's quite a large data set. The task is the following. You have the smart DPT 3.5 thing, generate a bunch of text, and humans label which one seems the best. So ranking. You ask it a question.

For example, you could do generate a joke in the style of Joe Rogan.

And you have a label, they have five options, and you have a label. Does it mention dick and pussy?

I don't know how exactly, but you get it to rank.