Information Theory for Language Models: Jack Morris

**SPEAKER_1** (0:03)
Hello, this is Latent Space Just Swicks today with our special guest, Jack Morris. A guest from Columbia, that's your affiliation right now.

**Jack Morris** (0:12)
Cornell. It's actually confusing because I'm in the New York City outposts of Cornell, so you have the city right, but it's Cornell Tech, which is like a small Cornell campus in New York.

**SPEAKER_1** (0:28)
I just, you're a student of Sasha Rush, who teaches at Cornell, so I shouldn't have made that connection. Okay, yeah, I'm sorry. Well, that's a horrible mistake to make right off the bat, but you're one of, look, you're one of, there are not that many PhD students that make an impact with their research. The last time someone like this happens was Shunyu from Princeton, and he joined the OpenAI operator team quite shortly after he graduated. So you're one of those high-profile PhD students, at least, that's coming out of the program, and I figured it was a good time to just talk about your work, and also the fact that you're looking for which lab you're going to join. That's a whole interesting meta discussion, especially with the insane market for AI talent these days. What's it like to be an AI grad student these days?

**Jack Morris** (1:21)
Yeah, and thanks for having me. I guess maybe we can go back to when things first started, or put yourself in my shoes. In 2017, 2018, I really learned a lot about machine learning, and I went to a state university. It's a good school, but they didn't have a deep learning research department or anything. They had people doing it, but it was just not as big at that time. But I was getting really interested in those topics, especially as applied to language. Then in 2019, I was starting to do research, and I think thinking about my career. I mean, at that point, I was 20, 21 I was thinking about where do I want to be career-wise, or who's doing the coolest stuff right now, like looking at what kind of stuff is coming out at that time. I mean, I think AlphaGo, I thought AlphaGo was really good. At that time, I was playing a lot with BERT and BERT-based models, so Google, DeepMind, they're doing great work. GPT-2, GPT-1 from OpenAI were interesting, but I think most people were into BERT at that time. I still have a soft spot for that parameter class of 100 million to 1 billion scale models. But this is all to say, I think at that time, I felt like the people doing a lot of the most impactful work were professors and PhD students, just a ton of interesting ideas being explored and cool opportunities in academia.
So I ended up applying to grad school. Well, at first, I did this Google AI residency program, which was mostly during the pandemic, like 2020 and then 2021 And then I was also applying to grad school, started grad school in 2021 That's still what was going on at that time. Like around when, I guess, GPT-3, 175 billion had been released, but not InstructGBT. So like we had pre-training and sort of the science of pre-training was emerging, but that's where the models were. And I still think like, I'm glad that I went to grad school and like I had a great experience, but the last five years have changed a lot. Like the whole meta has shifted, you know, like the kind of power dynamics are completely different. The ideas are coming from different places. Most stuff is open, now most stuff is not open. The types of questions people are asking are different. And so, yeah, I mean, for better or for worse, I did go to do the full grad school thing and here I am. It's been really interesting perspective watching the science kind of emerge with the products. Like the biggest thing that happened by far was like ChatGPT coming out, which was right in the middle, like, what? 2022 before Christmas, like November. I remember that year, like my grandma was asking me about it. And that's when it hit me like, oh, this is actually becoming like a real area that people will know about and understand. Like I was trying to explain it to my parents. And that's when I think things really started to change in terms of the types of questions you wanted to ask can't always be answered with academic resources. So a lot of the like fundamental kind of like boundary pushing and AI science moved into companies.

**SPEAKER_1** (4:37)
That was the year when like, you know, just around Europe as well, everyone in NLP and deep learning were like very confused at like, I think some people were like kind of expecting this already in the sense that they had, they were obviously more clued in to large language models. But I think that the sheer amount of consumer level interests that had that was around at the time in 2022, that completely changed the world. Like now we're just like in a different sphere. Did you have to pivot your research? Or were you already, you just went from BERT to like other stuff? You've done a lot of embeddings work.