The Future of the Transformer Part 1 with Trey Kollmer| H100 Chips will Supercharge AI Hardware

**Erik Torenberg** (0:00)
Turpentine is a network of podcasts, newsletters, and more covering tech, business, and culture, all from the perspective of industry insiders and experts.
We're the network behind the show you're listening to right now.
At Turpentine, we're building the first media outlet for tech people by tech people. We have a slate of hit shows across a range of topics and industries, from AI with Cognitive Revolution, to Econ 102 with Noah Smith. Our other shows drive the conversation in tech with the most interesting thinkers, founders, and investors like Moment of Zen and my show Upstream. We're looking for industry leading hosts and shows along with sponsors. If you think that might be you or your company, email me at erikaturpentine.co. That's E-R-I-K at turpentine.co.

**Nathan Labenz** (0:46)
With your 22,000 H100s, you could reach GPT-4 scale compute in five days.
They tried introducing a sandbox flag into the code to be improved. Sure enough, the, you know, the improver would do things like remove the sandbox flag. And, you know, I don't want to anthropomorphize this too much, but you can imagine a human doing this. And I don't really know what that sandbox flag does, but it's probably slowing me down. Hello, and welcome to The Cognitive Revolution, where we interview visionary researchers, entrepreneurs, and builders working on the frontier of artificial intelligence.
Each week, we'll explore their revolutionary ideas, and together we'll build a picture of how AI technology will transform work, life, and society in the coming years. I'm Nathan Labenz, joined by my co-host, Eric Thornburg.

**Trey Kollmer** (1:42)
I think in general, the writers were very happy with where the deal landed. On the AI stuff, it seemed like it ended up around where it was the month before, maybe with more, a more probably complete fleshed out legal understanding that the studios can't use generative AI to cut out writers out of credit or to, you know, turn their first drafts into second drafts.
But on training on our scripts, I think the final phrase they told us is that we retain like our right to assert like our rights relative to it. So I think we got nothing on the training and the argument from the studios we were told is they were like, well, open AI and all these other companies are training on your scripts. So, I mean, how can you ask us to agree not to? But some sense that I think the guild might try arbitration or other just like other venues to try to, you know, stake out some rights to us. But I think that's all pretty unclear how that's going to go.

**Nathan Labenz** (2:45)
Let's get to it then. So I think today I'm kind of thinking of structuring this as like the future of the transformer.
I was listening to last week in AI and Jeremy, who's one of the co-hosts there, who I'm a big fan of, throughout this number that the H100 does, you know, he said 100 trillion operations per second. I'm not super well versed in this, you know, in all the details, because there's a lot of little precision points on the definitions. And then, you know, I think it all kind of gets washed away when you start to put these H100s into clusters. And then it's like, you kind of have a theoretical max of what the machines can do. But then you also have to engineer, you know, obviously pretty substantially to get anything close to the theoretical max. So there's some of the research that we're going to touch on today kind of speaks to getting more out of hardware. But for starters, it was like, first of all, wow, that is an insane number. 100 trillion operations per second. That's like unfathomable, right? But then I was also kind of thinking, well, that starts to give an interesting angle on what does it take to train a GPT-4 in today's world? So with the caveat again that all these are pretty rough numbers and I would invite any listeners to give feedback on anything I'm kind of simplifying too much. But GPT-4 is said to be trained on approximately 10 to the 24th flops, 10 to the 24th. And 100 trillion is 10 to the 14th. So I created just a super simple little toy spreadsheet where I was like, all right, let's imagine we have a target scale of how much compute is going to go into a frontier model. And I have it just defaulted to 10 to the 24th to represent approximately GPT-4, even though we don't know exactly what that number is. And then a device flops, which for approximately to 100, I'm saying that's 10 to the 14th. That means you have 10 to the 10 seconds of H100 time that you have to run, assuming you're making full-ish use of it, to have enough flops to train a GPT-4 class model. So I was just like, okay, well, what does that look like in actual human time?