Interview with Gavin Uberti & Robert Wachen (Etched)

John Coogan (0:00)

I feed the robot a million times more words, and it just learns how to walk from a book.

Gavin Uberti (0:06)

Unironically, yes.

Robert Wachen (0:08)

A majority of the content of the internet is going to be AI generated.

Gavin Uberti (0:11)

99% of the world's population will have no idea.

John Coogan (0:14)

That's crazy.

Gavin Uberti (0:15)

I think we're going to see a long period of more than 10% GDP growth.

John Coogan (0:19)

Human population in 2060, over under 10 billion.

Gavin Uberti (0:23)

I worry under. There are not enough great thinkers, there are not enough great scientists.

John Coogan (0:27)

United States unemployment in 2030, higher than 10% or lower than 10%?

Gavin Uberti (0:33)

I think the much more exciting question is.

John Coogan (0:37)

Thanks for being here.

Can we start with some introductions? Who are you?

Gavin Uberti (0:40)

Yes, I'm Gavin. I'm a CEO and one of the co-founders of Etched.

Robert Wachen (0:45)

I'm Robert, also co-founder and CEO of Etched.

John Coogan (0:47)

So what is Etched?

Gavin Uberti (0:49)

Etched is a company building highly specialized chips just for transformers.

In the past, there are many kinds of AI models. Convolutional networks, recurrent neural networks, and so on and so forth. But that's changed. Now, all the big language models, and also things like Stable Diffusion 3 and Sora, are transformers.

John Coogan (1:10)

And also like Tesla self-driving.

Gavin Uberti (1:11)

That too. And even things like a Fault 3 are now being kind of transformer inspired.

John Coogan (1:16)

Sure.

Gavin Uberti (1:17)

And right now, all these transformer models run on general purpose AI hardware, like Nvidia GPUs or Google TPUs. But if you're willing to burn a single architecture onto a chip, you can be way more performant, more than an order of magnitude.

John Coogan (1:33)

Okay, so is this like some sort of refutation of the bitter lesson, Rich Hutton? Because the bitter lesson, for those who don't know, maybe you can give a summary of that and then explain why it makes sense in this specific case to do what you're doing.

Gavin Uberti (1:47)

Well, the bitter lesson states that the only techniques in AI that work are those that leverage Moore's Law and leverage scale. Ones that get better as the models get bigger. And that's the way we've gone from GPT-2 level models to the current generation. If you look at them from a pure architectural perspective, GPT-2 and modern open source models like Lama 3 are very similar. Exact same structure besides from some normalization stuff and positional encodings.

But the difference is that Lama 3 has been trained for 10 to the 4 more flops, or literally 50,000 times more compute. They just made the damn thing bigger, and it got much better.

And transformers follow the bitter lesson because they're able to leverage scale really effectively.

John Coogan (2:37)

I feel like there is a take on the bitter lesson that's like you should purely focus, put all of your chips on just more H100s, and you don't need custom software, you don't need custom hardware, because you can just throw more and more energy, more and more data, and that's the lesson. Is that not the case?

Gavin Uberti (2:56)

Well, it depends on the scale at which you're operating. The bitter lesson says that if you're going from one million to $10 million worth of compute, you should do exactly this.

But only a very small fraction of an H100 is used for compute. About 5% of the total area is spent on the floating point multiply add blocks that make up these FLOPs. The vast majority is used for just moving memory around the chip. Any flexible chip must have this property.

John Coogan (3:25)

Okay, so what makes a transformer special? Can you do like a ELI-5 on transformers, and why you think they're gonna stick around forever?

Gavin Uberti (3:33)

Well, the thing that makes a chip like our special is that when you have models that cost much more than the cost to build custom silicon, like say $100 million or a billion dollars, then suddenly the idea of building a single chip just for that kind of model makes sense because you can fit way more flops on the same silicon and get way higher utilization.

John Coogan (3:54)

Are there lessons from like Bitcoin mining?

Gavin Uberti (3:57)

Oh, absolutely.

John Coogan (3:58)

Walk me through those.

Gavin Uberti (3:59)

That used to be the case that Bitcoin was mined on GPUs, which did a much better job than CPUs.

But once the market for Bitcoin mining became larger than $10, $100 million a year, we saw ASICs emerge on the market. And these ASICs were only able to do one very narrow thing, and that was compute the SHA hashes used to mine Bitcoin. If another cryptocurrency had taken off, or Bitcoin had fallen out of fashion, they would not have any use anymore. But at that one task, they were 20 times better than the best GPUs.