TinyGPU, Massive Learning, with Adam Majmudar

**Erik Torenberg** (0:01)
Hey, everyone, Erik here. We're really excited about a new AI show from Turpentine called Autopilot, hosted by Will Summerlin.
This podcast explores the adoption and rollout of AI in the industries that drive the economy and the dynamic tech founders bringing rapid scalable change to slow moving industries from law to hardware to aviation. Will interviews founders backed by Benchmark, Greylock, YC and more to learn how they're automating at the frontiers and entrenched industries. Click on the link in the description to subscribe to Autopilot.

**Adam Majmudar** (0:31)
So these special registers are registers that you can't write to. The GPU itself handles writing to them. And that means that the GPU itself supplies this code execution environment with like, hey, you're this block number.
There's this many threads in each block and you're also this thread number in this block.
And so the hideable job of the GPU now is like, okay, well, how do we manage how to divide up these threads into the available compute cores, wait for them to complete their job, and then give them more jobs when they're done completing their work. The analogy is like, if aliens come to Earth and they find the computer, like how are they gonna understand it? Are they gonna break it down and look at all the connections, the transistors and see what's happening? No, they're not gonna understand it by doing that. They have to come to some understanding of their own and then test it against the computers that they have. It's a bit of a massively over complex analogy for going through a GitHub repo.

**Nathan Labenz** (1:25)
Hello and welcome to The Cognitive Revolution, where we interview visionary researchers, entrepreneurs and builders working on the frontier of artificial intelligence. Each week we'll explore their revolutionary ideas and together we'll build a picture of how AI technology will transform work, life and society in the coming years. I'm Nathan Labenz joined by my co-host Erik Torenberg. Hello and welcome back to The Cognitive Revolution. Today, I'm excited to share my conversation with Adam Majmudar, creator of the TinyGPU project in which Adam set out to speed run the creation of a GPU from scratch in just a few weeks, despite having no previous experience designing GPUs.
The result is truly remarkable and a great educational resource for anyone who wants a better understanding of the foundational technology on which today's frontier AIs are trained and run.
In this episode, Adam walks us through the GPU architecture layer by layer, from the tools he used to learn and design, to the GPU programming paradigm of single instruction multiple data, to the process of hardware implementation and verification, in a way that I think will be illuminating for just about anyone who's not already an expert in this critical industry. Along the way, I think this episode has a couple big picture lessons to teach us as well. First, it's a powerful reminder of the multi-generational technology stack that all of today's AI breakthroughs are built upon. Adam was able to do this work thanks to the compounding progress in chip design and design automation software that thousands, if not millions, of engineers have delivered over decades.
Second, it shows just how much AI itself is accelerating the learning process for individuals. Adam relied heavily on ChatGPT and Claude to help guide his learning journey. Whenever he got stuck, he would propose hypotheses to the AI and get feedback, enabling him to make much faster progress than learning solely from static resources. I found it particularly interesting to hear him describe today's leading models as having a sort of intuition for science and engineering, which transcends online resources in important ways. I've experienced this myself, most recently while trying to bring myself up to speed on the intersection of AI and biology, and separately when planning a macOS app. I think it's a truly important capability, which when combined with the agentic capabilities that we expect from next-generation models suggests potential for genuinely transformative impact in the near future.
Finally, beyond the technical content itself, I came away incredibly impressed with Adam as a thinker, builder, and communicator. The depth of understanding that he demonstrates reflects a level of curiosity, resourcefulness, and relentless drive to figure things out that will surely enable him to make a real impact in whatever domain he decides to tackle next. Personally, I came away from this conversation with a much stronger intuition for how GPUs actually work under the hood, and I'll bet this episode will scratch a lingering itch for many, many people. One note, if you're listening to the audio-only podcast feed, we do use a screen share in this episode and refer to a number of diagrams and code samples from the TinyGPU GitHub repository. With that in mind, I would recommend the YouTube version of this episode if that's convenient for you, though we'll also have a link in the show notes in case you want to follow along on your own. As always, if you're finding value in the show, I'd appreciate it if you'd take a moment and share it with friends. And if you do have any feedback, you can reach us via our website, cognitiverevolution.ai, or by DMing me on your favorite social network. Now, without further ado, please enjoy this whirlwind tour through GPU architecture with TinyGPU creator Adam Majmudar. Adam Majmudar, creator of the TinyGPU project, welcome to The Cognitive Revolution.