Academia is for Ambition — Alex Zhang, MIT

Swyx (0:03)

All right, we're here in the studio with Alex Zhang.

I guess most famously of RLMs, but you have a few other affiliations. Welcome to the show.

Alex Zhang (0:12)

Yeah, thank you for having me.

SPEAKER_3 (0:13)

Yeah.

Swyx (0:13)

I guess GPU Mode as well?

Alex Zhang (0:15)

Yes, GPU Mode as well.

Swyx (0:16)

You were shepherded in by Mark Saroufim, not everyone gets that kind of welcome.

SPEAKER_3 (0:19)

Yeah.

Alex Zhang (0:20)

I'm very, very close to all the people in GPU Mode. So we often end up working together in various capacities, like even beyond just GPU Mode itself. Yeah.

Swyx (0:30)

Can we explain that so people who are not that close don't know about this? It's a Discord, it used to be focused on, I guess, CUDA Mode and then generalize a little bit. It was started by Mark.

To me, it's like the hiring pipeline of the PyTorch team, and then he left PyTorch.

Alex Zhang (0:47)

Yeah.

So it started actually around when I was in college in like 2023 I think it was started by Mark, Andreas, and Jeremy Howard. The original premise was just like, it was a Discord dedicated to learning how to write GPU kernels, and they had like lectures. That was basically the extent of it. And I got interested in it because I was writing GPU kernels. It was actually out of like pure chance. I was interning at Snapchat at the time.

Swyx (1:22)

Rexis.

Alex Zhang (1:23)

Yeah, I was very bored with Rexis. So they had a project where like they were interested in writing. It was this paper called Infinite Attention. It was like a Google paper.

Swyx (1:35)

Yes, we've covered it on Paper Club.

Alex Zhang (1:36)

Yes, yes. So I was interested in whether or not you could write specialized kernels for it at Snapchat. Nothing really came of it, but I joined GPU Mode at the time. It was called CUDA Mode. I think for like legal reasons or something, they changed the name. But I met Mark, I met Matei, I met a bunch of other people that were very involved in the community. Then Mark had pitched this idea called Popcorn, which was now what you see as the leaderboard today. But the general idea was like, I think all of us had this intuition that GPU programming is very similar to, if you guys have done competitive programming.

I don't mean to say they're transferable skills.

Yeah, yeah, yeah, and there's actually a surprisingly small space of optimizations that people do, and there's actually not that many kernels per se that people are interested in optimizing. So we had this thought that if you had enough data, in the same way that Codeforces, there's millions of problems. If you could do this with GPU code, you could scale and automate GPU kernel development, which for researchers is a huge deal. Because I think one of the bigger bottlenecks, if you look at Mamba, for example, they released the paper with kernels because otherwise you can't really use it in any meaningful way. Not everyone has a Tri Dao on their team. So we're very interested in this. KernelBench spawned from that too, of can we get LLMs to automate GPU kernel code? I think that was a very, very fun time. It was between college and my PhD.

I had a really pleasant time doing stuff with GPU mode. Now, I kind of just help with the lectures sometimes. I'm not as involved and I think in general, we don't have as many competitions as we used to. But yeah, I still keep in touch a lot with everyone there.

Swyx (3:27)

Is there a friendly rivalry? Because I think the previous community that used to do this was like MLSys, MLPerf thing. Is there a friendly rivalry? Is this just new generation MLPerf or what's going on?

Alex Zhang (3:38)

The nice thing about GPU mode is that it is also a community in the sense that a lot of the lectures are very easy enough for a beginner to follow and ask questions and things like that. And the competitions are somewhat, not secondary, but you can participate in them to learn. I think with a lot of MLSys, MLPerf benchmarks, for the most part, only serious labs and companies participate. Like seriously in them at least. That was my understanding of it. I could be wrong. But I think also beyond GPU mode now, one thing that has been really exciting is there's a lot more websites and people that work on hosting competitions.

I think there's this website called LeetGPU or something, and it's like LeetCode for GPU problems. There's other ones too that I've seen. We've seen, there's many that have spawned and talked on GPU mode. It's very exciting in the sense that I think GPU programming used to be super niche. When I was interested in it, and the only reason I got interested in it was Tri Dao gave a talk at Princeton because he was applying for faculty. I mean, he is faculty there now, but I listened to his talk on Flash Tension in 2023, and I was like, wow, this is the coolest thing ever. I was like, this is what everyone should be working on. I guess VLM and stuff had come out too, and it was like, we should be writing kernels. But now, it's like everyone writes kernels.