RWKV: Reinventing RNNs for the Transformer Era — with Eugene Cheah of UIlicious

**swyx** (0:00)
Hey, listeners. Today, we have a very special episode for you. There's been a recent paper on the top 10 open challenges in LLM research that has consolidated a lot of intense debate. Today, we're going to talk about RWKV models, receptance-weighted key-value models, with Eugene Cheah, who is both part of the core RWKV team, CTO of a low-code AI test automation platform, and active member of the Latent Space Discord. The RWKV architecture has the potential to solve three of the top 10 open LLM challenges, increasing context length, making LLMs faster and cheaper, and designing a new model architecture. What is particularly appealing about it is that it does so by reviving the recurrent neural network, which even I have argued has been obsolete by the Transformer. It rejects the idea that attention is all you need, and replaces multi-head attention and feed-forward networks with new concepts called a time mix and channel mix, respectively.
It has been trained up to 14 billion parameters, they're getting help from Eleuther and Stability AI to scale up even more, and shows competitive results on reasoning benchmarks, the same benchmarks we covered in our benchmarks 101 episode, with similar size models, yet with linear costs and speed curves instead of quadratic ones.
In a way, RWKV are promising the room temperature superconductor of LLM architectures. In other words, the parallelizability and performance of Transformers without the quadratic cost. Obviously, the topic of what happens after the Transformer is in the finance terminology what we call a low delta outcome. It's probably not going to happen, but if it does, it will be very, very big. We even discussed this a little bit in our episode with Jonathan Frankle of MosaicML. But since the RWKV paper was published, the idea has been somewhat independently validated with Microsoft Research putting out the RetNet, or the Retentive Network, which has similar veins of what it's trying to do. And it also, of course, competes with other alternatives to the Transformer, like the state space models coming out of Chris Ray's group at Stanford, S4, H3, and the Monarch Mixer that was recently announced. However, RWKV is so far the most validated of all these ideas because it has already trained up to 14 billion parameters with multiple models that you can download and generate text with today. As podcasters, we want to be the first place that you hear about new things in AI that you'll be using in work or personal life as AI engineers and enjoyers. So this presents us with a problem. We have to be early on consequential topics and things, but also high signal. Some of our favorite compliments so far, which, by the way, I've added to our About page if you want to check that. Your pods are a legitimate highlight of life for me. They're amazing from McKay Wrigley and from the AI Safety Memes Twitter account, which is always fun. They just simply said, we're the highest signal pod for them. So we're very proud of this and want to keep it up while taking risks because even though we crossed a quarter million downloads just five months into the podcast, we're still very young and trying to figure out what kind of podcast we want to be and what kind of audience we want to have.
Today is going to be one of the riskier pods for a few reasons.
One, it is the first pod we're doing on a non-traditional architecture with no large Western institutional backing. Two, it is the first pod we recorded outside of the US without a regular studio, so the audio is not as good. And three, finally, it is the first pod we're doing with my Singaporean accent. I'll address each of these in turn. One, there's a significant institutional bias in Western media coverage when it comes to AI in that if you don't come from Stanford or Oxford or you don't work at Google or Facebook, your ideas have trouble getting attention. However, by far, the most admired organization that our guests have repeatedly mentioned is Eleuther AI, which spun up as a decentralized discord community that independently trained the first GPT class LLMs without a prestigious background. Eleuther has since spun off organizations like Stability AI and Conjecture, but I suspect that the RWKV community is working like the early days of Eleuther and we have a rare opportunity to capture an oral history of it live rather than two years after the fact.
Two, part of the latent space magic is that we try to get to know our guests and be in person with them to establish rapport like we did in New York in our Notion AI episode with Linus Lee. Despite the lower audio quality, I think we got a much better interview with Eugene because we were able to interact with each other in person.