The 10,000x Yolo Researcher Metagame — with Yi Tay of Reka

**AI Charlie** (0:05)
Welcome back, friends. It's only been a week since the World's Fair, and it was incredible gathering the community to see the latest and greatest in AI engineering. You can catch up now on the four live stream track days on the AI engineer YouTube, and our team is busy editing the remaining workshops and five other tracks, including the surprisingly popular AI leadership track. Thank you all for your support, and stay tuned for news about the next event, the 2025 AI engineer summit.
Last week, we did a very special deep dive with Josh and John of IMBU and Databricks Mosaic on training LLMs and setting up massive GPU clusters. And today, we're pleased to follow that up with a very special conversation with Yi Tay, formerly tech lead of Palm2 at Google Brain, and now chief scientist of Reka AI. Reka's largest model, Reka Core, was at launch, the fifth-best model in the world, and the only GPT4-class model not trained by a big lab like OpenAI, Google, Anthropic or Meta. In fact, while Google Gemini has 950 co-authors, Reka only has 20 employees, with up to five people actually working on pre-training.
Swix was excited to return to Singapore to delve into Yi Reka and building a new AI model lab outside of Silicon Valley. Stay tuned to the very end for a special bonus clip from Yi's recent appearance at the Tech in Asia meetup, for his spiciest take on why senior management is overrated and why this is the time to build up senior 10,000 ex-individual contributors like himself. Watch out and take care.

**swyx** (1:48)
Welcome Yi Tay to Latest Space. This is a long time coming, but I'm so excited to have you here.

**Yi Tay** (1:52)
Yeah, thanks for inviting and excited to be here.

**swyx** (1:57)
So you're interesting to research and introduce. You are now Chief Scientist of Reka, which is a super interesting model lab. But before that, you were at Google Brain. You were architecture co-lead on Palm 2 You were inventor of UL2. You're co-contributor on Flan. You're a member of the Bard Core team. And you also did some work on generative retrieval.
That's very, very illustrious three-year career at Google Brain.

**Yi Tay** (2:19)
Thanks, thanks, thanks.

**swyx** (2:20)
And then since then, Reka, you joined in March 2023, announced the $58 million Series A in June 2023 I don't know if you know the post-money valuation or the pre-money valuation is public. So it's crunch basis is 250-something million. So you don't even have to leak, it's on the internet.
Reka's stated goals were to work on universal intelligence, including general purpose multimodal and multilingual agents, self-improving AI and model efficiency. In February, you released Reka Flash. In April, you released Reka Core and Edge. And then most recently, you released Vibevile.
Is that a good summary of the last six years?

**Yi Tay** (2:54)
No, it's not five, four years.

**swyx** (2:55)
Four years, yeah. Oh my God.
We're talking about AI.

**Yi Tay** (2:58)
Yeah, I was like wondering like, since when did I like step into a time machine or something?

**swyx** (3:02)
Yeah, okay. So can we just talk about your transition into, you did your PhD and we can talk about your PhD, transition into brain and research and all that. I saw you do some work on recommender systems. I saw you do some work on quaternions.
What the fuck was that?

**Yi Tay** (3:17)
Okay, let's forget about that.

**swyx** (3:19)
Describe your path into modern LMs, because you didn't start there.

**Yi Tay** (3:25)
I think the world also didn't start there. I joined Google in 2019, end of 2019, and the world looked really different at that time.
I think that was around the time the first GPT was released by GPT-1 or something was released by OpenAI.
So research, ML research and NLP research looked very different at that time. So I was mostly, I identify as a language researcher. I don't like to use the word NLP, Jason will kill me if I use the word NLP, but I was like, okay, a language researcher. But I was more like an architecture kind of researcher. And when I joined Google, I was also, I continued on as a model architecture research. I worked a lot on efficient transformers.

**swyx** (4:04)
That was your first viral paper?

**Yi Tay** (4:05)
Yeah, and I worked long range arena. I spent quite a lot of time looking of could we do without attention? There was a synthesizer paper back in 2020 I think that was my early days in Google.
At that point of time, transformers research was mainly WMT, machine translation and pop-exity and stuff like that. It's not really about... I think a few short learning and few short in context learning came only about when GPT3 came out and beyond. I think that at that time, the meta, I would say, the meta looked really different.