Meta's MEGABYTE Revolution with Lili Yu of Meta AI

**Erik Torenberg** (0:00)
Turpentine is a network of podcasts, newsletters, and more covering tech, business, and culture, all from the perspective of industry insiders and experts.
We're the network behind the show you're listening to right now.
At Turpentine, we're building the first media outlet for tech people by tech people. We have a slate of hit shows across a range of topics and industries, from AI with Cognitive Revolution, to Econ 102 with Noah Smith. Our other shows drive the conversation in tech with the most interesting thinkers, founders, and investors like Moment of Zen and my show Upstream. We're looking for industry-leading hosts and shows along with sponsors. If you think that might be you or your company, email me at erik.turpentine.co. That's E-R-I-K at turpentine.co.

**Lili Yu** (0:45)
To model a 600 by 600 image, you have to have the one million tokens, and current architecture just cannot support it.
That naturally introduces a different problem. We need a new architecture to solve this, and that's why we have this very efficient way of modeling, involve multi-scale transformer. We're very excited about be able to directly in the future model the format of any file, any input, any modality. I think that's a really exciting direction for us.

**Nathan Labenz** (1:19)
Hello, and welcome to The Cognitive Revolution, where we interview visionary researchers, entrepreneurs, and builders working on the frontier of artificial intelligence. Each week, we'll explore their revolutionary ideas, and together we'll build a picture of how AI technology will transform work, life, and society in the coming years. I'm Nathan Labenz, joined by my co-host, Erik Torenberg. Hello, and welcome back to The Cognitive Revolution. Today, my guest is Lili Yu, research scientist at Meta, and author of the recent hit paper Megabyte, predicting million-byte sequences with multiscale transformers. I first encountered this paper via a viral tweet from OpenAI's Andre Karpathy, who called it a promising way to potentially move language models beyond tokenization. If you're not familiar with the term tokenization, in the context of language models, is the process of breaking natural language down into a fixed vocabulary of frequently occurring strings. In the case of GPT-3 and GPT-4, this vocabulary consists of more than 50,000 words, word parts, numbers, and symbols. All language model inputs are chopped up into these tokens prior to embedding into numbers, and all next token predictions are selected from this set of tokens as well. In other modalities, there are other conceptions of tokens. Image tokens, for example, might represent small squares within the larger image.
Tokenization is used because existing AI architectures struggle to work with really long sequences of data. With the best performing methods available today, the compute costs ultimately get out of control, and there is a GPU shortage, as you may know. Since sequence length is limited, some higher level, more semantic compression of the data is necessary, and tokenization is the first pass, still sort of hacky way that that's currently done. However, it does cause a bunch of problems, which Karpathy explains a bit in the tweet and also links to deeper reading on, if you're interested. Suffice it to say for our purposes that it gets super weird. So weird, in fact, that there is a redditor whose username has been immortalized as part of GPT-3 vocabulary, and there are other super rare tokens that famous models seemingly can't be made to say at all.
Karpathy summed it up by saying, the list goes on, TLDR, everyone should hope that tokenization could be thrown away. Maybe even more importantly, we may find general purpose strategies for multi-scale training in the process. And that's what we're talking to Lili about today, because in her strategy to eliminate the need for tokenization, she just might have made a more fundamental contribution. I'm not an algorithm expert. As you'll hear, I ask some pretty simple questions. But recently, I have come to believe that, given the presence of web scale data and web scale compute, it was really only a matter of time until somebody figured out a workable algorithm. Transformers are just one architecture, as the human brain is just one architecture, and neither is the end of history. The basic idea of this research is that the megabyte architecture operates at multiple scales. Unlike in a single transformer where all the layers tend to be the same size, in the megabyte architecture, information is first encoded in patches, then there is a global model that shuffles all the information around, and then the final byte level predictions are again made by separate local patch models, which can be run in parallel.
It seems like this architecture just might have it all. For starters, yes, the byte level prediction eliminates tokenization. Now the model looks at everything as raw bytes, and it's always predicting just the next byte. There are only 256 possible bytes, just two to the eight, so each one is ultimately just a series of eight zeros and ones. It's kind of crazy low level, to me at least, if you think about it that way.