Transformers: The Discovery That Sparked the AI Revolution

**Ankit Gupta** (0:00)
Nearly every state-of-the-art AI system, whether it's ChatGPT, Claude, Gemini, or Grok, is built on the same underlying model architecture, the Transformer. But where do the Transformer architecture come from, and what can its development teach us about the way breakthroughs in AI happen? Let's dive in.
A transformer is a neural network that uses self-attention to take input data, like text or images, model the relationships between that data, and finally generate outputs, like meaningful text responses, translations or classifications. Many people know that the original transformer architecture was introduced in a now famous 2017 paper from Google called Attention Is All You Need. But what you might not know about are the breakthroughs that made this overnight success possible. There are three key developments that we'll discuss today. Long short-term memory, seek to seek with attention, and then finally, transformers. Let's start with long short-term memory networks or LSTMs. One of the core challenges motivating early AI research was to get neural networks to understand sequences. Natural language is inherently sequential. The meaning of a word depends on what comes before it or after it, and understanding an entire sentence requires maintaining context across many words. Early architectures like feedforward neural networks process each input in isolation, and so they weren't capable of understanding context, or they required looking at inputs of a fixed length. So researchers developed recurrent neural networks, or RNNs, as a solution to this. In simple terms, an RNN iterates over the inputs in order one at a time, and consumes the previous outputs as additional input at each step. So if an input is of length N, there are N feedforward pass steps. And as a result, during the backwards pass, the gradient with respect to the early inputs is the result of N matrix multiplications.
Now in practice, this meant that we often face a problem called vanishing gradients. The early inputs in a sequence had less and less influence on the network's output as the sequence grew longer because it went through these multiple matrix multiplications. Gradients, which are the signals used to adjust weights during training, would fade to near zero as they were passed backwards through time. In the 1990s, Hockreiter and Schmidhuber proposed a solution of this. It was called the Long Short-Term Memory Network, or LSTMs. LSTMs were a type of RNN that attempted to fix the vanishing gradient problem by introducing gates, which could learn what information to keep, update or forget. This made it possible to learn long-range dependencies, something vanilla RNN struggled with. But LSTMs were too expensive to train at scale in the 90s, and so progress stalled. Now you fast forward to the early 2010s, and GPU acceleration, better optimization techniques and new large-scale datasets brought LSTMs back into the spotlight. Suddenly, this relatively old architecture was viable again, and it began to dominate natural language processing. LSTMs were quickly adopted for everything from speech recognition to language modeling. In these years, NLP and computer vision were actually somewhat separate worlds. RNNs and LSTMs in particular were preeminent in language tasks, while convolutional neural networks, or CNNs, were winning in vision. But the basic question motivating both NLP and computer vision was the same. How do you model sequences? How do you let those models capture a structure that spans time or space? LSTMs were a huge step forward, but they still had limitations. The most fundamental was something called the fixed length bottleneck. Here's how most early LSTM systems worked. For sequence-to-sequence tasks like translation, you would take the input sentence, feed it into an encoder LSTM, and boil the input down to a single fixed size vector. Then a decoder LSTM would take that vector and try to construct the target sentence word by word. This yielded impressive results on the benchmarks of that era. But in practice, that single vector was still unable to accurately capture the meaning of long or complex sentences. Also, there wasn't a great way to encode the concept of order into a fixed size vector. This was very important in translation tasks. For example, in English, we put adjectives before nouns, and in Spanish, we often place adjectives after nouns. You'd see this in performance. These models worked okay on short inputs, but they quickly fell apart as sequences got longer. And truthfully, this was more than a performance issue. It pointed to a deeper architectural problem. Allowing the decoder to only see one static summary of the input was a fundamental limitation. Why not give it access to all of the intermediate information that the encoder saw? This sort of insight is what gave rise to the next big leap. In 2014, a paper introduced what would become the new standard for sequence translation, sequence to sequence, or seek to seek models, with attention. Like before, the core idea was to train two neural networks jointly. An encoder, which reads the input sequence and builds a representation of it, and a decoder, which generates the output sequence one step at a time. Both models were LSTMs, and crucially, they were trained together end to end. But there was a key insight that enabled this performance jump. Attention. Even though seek to seek used a fixed length vector, researchers realized that if you could let the decoder look back or attend to the encoder's hidden states, you could let the model learn how to align parts of the input to parts of the output.