Starting a series of posts to explain path to linear Attention.
Before transformers, tasks like translation and classification mainly used recurrent models (RNNs). In 2014, the Attention mechanism appeared. It allowed us not just to read a text sequentially, but to look at all input tokens and assess which of them were important for generation.
➡️ How it worked?
A bidirectional RNN encoded the input sequence → for each decoder step, it calculated the relevance of input tokens → obtained weights via softmax → based on them, it formed a context for generating the next token.
This led to a significant improvement in machine translation quality. However, the main problem with RNNs remained - they performed poorly on long sequences. To "understand" a word, the models had to process the entire text and reach it.
Transformers became the next step in evolution
To see the entire sequence at once and better model the dependencies between tokens, a number of changes were made:
• We abandoned recurrence - the sequence is calculated in parallel.
• Added self-attention - in addition to the encoder, attention now starts to be applied directly to the decoder.
• Added Bahdanau Attention for expressiveness - instead of a single-layer perceptron, a dot product of the trainable Q,K,V matrices is used.
However, a new problem arose: Attention has quadratic complexity in terms of sequence length. This means that as the context increases, memory and computations grow very quickly.
This was attempted to be fixed in various ways: reducing the number of heads to save cache; calculating on a portion of the sequence; creating kernels for efficient Attention calculation (for example, Flash Attention).
These methods accelerated the calculations, but didn't change the Attention formula itself.
More about how this limitation was overcome in next post.
••••••••••••••••••••••••••••••••••••••
🤖 Data & ML | @DataXplore