This is a continuation of the series of posts about the path of linear Attention. Last time, we found out that transformers have quadratic complexity, which makes them poorly scalable for long sequences and requires a lot of memory.
➡️ How it works?
📌 Linear Attention first changed the approach to computation. It allowed to decompose the kernel function and instead of exp(QKᵀ) use φ(Q) · φ(K). As a result:
• rearrange the calculations;
• first calculate K · V;
• then apply Q.
But the main thing: the complexity became linear in the length of the sequence (instead of quadratic). The model worked faster and more efficiently in terms of memory, but at the same time lost accuracy.
The usual Attention stores tokens separately, while linear Attention aggregates the information into a general "summary" of the context. Thanks to this, the model understands the general meaning well, but poorly remembers the details.
The task was: to maintain the efficiency of linear Attention, but to return local context.
📌 This problem was partially solved by the RWKV architecture. It did not abandon the idea of compact memory, but added mechanisms that make it more sensitive to the current context:
• Token shift
Instead of considering a token in isolation, the model mixes it with the previous state. Therefore, each new step contains information about the nearest context.
• Memory management
Memory in RWKV does not just accumulate. At each step, some of the old information is forgotten, and new information is added. If nothing is forgotten, the memory will quickly turn into noise. And if forgotten too aggressively — the context will be lost. The model learns to find a balance between extremes on its own.
• Gate at retrieval
When it's necessary to retrieve information from memory, a gate is used. It works like a filter: it looks at the current token and decides which parts of the memory are important now and which can be ignored.
RWKV became a kind of hybrid of previous ideas. It did not make the model as accurate as classic Attention, but at the same time returned local context. In parallel with it, other architectures were developing: SSM and Mamba.
Will tell more about them in the next posts.
••••••••••••••••••••••••••••••••••••••
🤖 Data & ML | @DataXplore