Researchers proposed to rethink one of basic ideas of deep neural networks - residual connections.
Until now, such connections worked very simply: each layer adds its result to previous layers through fixed accumulation.
But this approach has problems:
• information gradually "blurs"
• hidden state increases
• harder for models to extract important representations from previous layers
🟢 What Attention Residuals solves?
Instead of the usual addition of layers, the model uses attention to select which representations from previous layers are really needed.
SIMPLY PUT:
the model itself decides which layers of the past to "return to".
What this gives?
• the network can selectively extract the necessary representations
• the effect of "blurring" information is reduced
• the growth of hidden states is controlled
• the depth of the network is used much more efficiently
To make this work on large models, the authors proposed a mechanism Block AttnRes.
It divides the network into blocks and applies compressed attention between them, making cross-layer attention scalable.
According to the results of experiments:
• about 1.25× gain in computations
• <2% increase in latency on inference
• a steady increase in quality on downstream tasks
The method was tested on the architecture Kimi Linear:
• 48B parameters in total
• 3B active parameters
If idea is confirmed on large LLMs, this could become a new standard for deep neural network architectures. GitHub_PDF
••••••••••••••••••••••••••••••••••••••
🤖 Data & ML | @DataXplore