Since ResNet with x + F(x) in 2015, deep residual connections have remained the only channel for inter-layer communication.
On paper, models look deep. But many layers "learn to be silent": as information accumulates, the signal becomes blurred and lost.
➡️ The bottleneck isn't computations within layers, but communication between them.
All previous approaches - DenseNet, DenseFormer, Hyper-Connections, MUDDFormer - answer the same question: "how to better merge the outputs of layers?"
Better coefficients. More channels. Adaptive weights.
But all of this is accumulation. And this is a categorical mistake.
Inter-layer communication should be extraction, not accumulation.
Request = "what do I need".
Key = "what do I have".
Both sides participate.
Layer 152 shouldn't "strain" to hear Layer 3 in the general chorus. It should simply ask him: "what did you say?"
Problem: naive implementation of depth attention took 44,924 ms for forward and backward passes. Too slow.
Introducing Flash Depth Attention (FDA) - a hardware-efficient core that accelerates depth attention by more than 40,000 times, making full-scale extraction of depth suitable for large-scale training.
Classic transformer pipeline: residual connections → sequential attention → residual connections → fully connected layer.
Pipeline with Flash Depth Attention (FDA): depth attention → sequential attention → depth attention → fully connected layer.
Next - Mixture-of-Depths Attention (MoDA): combining depth and sequential extraction into a single softmax.
Each head simultaneously accesses the KV of the current layer (sequentially) and the KV of all previous layers (depth-wise).
One operation, two dimensions of extraction.
Results: the model actively uses inter-layer extraction, the "attention sink" effect disappears, MoDA improves the baseline model OLMo2 on all metrics.
The first half of architecture development was about scaling components.
The second - about scaling communication.
Welcome to second half: Article, Blog (recommended), Code
••••••••••••••••••••••••••••••••••••••
🤖 Data & ML | @DataXplore