Allen Institute published OLMo Hybrid 7B - a model that is built on the alternation of Gated DeltaNet layers and standard attention in a 3:1 ratio. This architecture solves more subtasks from the training data with fewer tokens, which directly reduces the need for data during training.
🟢 Why it matters? (With tests)
Gated DeltaNet is an RNN with an extension in the form of negative transition matrix values. This small change in the rule for updating the internal state allows Gated DeltaNet layers to implement the dynamics of pairwise element permutation and, due to this, solve state tracking tasks that are inaccessible to pure transformers.
In OLMo Hybrid, Ai2 showed that hybrid models are more expressive than the sum of their parts. There is a class of tasks (let's call them state tracking with memory access) that neither pure transformers nor pure RNNs can solve, but the hybrid copes with them even with a simple alternation of layer types.
Ablation experiments from 60M to 1B parameters showed that GDN is consistently better than Mamba2 both in pure and hybrid versions, even and uniform alternation of layers is better than concentrating attention in the middle of the network, and a 3:1 ratio is the optimal balance between quality and computational cost on medium and large scales.
➜ TESTS
☞ On MMLU, OLMo Hybrid achieves the same accuracy as OLMo 3 7B, using 49% fewer tokens; on the Common Crawl cut - 35% less.
☞ Data efficiency ratio of the hybrid is 83.7 compared to 94.9 for the transformer.
☞ Data savings increase with the size of the model: approximately 1.3 times for 1B parameters and 1.9 times for 70B.
After retraining and adaptation to long context, OLMo Hybrid outperforms OLMo 3 in all evaluation categories. On RULER with 64k tokens - 85.0 versus 70.9 for the base model.
Article, Set of models, Technical report
#AI #ML #LLM #OLMoHybrid #Ai2
••••••••••••••••••••••••••••••••••••••••••••••
🤖 Data & ML | @DataXplore