Hybrid architectures are becoming increasingly popular, so more attention is being paid to creating next generation of linear models.
🟡 What did the researchers propose?
In Mamba-3, researchers proposed several new ideas around SSM (state space models), which significantly increase the model's ability to learn without losing speed.
As a result, Mamba-3 shows higher performance than previous linear models, including: Mamba-2 and Gated DeltaNet
And improvements are observed in all model sizes.
Interesting fact: this is the first version of Mamba created by students.
Main authors of the project: Aakash Lahoti, Kevin Li, Berlin Chen, Caitlin Weng, Tri Dao
🟢 What's new in Mamba-3?
1️⃣ Improved SSM discretization
A new discretization scheme has been proposed that better simulates convolution and makes the model more expressive.
It also allowed to get rid of short convolution, which was previously used in linear models.
2️⃣ Complex state transitions
State transitions now use complex numbers, which significantly improves the model's ability to track the state.
This solves the problems of previous versions of Mamba, which, for example, could not solve even some simple tasks like bit parity.
3️⃣ MIMO architecture instead of SISO
The model has moved from the structure of SISO (single input single output) to MIMO (multiple input multiple output).
This allows better use of hardware during inference, which gives an additional boost in performance.
Also, the architecture has been slightly updated and is closer to modern language models.
For example, components like BC / QK normalization have been added.
Mamba-3 significantly strengthens linear models and makes them even more competitive with transformers, especially in language modeling tasks.
Paper, Code, Blog
••••••••••••••••••••••••••••••••••••••
🤖 Data & ML | @DataXplore