An experimental attention mechanism for transformers
MEA offers an alternative to classic softmax-attention. Instead of normalization via softmax, a matrix exponential is used, which allows modeling more complex, high-order interactions between tokens.
🟢 How it works?
IDEA:
Attention is formulated as exp(QKᵀ), and the calculation of the exponential is approximated by a truncated series. This makes it possible to calculate attention linearly along the length of the sequence, without creating huge n×n matrices.
What does this provide
- More expressive attention compared to softmax
- Higher-order interactions between tokens
- Linear complexity in memory and time
- Suitable for long contexts and research architectures
The project is at the intersection of Linear Attention and Higher-order Attention and is of a research nature. This is not a ready-made replacement for standard attention, but an attempt to expand its mathematical form.
For ML researchers and engineers who are studying new forms of attention, alternatives to softmax, and architectures for long sequences.
GitHub Not for production yet
••••••••••••••••••••••••••••••••••••••
🤖 Data Science, ML & Big Data with @DataXplore