https://arxiv.org/abs/2302.10866
https://github.com/HazyResearch/safari
Convolutional LMM, hmmm.
> reaching Transformer quality with a 20% reduction in training compute required at sequence length 2K. Hyena operators are twice as fast as highly optimized attention at sequence length 8K, and 100x faster at sequence length 64K.