13+ attention mechanisms worth knowing:

▪️ self-attention
▪️ cross-attention
▪️ causal attention
▪️ linear attention
▪️ attention with softmax
▪️ sliding window (local attention)
▪️ global attention
▪️ FlashAttention
▪️ multi-headed attention (MHA)
▪️ multi-query attention (MQA)
▪️ grouped attention by queries (GQA)
▪️ multi-headed latent attention (MLA)
▪️ interleaved heads attention (IHA)

Slim Attention, KArAt, XAttention, Mixture-of-Depths Attention (MoDA)
https://t.me/DataScienceM