Can Looped Transformers still help when FLOPs, parameters, and KV cache are all matched?

Bytedance introduced SMELT: Scaling Laws for Compute-Matched MoE Looped Transformers.

The answer is yes. And the advantage grows with scale.