Jointly announcing EAGLE-3 with SGLang: Setting a new record in LLM inference acceleration!
- 5x🚀than vanilla (on HF)
- 1.4x🚀than EAGLE-2 (on HF)
- A record of ~400 TPS on LLama 3.1 8B with a single H100 (on SGLang)
- 1.65x🚀in latency even for large bs=64 (on SGLang)
- A new scaling law: more training data, better speedup
- Apache 2.0
Paper: https://arxiv.org/abs/2503.01840
Code: https://github.com/SafeAILab/EAGLE
SGLang version: https://github.com/sgl-project/sglang/pull/4247
@Machine_learn
Post #3500
2.19K
- ❤ 1
- 👍 1