Hugging Face (Twitter)
RT @remi_or_: Late Christmas gift for RL people 🎄
We are adding support for parallel decoding in transformers continuous batching!
You can now decode as many streams as you want from one prompt, which changes the game for long contexts 📜
but since a picture is worth 1024 words:
Post #2137
20
