We're open-sourcing the Unigram tokenizer we rebuilt to reduce CPU utilization by 5-6x.
Small rerankers and embedders run in single-digit milliseconds on GPU, making CPU tokenization a meaningful share of total latency.
https://github.com/perplexityai/pplx-garden
Read more about improving Unigram tokenizer CPU performance on our blog:
https://research.perplexity.ai/articles/improving-unigram-tokenizer-cpu-performance
Post #248
1.83K
- ❤ 3
- 🤡 2