Meet the world's fastest tokenizer implementation, Gigatoken
Gigatoken can tokenize the entire internet in <7 hours on one machine.
Specifically, this refers to tokenizing Common Crawl on a dual socket AMD Epyc server, reaching 24GB/s of throughput.
Gigatoken is ~500-1000x faster than HuggingFace, and ~100x faster than OpenAI's tiktoken for most tokenizer definitions on most machines.
Post #4373
670
