TGViewer
Hugging Face Hugging Face @huggingface · 303 subscribers
Post #2019 13
Hugging Face (Twitter)

RT @itazapo: Transformers v5 redesigns tokenization.

In this blog post we talk about:
> tokenization crash course
> tokenizers and transformers - the bridge
> v5 tokenizers backend

Major shoutout to @karpathy for his BPE video that got me interested in tokenization in the first place.
Open in Telegram All posts from @huggingface
More from @huggingface
  1. Aug 27, 2026AI telegram stickers. — @deepstickbot
  2. Aug 27, 2026Post #2316
  3. May 10, 2026AI telegram bot. — @aigram
  4. Feb 17, 2026Hugging Face (Twitter) RT @evijit: Today, @evaluatingevals is introducing Every Eval Ever,…
  5. Feb 17, 2026Hugging Face (Twitter) RT @Cohere_Labs: Introducing ✨Tiny Aya✨, a family of massively mult…
  6. Feb 17, 2026Hugging Face (Twitter) RT @Cohere_Labs: Very special to work with our @huggingface friends…
Threads Profile ViewerView any public Threads profile without an account.Open ThreadLook →Writing with AI? Make it sound human.Metric37 rewrites AI drafts so they read naturally. Free AI detector, 1,500 words free.Try Metric37 →