Demo
the-tokenizer-playground
Experiment with and compare different tokenizers · running · License unknown
- HF likes
- 709
- Status
- running
- License
- Unknown
- SDK
- static
Hugging Face
Xenova/the-tokenizer-playground
- SpaceXenova/the-tokenizer-playground
- SDKstatic
- Likes709
- LicenseUnknown
- Updated2024-10-31T20:47:29.000Z
Technical tags
Try this demo
Decision details
Experiment with and compare different tokenizers
- Availabilityrunning
- Last checked2026-09-28T22:15:39.214029+00:00
- LicenseUnknown
- Publisher (host)Xenova
- Formatstext
- RequirementsNone detected
- Example inputPrompt / chat message
- Example outputGenerated text
Underlying models
12 · not the same as publisher
-
Xenova/gpt-4 underlying model
-
Xenova/text-davinci-003 underlying model
-
Xenova/gpt-3 underlying model
-
Xenova/grok-1-tokenizer underlying model
-
Xenova/claude-tokenizer underlying model
-
Xenova/mistral-tokenizer underlying model
-
Xenova/gemma-tokenizer underlying model
-
Xenova/llama-tokenizer underlying model
-
Xenova/c4ai-command-r-v01-tokenizer underlying model
-
Xenova/t5-small underlying model
-
Xenova/bert-base-cased underlying model
-
xai-org/grok-1 underlying model