Online LLM Tokenizer

A pure Javascript tokenizer running in your browser that can load tokenizer.json and tokenizer_config.json from any repository on Huggingface. You can use it to count tokens and compare how different large language model vocabularies work. It's also useful for debugging prompt templates. If you are wondering why are there so many models under Xenova, it's because they work for HuggingFace and re-upload just the tokenizers, so it's possible to load them without agreeing to model licences.