Tokenizers v1 RC
A release-candidate rewrite of Hugging Face's Tokenizers library that is 3-30x faster while producing identical token IDs to v0.23.
Overview
Hugging Face published a release candidate of Tokenizers v1.0.0 on September 21, 2026, producing the same token IDs as v0.23 while running 3 to 30 times faster single-threaded on an Apple M4 Max. The rewrite introduces 'Bitcannon,' which replaces regex splitting with SIMD bitstream operations for GPT-2, cl100k, o200k, Tekken, and DeepSeek tokenizer families; a thread-local word cache that avoids reprocessing repeated words; a workspace split so applications link only the components they need; native parallelism without single-lock queuing; and reusable scratch buffers that cut repeated allocations. The full 1.0.0 release is planned to add more model families, simplified Python bindings, and inference-only C/C++ bindings.
This is a sourced directory entry. Features and capabilities are reported in the linked sources; this entry does not document independent Aymorphic testing.
Key features
- 3-30x single-threaded speedup vs. v0.23 while preserving identical token IDs
- Bitcannon: SIMD bitstream splitting replacing regex for GPT-2, cl100k, o200k, Tekken, and DeepSeek tokenizers
- Thread-local word cache avoiding reprocessing of repeated words
- Workspace split so apps link only needed components
- Native parallelism without single-lock thread queuing
Use cases
- High-throughput tokenization for LLM training and inference pipelines
- Rust applications wanting a lighter-weight tokenizer dependency
- Migration path to a faster tokenizer with no output drift from v0.23
Pricing
Free and open source; install the release candidate via `cargo add tokenizers --pre`.
Pricing reflects the source record and may change. Confirm current terms with the provider.
Dates and sources
- Released
- 21 Sept 2026
- Record updated
- 24 Sept 2026
- Record last checked
- 24 Sept 2026