Issues · huggingface/tokenizers · GitHub

ByteLevelBPETokenizer output seems weird
#203 · seyyaw opened on Mar 24, 2020
2
Training a model from in-memory data
#198 · loicbarrault opened on Mar 18, 2020
1

Labels Milestones

Precompiled normalizer uses grapheme-level shortest-prefix matching and diverges from SentencePiece on NFD input

#2116

· Incheonkirin opened

on Jun 17, 2026

Proposal: emit UserWarning from Split.pre_tokenize_str() when it produces zero matches (catches silent str vs Regex confusion)

#2109

· vamsin07 opened

on Jun 16, 2026

[Security] Load-time process abort when loading a malicious tokenizer.json (BPE merge buffer overrun)

#2094

· geo-chen opened

on Jun 10, 2026

Release 0.23.1 introduces breaking change in Encoding.overflowing

#2091

· alistairewj opened

on Jun 8, 2026

Two incorrect type annotations affect input typing to tokenizers.Tokenizer.encode()

#2088

· JEHoctor opened

on Jun 7, 2026

Reproducible WordPiece encode bench shows ~5x speedup on 123B fixture — methodology question for maintainers

#2075

· SMC17 opened

on May 28, 2026

Unigram trainer: prune loss uses `alternatives.len()` instead of `alternatives[id].len()`

#2069

· hunter-heidenreich opened

on May 24, 2026

BPE trainer: i32 overflow in pair_counts corrupts merge ordering on large code corpora

#2058

· petyb opened

on May 14, 2026

Publish error in npm repository, Node.js binary bindings

#2045

· uasan opened

on Apr 28, 2026

It is important to understand the real cause of the problem.

#2027

· Nearbe opened

on Apr 18, 2026

Feature request: compact vocabulary remapping for domain-specific training with pretrained tokenizers

#2023

· saslifat-gif opened

on Apr 14, 2026

[RFC] Korean Tokenization: Jamo Decomposition as Pre-tokenizer (A Blueprint for Compositional Scripts)

Feature Request

#1975

· nicezic opened

on Mar 22, 2026