Skip to content

Pull requests: huggingface/tokenizers

New pull request New
89 Open 850 Closed
89 Open 850 Closed
Author
Filter by author
Loading
Label
Filter by label
Loading
Use alt + click/return to exclude labels
or ⇧ + click/return for logical OR
Projects
Filter by project
Loading
Milestones
Filter by milestone
Loading
Reviews
Filter by reviews
No reviews Review required Approved review Changes requested
Assignee
Filter by who’s assigned
Assigned to nobody Loading
Sort
Sort by
Newest Oldest Most commented Least commented Recently updated Least recently updated Best match
Most reactions
👍 👎 😄 🎉 😕 ❤️ 🚀 👀

Pull requests list

BPE trainer: inherit continuing_subword_prefix/end_of_word_suffix from the model
#2108 opened Jun 13, 2026 by discobot Loading…
1
Persist encode_special_tokens across save/load
#2107 opened Jun 13, 2026 by discobot Loading…
1
Fix Unigram trainer prune loss to use per-piece alternative count
#2106 opened Jun 13, 2026 by NahButch Loading…
1
Fix i32 overflow in BPE trainer pair counts
#2105 opened Jun 13, 2026 by NahButch Loading…
1
Return an error instead of panicking on out-of-range BPE merges
#2104 opened Jun 12, 2026 by NahButch Loading…
1
ci: cache HF test data and authenticate Hub downloads to avoid rate limits
#2102 opened Jun 12, 2026 by SBrandeis Contributor Loading…
2
perf: add benchmarks for the decode API
#2099 opened Jun 12, 2026 by SBrandeis Contributor • Draft
1
Fix empty Encoding.overflowing when truncation is enabled
#2098 opened Jun 12, 2026 by discobot Loading…
1
perf: do not allocate for alignments in encode_fast
#2097 opened Jun 11, 2026 by SBrandeis Contributor • Draft
4
chore(deps-dev): bump shell-quote from 1.8.3 to 1.8.4 in /tokenizers/examples/unstable_wasm/www dependencies Pull requests that update a dependency file javascript Pull requests that update Javascript code
#2096 opened Jun 10, 2026 by dependabot Bot Loading…
1
chore(deps): bump shell-quote from 1.8.1 to 1.8.4 in /bindings/node dependencies Pull requests that update a dependency file javascript Pull requests that update Javascript code
#2093 opened Jun 9, 2026 by dependabot Bot Loading…
1
Reduce BPE merge update allocations and add file progress logs
#2092 opened Jun 8, 2026 by voidful Loading…
1
fix(typing): correct encode() input typing (PreTokenizedInputSequence tuple + stub Any)
#2089 opened Jun 7, 2026 by Anai-Guo Loading…
1
2
fix(bpe): widen pair_counts to i64 + add overflow regression test (#2058)
#2087 opened Jun 6, 2026 by pjdurden Loading…
1
6
ByteLevel: single-pass byte-level transform (apply_byte_map)
#2086 opened Jun 5, 2026 by dmatth1 Loading…
Avoid walking sparse vocab holes during serialization
#2085 opened Jun 4, 2026 by dfgvaetyj3456356-hash Loading…
1
Validate BPE prefix merges without unchecked UTF-8
#2082 opened Jun 1, 2026 by dfgvaetyj3456356-hash Loading…
1
Complete incomplete Viterbi lattice tests
#2081 opened Jun 1, 2026 by eunseo9311 Contributor Loading…
1
Add group capture support for Replace normalizer in Rust and Python
#2080 opened May 30, 2026 by ander-db Loading…
1
Batch encode: coarsen rayon tasks with with_min_len
#2077 opened May 28, 2026 by sebpop Contributor Loading…
4
bindings & bench: use mimalloc as global allocator on tested targets
#2073 opened May 26, 2026 by sebpop Contributor Loading…
4
chore: enable Dependabot weekly GitHub Actions bumps dependabot
#2071 opened May 26, 2026 by hf-dependantbot-rollout Bot Loading…
1
Fix Unigram trainer prune loss to use per-piece alternative count
#2070 opened May 24, 2026 by hunter-heidenreich Loading…
1
chore(deps): bump qs and express in /tokenizers/examples/unstable_wasm/www dependencies Pull requests that update a dependency file javascript Pull requests that update Javascript code
#2067 opened May 22, 2026 by dependabot Bot Loading…
1
Make BPE/WordPiece training deterministic
#2066 opened May 22, 2026 by ATOM00blue Loading…
1
5
ProTip! Updated in the last three days: updated:>2026-06-12.