Skip to content

Pull requests: huggingface/tokenizers

New pull request New
10 Open 657 Closed
10 Open 657 Closed
Author
Filter by author
Loading
Label
Filter by label
Loading
Use alt + click/return to exclude labels
or ⇧ + click/return for logical OR
Projects
Filter by project
Loading
Milestones
Filter by milestone
Loading
Reviews
Filter by reviews
No reviews Review required Approved review Changes requested
Assignee
Filter by who’s assigned
Sort
Sort by
Newest Oldest Most commented Least commented Recently updated Least recently updated Best match
Most reactions
👍 👎 😄 🎉 😕 ❤️ 🚀 👀

Pull requests list

Fix encode_batch and encode_batch_fast to accept ndarrays again
#1679 opened Nov 8, 2024 by diliop Loading…
Decode stream python
#1678 opened Nov 7, 2024 by Narsil Loading…
1
Adding an API for decode streaming.
#1677 opened Nov 6, 2024 by Narsil Loading…
12
Testing ABI3 wheels to reduce number of wheels
#1674 opened Nov 6, 2024 by Narsil Loading…
7
Fix normalized string append after clear #1636
#1660 opened Oct 20, 2024 by Anantha-Kandrapu Loading…
Support updating template processors
#1652 opened Oct 14, 2024 by ArthurZucker • Draft
1
[WIP] free speed/mem optimizations with ahash, dary_heap, and compact_str
#1618 opened Aug 21, 2024 by mjbommar Loading…
🚨 breaking: Fix training with special tokens
#1617 opened Aug 19, 2024 by ArthurZucker Loading…
1
2
Fast regex
#1605 opened Aug 8, 2024 by ArthurZucker • Draft
1
[Feature] support Assign token to update the content of a token Stale
#1570 opened Jul 12, 2024 by ArthurZucker • Draft
1 task
1
5
ProTip! Type g i on any issue or pull request to go back to the issue listing page.