Code

Add unigram bytefallback#1217

Code

Merged

Narsil merged 52 commits intohuggingface:mainhuggingface/tokenizers:mainfrom ArthurZucker:add-unigram-byte-fallbackArthurZucker/tokenizers:add-unigram-byte-fallbackCopy head branch name to clipboardJun 26, 2023

+7,262-6,526

Commits

Commits on Apr 12, 2023

• current updates will go red

  ArthurZucker

  committed
  044fb41

  Copy full SHA for 044fb41

Commits on Apr 14, 2023

• cargo fmt

  ArthurZucker

  committed
  f26b0b7

  Copy full SHA for f26b0b7

• npm install

  ArthurZucker

  committed
  f8c6c47

  Copy full SHA for f8c6c47

Commits on May 30, 2023

• Merge branch 'main' of https://github.com/huggingface/tokenizers into add-unigram-byte-fallback

  ArthurZucker

  committed
  ac7529a

  Copy full SHA for ac7529a

Commits on Jun 6, 2023

• refactor train for unigram to allow bytefallbakc (breaking)

  ArthurZucker

  committed
  ce61a40

  Copy full SHA for ce61a40

• fmt

  ArthurZucker

  committed
  b327540

  Copy full SHA for b327540

• nits

  ArthurZucker

  committed
  5e13667

  Copy full SHA for 5e13667

Commits on Jun 20, 2023

• Merge branch 'main' of https://github.com/huggingface/tokenizers into add-unigram-byte-fallback

  ArthurZucker

  committed
  e9e42e8

  Copy full SHA for e9e42e8

• update

  ArthurZucker

  committed
  92b6490

  Copy full SHA for 92b6490

• add a proper test

  ArthurZucker

  committed
  0fef053

  Copy full SHA for 0fef053

Commits on Jun 21, 2023

• fix encode optimised fallback + add trainer arg

  ArthurZucker

  committed
  dfd36ff

  Copy full SHA for dfd36ff

• fixes

  ArthurZucker

  committed
  c72eac1

  Copy full SHA for c72eac1

• fixes

  ArthurZucker

  committed
  3323956

  Copy full SHA for 3323956

• fix tests

  ArthurZucker

  committed
  03b5be5

  Copy full SHA for 03b5be5

• add test

  ArthurZucker

  committed
  c97fd26

  Copy full SHA for c97fd26

• fmt

  ArthurZucker

  committed
  4375f07

  Copy full SHA for 4375f07

• fix rust test

  ArthurZucker

  committed
  cc2f12d

  Copy full SHA for cc2f12d

• update python bindings

  ArthurZucker

  committed
  00e3a3d

  Copy full SHA for 00e3a3d

• update

  ArthurZucker

  committed
  8834796

  Copy full SHA for 8834796

• pub is okay and needed

  ArthurZucker

  committed
  005698a

  Copy full SHA for 005698a

• more fix

  ArthurZucker

  committed
  474d31e

  Copy full SHA for 474d31e

• cleanup

  ArthurZucker

  committed
  c2881b2

  Copy full SHA for c2881b2

• remove useles id

  ArthurZucker

  committed
  ad6f524

  Copy full SHA for ad6f524

• MissingUnkId error

  ArthurZucker

  committed
  29601fb

  Copy full SHA for 29601fb

• nits

  ArthurZucker

  committed
  739790d

  Copy full SHA for 739790d

• fix offset

  ArthurZucker

  committed
  fa5b6a6

  Copy full SHA for fa5b6a6

• add a test in python

  ArthurZucker

  committed
  0684008

  Copy full SHA for 0684008

• update src bindings

  ArthurZucker

  committed
  d5f37bd

  Copy full SHA for d5f37bd

• remove bytefallback from trainer

  ArthurZucker

  committed
  263376f

  Copy full SHA for 263376f

• styling

  ArthurZucker

  committed
  60c2dc0

  Copy full SHA for 60c2dc0

• update pckg

  ArthurZucker

  committed
  e01fbae

  Copy full SHA for e01fbae

• lint

  ArthurZucker

  committed
  112d423

  Copy full SHA for 112d423

• fmt

  ArthurZucker

  committed
  9320717

  Copy full SHA for 9320717

• stup with dev

  ArthurZucker

  committed
  6476410

  Copy full SHA for 6476410

Commits on Jun 22, 2023

• update code based on review

  ArthurZucker

  committed
  dc7cced

  Copy full SHA for dc7cced

• remove unused function

  ArthurZucker

  committed
  41f2a7e

  Copy full SHA for 41f2a7e

• udpate python test to compare ids

  ArthurZucker

  committed
  1cd282b

  Copy full SHA for 1cd282b

• fix option bool issues

  ArthurZucker

  committed
  949bcb4

  Copy full SHA for 949bcb4

• final fix

  ArthurZucker

  committed
  03aacd8

  Copy full SHA for 03aacd8

• clippy

  ArthurZucker

  committed
  e44b03a

  Copy full SHA for e44b03a

• fix npm isntall

  ArthurZucker

  committed
  ea02db0

  Copy full SHA for ea02db0

• update

  ArthurZucker

  committed
  5ad1d63

  Copy full SHA for 5ad1d63

• update test

  ArthurZucker

  committed
  8bf84cf

  Copy full SHA for 8bf84cf

• more in depth testing

  ArthurZucker

  committed
  6c3ea53

  Copy full SHA for 6c3ea53

• Lint

  ArthurZucker

  committed
  a04408c

  Copy full SHA for a04408c

• last attempt to fix node

  ArthurZucker

  committed
  7fc68a3

  Copy full SHA for 7fc68a3

• update node bindings

  ArthurZucker

  committed
  e1b7a33

  Copy full SHA for e1b7a33

• fmt

  ArthurZucker

  committed
  16f9619

  Copy full SHA for 16f9619

Commits on Jun 23, 2023

• Update tokenizers/src/models/unigram/model.rs

  Show description for 003d284
  

  ArthurZucker

  and

  Narsil

  authored
  003d284

  Copy full SHA for 003d284

Commits on Jun 26, 2023

• update based on review

  ArthurZucker

  committed
  2451f8c

  Copy full SHA for 2451f8c

• simpler test

  ArthurZucker

  committed
  4b90431

  Copy full SHA for 4b90431

• lint

  ArthurZucker

  committed
  58912da

  Copy full SHA for 58912da