Code

Fixing OPT fast tokenizer option.#18753

Code

Merged

Narsil merged 4 commits intohuggingface:mainhuggingface/transformers:mainfrom Narsil:fix_opt_fastNarsil/transformers:fix_opt_fastCopy head branch name to clipboardSep 15, 2022

+70-13

Commits

Commits on Sep 15, 2022

• Fixing OPT fast tokenizer option.

  Narsil

  committed
  73b8932

  Copy full SHA for 73b8932

• Remove dependency on pt.

  Narsil

  committed
  2c248c6

  Copy full SHA for 2c248c6

• Move it to GPT2 tokenization tests.

  Narsil

  committed
  9ca6e93

  Copy full SHA for 9ca6e93

• Added a few tests.

  Narsil

  committed
  1e4b9ee

  Copy full SHA for 1e4b9ee