Code

Allow add_tokens for ESM#28535

Code

Merged

Rocketknight1 merged 5 commits intomainhuggingface/transformers:mainfrom allow_esm_add_tokenshuggingface/transformers:allow_esm_add_tokensCopy head branch name to clipboardJan 19, 2024

+27-10

Commits

Commits on Jan 16, 2024

• Allow non-special tokens to be added

  Rocketknight1

  committed
  8d4cfeb

• Add test, fix token adding code

  Rocketknight1

  committed
  6acba9b

Commits on Jan 17, 2024

• Revert changes to id_to_token and token_to_id

  Rocketknight1

  committed
  b3fa7e1

Commits on Jan 18, 2024

• Update the ESM tokenizer to be a bit more standardized

  Rocketknight1

  committed
  7fdb3f2

Commits on Jan 19, 2024

• Update src/transformers/models/esm/tokenization_esm.py

  Show description for 2dca270
  

  Rocketknight1

  and

  ArthurZucker

  authored
  2dca270