Code

Trainer improvements#519

Code

Merged

n1t0 merged 11 commits intomasterhuggingface/tokenizers:masterfrom trainer-improvementshuggingface/tokenizers:trainer-improvementsCopy head branch name to clipboardNov 20, 2020

+901-363

Commits

Commits on Nov 20, 2020

• Rust - Trainer::process_tokens has a default impl

  n1t0

  committed
  7a6423e

  Copy full SHA for 7a6423e

• Add WordLevel trainer

  n1t0

  committed
  5a6e91b

  Copy full SHA for 5a6e91b

• A Model can return its associated Trainer

  n1t0

  committed
  7dd8999

  Copy full SHA for 7dd8999

• Python - Make the trainer optional on Tokenizer.train

  n1t0

  committed
  9bd12fd

  Copy full SHA for 9bd12fd

• Train Model in place

  Show description for a4b6588
  n1t0

  committed
  a4b6588

  Copy full SHA for a4b6588

• PyModel uses a RwLock to allow modifications

  n1t0

  committed
  b401c3b

  Copy full SHA for b401c3b

• Test BPE keeping its options after training

  n1t0

  committed
  3983bce

  Copy full SHA for 3983bce

• Generate pyi, fix tests and clippy warnings

  n1t0

  committed
  e70a66c

  Copy full SHA for e70a66c

• Node - Trainers train the Model in-place

  n1t0

  committed
  22dabea

  Copy full SHA for 22dabea

• Node - Add WordLevelTrainer

  n1t0

  committed
  1c45b1c

  Copy full SHA for 1c45b1c

• Make sure TrainerWrapper can only train the right Model

  n1t0

  committed
  2410452

  Copy full SHA for 2410452