You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
When training a Tokenizer, its Model gets replaced after training since the Trainer generates a new Model. This has several limitations:
When the Model being replaced has been customized (dropout, unk_token, ...), we lose all of this when we replace it (cf BPE dropout not working as expected #201)
In Python, if we keep a reference to the model added to the Tokenizer, this reference does not point to the actual model used by the Tokenizer after training.
Goal
Change the Trainer to actually train to Model in-place.
The text was updated successfully, but these errors were encountered:
Current state
When training a
Tokenizer
, itsModel
gets replaced after training since theTrainer
generates a newModel
. This has several limitations:Model
being replaced has been customized (dropout, unk_token, ...), we lose all of this when we replace it (cf BPE dropout not working as expected #201)Tokenizer
, this reference does not point to the actual model used by theTokenizer
after training.Goal
Change the
Trainer
to actually train toModel
in-place.The text was updated successfully, but these errors were encountered: