We read every piece of feedback, and take your input very seriously.
To see all available qualifiers, see our documentation.
fd316bd
tokenizer.train(files, trainer)
TemplateProcessing
WordLevel
Unigram
nmtNormalizer
precompiledNormalizer
templateProcessing
digitsPreTokenizer
splitPreTokenizer
behavior
punctuationPreTokenizer
fromPretrained