Skip to content

Auto-select Tokenizer
Compare
Choose a tag to compare
@zredlined zredlined released this 20 Apr 20:33
· 177 commits to master since this release
v0.15.5
a15f0f8
This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
GPG key ID: 4AEE18F83AFDEB23
Expired
Learn about vigilant mode.

Automatically select character-based tokenization over SentencePiece if vocab_size is set to zero.

Assets 2
Loading