Releases: daulet/tokenizers
Releases · daulet/tokenizers
v0.4.2
- Make
libtokenizers.a
location relative to the source location;
- Handle empty inputs to encode/decode;
v0.4.1
- Much smaller go package due to minimal test config
v0.4.0
- Support encoding truncation
- Fix compiler warning
v0.3.4
- Release correct x86_64 artifact
v0.3.2
- Release darwin arm64 artifact
v0.3.1
- Artifacts no longer in nested dirs
v0.3.0
First release with pre-built binaries
v0.2.1
- Make
tokenizers.FromBytes()
less flaky
v0.2.0
- Add support loading tokenizer config from bytes to enable go file embedding.