Skip to content

Releases: daulet/tokenizers

v0.4.2

07 May 22:25
Compare
Choose a tag to compare
  • Make libtokenizers.a location relative to the source location;
  • Handle empty inputs to encode/decode;

v0.4.1

05 May 02:25
Compare
Choose a tag to compare
  • Much smaller go package due to minimal test config

v0.4.0

04 May 22:13
Compare
Choose a tag to compare
  • Support encoding truncation
  • Fix compiler warning

v0.3.4

04 May 03:40
Compare
Choose a tag to compare
  • Release correct x86_64 artifact

v0.3.3

04 May 03:28
Compare
Choose a tag to compare
  • Add x64 artifact

v0.3.2

03 May 00:43
Compare
Choose a tag to compare
  • Release darwin arm64 artifact

v0.3.1

02 May 23:50
Compare
Choose a tag to compare
  • Artifacts no longer in nested dirs

v0.3.0

02 May 23:28
Compare
Choose a tag to compare

First release with pre-built binaries

v0.2.1

07 Apr 23:34
Compare
Choose a tag to compare
  • Make tokenizers.FromBytes() less flaky

v0.2.0

03 Apr 22:21
Compare
Choose a tag to compare
  • Add support loading tokenizer config from bytes to enable go file embedding.