Closed
Description
The current tokenizer uses a deprecated tf.keras.preprocessing.text.Tokenizer
, which is only available to Python. This means that the tokenizer should be made available to the plugin using a REST API, which adds overhead. The tokenizer should be replaced by something like sentencepiece that is also available to Go: go-sentencepiece-encoder.