Skip to content

Change tokenizer to make it work with both Go and Python #7

Closed
@mostafa

Description

@mostafa

The current tokenizer uses a deprecated tf.keras.preprocessing.text.Tokenizer, which is only available to Python. This means that the tokenizer should be made available to the plugin using a REST API, which adds overhead. The tokenizer should be replaced by something like sentencepiece that is also available to Go: go-sentencepiece-encoder.

Related

Metadata

Metadata

Assignees

Labels

enhancementNew feature or request

Type

No type

Projects

No projects

Relationships

None yet

Development

No branches or pull requests

Issue actions