Tokenizer supports multiple encodings, compatible with .Net Standard 2.0#218

Open

Frogley wants to merge 4 commits intobetalgo:devfrom Frogley:feature/Tokenizer-multiple-encodings-DotnetStandard2.0

+201,581-19

Commits on Mar 20, 2023

Merge pull request betalgo#187 from betalgo/dev
kayhantolga
authored

Commits on Mar 28, 2023

Commits on Apr 7, 2023

1. Add the OpenAI.GPT3.Tokenizer.TikToken.Tokenizer, supports multiple encoding: r50k_base, p50k_base, cl100k_base; supports encode and decode method.
Frogley
committed