Make ITextEmbeddingGenerator.CountTokens and ITextGenerator.CountTokens ValueTask<int> #630
Replies: 5 comments
-
|
We tried making it async during the initial implementation, but it would affect the speed and complexity of the text chunker that would need quite a bit of rewrite, and raise questions about the usage. Counting tokens is currently meant to be fast&free, e.g. we use CountTokens even for logging statements. If we modify that assumption, we'll need to reassess each use case to avoid unnecessary calls and unforeseen expenses. |
Beta Was this translation helpful? Give feedback.
-
|
OK, well what I'm doing is creating an implementation of the LLama API (native not emulated). I can't find a good LLama token counter for C# that will do it without the API call. Suggestions? |
Beta Was this translation helpful? Give feedback.
-
|
IIRC Llama uses SentencePiece, anything available in that direction? |
Beta Was this translation helpful? Give feedback.
-
|
Seems like you guys have one? |
Beta Was this translation helpful? Give feedback.
-
Did you end up going with this? I'm facing this exact issue right now and haven't found a good solution. I'm still doing it async, but just using I don't exactly see why this method can't be async |
Beta Was this translation helpful? Give feedback.
-
Right now these are synchronous, but if you're using an online service to implement these (i.e. LLama.cpp server) then these need to be able to return async responses. Having it return ValueTask would be greatly helpful.
Inversely GenerateEmbeddingAsync could be ValueTask as well.
Beta Was this translation helpful? Give feedback.
All reactions