You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Currently, Tokenizer returns counts strictly based on tokens.
However, there are scenarios where library authors may want / need to implement their own custom token counting function.
One such scenario is providing token counts for image inputs. In such cases, AI service providers provide an arbitrary way of calculating cost based on a fixed token count.
Currently, Tokenizer returns counts strictly based on tokens.
However, there are scenarios where library authors may want / need to implement their own custom token counting function.
One such scenario is providing token counts for image inputs. In such cases, AI service providers provide an arbitrary way of calculating cost based on a fixed token count.
The text was updated successfully, but these errors were encountered: