Total completion token count

[oferavnery]

With the new functions defined as part of the request, combined with streaming output, it is becoming very tricky and probably inaccurate to measure total messages + functions + response tokens. I'm building an app that charges end users by their token usage and I want to be as close as possible to what OpenAI logs and charges.

The wish is to incorporate a way to easily get to those total response / request token counts both in streaming and standard mode.

Thanks! Awesome progress!

[oferavnery]

Anybody else is looking to get total token count including request and response as part of the API functionality?

[kayhantolga]

Usage should be available in response messages. Are you looking for something else?

[oferavnery]

I don't think it's available in streaming at all. Also, tokens should be counted for the Request body as well as for the response, including functions and messages, and I don't think there's a good way to do that too.

[kayhantolga]

You are correct, the Stream Endpoint does not provide usage information. I believe that PromptTokens refers to the request tokens, while CompletionTokens refers to the response tokens, which are used to calculate overall usage.

Another option is to use the Tokenizer, but currently, it is not working perfectly.

[richardtallent-erm]

Is there a way to access the raw JSON sent and received, so we can use a tokenizer to get a more accurate token count? The prompt text and completion text isn't the whole story, apparently.