AI does not support receiving input and output tokens used

[rpostulart]

Is your feature request related to a problem? Please describe.
I would like to track and limit the token usage per person. Therefor I need to receive the used tokens per input and output

Describe the solution you'd like
I want meta data to be returned:

"usage": {
"inputTokens": number,
"outputTokens": number,
"totalTokens": number
}

in the docs: https://docs.aws.amazon.com/bedrock/latest/APIReference/API_runtime_ConverseStream.html#API_runtime_ConverseStream_ResponseElements

[atierian]

Thanks for opening this feature request @rpostulart.
I'm transferring it over to the amplify-backend repo.

Providing access to the Bedrock request metadata and allowing per user token based throttling is something we're thinking about.

A few questions to help us better understand your use case:

• Is this for conversation routes, generation routes, or both?
• What criteria are you expecting to use to determine if a user request should be throttled? Is it n tokens within last x time frame? Something else?

[rpostulart]

• It is for conversations!
• I would like to set a max tokens on user level in DynomaDB and for each request if there are sufficient tokens, it not then deny the request, otherwise continue and receive the usedTokens and update the Dynamodb