[Feature]: Proxy - Spend Tracking, Set Global Budget For the Proxy + Reset the budget

[ishaan-jaff]

The Feature

• Set, Reset budgets for Global Proxy LLM Calls
  • Global Level Budget Management for the Proxy was the most important to him
  • Said user level budgets was a nice to have

Motivation, pitch

Twitter / LinkedIn details

No response

[krrishdholakia]

Need to:

• store this value in a db table
• use this value in user_api_key_auth
• update this value in track_cost_callback

[krrishdholakia]

Thinking

Approach 1:

• Save the max budget in the config table (also add a reset at, and budget duration columns to it)

Approach 2:

• Save the max budget for the proxy in the tokens table (lets us extend usage for proxy to globally support all controls that keys currently do)

Concerns:

• approach 2 would put a token in the db. This means somebody could use that token to auth into the proxy and make llm api calls. Ideally we could track this spend, but not have this be a key you can use to auth into the proxy.

Approach 3:

• Save the max budget for the proxy in the user table -> same benefits of token table, but can't be used to make llm api calls.

---

Concern: How do we prevent slowing things down a lot with 3 queries in user api key auth (user budget, key budget, proxy budget)

Solution: Looks like we can run a raw query in user api key auth - RobertCraigie/prisma-client-py#673 (comment)

This should let us get the user, key and proxy budget in 1 query

[krrishdholakia]

Initial PR which adds user based budgets which reset is merged.

Now need to add:

• cost tracking support for each call
• limiting support in user_api_key_auth

[krrishdholakia]

This is now live in v1.19.2.

Docs - https://docs.litellm.ai/docs/proxy/users#set-budgets