[22/01/2024 - 29/01/2024] New Models/Endpoints/Providers/Improvements

[ishaan-jaff]

Budget Management

• Add testing for proxy_server.track_cost_callback - writing to SpendLogs Table Add testing for proxy_server.track_cost_callback - writing to SpendLogs Table[Feature]:  #1584
• Allow Users to Set Budgets per user per month [Feature]: Allow Users to Set Budgets per user per month #1544
• See (By User) spend this month [Feature]: (admin ui) See (By User) spend this month #1545
• See (By User) to see total requests, Total tokens used (completion, embedding), spend,
• Set Global Budget For the Proxy + Reset the budget [Feature]: Proxy - Spend Tracking, Set Global Budget For the Proxy + Reset the budget  #1585
• Allow key budgets to be reset [Feature]: Allow key budgets to be reset  #1567
• Update Budgets per user (allow users to reset budgets) [Feature]: Update Budgets per user (allow users to reset budgets)  #1546
• Allow setting budgets per key [Feature]: Allow setting budgets per key  #1548
• Allow setting tpm/rpm limits per key [Feature]: Allow setting tpm/rpm limits per key  #1550
• Slack Alerts for Budgets [Feature]: Add slack alerting when Keys near their max budget #1612
• End User Emails when a key is near its budget

Spend Tracking

• Support input/output cost per second - custom pricing [Feature]: Support input/output cost per second - custom pricing  #1374
• Emit spend as part of logging call [Feature]: Emit spend as part of logging call #1547
• Log token usage in spend logs [Feature]: Track token usage per key (spend logs?)  #1549
• Get spend logs by key [Feature]: Get spend logs by key  #1582
• Allow specifying key alias [Feature]: Allow specifying key alias (view-only)  #1624
• Support bedrock region pricing in completion_cost - [Feature]: Support bedrock region pricing in completion_cost()  #1565
• Completion cost support for image gen models - [Feature]: completion_cost() support for image generation  #1558
• add Region-specific amazon text embedding pricing - [Feature]: Add region-specific amazon text embedding pricing to model cost map  #1566
• Allow local model cost maps for proxy [Feature]: local-only load model cost #1434
• Allow users to access litellm ui + proxy using docker compose up [Feature]: Make it easy to use UI with Proxy - Static Web App #1631

Misc

• real Sagemaker streaming support - [Feature]: Support real sagemaker streaming #1384
• Gemini Pro - VertexAI function calling when streaming support - [Feature]: Support vertex ai function calling w/ streaming #1263
• Allow users to add their own logic to /key/generate [Feature]: Proxy - Allow users to add their own logic to /key/generate #1462
• Caching improvement for embedding calls - [Feature]: caching individual texts from batch call #1350
• Make Authentication Errors OpenAI compatible: [Feature]: Make Proxy Authentication Errors OpenAI Compatible  #1554
• Auto infer known model names for health checks: [Feature]: auto-infer model mode for health checks (if known)  #1580
• Support vertex project and location being set dynamically: [Feature]: Support vertex location and project being passed in completion call  #1613
• Reduce proxy cold start times - [Feature]: Reduce proxy cold start times  #1636

[krrishdholakia]

Known Backlog:

1. OpenMeter spend tracking support - [Feature]: Add openmeter support for spend tracking across llm providers #1268
2. tracking - [Feature]: Support AirLLM #997
3. tracking - [Feature]: Integrate with mlx #1188 cc: @r1cc4rd0m4zz4
4. tracking - [Feature]: Support exllama v2  #1192
5. yandexgpt support requested by 2 users - [Feature]: Add support for YandexGPT  #1254
6. Aphrodite engine endpoint - https://github.com/PygmalionAI/aphrodite-engine, feat: add aphrodite support #1153
7. TheBloke/NexusRaven-V2-13B-GGUF function calling support [Feature]: Support Nexusflow/NexusRaven-V2-13B function calling  #1060 cc: @homanp
8. Predibase lorax support - [Feature]: Add support for Predibase Lorax apis [Feature]: Add support for Predibase Lorax apis  #1253
9. Deepsparse CPU inference support - docker access denied error neuralmagic/deepsparse#1452
10. Consistent format for model prices and context window - [Feature]: Consistent format for model_prices_and_context_window file  #1375
11. Vertex AI model garden support - Model Garden custom endpoint support? [Feature]:  #920

[ishaan-jaff]

New one here: #1665