v1.18.4

What's Changed

[Feat] Proxy - Add Spend tracking logs by @ishaan-jaff in #1498

New SpendTable when Using LiteLLM Virtual Keys - Logs API Key, CreatedAt Date + Time, Model, Spend, Messages, Response
Docs to get started: https://docs.litellm.ai/docs/proxy/virtual_keys

[Feat] Proxy - Track Cost Per User (Using user passed to requests) by @ishaan-jaff in #1509

• Proxy Server Track Cost Per User
  Request:

curl --location 'http://0.0.0.0:8000/chat/completions' \
        --header 'Content-Type: application/json' \
        --header 'Authorization: Bearer sk-RwPq' \
        --data ' {
        "model": "BEDROCK_GROUP",
        "user": "litellm-is-awesome-user",
        "messages": [
            {
            "role": "user",
            "content": "what llm are you-444"
            }
        ],
        }'

Cost Tracked in LiteLLM Spend Tracking DB

Notes:

• If a user is passed to the request the proxy tracks cost for it
• If the user does not exist in the User Table, we make a new user with the spend

feat(parallel_request_limiter.py): add support for tpm/rpm rate limits for keys by @krrishdholakia in #1501

Full Changelog: v1.18.3...v1.18.4

v1.18.3

What's Changed

• [Feat] /key/generate - create keys withteam_id by @ishaan-jaff in #1500
  Set team_id when creating users, keys
  docs: https://docs.litellm.ai/docs/proxy/virtual_keys#request

curl 'http://0.0.0.0:8000/key/generate' \
--header 'Authorization: Bearer <your-master-key>' \
--header 'Content-Type: application/json' \
--data-raw '{
  "models": ["gpt-3.5-turbo", "gpt-4", "claude-2"],
  "team_id": "core-infra"
}'

• (fix) read azure ad token from optional params extra body by @krrishdholakia in e0aaa94

Full Changelog: v1.18.2...v1.18.3

v1.18.2

What's Changed

• [Test+Fix] /Key/Info, /Key/Update - Litellm unit test key endpoints by @ishaan-jaff in #1496
• fix(ollama_chat.py): use tiktoken as backup for prompt token counting by @puffo in #1495
• fix(parallel_request_limiter.py): decrement count for failed llm calls by @krrishdholakia in 1ea3833
• fix(proxy_server.py): show all models user has access to in /models by @krrishdholakia in c8dd36d

New Contributors

• @puffo made their first contribution in #1495

Full Changelog: v1.18.1...v1.18.2

v1.18.1

What's Changed

• Altered requirements.txt to require pyyaml 6.0.1 which will resolve #1488 by @ShaunMaher in #1489
• Update s3 cache to support folder by @duarteocarmo in #1494

New Contributors

• @ShaunMaher made their first contribution in #1489
• @duarteocarmo made their first contribution in #1494

Full Changelog: v1.18.0...v1.18.1

v1.18.0

What's Changed

https://docs.litellm.ai/docs/simple_proxy

• [Feat] Proxy - Access Key metadata in callbacks by @ishaan-jaff in #1484
  • Access Proxy Key metadata in callbacks
  • Access Endpoint URL in calbacks - you can see if /chat/completions, /embeddings, /image/generation etc is called
  • Support for Langfuse Tags, We log request metadata as langfuse tags

PS. no keys leaked - these are keys to my local proxy

Support for model access groups

Use this if you have keys with access to specific models, and you want to give all them access to a new model.

You can now assign keys access to model groups, and add new models to that group via the config.yaml - https://docs.litellm.ai/docs/proxy/users#grant-access-to-new-model

curl --location 'http://localhost:8000/key/generate' \
-H 'Authorization: Bearer <your-master-key>' \
-H 'Content-Type: application/json' \
-d '{"models": ["beta-models"], # 👈 Model Access Group
            "max_budget": 0,}'

Langfuse Tags logged:

* feat(proxy_server.py): support model access groups by @krrishdholakia in https://github.com//pull/1483

Full Changelog: v1.17.18...v1.18.0

What's Changed

• [Feat] Proxy - Access Key metadata in callbacks by @ishaan-jaff in #1484
• feat(proxy_server.py): support model access groups by @krrishdholakia in #1483

Full Changelog: v1.17.18...v1.18.0

v1.17.18

What's Changed

• [Fix+Test] /key/delete functions by @ishaan-jaff in #1482 Added extensive testing + improved swagger

Full Changelog: v1.17.17...v1.17.18

v1.17.17

What's Changed

• [Test] Proxy - Unit Test proxy key gen by @ishaan-jaff in #1478

Testing + fixes for: https://docs.litellm.ai/docs/proxy/virtual_keys

1. Generate a Key, and use it to make a call
2. Make a call with invalid key, expect it to fail
3. Make a call to a key with invalid model - expect to fail
4. Make a call to a key with valid model - expect to pass
5. Make a call with key over budget, expect to fail
6. Make a streaming chat/completions call with key over budget, expect to fail
7. Make a call with an key that never expires, expect to pass
8. Make a call with an expired key, expect to fail

Full Changelog: v1.17.16...v1.17.17

v1.17.16

Full Changelog: v1.17.15...v1.17.16

v1.17.15

What's Changed

• [Feat] Support Azure GPT-4 Vision Enhancements by @ishaan-jaff in #1475

Usage - with Azure Vision enhancements

Docs: https://docs.litellm.ai/docs/providers/azure#usage---with-azure-vision-enhancements

Note: Azure requires the base_url to be set with /extensions

Example

base_url=https://gpt-4-vision-resource.openai.azure.com/openai/deployments/gpt-4-vision/extensions
# base_url="{azure_endpoint}/openai/deployments/{azure_deployment}/extensions"

Usage

import os 
from litellm import completion

os.environ["AZURE_API_KEY"] = "your-api-key"

# azure call
response = completion(
            model="azure/gpt-4-vision",
            timeout=5,
            messages=[
                {
                    "role": "user",
                    "content": [
                        {"type": "text", "text": "Whats in this image?"},
                        {
                            "type": "image_url",
                            "image_url": {
                                "url": "https://avatars.githubusercontent.com/u/29436595?v=4"
                            },
                        },
                    ],
                }
            ],
            base_url="https://gpt-4-vision-resource.openai.azure.com/openai/deployments/gpt-4-vision/extensions",
            api_key=os.getenv("AZURE_VISION_API_KEY"),
            enhancements={"ocr": {"enabled": True}, "grounding": {"enabled": True}},
            dataSources=[
                {
                    "type": "AzureComputerVision",
                    "parameters": {
                        "endpoint": "https://gpt-4-vision-enhancement.cognitiveservices.azure.com/",
                        "key": os.environ["AZURE_VISION_ENHANCE_KEY"],
                    },
                }
            ],
)

Full Changelog: v1.17.14...v1.17.15

v1.17.14

Fixes bug for mistral ai api optional param mapping

Full Changelog: v1.17.13...v1.17.14