add custom open ai models to asyncio call

[PrathamSoni]

No description provided.

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
litellm	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Nov 10, 2023 5:03am

[ishaan-jaff]

@PrathamSoni did you test this locally with anyscale ?

[PrathamSoni]

Yeah, it fixes the FD error asw:

import psutil
%load_ext dotenv
%dotenv

def print_open_files():
    pid = os.getpid()
    proc = psutil.Process(pid)

    print(f"Number of file descriptors for process {pid}: {proc.num_fds()}")

import requests
import litellm
import asyncio
# litellm.set_verbose=True


async def make_request(i):
    response = await litellm.acompletion(
        model="anyscale/mistralai/Mistral-7B-Instruct-v0.1",
        messages=[
                {"role": "system", "content": "You're a chat bot"},
                {"role": "user", "content": "Whats 1+1"},
            ]
    )

    return response

print("init")
print_open_files()

for i in range(10):
    with requests.Session() as custom_session:
        adapter = requests.adapters.HTTPAdapter(pool_connections=100, max_retries=3)
        custom_session.mount('http://', adapter)
        custom_session.mount('https://', adapter)
        litellm.client_session = custom_session
        tasks = [make_request(j) for j in range(10)]
        res = await asyncio.gather(*tasks)
    print_open_files()

print_open_files()

init
Number of file descriptors for process 77355: 74
Number of file descriptors for process 77355: 83
Number of file descriptors for process 77355: 75
Number of file descriptors for process 77355: 76
Number of file descriptors for process 77355: 76
Number of file descriptors for process 77355: 75
Number of file descriptors for process 77355: 78
Number of file descriptors for process 77355: 75
Number of file descriptors for process 77355: 75
Number of file descriptors for process 77355: 77
Number of file descriptors for process 77355: 76
Number of file descriptors for process 77355: 76

Update, don't actually need to have the custom_session either.

[ishaan-jaff]

lgtm ! our ci/cd pipeline has been having some trouble so will try to get this in the pypi package in 12 hrs