feat: add LiteLLM support #549

aleclarson · 2024-04-11T22:19:21Z

🚨 Feedback wanted!

When all of --openai-api-key, --openai-api-base, and --openrouter are not passed and the user has installed LiteLLM with pip install litellm, Aider will default to using the LiteLLM client, which has support for many models, including Claude 3, Gemini 1.5 Pro, and Command R+.

You also need to install model-specific clients, which LiteLLM will use under the hood. For example, you'll need to pip install anthropic to use Claude 3. To use Gemini 1.5 Pro, you'll need to pip install google-generativeai first.

Since LiteLLM loads the nearest .env file, it's recommended to place your model-specific API key in there. For example, LiteLLM expects an ANTHROPIC_API_KEY environment variable to exist when using Claude 3. Of course, you can define the environment variable however you please, so an .env file is optional. Find your preferred model on the LiteLLM supported models page to determine which environment variable is expected by LiteLLM based on which model you're using.

Does this add dependencies to Aider?
No, the user must run pip install litellm for Aider to support LiteLLM.
Can I use any model supported by LiteLLM?
Yes, the supported models are loaded directly from LiteLLM's Github repository. As long as you keep LiteLLM up-to-date, any new models can be used as long as LiteLLM supports them.
What model is used for the summarizer and repo-map?
If you're using Claude 3 Opus or GPT-4, Aider will use GPT-3.5 Turbo for the chat summarizer and repo-map features. Otherwise, the --model you defined is what's used.
Has this been tested extensively?
No, I'm looking for people to try it out and leave feedback on this pull request.

Model aliases

When defining the --model name, you can use these aliases:

# claude-3
opus   => claude-3-opus-20240229
sonnet => claude-3-sonnet-20240229
haiku  => claude-3-haiku-20240307

# gemini-1.5-pro
gemini => gemini-1.5-pro-preview-0409

# gpt-3.5
gpt-3.5           => gpt-3.5-turbo-0613
gpt-3.5-turbo     => gpt-3.5-turbo-0613
gpt-3.5-turbo-16k => gpt-3.5-turbo-16k-0613

# gpt-4
gpt-4     => gpt-4-0613
gpt-4-32k => gpt-4-32k-0613

The gpt-3.5 and gpt-4 aliases were already supported, but they're also supported when using the --litellm flag or when LiteLLM is used by default.

Additional improvements

I've added some environment variables to improve .env file support, so I can run aider with dotenv run aider to use them:

the AIDER_MODEL environment variable is now supported
the AIDER_EDIT_FORMAT environment variable is now supported

Each model provided by LiteLLM has its own environment variable for the API key. It‘s recommended to create a .env file and place your API key there, which LiteLLM will automatically load. Environment variables for popular LLMs: - ANTHROPIC_API_KEY - GEMINI_API_KEY - COHERE_API_KEY - MISTRAL_API_KEY

When the `model` is defined, but the openai-api-key or openai-api-base values are undefined, assume the user wants to use --litellm (but only if the litellm package can be found).

paul-gauthier · 2024-04-12T21:19:24Z

This is looking really great, thanks for putting in all the work!

My main question is why not just use litellm, and ditch the openai client? Direct calls to openai are handled by litellm, as are OpenRouter and Azure. I had been planning on going down that path with a litellm integration.

nkeilar · 2024-04-15T11:43:40Z

aider/models/model.py


+ if client and not hasattr(client, "base_url"):


Not sure about this... I setup LiteLLM in docker on a VM for my team to use, the idea being we can setup the models in one place any everyone has access (and all our projects), but this would require setting a URL or I guess a hacky port forward, would be nice to be able to set a URL for litellm in-case its not running locally.

This condition doesn't preclude setting a base_url for LiteLLM, but I understand why you'd think that. It's just duck-typing to detect LiteLLM client, which doesn't have a base_url property. There's probably a better way (with more clarity) to detect that, so I'll see what I can do.

aleclarson · 2024-04-15T17:10:43Z

My main question is why not just use litellm, and ditch the openai client?

That could make sense in the future. For now, it's better to make the transition smooth for everyone by still allowing the previous usage, just in case there's a subtle difference between the two approaches? Also, I figured it was easier to get a less drastic change merged sooner.

This prevents any confusion about the check‘s purpose.

aleclarson · 2024-04-16T18:53:10Z

aider/main.py

+ # Support LITELLM_API_KEY, LITELLM_BASE_URL, etc.
+ litellm_kwargs = {
+ key.replace("LITELLM_", "").lower(): value
+ for key, value in os.environ.items()
+ if key.startswith("LITELLM_")
+ }
+
+ from litellm import LiteLLM
+ client = LiteLLM(**litellm_kwargs)


@nkeilar please try setting LITELLM_BASE_URL environment variable and let me know if that works for you

LITELLM_BASE_URL=http://localhost:4000 aider --model gemini

paul-gauthier · 2024-04-17T22:59:21Z

I really appreciate the work you put in here. Just wanted to give you a heads up that I've also got a litellm branch going. I am leaning towards a wholesale replacement of the openai client with litellm. I'm far enough into my branch now that it looks feasible.

You can see my work in progress in #564.

aleclarson added 3 commits April 11, 2024 18:04

feat: default to litellm in certain cases

6804028

When the `model` is defined, but the openai-api-key or openai-api-base values are undefined, assume the user wants to use --litellm (but only if the litellm package can be found).

fix: cache the litellm supported models data

8e0ef61

aleclarson mentioned this pull request Apr 11, 2024

Support for other LLMs, local LLMs, etc #172

Closed

fix: don‘t set --model default until later

3e28efe

aleclarson force-pushed the feat/litellm branch from 4cfbb70 to 6c4660f Compare April 12, 2024 00:27

fix: gemini 1.5 pro alias

399bc7e

aleclarson force-pushed the feat/litellm branch from 6c4660f to 399bc7e Compare April 12, 2024 00:33

aleclarson added 4 commits April 12, 2024 16:42

fix: remove Model.create calls from aider/models/__init__.py

e83bf6e

fix: reuse litellm model_cost map and update its backup periodically

edd88e6

chore: clean up

fa4cbaf

fix: stop using pkg_resources

41ca019

aleclarson force-pushed the feat/litellm branch from d1ce887 to 41ca019 Compare April 12, 2024 20:53

nkeilar reviewed Apr 15, 2024

View reviewed changes

aleclarson added 4 commits April 16, 2024 14:50

fix: set LiteLLM log level to ERROR

2484e6a

feat: support LITELLM_API_KEY, LITELLM_BASE_URL, etc

ad53881

fix: check client typename instead of duck typing

aa8bca6

This prevents any confusion about the check‘s purpose.

fix: import order issue

164e4af

aleclarson commented Apr 16, 2024

View reviewed changes

fix: stop using every LITELLM_ variable

5613cc9

paul-gauthier closed this Apr 27, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: add LiteLLM support #549

feat: add LiteLLM support #549

aleclarson commented Apr 11, 2024 •

edited

Loading

paul-gauthier commented Apr 12, 2024

nkeilar Apr 15, 2024

aleclarson Apr 15, 2024

aleclarson commented Apr 15, 2024

aleclarson Apr 16, 2024 •

edited

Loading

paul-gauthier commented Apr 17, 2024

feat: add LiteLLM support #549

feat: add LiteLLM support #549

Conversation

aleclarson commented Apr 11, 2024 • edited Loading

🚨 Feedback wanted!

Model aliases

Additional improvements

paul-gauthier commented Apr 12, 2024

nkeilar Apr 15, 2024

Choose a reason for hiding this comment

aleclarson Apr 15, 2024

Choose a reason for hiding this comment

aleclarson commented Apr 15, 2024

aleclarson Apr 16, 2024 • edited Loading

Choose a reason for hiding this comment

paul-gauthier commented Apr 17, 2024

aleclarson commented Apr 11, 2024 •

edited

Loading

aleclarson Apr 16, 2024 •

edited

Loading