Support for other LLMs, local LLMs, etc #172

paul-gauthier · 2023-08-02T15:25:31Z

This issue is a catch all for questions about using aider with other or local LLMs. The text below is taken from the FAQ.

Aider provides experimental support for LLMs other than OpenAI's GPT-3.5 and GPT-4. The support is currently only experimental for two reasons:

GPT-3.5 is just barely capable of editing code to provide aider's interactive "pair programming" style workflow. None of the other models seem to be as capable as GPT-3.5 yet.
Just "hooking up" aider to a new model by connecting to its API is almost certainly not enough to get it working in a useful way. Getting aider working well with GPT-3.5 and GPT-4 was a significant undertaking, involving specific code editing prompts and backends for each model and extensive benchmarking. Officially supporting each new LLM will probably require a similar effort to tailor the prompts and editing backends.

Numerous users have done experiments with numerous models. None of these experiments have yet identified other models that look like they are capable of working well with aider.

Once we see signs that a particular model is capable of code editing, it would be reasonable for aider to attempt to officially support such a model. Until then, aider will simply maintain experimental support for using alternative models.

More information

For more information on connecting to other models, local models and Azure models please see the FAQ.

There are ongoing discussions about LLM integrations in the aider discord.

Here are some GitHub issues which may contain relevant information.

Explosion-Scratch · 2023-08-02T15:33:26Z

Why not have a unified API that you could provide then a plugin system to integrate other LLMS, then you can just provide gpt-3.5 and gpt-4 plugins officially yourself.

paul-gauthier · 2023-08-02T16:12:47Z

Per the FAQ and info above in the issue, there are already some useful hooks for connecting to other LLMs. A number of users have been using them to experiment with local LLMs. Most reports haven't been very enthusiastic about how they perform with aider compared to GPT-3.5/4.

Aider does have a modular system for building different "coder" backends, allowing customization of prompts and edit formats.

So all the raw materials seem to be available to get aider working with specific new models. I'm always happy to advise as best I can. And if you see evidence that a particular model has the potential to work well with aider, that would exciting news.

aldoyh · 2023-08-03T05:53:13Z

Great work, has anyone confirmed a successful usage with other local LLMs? It's not about cheapness, we still don't have access to ChatGPT API nor being able to pay for any alternatives.

jmdevall · 2023-08-03T07:35:04Z

hello! great news! there is another contender in the ecosystem of open source llm-coders

Yesterday I tested quantized version https://huggingface.co/TheBloke/NewHope-GGML (The bloke also released GPTQ version). Running in local using ooba textgen and activating opeanai extension. What I read is that is a llama2 model finetuned similar to wizardcoder

Seems to be better than wizardcoder but still needs effort to adjust prompts to run and be usable. Upsss... The group that originally released the model had to remove it because they realized that some data with which the quality was evaluated had slipped into the data with which they trained the model, so that they were giving comparative results that they weren't real. Even so, the quantized model of the_block is still there and can be downloaded and tested.

tmm1 · 2023-08-03T15:37:05Z

The NewHope model was retracted because it was contaminated with test data causing overfit.

https://twitter.com/mathemagic1an/status/1686814347287486464?s=46&t=hIokEbug9Pr72tQFuXVULA

So far none of the models come close to gpt and cannot follow instructions well enough to work with aider.

joshuavial · 2023-08-03T22:31:30Z

@aldoyh

#170 might be of interest - are you able to access the openrouter apis?

strangelearning · 2023-08-04T14:18:40Z

Hey @paul-gauthier, while this question isn’t directly related to using other LLMs, I was wondering your advice for where to poke around to embed additional context into the prompt.

My friend and I are putting together a context embedding for some relevant up to date developer documentation and would love to try aider in conjunction with that context.

Thanks for this tool!

aldoyh · 2023-08-04T16:52:09Z

To reply I had to research that and it's my answer.
No, didn't know about it and going to check it out.. but even if I deployed a LocalAI, isn't there a way to make Aider look at it?

apcameron · 2023-08-04T22:47:09Z

@aldoyh Have a look at my comments in #138

JimmysDev · 2023-08-12T21:31:18Z

Appreciate your work on this project. This represents one of the biggest missing pieces between LLM and real code writing utility, so well done.

I read the bits about how hard it is to add new models, I just want to request you take a look at Claude 2. v2 added some big improvements on the code side, and while it is still dumber than GPT-4, it has a more recent knowledge cutoff. Plus massive context.

I understand you are working around context limitations with ctags, but it could be interesting to see if there is an advantage to being able to load the entire project in context with Claude. For example, it may be better at answering high level questions, or writing features that are described in more abstract terms. But regardless, I think that Claude is hot on the heels of GPT-4, and if the reporting on it being a 52B model is true then it is already significantly smarter (pound for pound).

Just my 2c anyway

paul-gauthier · 2023-08-14T10:59:12Z

I agree that the Claude models sound like they are the most likely to be capable of working well with aider. I have been waiting for an api key for months unfortunately. My impression is that it is very difficult to get a key, which limits the benefits of integrating Claude into aider. Not many folks could use it.

linvest21 · 2023-08-14T12:01:01Z

Paul, perhaps try openrouter, which seem sidestep the key issue, and give access to claude directly..

…

On Mon, Aug 14, 2023, 6:59 AM paul-gauthier ***@***.***> wrote: I agree that the Claude models sound like they are the most likely to be capable of working well with aider. I have been waiting for an api key for months unfortunately. My impression is that it is very difficult to get a key, which limits the benefits of integrating Claude into aider. Not many folks could use it. — Reply to this email directly, view it on GitHub <#172 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/BAW75LIGJJWEWORRNCA4TY3XVIAI5ANCNFSM6AAAAAA3BOLHYA> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

paul-gauthier · 2023-08-14T12:06:50Z

Paul, perhaps try openrouter, which seem sidestep the key issue, and give access to claude directly..

Yes, I am aware of openrouter. But that is a confusing extra layer to explain to users. Most users won't have direct Claude api access. And I won't be able to test aider directly against the Claude api. It's all sort of workable, but far from ideal.

Explosion-Scratch · 2023-08-14T12:28:08Z

Paul, perhaps try openrouter, which seem sidestep the key issue, and give access to claude directly..

Yes, I am aware of openrouter. But that is a confusing extra layer to explain to users. Most users won't have direct Claude api access. And I won't be able to test aider directly against the Claude api. It's all sort of workable, but far from ideal.

I like the idea of an easily extendable system, e.g a flag (--bot llama5) that exports a class with this structure:

export default class LLaMA5 {
   requirements = [
      {
         id: "apiKey",
         name: "API Key",
         type: "string",
         required: true,
      }
   ];
   constructor({apiKey}){
      
   }
   createConversation() --→ ConvoParams
   sendMessage(message, {conversation: ConvoParams, progress, done})
   // Other optional methods such as deleteConversation, deleteMessage, editMessage, retryMessage, etc
}

This would be easily inspectable by Aider to check if this bot supports retrying, editing, etc as well as supporting the required parameters.

paul-gauthier · 2023-08-14T12:43:50Z

https://xkcd.com/927/

Egalitaristen · 2023-08-22T13:49:33Z

When it comes to the coding capabilities of local LLMs, I believe that the HumanEval (pass@1) is the most important metric.

The leaderboard lists "Starcoder-16b" as the best open model with a score of 0.336 compared to GPT-4 score of 0.67.

But we also have 2 GPT-3.5 models at 0.48/0.46. But here's the thing, There's also the "WizardLM-70B-V1.0" which no one has added to the leaderboards yet but it actually has a higher score than GPT-3.5 at 0.506.

I don't have a machine powerful enough to run it but I think that with minor tweaking it should perform as well as GPT-3.5

All this being said, I'm not a dev and I haven't tested any of this and honestly don't fully understand all the steps that autonomous agents like Aider take to get it to work.

Just though that I'd mention it in case it's useful to someone.

And Paul, great work with everything here. Really cool

Edit

There's also the WizardCoder-15B-V1.0 with a score of 0.573 which was what I originally came to inform about but somehow forgot along the way while checking sources.

J-DTurner · 2023-08-23T09:26:33Z

I agree that the Claude models sound like they are the most likely to be capable of working well with aider. I have been waiting for an api key for months unfortunately. My impression is that it is very difficult to get a key, which limits the benefits of integrating Claude into aider. Not many folks could use it.

Hey Paul,

I'd be happy to lend you my API key to use for testing. There's a max of 1 call at a time, so if you can deal with that limitation - all good!

paul-gauthier · 2023-08-23T22:07:40Z

I'd be happy to lend you my API key to use for testing.

Thanks @JamesSKR. I have a loaner API key already. But again, so few people have Claude API access that it's not going to be very impactful to get aider working with Claude. Almost no one could use it. I definitely want to experiment with Claude, but it's not super high priority right now for that reason.

paul-gauthier · 2024-03-08T16:38:21Z

The new claude-3-opus-20240229 model got the highest score ever on aider's benchmark, completing 68.4% of the tasks. While Opus got the highest score, it was only a few points higher than the GPT-4 Turbo results. Given the extra costs of Opus and the slower response times, it remains to be seen which is the most practical model for daily coding use.

https://aider.chat/2024/03/08/claude-3.html

AutoJenny · 2024-03-10T16:29:59Z

Is there a tldr; for this thread? I'd love to use Aider with a local LLM for coding, either through LMstudio or directly. I'm able to put a bit of work into making it work, but would prefer not to spend days on it. What would anyone recommend? I'm on a 64gb Mac Studio btw with a full LAMP stack.

xor2003 · 2024-03-15T23:23:42Z

openhermes-2.5-mistral-7b-16k.Q8_0.gguf kind of working. But it does not provide filename as requested. what I understood: ChatML should be supported by model. So it is models like Dolphin and Open*

kit678 · 2024-04-01T04:15:39Z

Just passing through, and thought to add my 2c:

Interesting project, seems like it would be useable for security conscious companies if it supported private / open LLM's

Managed to get this "sort-of" working with Ollama using the codellama model, in a few minutes.

It's worth checking the network graph - there are forks that are adding features that would simplify adding additional LLMs. eg tastyblob@cdd8b36

Seems like instructor could / should be utilised to do much of the heavy lifting.

/ 2c

@macropin What was your aider syntax for using local model via ollama ? I am trying to run aider with codellama via ollama, but lost at mentioning the model name while invoking aider in cli. I have this so far:
aider --openai-api-base http://localhost:11434 --openai-api-key fake-key --model <?>

What am I missing? Thanks in advance

cosmojg · 2024-04-02T17:19:21Z

Accessing claude-3-opus-20240229 through LiteLLM doesn't seem to work right.

$ aider --openai-api-base "http://0.0.0.0:4000" --edit-format diff
Aider v0.27.0
API key supports: anthropic/claude-3-opus-20240229
API key does not support gpt-4-1106-preview, falling back to gpt-3.5-turbo-0125

$ aider --openai-api-base "http://0.0.0.0:4000" --model anthropic/claude-3-opus-20240229 --edit-format diff
Aider v0.27.0
Traceback (most recent call last):
  File "/Users/cosmoguerini/micromamba/envs/cosmosis/lib/python3.12/site-packages/aider/models/openai.py", line 124, in __init__
    self.tokenizer = tiktoken.encoding_for_model(true_name)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/cosmo/micromamba/envs/cosmosis/lib/python3.12/site-packages/tiktoken/model.py", line 101, in encoding_for_model
    return get_encoding(encoding_name_for_model(model_name))
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/cosmo/micromamba/envs/cosmosis/lib/python3.12/site-packages/tiktoken/model.py", line 88, in encoding_name_for_model
    raise KeyError(
KeyError: 'Could not automatically map anthropic/claude-3-opus-20240229 to a tokeniser. Please use `tiktoken.get_encoding` to explicitly get the tokeniser you expect.'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/cosmo/micromamba/envs/cosmosis/bin/aider", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/Users/cosmo/micromamba/envs/cosmosis/lib/python3.12/site-packages/aider/main.py", line 592, in main
    main_model = models.Model.create(args.model, client)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/cosmo/micromamba/envs/cosmosis/lib/python3.12/site-packages/aider/models/model.py", line 28, in create
    return OpenAIModel(name)
           ^^^^^^^^^^^^^^^^^
  File "/Users/cosmo/micromamba/envs/cosmosis/lib/python3.12/site-packages/aider/models/openai.py", line 126, in __init__
    raise ValueError(f"No known tokenizer for model: {name}")
ValueError: No known tokenizer for model: anthropic/claude-3-opus-20240229

krrishdholakia · 2024-04-02T19:38:07Z

hey @cosmojg the error looks like it's being caused by aider using tiktoken

why not set the model name as gpt-3.5-turbo on aider

and let litellm proxy handle the translation

litellm --model anthropic/claude-3-opus-20240229

You can call it any model on aider, and it'll automatically translate it to anthropic claude 3 for you

garthtrickett · 2024-04-03T12:52:58Z

@paul-gauthier

https://github.com/leona/helix-gpt/blob/master/src/providers/github.ts

This add-on lets you use copilot-x's chat feature within helix editor.
I think copilot chat uses gpt-4 under the hood as the github.ts file contains
model: "gpt-4",
If this could be added to aider then we would be able to do unlimited calls to the API for the fixed cost of the github copilot x subscription.

nkeilar · 2024-04-03T13:31:17Z

@paul-gauthier

https://github.com/leona/helix-gpt/blob/master/src/providers/github.ts

This add-on lets you use copilot-x's chat feature within helix editor.

I think copilot chat uses gpt-4 under the hood as the github.ts file contains
model: "gpt-4",
If this could be added to aider then we would be able to do unlimited calls to the API for the fixed cost of the github copilot x subscription.

Wow! I'm already paying for this but prefer to use aider. The main reason I don't use aider more is because of the cost using OpenAI gpt-4 tokens. This would let us legit use copilot. Mind blown.

cosmojg · 2024-04-03T18:13:54Z

@krrishdholakia Wow, thanks! Hilariously, you made me realize that I was having a bit of a brainfart. I had already tried setting the model to gpt-3.5-turbo, saw that it worked, but worried that the --model flag would override the --edit-format flag and force the use of aider's wholefile prompts instead of the udiff or editblock prompts. Your reply inspired me to go and try it again, and I only now just noticed the line confirming the use of the specified format: Model: gpt-3.5-turbo using diff edit format

For my fellow fishsticks, I recommend setting up the following Fish alias to use Aider with Claude via LiteLLM:

alias -s 'cider' 'aider --openai-api-base "http://0.0.0.0:4000" --model "gpt-3.5-turbo" --edit-format "diff"'

rambalachandran · 2024-04-05T11:47:45Z

One straightforward way to support a wide range of LLMs might be to support OpenRouter ( https://openrouter.ai/)

kit678 · 2024-04-06T00:50:11Z

One straightforward way to support a wide range of LLMs might be to support OpenRouter ( https://openrouter.ai/)

aider supports openrouter. has been doing so for quite sometime.

rambalachandran · 2024-04-06T11:07:39Z

Sorry just started using aider. Its docs just asked for OPENAI_API_KEY. So assumed it just supported openai.

cosmojg · 2024-04-09T21:35:48Z

@krrishdholakia Alas, it seems that running aider with --model "gpt-3.5-turbo fails to properly generate a working repomap for Claude to reference. Fortunately, I've come up with a better workaround (at least until Claude is officially supported).

After installing aider with pip, navigate to the directory in which it was installed, and edit the openai_aliases dictionary on line 108 of ../site-packages/aider/models/openai.py like so:

Before:

https://github.com/paul-gauthier/aider/blob/3ab49f1df351d947d76d5e333326f9fd416e5f20/aider/models/openai.py#L108-L116

After:

openai_aliases = {
    # claude-3-opus
    "anthropic/claude-3-opus-20240229": "gpt-4-vision-preview",
    # gpt-3.5
    "gpt-3.5-turbo": "gpt-3.5-turbo-0613",
    "gpt-3.5-turbo-16k": "gpt-3.5-turbo-16k-0613",
    # gpt-4
    "gpt-4-turbo-preview": "gpt-4-0125-preview",
    "gpt-4": "gpt-4-0613",
    "gpt-4-32k": "gpt-4-32k-0613",
}

After adding the above dictionary entry for Claude, you should be able to run the following without issue:

aider --openai-api-base "http://0.0.0.0:4000" --model "anthropic/claude-3-opus-20240229"

Not only will this result in a properly generated repomap, but it will also set sane defaults with regard to context size (i.e., ~100,000 tokens) and edit format (i.e., diff edit blocks). I haven't tested whether images work like they do with gpt-4-vision-preview, but in theory, if litellm can handle the API translation, I think they might!

aleclarson · 2024-04-11T22:20:21Z

Looking for testers

See #549

This PR adds LiteLLM support. See the PR for more details.

What is LiteLLM?
It's a Python library that gives you wayyy more models to choose from (including Claude 3, Gemini 1.5 Pro, and Command R+) by generalizing all of them to have the same API as OpenAI's client.

How does your PR work?
By running pip install litellm, you'll have access to every model supported by LiteLLM when using Aider.

e-nouri · 2024-04-15T11:28:20Z

Thank you @paul-gauthier for this. Is there a way to use AWS Bedrock deployed models with aider ?

paul-gauthier · 2024-04-15T19:58:50Z

@e-nouri not directly, unless Bedrock provides an openai compatible api?

e-nouri · 2024-04-23T13:25:50Z

Is it possible to have all the endpoints that are used from the openai api ? this is the openapi for openai :D https://raw.githubusercontent.com/openai/openai-openapi/master/openapi.yaml so many routes to implement on aws using aws services S3, Bedrock, etc. I am looking through the code meanwhile.

krrishdholakia · 2024-04-23T14:47:14Z

@paul-gauthier i noticed you use litellm - we already support openai-compatible bedrock - https://docs.litellm.ai/docs/providers/bedrock

paul-gauthier · 2024-04-23T18:00:35Z

Yes, the latest version of aider connects to providers using litellm. So openai-compatible bedrock should work out of the box.

See aider's LLM docs here: https://aider.chat/docs/llms.html

paul-gauthier · 2024-04-29T14:29:47Z

FYI, I'm thinking about closing this issue. In the last couple of months there has been an explosion of much more capable models, both in the cloud and for local use. And Aider now has first class support for working with many of them as of v0.29.0.

keesj · 2024-04-29T14:51:55Z

I have been watching this issue over the last couple of month. IMHO litlelm is a bit of a hack. I would like to see direct support from ollama with some suggested models/prompts

keesj · 2024-04-29T14:53:32Z

I stand corrected. this is now well documented in here https://aider.chat/docs/llms.html#ollama

krrishdholakia · 2024-04-29T15:03:59Z

hi @keesj would love feedback on this - any specific reason why you think litellm is not a good solution for your use-case?

Would love to improve here.

paralin · 2024-04-29T15:48:51Z

Now that litellm and opus are working I'd say this could be closed as completed! Other models can be new issues.

joshuavial · 2024-04-30T00:09:43Z

agree with closing the issue, with ollama and litellm folks have got plenty of options for interfacing with local models and the work of making coders performant with individual models is too large and ongoing for one ticket.

keesj · 2024-04-30T07:03:37Z

hi @keesj would love feedback on this - any specific reason why you think litellm is not a good solution for your use-case?

Would love to improve here.

I am quite sure this is a personal preference and probably not the right issue to discuss but here we are. In my (few) experiments running litellm was this extra magic step I did to get things working but the extra level of abstraction and OPENAI API combability and not on top of my list. I think I would not feel that way if litellm was integrated (e.g. was not running separately). I hope this helps.

paul-gauthier · 2024-04-30T15:01:02Z

Ok, I'm going to close this issue. Feel free to open new issues related to aider working with specific other LLMs. But of course, please check the docs which have a fair amount of useful info on this topic:

https://aider.chat/docs/llms.html

paul-gauthier added the enhancement New feature or request label Aug 2, 2023

This was referenced Aug 2, 2023

Issues getting oobabooga/text-generation-webui to work with aider #162

Closed

Module to use Anthropic's Claude #147

Closed

No more extra pay for chatgpt please #138

Closed

try aider with anthropic claude-instant-100k #7

Closed

paul-gauthier mentioned this issue Aug 2, 2023

Prompting strategies #110

Closed

paul-gauthier pinned this issue Aug 2, 2023

joshuavial mentioned this issue Aug 3, 2023

Is there a way to keep existing code local? #176

Closed

This was referenced Aug 4, 2023

Need support for llama2 70B or wizardCode #179

Closed

where to poke around to embed additional context into the prompt. #180

Closed

joshuavial mentioned this issue Aug 7, 2023

Thoughts on internal architecture #181

Closed

apcameron mentioned this issue Aug 21, 2023

Adding support to run without API Key. #212

Closed

aleclarson mentioned this issue Apr 8, 2024

Anthropic without OpenRouter middle man #538

Closed

paul-gauthier unpinned this issue Apr 29, 2024

paul-gauthier closed this as completed Apr 30, 2024

Support for other LLMs, local LLMs, etc #172

Support for other LLMs, local LLMs, etc #172

Comments

paul-gauthier commented Aug 2, 2023 • edited Loading

More information

Explosion-Scratch commented Aug 2, 2023

paul-gauthier commented Aug 2, 2023 • edited Loading

aldoyh commented Aug 3, 2023

jmdevall commented Aug 3, 2023 • edited Loading

tmm1 commented Aug 3, 2023

joshuavial commented Aug 3, 2023

strangelearning commented Aug 4, 2023

aldoyh commented Aug 4, 2023

apcameron commented Aug 4, 2023

JimmysDev commented Aug 12, 2023 • edited Loading

paul-gauthier commented Aug 14, 2023

linvest21 commented Aug 14, 2023 via email

paul-gauthier commented Aug 14, 2023

Explosion-Scratch commented Aug 14, 2023 • edited Loading

paul-gauthier commented Aug 14, 2023

Egalitaristen commented Aug 22, 2023 • edited Loading

J-DTurner commented Aug 23, 2023

paul-gauthier commented Aug 23, 2023

paul-gauthier commented Mar 8, 2024

AutoJenny commented Mar 10, 2024

xor2003 commented Mar 15, 2024

kit678 commented Apr 1, 2024

cosmojg commented Apr 2, 2024 • edited Loading

krrishdholakia commented Apr 2, 2024

garthtrickett commented Apr 3, 2024

nkeilar commented Apr 3, 2024

cosmojg commented Apr 3, 2024 • edited Loading

rambalachandran commented Apr 5, 2024

kit678 commented Apr 6, 2024

rambalachandran commented Apr 6, 2024

cosmojg commented Apr 9, 2024 • edited Loading

Before:

After:

aleclarson commented Apr 11, 2024 • edited Loading

Looking for testers

e-nouri commented Apr 15, 2024

paul-gauthier commented Apr 15, 2024

e-nouri commented Apr 23, 2024

krrishdholakia commented Apr 23, 2024

paul-gauthier commented Apr 23, 2024

paul-gauthier commented Apr 29, 2024

keesj commented Apr 29, 2024

keesj commented Apr 29, 2024

krrishdholakia commented Apr 29, 2024 • edited Loading

paralin commented Apr 29, 2024

joshuavial commented Apr 30, 2024

keesj commented Apr 30, 2024 • edited Loading

paul-gauthier commented Apr 30, 2024

paul-gauthier commented Aug 2, 2023 •

edited

Loading

paul-gauthier commented Aug 2, 2023 •

edited

Loading

jmdevall commented Aug 3, 2023 •

edited

Loading

JimmysDev commented Aug 12, 2023 •

edited

Loading

Explosion-Scratch commented Aug 14, 2023 •

edited

Loading

Egalitaristen commented Aug 22, 2023 •

edited

Loading

cosmojg commented Apr 2, 2024 •

edited

Loading

cosmojg commented Apr 3, 2024 •

edited

Loading

cosmojg commented Apr 9, 2024 •

edited

Loading

aleclarson commented Apr 11, 2024 •

edited

Loading

krrishdholakia commented Apr 29, 2024 •

edited

Loading

keesj commented Apr 30, 2024 •

edited

Loading