openrouter.ai api

[ixiion001]

Dear Paul please consider options to integrate.

openrouter.ai
https://openrouter.ai/docs
since their api enables to use gpt-4-32k and claude-v2 api, with no additional cost.

thanks

[paul-gauthier]

Agreed, this looks like a potentially very useful integration.

[joshuavial]

I've done some experimenting and gotten aider to work against the openrouter api - main...joshuavial:aider:openrouter-hacks has some quick and dirty changes that let me mess around

Main noticings are

• llama 2 wasn't working, would break after generating a few lines
• google palm 2 codechat generated nice code, but really didn't listen to directions about file format that much, my gut feel is it will need a custom coder, but could perform well against it. No streaming so it felt slower, even though it was pretty quick
• claude v2 could create diffs fairly reliably and had good speed, I didn't test the full 100k context but overall I got useful code out of it without too much wrangling.
• claude v1 was very fast, but felt more fragile than gpt-3.5 - might be good for analyzing a large code base
• both claudes are whiny and would sometimes get into a "I can't ethically create code without more permission / context" particularly the v1. I was just making cli games so it felt like they are a bit oversensitive on the safety front.

I didn't mess with the gpt-4-32k context but it would be pretty easy to test.

I've seen comments in the openrouter discord about rate limits being proportional to the amount of credits you buy, it's a work in progress though but at the moment rate limits are unclear. It also sounds like they are throttled on the amount of gpt-4-32k access they get so I would expect that to be a bit patchy.

But overall results felt promising and I think it would be a good feature to implement.

If you want to play around then you can use the code above as a starting point (still needs a bunch of work to make it robust) and then you can set your openai api key and base and use commands like aider --model google/palm-2-codechat-bison --edit-format whole and aider --model anthropic/claude-2 --edit-format diff to see how the different models perform.

[r1di]

Im on the latest dev build v0.13.1-dev(with#199 merged), when i try to use 'meta-llama/codellama-34b-instruct' it says Repo-map: disabled
Is it because openrouter models arent fully supported yet?

[joshuavial]

@r1di currently it will only use the repo_map for gpt4 and claude2 which was a bit of an arbitrary choice. I think it would be a good enhancement if it let you add the repo_map when you wanted, happy to add it in at some stage.

[paul-gauthier]

I have tested gpt-3.5 with the repo map, and it confuses the model and makes things worse.

That said, a manual switch to force the repo-map to be used does sound theoretically useful. Just not sure there currently exist any models for which it would be smart to invoke that switch.

[NightMachinery]

How should I use aider with Claude 3 from OpenRouter? Is there a doc for this?

[paul-gauthier]

https://aider.chat/2024/03/08/claude-3.html