-
Notifications
You must be signed in to change notification settings - Fork 1.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Enhancement] Please integrate newest models #20
Comments
You should be able to try it with And yes, I'll be investigating the functions too. |
Sorry, you'll need to pull the latest build from GitHub for that to work. For now it just treats it like 3.5 with a bigger context window. So repo maps are disabled and it does not try and use a diff based output format. I'll be experimenting to see if either of those restrictions can be relaxed with this new model. |
I'm playing with the new 3.5 16k model. I asked the LLM to create a new file, but aider keeps complaining that "Malformed ORIGINAL/UPDATE blocks, retrying..." EDIT: I see the underlying error is "is not one of". I guess the 35 update code path doesn't handle new files correctly |
Sorry to hear you're having troubles. Are you able to paste some example output showing the problem? Often it's easiest to open up |
Ah I see your reference to "is not one of" and yes, this looks like it is related to trying to create a new file. Aider doesn't currently support that for the 3.5 models. A workaround is to just make an empty file and I will try and make this more automatic, similar to how aider handles file creation in gpt-4 chats. |
Thanks for the workaround. Could you also add support for this?
Ideally these new model releases could be used directly without requiring stubs in the python code |
I'm continuing to have "is not one of" issues. I asked for a modification for a specific file but the code block wasn't tagged in the parsable format.
I wonder if it makes sense to integrate the new functions api instead, and offer the LLM functions to create and modify files, with json arguments that explicitly include a filename property. |
Ya, support for gpt-3.5 models is experimental. It looks like you might have access to gpt-4? If so, it is much more reliable about following directions for including code/edits in its responses. I will certainly be investigating how to use the new functions capabilities in aider! Both for 3.5 and 4. Also, I updated aider to be more flexible about openai's models. It should now support any model that your key has access to, including gpt-4-32k-0613. |
It would be good to refactor the code to use langchain so other models from different providers can more easily be integrated / experimented with in the future. |
Hi @tmm1, I just released version 0.7.0, which has more robust support for GPT-3.5. Aider is now much more permissive at recognizing which files to edit when 3.5 returns malformed instructions (as it often does). I did some benchmarking, and this release of aider performs better at coding editing competency with 3.5 compared to the prior release. Let me know if you have a chance to try it out and if you see improvements.
I also roughed in an coding backend for 3.5 which uses the new functions api. Surprisingly it benchmarks significantly worse than the current editing format that relies on markdown backtick fenced codeblocks. I'm going to continue to experiment here, and will hopefully share some quantitative learnings at some point. |
Thanks for the feedback @tomrobinsond. I agree it would be nice for aider to support a larger variety of LLMs. Other users have used the https://github.com/go-skynet/LocalAI One thing to keep in mind is that it generally requires some model-specific tuning to get prompts and editing formats working well. For example, GPT-3.5 and GPT-4 use very different prompts and editing formats in aider right now. So I imagine adopting new LLMs will require a similar effort to tailor the prompting and edit formats. The latest version of aider includes a major refactor that makes it easy to develop, benchmark and manage a collection of many "coding backends". This should make it easier tailor aider to additional LLMs in the future. |
Hi @paul-gauthier |
I have access to GPT4 32k but it still limits my tokens to 8k, can I select the model somewhere? |
@mallefitzo thanks for reporting this problem. And congrats for having access to gpt-4-32k! What output is telling you that aider is using gpt-4-32k but only treating it as an 8k context window? Aider parses the context window size right out of the model name. So it should be assuming 32k. |
@paul-gauthier this is the error I'm getting: |
Thanks for the screenshot @mallefitzo, super helpful. Are you running aider with
Obviously my attempt to run with gpt-4-32k fails. But what status lines do you get? |
@paul-gauthier ahhh! It shows me this now (I assumed, it would pick the highest model automatically). So I suppose, this will enable the 32k tokens? Thanks for your help! |
Glad to hear you have it working now @mallefitzo Aider defaults to plain old gpt-4. If that isn't available it falls back to gpt-3.5-turbo-16k. I didn't want to automatically select gpt-4-32k because of the cost. I figured it would be safer to let folks choose it explicitly. |
Also @mallefitzo you can use the |
I'm going to close this issue since aider seems to be properly supporting all the recently released OpenAI models. Please feel free to reopen or start a new issue if you are having problems. @tmm1 you were asking about supporting the new functions api. I did some benchmarking, and using functions actually seems to make code editing less reliable with both GPT-3.5 and GPT-4. I wrote up some notes here: |
Hello Paul,
could you integrate gpt3.5 16k and the new versions of gpt-4 and gpt-3.5-turbo?
https://openai.com/blog/function-calling-and-other-api-updates
Will be interesting to test out GPT-4 with 8k context vs GPT-35 with 16k context.
And maybe you could utilize the functions also?
Greetings
The text was updated successfully, but these errors were encountered: