Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Enable CEDARScript edit format #1961

Draft
wants to merge 10 commits into
base: main
Choose a base branch
from
Draft

Enable CEDARScript edit format #1961

wants to merge 10 commits into from

Conversation

elifarley
Copy link

@elifarley elifarley commented Oct 7, 2024 •
edited
Loading

Intro

This PR adds cedarscript as an optional edit format, best suited for handling refactorings of larger files.
When used by the editor LLM in the architect/editor mode, it should work great with any file size.

How to install Aider+CEDARScript from this PR

pip install --upgrade --force-reinstall \
git+https://github.com/elifarley/aider@cedarscript \
aider-chat

Further details:

  1. CEDARScript
  2. Aider Integration - How to use it

Refactoring Benchmarks

Gemini 1.5 Flash

Notice that Sonnet 3.5 is still a tad better at pass_rate, but loses to Gemini Flash in percent_cases_well_formed:

Sonnet 3.5:

  pass_rate_1: 92.1
  percent_cases_well_formed: 91.0
More...

Gemini 1.5 Flash:

- dirname: 2024-11-04-06-01-55--gemini-1.5-flash-refactoring-cedarscript-i1.1.3-e1.1.7
  test_cases: 89
  model: gemini/gemini-1.5-flash-latest
  edit_format: cedarscript
  commit_hash: 82ac553
  pass_rate_1: 84.3
  pass_rate_2: 84.3
  pass_rate_3: 85.4
  percent_cases_well_formed: 93.3
  error_outputs: 134
  num_malformed_responses: 28
  num_with_malformed_responses: 6
  user_asks: 35
  lazy_comments: 0
  syntax_errors: 6
  indentation_errors: 3
  exhausted_context_windows: 0
  test_timeouts: 0
  command: aider --model gemini/gemini-1.5-flash-latest
  date: 2024-11-04
  versions: 0.60.2.dev
  seconds_per_case: 17.1
  total_cost: 0.6229

Comparison: Gemini 1.5 Flash using whole vs CEDARScript

Performance Highlights:

  • 48% of tests (43 total) showed improvements
  • 103% increase in Pass 1 success rate (75 tests)
  • Test duration reduced by 93% (from 5:17:26 to 0:25:17)
  • Token efficiency greatly improved:
  • Sent tokens: -37% (7.59M)
  • Received tokens: -96% (180K)
  • Error reduction:
  • Error outputs: -94% (35 total)
  • Malformed outputs: -94% (6 cases)
  • Syntax errors: -85% (3 cases)
  • Indent errors eliminated (100% reduction)

image

Gemini 1.5 PRO

More... 
- dirname: 2024-11-04-06-57-53--gemini-1.5-pro-refactoring-cedarscript-i1.1.3-e1.1.7
  test_cases: 89
  model: gemini/gemini-1.5-pro-latest
  edit_format: cedarscript
  commit_hash: 82ac553
  pass_rate_1: 79.8
  pass_rate_2: 80.9
  pass_rate_3: 83.1
  percent_cases_well_formed: 79.8
  error_outputs: 329
  num_malformed_responses: 43
  num_with_malformed_responses: 18
  user_asks: 49
  lazy_comments: 0
  syntax_errors: 19
  indentation_errors: 6
  exhausted_context_windows: 0
  test_timeouts: 0
  command: aider --model gemini/gemini-1.5-pro-latest
  date: 2024-11-04
  versions: 0.60.2.dev
  seconds_per_case: 51.3
  total_cost: 38.4399

Comparison: Gemini 1.5 PRO using diff-fenced vs CEDARScript

Performance Highlights:

  1. 65% of tests (58 total) showed improvements
  2. 60% (53 tests) now passing, with only 1% (1 test) newly failing
  3. Test duration reduced by 53% (from 2:43:22 to 1:16:01)
  4. Token metrics:
  5. Received tokens: -68% (123.5K)
  6. Sent tokens: +38% (11M)
  7. Error improvements:
  8. Error outputs: -62% (49 total)
  9. Context exhausts: -76% (43 cases)
  10. Syntax errors: -94% (6 cases)
  11. Indent errors eliminated (100% reduction)

Success Distribution:

  • Pass 1 rate increased dramatically: +344% (71 tests)
  • Failed tests reduced by 78% (now only 15)
  • Stable tests: 34% (30 tests)
  • 18% maintained pass status (16 tests)
  • 16% remained in failed state (14 tests)

image

OpenAI GTP-4o

More... 
- dirname: 2024-11-04-13-33-35--gpt-4o-mini-refactoring-cedarscript-i1.1.3-e1.1.7
  test_cases: 89
  model: openrouter/openai/gpt-4o-mini
  edit_format: cedarscript
  commit_hash: 82ac553
  pass_rate_1: 32.6
  pass_rate_2: 37.1
  pass_rate_3: 39.3
  percent_cases_well_formed: 20.2
  error_outputs: 571
  num_malformed_responses: 435
  num_with_malformed_responses: 71
  user_asks: 144
  lazy_comments: 0
  syntax_errors: 5
  indentation_errors: 86
  exhausted_context_windows: 0
  test_timeouts: 0
  command: aider --model openrouter/openai/gpt-4o-mini
  date: 2024-11-04
  versions: 0.60.2.dev
  seconds_per_case: 116.7
  total_cost: 0.0000

Comparison: Gemini 1.5 PRO vs GPT-4o (both using CEDARScript)

image

DeepSeek

More... 
- dirname: 2024-11-04-16-38-33--deepseek-refactoring-cedarscript-i1.1.3-e1.1.7
  test_cases: 89
  model: openrouter/deepseek/deepseek-chat
  edit_format: cedarscript
  commit_hash: 82ac553
  pass_rate_1: 89.9
  pass_rate_2: 89.9
  pass_rate_3: 91.0
  percent_cases_well_formed: 87.6
  error_outputs: 157
  num_malformed_responses: 63
  num_with_malformed_responses: 11
  user_asks: 6
  lazy_comments: 0
  syntax_errors: 0
  indentation_errors: 0
  exhausted_context_windows: 4
  test_timeouts: 0
  command: aider --model openrouter/deepseek/deepseek-chat
  date: 2024-11-04
  versions: 0.60.2.dev
  seconds_per_case: 111.9
  total_cost: 0.0000

Comparison: Gemini 1.5 PRO vs DeepSeek

--- 2024-11-04-06-57-53--gemini-1.5-pro-refactoring-cedarscript-i1.1.3-e1.1.7
+++ 2024-11-04-16-38-33--deepseek-refactoring-cedarscript-i1.1.3-e1.1.7

@@ ============= TEST STATUS CHANGES ============ @@

+ IMPROVED     :  13 ( 15% of total) +
++  Now PASSES :  10 ( 11% of total) +
+        Minor :   3 (  3% of total) +

- WORSENED     :   4 (  4% of total) +
--  Now FAILED :   3 (  3% of total) +
-        Minor :   1 (  1% of total) +

# STABLE       :  72 ( 81% of total) ++++++++
#+      PASSED :  67 ( 75% of total) +++++++
#-      FAILED :   5 (  6% of total) +

@@ ============ Success Distribution =========== @@
#            FAILED:          8 (        -7,  -47%) ----
#          Pass   1:         80 (        +9,  +13%) +
#          Pass   2:          0 (        -1, -100%) ----------
#          Pass   3:          1 (        -1,  -50%) -----

@@ ============ PERFORMANCE METRICS  =========== @@
# MODEL            : deepseek-chat (was gemini-1.5-pro-latest)
# EDIT FORMAT      : cedarscript 
# TOTAL TEST COUNT :         89 
# Max attempt count:          3
# DURATION hh:mm:ss:    2:46:03 (+  1:30:01, +118%) +++++++++++
# COST ($)         :       0.00 N/A
# TOKENS SENT      : 36,020,000 (+25,064,000, +229%) ++++++++++++++++++++
# TOKENS RECEIVED  :    122,739 (      -772,   -1%) -
# TIMEOUTS         :        157 (      -172,  -52%) -----
# ERROR OUTPUTS    :          6 (       -43,  -88%) --------
# USER ASKS        :          4 N/A
# CONTEXT EXHAUSTS :         63 (       +20,  +47%) ++++
# MALFORMED        :          0 (       -19, -100%) ----------
# SYNTAX ERRORS    :          0 (        -6, -100%) ----------
# INDENT ERRORS    :          0 N/A
# LAZY COMMENTS    :          0 N/A

Editing Benchmarks

Gemini 1.5 PRO

More... 
- dirname: 2024-11-05-13-05-27--gemini-1.5-pro-editing-cedarscript-raw-i1.2.1-e1.2.0
  test_cases: 133
  model: gemini/gemini-1.5-pro-latest
  edit_format: cedarscript-raw
  commit_hash: 2ee4c9e
  pass_rate_1: 47.4
  pass_rate_2: 61.7
  pass_rate_3: 66.2
  percent_cases_well_formed: 88.7
  error_outputs: 157
  num_malformed_responses: 77
  num_with_malformed_responses: 15
  user_asks: 34
  lazy_comments: 0
  syntax_errors: 3
  indentation_errors: 4
  exhausted_context_windows: 0
  test_timeouts: 2
  command: aider --model gemini/gemini-1.5-pro-latest
  date: 2024-11-05
  versions: 0.60.2.dev
  seconds_per_case: 31.3
  total_cost: 27.0067

Comparison: Gemini 1.5 PRO using whole vs CEDARScript

--- 2024-10-22-03-16-35--gemini1.5-pro-editing-whole
+++ 2024-11-05-13-05-27--gemini-1.5-pro-editing-cedarscript-raw-i1.2.1-e1.2.0

@@ ============= TEST STATUS CHANGES ============ @@

  # STABLE      :  89 ( 67%) ++++++
  #+      PASSED:  58 ( 44%) ++++
  #-      FAILED:  31 ( 23%) ++

  + IMPROVED    :  22 ( 17%) +
  ++  Now PASSED:  15 ( 11%) +
  +        Minor:   7 (  5%) +

  - WORSENED    :  22 ( 17%) +
  --  Now FAILED:  14 ( 11%) +
  -        Minor:   8 (  6%) +

  ∂      nP - nF:   1 (  1%) +

@@ ============ Success Distribution =========== @@
#            FAILED:         45 (        -1,   -2%) -
#          Pass   1:         63 (        +2,   +3%) +
#          Pass   2:         19 (        +0,   +0%)
#          Pass   3:          6 (        -1,  -14%) -

@@ ============ PERFORMANCE METRICS  =========== @@
# MODEL            : gemini-1.5-pro-latest 
# EDIT FORMAT      : cedarscript-raw (was whole)
# TOTAL TEST COUNT :        133 
# Max attempt count:          3
# DURATION hh:mm:ss:    1:09:17 (+  0:29:42,  +75%) +++++++
# COST ($)         :       2.00 N/A
# TOKENS SENT      :  7,691,000 (+7,082,058, +1163%) ++++++++++++++++++++
# TOKENS RECEIVED  :    127,280 (   +37,135,  +41%) ++++
# TIMEOUTS         :        157 (      +142, +947%) ++++++++++++++++++++
# ERROR OUTPUTS    :         34 (       -24,  -41%) ----
# USER ASKS        :          0 N/A
# CONTEXT EXHAUSTS :         77 (       +62, +413%) ++++++++++++++++++++
# MALFORMED        :          3 N/A
# SYNTAX ERRORS    :          4 N/A
# INDENT ERRORS    :          0 N/A
# LAZY COMMENTS    :          0 N/A


@@ Improved, now PASSED (15) @@
++ [-3 -> 0] (          ----------) (++++++++++++++++++++) beer-song
++ [-3 -> 1] (--------------------) (++++++++++++++++++++) clock
++ [-3 -> 1] (        ------------) (++++++++++++++++++++) dominoes
++ [-3 -> 1] (       -------------) (++++++++++++++++++++) hexadecimal
++ [-3 -> 1] (      --------------) (++++++++++++++++++++) house
++ [-3 -> 0] (   -----------------) (++++                ) matching-brackets
++ [-3 -> 2] (              ------) (++++++++++++++++++++) phone-number
++ [-3 -> 1] (                ----) (++++++++++++++++++++) pig-latin
++ [-3 -> 1] (                  --) (++++++++++++++++++++) pythagorean-triplet
++ [-3 -> 0] (  ------------------) (+++++               ) reverse-string
++ [-3 -> 1] (              ------) (++++++++++++++++++++) satellite
++ [-3 -> 0] (        ------------) (++++++++++++++++++++) square-root
++ [-3 -> 0] (                 ---) (++++++++++++++++++++) sublist
++ [-3 -> 1] (            --------) (++++++++++++++++++++) twelve-days
++ [-3 -> 2] (                   -) (++++++++++++++++++++) word-count

Gemini 1.5 Flash

More... 
- dirname: 2024-11-04-01-11-29--gemini-1.5-flash-editing-cedarscript-i1.1.3-e1.1.6
  test_cases: 133
  model: gemini/gemini-1.5-flash-latest
  edit_format: cedarscript
  commit_hash: 698a9e5
  pass_rate_1: 32.3
  pass_rate_2: 34.6
  pass_rate_3: 37.6
  percent_cases_well_formed: 76.7
  error_outputs: 262
  num_malformed_responses: 156
  num_with_malformed_responses: 31
  user_asks: 159
  lazy_comments: 0
  syntax_errors: 18
  indentation_errors: 30
  exhausted_context_windows: 0
  test_timeouts: 3
  command: aider --model gemini/gemini-1.5-flash-latest
  date: 2024-11-04
  versions: 0.60.2.dev
  seconds_per_case: 39.0
  total_cost: 1.3111

Comparison: Gemini 1.5 Flash using whole vs CEDARScript

--- 2024-10-21-12-58-15--gemini-1.5-flash-editing-whole
+++ 2024-11-04-01-11-29--gemini-1.5-flash-editing-cedarscript-i1.1.3-e1.1.6

@@ ============= TEST STATUS CHANGES ============ @@

+ IMPROVED     :   9 (  7% of total) +
++  Now PASSES :   4 (  3% of total) +
+        Minor :   5 (  4% of total) +

- WORSENED     :  28 ( 21% of total) ++
--  Now FAILED :  25 ( 19% of total) +
-        Minor :   3 (  2% of total) +

# STABLE       :  96 ( 72% of total) +++++++
#+      PASSED :  38 ( 29% of total) ++
#-      FAILED :  58 ( 44% of total) ++++

@@ ============ Success Distribution =========== @@
#            FAILED:         83 (       +21,  +34%) +++
#          Pass   1:         43 (        -3,   -7%) -
#          Pass   2:          3 (       -12,  -80%) --------
#          Pass   3:          4 (        -6,  -60%) ------

@@ ============ PERFORMANCE METRICS  =========== @@
# MODEL            : gemini-1.5-flash-latest 
# EDIT FORMAT      : cedarscript (was whole)
# TOTAL TEST COUNT :        133 
# Max attempt count:          3
# DURATION hh:mm:ss:    1:26:32 (+  1:06:43, +337%) ++++++++++++++++++++
# COST ($)         :       3.00 (     +1.00,  +50%) +++++
# TOKENS SENT      : 13,928,000 (+13,147,158, +1684%) ++++++++++++++++++++
# TOKENS RECEIVED  :    889,294 (  +694,472, +356%) ++++++++++++++++++++
# TIMEOUTS         :        262 N/A
# ERROR OUTPUTS    :        159 (      +138, +657%) ++++++++++++++++++++
# USER ASKS        :          0 N/A
# CONTEXT EXHAUSTS :        156 N/A
# MALFORMED        :         18 N/A
# SYNTAX ERRORS    :         30 (       +27, +900%) ++++++++++++++++++++
# INDENT ERRORS    :          0 N/A
# LAZY COMMENTS    :          0 N/A

Haiku 3.5

More... 
- dirname: 2024-11-05-21-53-47--haiku3.5-editing-cedarscript-raw-i1.2.1-e1.2.0
  test_cases: 133
  model: openrouter/anthropic/claude-3-5-haiku
  edit_format: cedarscript-raw
  commit_hash: 2ee4c9e
  pass_rate_1: 60.2
  pass_rate_2: 72.2
  pass_rate_3: 77.4
  percent_cases_well_formed: 99.2
  error_outputs: 1
  num_malformed_responses: 1
  num_with_malformed_responses: 1
  user_asks: 5
  lazy_comments: 0
  syntax_errors: 0
  indentation_errors: 0
  exhausted_context_windows: 0
  test_timeouts: 4
  command: aider --model openrouter/anthropic/claude-3-5-haiku
  date: 2024-11-05
  versions: 0.60.2.dev
  seconds_per_case: 28.2
  total_cost: 5.1379

Comparison: Gemini 1.5 PRO using whole vs Haiku 3.5 using CEDARScript

--- 2024-10-22-03-16-35--gemini1.5-pro-editing-whole
+++ 2024-11-05-21-53-47--haiku3.5-editing-cedarscript-raw-i1.2.1-e1.2.0

@@ ============= TEST STATUS CHANGES ============ @@

# STABLE      :  87 ( 65%) ++++++
#+        PASS:  62 ( 47%) ++++
#-        FAIL:  25 ( 19%) +

+ IMPROVED    :  36 ( 27%) ++
++    Now PASS:  21 ( 16%) +
+        Minor:  15 ( 11%) +

- WORSENED    :  10 (  8%) +
--    Now FAIL:   5 (  4%) +
-        Minor:   5 (  4%) +

∂ nowPASS-FAIL:  16 ( 12%) +

@@ ============ Success Distribution =========== @@
#              FAIL:         30 (       -16,  -35%) ---
#          Pass   1:         80 (       +19,  +31%) +++
#          Pass   2:         16 (        -3,  -16%) -
#          Pass   3:          7 (        +0,   +0%)

@@ ============ PERFORMANCE METRICS  =========== @@
# MODEL            : claude-3-5-haiku (was gemini-1.5-pro-latest)
# EDIT FORMAT      : cedarscript-raw (was whole)
# TOTAL TEST COUNT :        133 
# Max attempt count:          3
# DURATION hh:mm:ss:    1:02:28 (+  0:22:52,  +58%) +++++
# COST ($)         :       4.00 N/A
# TOKENS SENT      :  4,328,000 (+3,719,058, +611%) ++++++++++++++++++++
# TOKENS RECEIVED  :    160,308 (   +70,163,  +78%) +++++++
# TIMEOUTS         :          1 (       -14,  -93%) ---------
# ERROR OUTPUTS    :          5 (       -53,  -91%) ---------
# USER ASKS        :          0 N/A
# CONTEXT EXHAUSTS :          1 (       -14,  -93%) ---------
# MALFORMED        :          0 N/A
# SYNTAX ERRORS    :          0 N/A
# INDENT ERRORS    :          0 N/A
# LAZY COMMENTS    :          0 N/A

@elifarley elifarley marked this pull request as draft October 7, 2024 20:22
@elifarley elifarley marked this pull request as ready for review October 7, 2024 20:28
@fry69
Copy link
Contributor

fry69 commented Oct 7, 2024 •
edited
Loading

I skimmed through your code and noticed:

  • changes to benchmark, model and docker files, please remove those from this PR or explain why they are needed
  • adding of rope dependency, is this needed for your code? I couldn't find a reference in the code after a quick search
  • this PR should only be about your new coder, everything else should come in its own PR IMHO

@elifarley
Copy link
Author

I skimmed through your code and noticed:

  • changes to benchmark and docker files, please remove those from this PR or explain why they are needed
  • adding of rope dependency, is this needed for your code? I couldn't find a reference in the code after a quick search

I'll re-check those items and update the PR.

Thank you!

@elifarley elifarley force-pushed the cedarscript branch 5 times, most recently from ba5c50a to 42cffd7 Compare October 8, 2024 00:14
fry69
fry69 reviewed Oct 8, 2024
View reviewed changes
requirements.txt Outdated
Copy link
Contributor

@fry69 fry69 Oct 8, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This file still needs to get regenerated to remove rope dependency and what it pulls in.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I did regenerate it. rope is now coming from this:

rope==1.13.0
   # via cedarscript-editor

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note: CEDARScript now uses Tree-Sitter instead of Rope to obtain the CST.

elifarley
elifarley commented Oct 8, 2024
View reviewed changes
benchmark/benchmark.py
cache_prompts=True,
suggest_shell_commands=False,
)
with change_dir(testdir):
Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we don't do this, the CEDARScript editor won't know which directory to consider as the base. We can remove all changes to this file, but then we won't be able to run benchmarks using cedarscript-g edit format.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should I drop changes to this file, and create a separate PR? Maybe it's better. This main PR won't be able to run benchmarks using cedarscript-g though.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is fine, since

  • the benchmark does not touch regular usage
  • it is more or less part of the new coder

but @paul-gauthier should make the call of course.

@paul-gauthier
Copy link
Collaborator

It's great that you are trying out new code editing prompts. It's a fun area to experiment.

But it seems premature to be proposing a PR for this, for a couple of reasons:

  1. All of aider's code editing formats are language-agnostic. They work on any text file. As far as I can tell this PR contains a python-only edit format. There's no existing mechanism to select the edit format based on the code language. I am reluctant to take on the maintenance burden of per-language editing formats, without a really compelling benefit.

  2. It appears this has only been benchmarked once, against one model and only on the refactoring benchmark. To adopt a new editing format, it would be important to see strong benchmark results on the main code editing benchmark -- ideally across multiple noteworthy models.

@elifarley elifarley marked this pull request as draft October 8, 2024 02:42
@elifarley
Copy link
Author

Agreed. Will try to get data for more models and the main code editing benchmark too.

@CLAassistant
Copy link

CLAassistant commented Oct 16, 2024 •
edited
Loading

CLA assistant check
All committers have signed the CLA.

@elifarley elifarley force-pushed the cedarscript branch 2 times, most recently from 8ab7a06 to 4aa8890 Compare October 21, 2024 05:08
@elifarley
Copy link
Author

elifarley commented Oct 21, 2024 •
edited
Loading

All of aider's code editing formats are language-agnostic. They work on any text file. As far as I can tell this PR contains a python-only edit format. There's no existing mechanism to select the edit format based on the code language. I am reluctant to take on the maintenance burden of per-language editing formats, without a really compelling benefit.

Now, just like Aider itself, I'm also using tree-sitter-languages, and I have setup tree-sitter queries for Python, Kotlin, PHP, Rust, Go, C++, C, Java, Javascript, Lua, FORTRAN, Scala and C#,
but only Python has been tested so far.

It appears this has only been benchmarked once, against one model and only on the refactoring benchmark. To adopt a new editing format, it would be important to see strong benchmark results on the main code editing benchmark -- ideally across multiple noteworthy models.

Got new refactoring benchmark results for Gemini 1.5 PRO & Flash.
The more cost-effective Gemini 1.5 Flash model, using CEDARScript, outperformed Claude 3.5 Sonnet.

However, for the main editing benchmark, Gemini 1.5 Flash shows poor performance:

pass_rate_4: 39.1
percent_cases_well_formed: 69.9

Gemini 1.5 PRO:

pass_rate_1: 42.1
percent_cases_well_formed: 85.7

I still need to improve the prompts to get it right.

@elifarley elifarley force-pushed the cedarscript branch 2 times, most recently from 6d03727 to 00c60cd Compare October 21, 2024 18:41
@elifarley elifarley marked this pull request as ready for review November 4, 2024 15:30
@elifarley elifarley requested a review from fry69 November 4, 2024 16:53
@elifarley elifarley marked this pull request as draft November 4, 2024 17:45
@devsoft-co-za devsoft-co-za mentioned this pull request Nov 8, 2024
Closed
@elifarley
Copy link
Author

elifarley commented Nov 9, 2024 •
edited
Loading

@paul-gauthier:

It appears this has only been benchmarked once, against one model and only on the refactoring benchmark. To adopt a new editing format, it would be important to see strong benchmark results on the main code editing benchmark -- ideally across multiple noteworthy models.

We now have benchmark results for:

Editing Benchmarks:

  1. Gemini 1.5 PRO
  2. Gemini 1.5 Flash
  3. Haiku 3.5

Refactoring Benchmarks:

  1. Gemini 1.5 Flash
  2. Gemini 1.5 PRO
  3. OpenAI GTP-4o
  4. DeepSeek

While the main editing benchmark results show degraded performance, the fact that the refactoring benchmark shows significantly better results means there's something worth investigating here.
If this is merged, people will more easily be able to test this editing format when they want, allowing us to gather more data points and further improve both Aider and its editing formats.

Of course, it's still possible to test it without merging. I'll try to provide detailed instructions for those who want to try out CEDARScript without the merge.

@gtrak
Copy link

gtrak commented Nov 11, 2024 •
edited
Loading

I was curious to try it, but ran into some issues: EDIT: commit 9e2a73d was able to run

installed via:

pip install git+https://github.com/CEDARScript/cedarscript-integration-aider
pip install git+https://github.com/elifarley/aider@cedarscript aider-chat

run command: aider --model ollama/qwen-2.5-coder-32b-instruct-q4_k_m --edit-format cedarscript

Traceback (most recent call last):
  File "/home/gary/.pyenv/versions/3.12.7/bin/aider", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/home/gary/.pyenv/versions/3.12.7/lib/python3.12/site-packages/aider/main.py", line 690, in main
    coder = Coder.create(
            ^^^^^^^^^^^^^
  File "/home/gary/.pyenv/versions/3.12.7/lib/python3.12/site-packages/aider/coders/base_coder.py", line 155, in create
    res = coder(main_model, io, **kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/gary/.pyenv/versions/3.12.7/lib/python3.12/site-packages/aider/coders/cedarscript_coder.py", line 43, in __init__
    super().__init__(*args, edit_format = self.edit_format, **kwargs)
  File "/home/gary/.pyenv/versions/3.12.7/lib/python3.12/site-packages/aider/coders/cedarscript_coder.py", line 10, in __init__
    super().__init__(*args, **kwargs)
  File "/home/gary/.pyenv/versions/3.12.7/lib/python3.12/site-packages/aider/coders/folder_coder.py", line 15, in __init__
    super().__init__(*args, **kwargs)
TypeError: Coder.__init__() got an unexpected keyword argument 'edit_format'

@devsoft-co-za
Copy link

I was curious to try it, but ran into some issues: EDIT: commit 9e2a73d was able to run

installed via:

pip install git+https://github.com/CEDARScript/cedarscript-integration-aider
pip install git+https://github.com/elifarley/aider@cedarscript aider-chat

run command: aider --model ollama/qwen-2.5-coder-32b-instruct-q4_k_m --edit-format cedarscript

Traceback (most recent call last):
  File "/home/gary/.pyenv/versions/3.12.7/bin/aider", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/home/gary/.pyenv/versions/3.12.7/lib/python3.12/site-packages/aider/main.py", line 690, in main
    coder = Coder.create(
            ^^^^^^^^^^^^^
  File "/home/gary/.pyenv/versions/3.12.7/lib/python3.12/site-packages/aider/coders/base_coder.py", line 155, in create
    res = coder(main_model, io, **kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/gary/.pyenv/versions/3.12.7/lib/python3.12/site-packages/aider/coders/cedarscript_coder.py", line 43, in __init__
    super().__init__(*args, edit_format = self.edit_format, **kwargs)
  File "/home/gary/.pyenv/versions/3.12.7/lib/python3.12/site-packages/aider/coders/cedarscript_coder.py", line 10, in __init__
    super().__init__(*args, **kwargs)
  File "/home/gary/.pyenv/versions/3.12.7/lib/python3.12/site-packages/aider/coders/folder_coder.py", line 15, in __init__
    super().__init__(*args, **kwargs)
TypeError: Coder.__init__() got an unexpected keyword argument 'edit_format'

I think the options have changed - try using aider --model ollama/qwen-2.5-coder-32b-instruct-q4_k_m --edit-format cedarscript-raw on the latest version.

Thanks for posting your install process by the way - unfortunately there is an issue for me on Manjaro Linux, not related to the code I think.

@elifarley
Copy link
Author

I'm working on a fix, guys!
And thanks for posting details of the installation, @gtrak !

@elifarley
Copy link
Author

elifarley commented Nov 13, 2024 •
edited
Loading

Note: you don't have to install this:

pip install \
git+https://github.com/CEDARScript/cedarscript-integration-aider

Since this PR already depends on it

@elifarley
Copy link
Author

elifarley commented Nov 13, 2024 •
edited
Loading

Fixed. Testing on an empty venv succeeded:

pip install --upgrade --force-reinstall \
git+https://github.com/elifarley/aider@cedarscript \
aider-chat

Aider v0.61.1.dev462+gab333361

Please check the updated installation instructions.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@fry69 fry69 Awaiting requested review from fry69

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
6 participants
@elifarley @fry69 @paul-gauthier @CLAassistant @gtrak @devsoft-co-za