Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

try aider with anthropic claude-instant-100k #7

Closed
ericries opened this issue May 27, 2023 · 20 comments
Closed

try aider with anthropic claude-instant-100k #7

ericries opened this issue May 27, 2023 · 20 comments
Labels
enhancement New feature or request

Comments

@ericries
Copy link

I would be very curious to see how aider performs with claude-instant, especially the 100k context variant. I think the latency and cost might be really different. Let me know if you're interested and don't have access

@paul-gauthier
Copy link
Collaborator

That would certainly be interesting to try.

I spent quite a bit of effort finding a prompt and syntax that GPT could use to output these sorts of code edits. I shared some notes recently on HN:
https://news.ycombinator.com/item?id=35950068

Thanks to that process I have become fairly convinced that GPT-4 is just barely competent to complete these sort of tasks. I spent far too much time trying to get gpt-3.5-turbo to work in this way. It was a lost cause for outputting any editing syntax -- it only worked reliably by outputting the entire modified source file. Which was slow and wasted the small context window.

So I think it would be fun to try with other LLMs, but I am not optimistic about it "just working".

@tiagoefreitas
Copy link

smol-developer-js is the most similar I found to aider (much better than the original smol-developer) and the developer says it works better with claude 100k than with GPT-4 due to the larger context window.

I tried it and works well, but didn't try aider yet.

@paul-gauthier
Copy link
Collaborator

That's good to know. I actually applied for Claude access yesterday, so I'll give it a try when I am able.

@paul-gauthier
Copy link
Collaborator

Looks like GitHub redacted your email. You can reach me at: aider at paulg.org.

@tiagoefreitas
Copy link

Sent, please check

@bitnom
Copy link

bitnom commented Jul 2, 2023

That's good to know. I actually applied for Claude access yesterday, so I'll give it a try when I am able.

Claude hasn't been working well for me. If I give it 25k of an API doc and ask it to restructure as a Python class (Setting output len to 50k), it outputs maybe one or two methods of a few hundred tokens and decides it's done.

@felixpie
Copy link

Today Anthropic released Claude 2 with API access to the public. My first tests were quite impressing. Cody also seem to use Claude 2 as their LLM https://about.sourcegraph.com/cody

@paul-gauthier
Copy link
Collaborator

@felixpie as far as I can tell, Anthropic only released Claude 2 to the public via their web chat UI? It appears that they are still waitlisting people for API access. I have not yet cleared the waitlist.

If I am mistaken and missed the public API signup please let me know.

@Explosion-Scratch
Copy link

Explosion-Scratch commented Jul 15, 2023

@paul-gauthier I saw your comment on hackernews, I'd be happy to work on integrating Claude with this project. Of course the Claude Client I created is still very much beta, unstable, etc, I'm planning to do a few things on it which would make it much more stable, integrating automated testing including file attachments, context length, etc.

Of course if Claude fails, it'd be simple to fall back to a more officially supported API, e.g. GPT3.5-Turbo

@felixpie
Copy link

felixpie commented Jul 16, 2023

@felixpie as far as I can tell, Anthropic only released Claude 2 to the public via their web chat UI? It appears that they are still waitlisting people for API access. I have not yet cleared the waitlist.

If I am mistaken and missed the public API signup please let me know.

@paul-gauthier Your are correct, I misunderstood their press release. Sorry.

@paul-gauthier
Copy link
Collaborator

I'd be happy to work on integrating Claude with this project.

@Explosion-Scratch nice work on the Claude API!

I'm hesitant to integrate via unofficial/unsupported endpoints which are intended for use by their web chat ui. It seems likely they will try and block or complicate such usage if it gains much popularity. But perhaps I am misunderstanding how you're accessing Claude?

@Explosion-Scratch
Copy link

Via their internal API, it seems these endpoints are fairly straightforward though, for instance https://claude.ai/api/append_message and many others like that (see the source code), as with ChatGPT it'd be very hard to block this type of traffic entirely, especially with User-Agent spoofing, headless browser and other such methods.

@Happ1ness-dev
Copy link

My first tests were quite impressing

Hello. I don't know if it's only for me, but my coding experience with claude-2 was pretty awful.

Even when you ask it to give you the whole code, without placeholders and stuff like that, it always returns code similar to that:

some_function():
    # implementation of some function

# ...

And this is just annoying. No matter how hard I try, I couldn't defeat it's love for placeholder/pseudo code comments.
Maybe it's just a skill issue though :D

Well, at least it can be a decent summarizer/explainer and can rhyme in Russian ¯⁠\⁠_⁠(⁠ツ⁠)⁠_⁠/⁠¯

@Happ1ness-dev
Copy link

Nevermind. By some miracle, I managed to make it write okay code, but it's still very shaky

@Explosion-Scratch
Copy link

I've been working on some prompts to make claude better handle placeholders and such, I'll give those here in a few mins once I get them to work better

@Explosion-Scratch
Copy link

Current prompt:

Act as an expert software engineer at Google. Be concise! First write out your initial code draft (no comments and no placeholders), then think through 4 thoughts and criticisms of the code you wrote (in bullet points). Make sure to use the latest syntax (e.g. ESM/ES6 in JavaScript, HTML5, CSS3), try not to rely on libraries if possible, don't include debug or comments or empty lines. Include checks and error handling to make sure it works properly. Make sure your code is readable and does not include any placeholders or psuedocode. After your draft, write out some final thoughts and then write your final version. Your code should be able to be run instantly. Unless explicitly requested make sure all functions are completely filled in. DONT INCLUDE ANY TODO COMMENTS. DONT INCLUDE PSUEDOCODE. DONT INCLUDE PLACEHOLDERS. Adhere to the user's request as closely as possible. Use a single file only. Try to make all functions pure. Remember to fill in any placeholders and psuedocode! Start your final version with "FINAL:". Put your result in a codeblock and give instructions on how set up and run it afterwards with a single shell commant (e.g. pnpm init && pnpm i express && code index.js).

Your first request is: Write a javascript program to play hangman in the terminal (node.js, use inquirer).

@Explosion-Scratch
Copy link

image Working pretty well so far!

@paul-gauthier paul-gauthier added the enhancement New feature or request label Jul 19, 2023
@Happ1ness-dev
Copy link

Working pretty well so far!

Can it handle editing existing code as well thought?

@Explosion-Scratch
Copy link

Thought you guys might be interested in this:

output.mp4

Basically Claude cli with prompt templates it:

  1. Writes a design document for the program you request ("cli math game with simple multiplication, division and subtraction node.js")
  2. Uploads that as a file on Claude's API
  3. Starts a new conversation, writes the implementation based on the design doc
  4. Critiques its implementation finding bugs and improving it
  5. Outputs the final code
  6. Writes code to files on your computer (parsed using JS regex contained in templates)

See this filefor more

@paul-gauthier
Copy link
Collaborator

Closing this for now. See #172 for more info.

paul-gauthier added a commit that referenced this issue May 15, 2024
# Aider chat conversation:

USER: will this pickup the github pages jekyll site that is in this repy? Like the _layouts _data _posts etc dirs?

ASSISTANT: Yes, the current Dockerfile and scripts should pick up the GitHub Pages Jekyll site, including directories like `_layouts`, `_data`, `_posts`, etc., because the `COPY . /srv/jekyll` command in the Dockerfile copies the entire current directory (which should include those directories) into the Docker container.

However, to ensure everything works correctly, let's make sure the Dockerfile and scripts are set up to handle the Jekyll site properly. Here are the steps:

1. Ensure the Dockerfile is correctly set up to copy the entire site and install dependencies.
2. Ensure the build and run scripts are correctly set up to build the Docker image and run the container.

Let's review the current state of the files and make any necessary adjustments.

scripts/Dockerfile
```python
# Use the official Jekyll image from Docker Hub
FROM jekyll/jekyll:latest

# Set the working directory
WORKDIR /srv/jekyll

# Copy the current directory contents into the container at /srv/jekyll
COPY . /srv/jekyll

# Install any needed packages specified in Gemfile
RUN bundle install

# Expose port 4000 to the host
EXPOSE 4000

# Run Jekyll server
CMD ["jekyll", "serve", "--host", "0.0.0.0"]
```

scripts/build.sh
```python
#!/bin/bash

# Build the Docker image
docker build -t my-jekyll-site -f scripts/Dockerfile .
```

scripts/run.sh
```python
#!/bin/bash

# Run the Docker container
docker run --rm -v "$PWD:/srv/jekyll" -p 4000:4000 my-jekyll-site
```

These files should work as expected. The `COPY . /srv/jekyll` command in the Dockerfile will copy all the necessary Jekyll site files, including `_layouts`, `_data`, `_posts`, etc., into the Docker container. The `bundle install` command will install any dependencies specified in the `Gemfile`.

To verify, you can run the following commands in your project directory:

```sh
bash scripts/build.sh
bash scripts/run.sh
```

This will build the Docker image and start a container that serves your Jekyll site on `http://localhost:4000`. If you encounter any issues, please let me know!

USER: I ran this command:

 bash scripts/build.sh

And got this output:

#1 [internal] load build definition from Dockerfile
#1 transferring dockerfile: 435B done
#1 DONE 0.0s

#2 [internal] load .dockerignore
#2 transferring context: 101B 0.0s done
#2 DONE 0.0s

#3 [internal] load metadata for docker.io/jekyll/jekyll:latest
#3 DONE 0.6s

#4 [1/4] FROM docker.io/jekyll/jekyll:latest@sha256:400b8d1569f118bca8a3a09a25f32803b00a55d1ea241feaf5f904d66ca9c625
#4 DONE 0.0s

#5 [internal] load build context
#5 transferring context: 250.53kB 0.2s done
#5 DONE 0.2s

#6 [2/4] WORKDIR /srv/jekyll
#6 CACHED

#7 [3/4] COPY . /srv/jekyll
#7 CACHED

#8 [4/4] RUN bundle install
#8 6.277 Could not locate Gemfile
#8 ERROR: process "/bin/sh -c bundle install" did not complete successfully: exit code: 10
------
 > [4/4] RUN bundle install:
#8 6.277 Could not locate Gemfile
------
Dockerfile:11
--------------------
   9 |
  10 |     # Install any needed packages specified in Gemfile
  11 | >>> RUN bundle install
  12 |
  13 |     # Expose port 4000 to the host
--------------------
ERROR: failed to solve: process "/bin/sh -c bundle install" did not complete successfully: exit code: 10

ASSISTANT: The error indicates that the `Gemfile` is not being found in the context being copied into the Docker container. This could be because the `Gemfile` is not in the root directory of your project or it might be missing.

To resolve this, we need to ensure that the `Gemfile` is present in the root directory of your project. If it is not, you should add it. If it is present, we need to make sure it is being copied correctly.

Let's first verify that the `Gemfile` is in the root directory of your project. If it is not, please add it. If it is, we can proceed with the current Dockerfile and scripts.

Assuming the `Gemfile` is present in the root directory, the Dockerfile should work as expected. If the `Gemfile` is in a different directory, please let me know its location so we can adjust the Dockerfile accordingly.

If the `Gemfile` is missing, you can create a basic one with the following content:

Gemfile
```ruby
source 'https://rubygems.org'
gem 'jekyll'
```

Once the `Gemfile` is in place, you can re-run the build script:

```sh
bash scripts/build.sh
```

If you encounter any further issues, please let me know!

USER: please make it

ASSISTANT: Let's create the `Gemfile` in the root directory of your project.

Gemfile
```ruby
<<<<<<< SEARCH
=======
source 'https://rubygems.org'
gem 'jekyll'
>>>>>>> REPLACE
```

This `Gemfile` specifies the source for Ruby gems and includes the `jekyll` gem, which is necessary for your Jekyll site.

After adding this `Gemfile`, you can re-run the build script:

```sh
bash scripts/build.sh
```

This should resolve the issue with the missing `Gemfile` and allow the Docker image to be built successfully. If you encounter any further issues, please let me know!
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

7 participants