adding autoedits support #5845

hitesh-1997 · 2024-10-08T13:17:08Z

Context

The PR enables a basic setup for auto-edits. The goal of the PR is for us to check the model quality by manually triggering the auto-edits and have a vibe check. The actual integration of next edits should be done with the existing autocomplete infra. So, this PR is supposed to live behind a feature flag just for the internal testing and model iterations.

The PR takes 4 lines surrounding the cursor and than propose a diff for those 4 lines.
The current implementation uses a fine-tuned gpt-4o-mini model. The name of the model is: ft:gpt-4o-mini-2024-07-18:sourcegraph-production::AFXNjNiC

Steps to run the autoedits (in debug mode):

Set the following following setting in the vscode config

"cody.experimental.autoedits": {
    "provider": "openai",
    "model": "ft:gpt-4o-mini-2024-07-18:sourcegraph-production::AGgXey7l",
    "apiKey": "<openai_token>",
    "tokenLimit": {
      "prefixTokens": 2500,
      "suffixTokens": 2500,
      "maxPrefixLinesInArea": 12,
      "maxSuffixLinesInArea": 5,
      "codeToRewritePrefixLines": 2, 
      "codeToRewriteSuffixLines": 3,
      "contextSpecificTokenLimit": {
        "recent-view-port": 1000,
        "diagnostics": 1000,
        "recent-copy": 1000,
        "jaccard-similarity": 1000,
        "recent-edits": 1000
      }
    }
  },

Press ctrl+shift+enter to trigger the
The debug console Cody by Sourcegraph will show the diff in the console.
The suggestion will also be shown on the UI as a ghost text and we can press tab to apply the changes and escape to reject the changes.

Test plan

Updated CI checks and manual testing. Please see a demo below

How.to.run.the.autoedits.mp4

valerybugakov · 2024-10-09T02:05:44Z

vscode/src/configuration.test.ts

            experimentalSupercompletions: false,
+            experimentalAutoedits: false,


Do you plan to use the existing supercompletions code in your work? If not, it would be great to remove it.

I think we can keep supercompletions for now, since some code might be usable, also this PR is just to iterate quickly on the model quality checks. The actual integration would anyways be with the existing autocomplete code like you suggested, so in future we might remove both these functionalities.

valerybugakov · 2024-10-09T03:10:19Z

vscode/src/autoedits/model-helpers.ts

+        const response = await axios.post(
+            'https://api.openai.com/v1/chat/completions',


Can we do it through the existing infrastructure using the Sourcegraph instance and Cody Gateway? We should have all the tools required to make these requests without introducing a separate HTTP client and API key management on the client.

I think the goal of this PR is to quickly check the model performance and will mostly live internally behind a feature flag. For any model we finetune and want to experiment with, would need to be added in the cody gateway allow list for client to actually work, and this would slow the quick iterations.

When we integrate the autoedits we should do it properly using Cody gateway etc. but I think for now, it would make sense to keep it as is, and spend time instead on model improvements and making the feature work with the smaller models :)

We can use the server-side model configuration for quick iterations. I just added the ft:gpt-4o-mini-2024-07-18:sourcegraph-production::AFXNjNiC model to the sg04 instance. Usage example:

curl 'https://sg04.sourcegraphcloud.com/.api/completions/stream?api-version=4&client-name=defaultclient&client-version=6.0.0' -H 'Content-Type: application/json' -H "Authorization: token $SRC_ACCESS_TOKEN" --data-raw '{ "maxTokensToSample": 4000, "messages": [ { "role": "user", "content": [ { "type": "text", "text": "what is in this file?" }, { "type": "file", "file": { "uri": "https://github.com/sourcegraph/cody/-/blob/src/index.ts", "content": "function helloworld() { console.log(`hello world`)" } } ] } ], "model": "openai::2024-02-01::gpt-4o-mini-autoedit", "stream": false }'

This should allow us to migrate to the existing infra and avoid adding additional tech debt to the codebase.

valerybugakov · 2024-10-09T03:14:44Z

vscode/src/chat/chat-view/ChatController.test.ts

@@ -284,7 +284,7 @@ describe('ChatController', () => {
        })
    })

-    test('send error', async () => {
+    test.only('send error', async () => {


abeatrix

(will continue the review tomorrow)

vscode/src/autoedits/model-helpers.ts

vscode/src/chat/chat-view/ChatController.test.ts

vscode/src/completions/context/context-mixer.test.ts

vscode/package.json

abeatrix · 2024-10-09T05:06:49Z

vscode/src/autoedits/prompt-provider.ts

+// ################################################################################################################
+
+// Some common prompt instructions
+const SYSTEM_PROMPT = ps`You are an intelligent programmer named CodyBot. You are an expert at coding. Your goal is to help your colleague finish a code change.`


Should we update it to match our preamble so that it'd work with Cody gateway later?

This makes sense, can you please add a link to the preamble we use. I will try that offline :)

arafatkatze · 2024-10-09T11:14:56Z

vscode/src/autoedits/prompt-provider.ts

+${lintErrorsPrompt}
+${recentCopyPrompt}
+${codeToRewritePrompt}
+${FINAL_USER_PROMPT}


Super cool!!!

From the way that llm focus works wouldn't we want to keep the CURRENT_FILE_INSTRUCTION at the very end or at the start or is that irrelevant coz of relatively smaller context here?

This makes sense Ara, I think this is something which worked offline but I am yet to do a proper evaluation of the prompt rendering. Also, since this is just experimental feature for now, we can keep it as is, based on the offline experiment and results, I can update the ordering :)

hitesh-1997 · 2024-10-09T12:26:17Z

Thanks @valerybugakov @abeatrix @arafatkatze for the review
The current PR is not the final version we expect to merge, I just wanted to save work so people can pull and quickly experiment with the feature. There will be a lot of changes in the PR so, I think we can ignore the detailed review for now :) since most likely they will change.
Also, added a comment in the description and marked the PR in draft stage to indicate the same :)

hitesh-1997 · 2024-10-10T19:28:16Z

Hi @valerybugakov
I am done with the first iteration of model changes, please feel free to review the PR :)

valerybugakov

Hey Hitesh! I reviewed only part of the updated PR today. I will complete it on Monday morning.

valerybugakov

Great work! Excited for v0 🎉

The only significant comment is related to using the existing pipeline for network requests. It should simplify the code, make it easier for teammates to dogfood this feature (no manual API key management), and ensure we don't spend time ripping it off later.

valerybugakov · 2024-10-14T09:24:52Z

vscode/src/completions/context/context-mixer.ts

+        private strategyFactory: ContextStrategyFactory,
+        dataCollectionEnabled = false


Let's convert it to an object to have new ContextMixer({ strategyFactory, dataCollectionEnabled: false }) instead of new ContextMixer(strategyFactory, false) for readability.

valerybugakov · 2024-10-14T09:27:53Z

vscode/src/autoedits/autoedits-provider.ts

+function convertTokensToChars(tokens: number) {
+    return tokens * 4
+}


Can we use tokensToChars from @sourcegraph/cody-shared?

valerybugakov · 2024-10-14T09:29:56Z

vscode/src/autoedits/prompt-provider.ts

+
+    getModelResponse(model: string, apiKey: string, prompt: PromptProviderResponse): Promise<string>
+}
+export class OpenAIPromptProvider implements PromptProvider {


Could you extract OpenAIPromptProvider and DeepSeekPromptProvider into separate files?

valerybugakov · 2024-10-14T09:33:06Z

vscode/src/completions/context/context-data-logging.ts

-                return new RecentEditsRetriever(10 * 60 * 1000)
+                return new RecentEditsRetriever({
+                    maxAgeMs: 10 * 60 * 1000,
+                })


abeatrix

Just tested the latest changes and noticed there is some formatting issue where the code is inserted at the unexpected position:

It works as expected if triggered in-line though so this might be some edge cases that can be looked into in follow-ups:

Not a blocker: it might be helpful if we could limit the change block to a smaller section since sometimes it would start at the function-level when the changes are only needed for a variable.

Since the feature is behind a feature flag and works as expected in most cases, approving to unblock after comments from @valerybugakov are addressed 😄

abeatrix · 2024-10-14T16:40:08Z

vscode/src/supercompletions/supercompletion-provider.ts

-        this.recentEditsRetriever = new RecentEditsRetriever(EDIT_HISTORY, workspace)
+        this.recentEditsRetriever = new RecentEditsRetriever(
+            {
+                maxAgeMs: EDIT_HISTORY,


Suggested change

maxAgeMs: EDIT_HISTORY,

maxAgeMs: EDIT_HISTORY_TIMEOUT,

nit: clear naming that matches the SUPERCOMPLETION_TIMEOUT

abeatrix · 2024-10-14T16:42:26Z

vscode/src/autoedits/renderer.ts

+    }
+
+    private logDiff(uri: vscode.Uri, codeToRewrite: string, predictedText: string, prediction: string) {
+        const predictedCodeXML = `<code>\n${predictedText}\n</code>`


not sure if this would be an issue but previously when we use <code> as the XML tags the LLM sometimes would misunderstood it as part of the code if the code is HTML.

this is to just log on the console, we are not pushing this to LLM in the prompt.
But this is a really nice insight, thanks for that :)

hitesh-1997 · 2024-10-16T10:42:08Z

Thanks for the review @abeatrix @valerybugakov
There is one outstanding comment still to address (using existing infra for model integration), Creating a seperate PR for that since the current PR is quite long. Will also try to address UI changes in the same.

valerybugakov reviewed Oct 9, 2024

View reviewed changes

abeatrix reviewed Oct 9, 2024

View reviewed changes

arafatkatze reviewed Oct 9, 2024

View reviewed changes

hitesh-1997 marked this pull request as ready for review October 10, 2024 16:25

hitesh-1997 marked this pull request as draft October 10, 2024 16:25

hitesh-1997 and others added 8 commits October 10, 2024 21:55

adding autoedits support

7230317

fix prompt templates as per fine-tuning

f63e4eb

fix ci test cases

e1caec2

logs diff at the cursor position

53bb858

add info on lines

602822d

display diff with hover provider

85d7d93

change model with fix indentation error

eb3f0a1

use config from the vscode settings

9a01733

hitesh-1997 force-pushed the hitesh/autoedits-integeration branch from bd36253 to 9a01733 Compare October 10, 2024 16:26

hitesh-1997 added 3 commits October 10, 2024 22:16

minor fix

6a3113e

minor fix

e488280

add decorations for rendering

890b3fb

hitesh-1997 marked this pull request as ready for review October 10, 2024 19:27

valerybugakov reviewed Oct 11, 2024

View reviewed changes

valerybugakov reviewed Oct 14, 2024

View reviewed changes

abeatrix approved these changes Oct 14, 2024

View reviewed changes

hitesh-1997 and others added 2 commits October 16, 2024 15:48

address pr comments

0d7dc7a

Merge branch 'main' into hitesh/autoedits-integeration

8d28c07

hitesh-1997 merged commit 9b9f64c into main Oct 16, 2024
20 checks passed

hitesh-1997 deleted the hitesh/autoedits-integeration branch October 16, 2024 10:42

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

adding autoedits support #5845

adding autoedits support #5845

hitesh-1997 commented Oct 8, 2024 •

edited

Loading

valerybugakov Oct 9, 2024

hitesh-1997 Oct 10, 2024

valerybugakov Oct 9, 2024

hitesh-1997 Oct 10, 2024 •

edited

Loading

valerybugakov Oct 14, 2024

valerybugakov Oct 9, 2024

hitesh-1997 Oct 10, 2024

abeatrix left a comment

abeatrix Oct 9, 2024

hitesh-1997 Oct 10, 2024

arafatkatze Oct 9, 2024

hitesh-1997 Oct 10, 2024

hitesh-1997 commented Oct 9, 2024

hitesh-1997 commented Oct 10, 2024

valerybugakov left a comment

valerybugakov left a comment

valerybugakov Oct 14, 2024

hitesh-1997 Oct 16, 2024

valerybugakov Oct 14, 2024

hitesh-1997 Oct 16, 2024

valerybugakov Oct 14, 2024

hitesh-1997 Oct 16, 2024

valerybugakov Oct 14, 2024

abeatrix left a comment

abeatrix Oct 14, 2024

hitesh-1997 Oct 16, 2024

abeatrix Oct 14, 2024

hitesh-1997 Oct 16, 2024

hitesh-1997 commented Oct 16, 2024

		experimentalSupercompletions: false,
		experimentalAutoedits: false,

		const response = await axios.post(
		'https://api.openai.com/v1/chat/completions',

		private strategyFactory: ContextStrategyFactory,
		dataCollectionEnabled = false

adding autoedits support #5845

adding autoedits support #5845

Conversation

hitesh-1997 commented Oct 8, 2024 • edited Loading

Context

Steps to run the autoedits (in debug mode):

Test plan

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hitesh-1997 Oct 10, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

abeatrix left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hitesh-1997 commented Oct 9, 2024

hitesh-1997 commented Oct 10, 2024

valerybugakov left a comment

Choose a reason for hiding this comment

valerybugakov left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

abeatrix left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hitesh-1997 commented Oct 16, 2024

hitesh-1997 commented Oct 8, 2024 •

edited

Loading

hitesh-1997 Oct 10, 2024 •

edited

Loading