Support streaming of partial tool data #278640

lramos15 · 2025-11-20T19:03:33Z

I'm not sure I love this flow, but I wanted each tool to be able to handle its own messaging while data streams in because it would likely know best. If we want to simplify the API we could just have one centralized streaming tool call processor in Copilot that handles everything before any tools have prepare called.

The current flow is

Stream gets a prepareToolInvocation pushed to it with partial stream data
Core calls handleToolStream with that data
The returned invocation message is then rendered as a progress message which updates the progress message rather than stacking messages

This reverts commit f502d22.

This reverts commit 8556686.

This reverts commit 4c4db3a.

vs-code-engineering · 2025-11-20T19:03:58Z

📬 CODENOTIFY

The following users are being notified based on files changed in this PR:

@roblourens

Matched files:

src/vs/workbench/contrib/chat/browser/chatListRenderer.ts

Copilot

Pull Request Overview

This PR adds support for streaming partial tool data during tool invocations. The implementation allows tools to provide real-time progress updates while their input data is being streamed, improving user experience by showing tool preparation status rather than a generic "working" indicator.

Key Changes:

Added handleToolStream method to the tool system for processing partial streaming data
Introduced IToolInvocationStreamContext and IStreamedToolInvocation interfaces for streaming context and results
Modified progress message handling to support replacing previous messages via replacesPreviousMessage flag
Updated ChatPrepareToolInvocationPart to include optional streamData field

Reviewed Changes

Copilot reviewed 13 out of 14 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
src/vs/workbench/contrib/chat/common/languageModelToolsService.ts	Added streaming interfaces and method signature for `handleToolStream`
src/vs/workbench/contrib/chat/browser/languageModelToolsService.ts	Implemented `handleToolStream` method with tool activation and error handling
src/vs/workbench/contrib/chat/common/chatService.ts	Added `replacesPreviousMessage` flag to `IChatProgressMessage` and `streamData` to `IChatPrepareToolInvocationPart`
src/vs/workbench/contrib/chat/common/chatModel.ts	Implemented logic to replace previous progress messages instead of appending
src/vs/workbench/contrib/chat/browser/chatListRenderer.ts	Updated "working" indicator logic to hide when `prepareToolInvocation` is present
src/vs/workbench/api/common/extHostTypes.ts	Updated `ChatPrepareToolInvocationPart` constructor to accept optional `streamData` parameter
src/vs/workbench/api/common/extHostTypeConverters.ts	Added conversion logic for `streamData` field
src/vs/workbench/api/common/extHostLanguageModelTools.ts	Implemented `$handleToolStream` method in extension host
src/vs/workbench/api/common/extHostChatAgents2.ts	Updated `prepareToolInvocation` to accept `streamData` parameter
src/vs/workbench/api/common/extHost.protocol.ts	Added `$handleToolStream` to protocol interface
src/vs/workbench/api/browser/mainThreadLanguageModelTools.ts	Added `handleToolStream` implementation in main thread
src/vs/workbench/api/browser/mainThreadChatAgents2.ts	Implemented progress chunk handling for tool stream invocations
src/vs/workbench/contrib/chat/test/common/mockLanguageModelToolsService.ts	Added stub implementation and updated signature for test mock

Copilot · 2025-11-20T19:04:45Z

src/vs/workbench/contrib/chat/browser/chatListRenderer.ts

 			((lastPart.kind === 'textEditGroup' || lastPart.kind === 'notebookEditGroup') && lastPart.done && !partsToRender.some(part => part.kind === 'toolInvocation' && !IChatToolInvocation.isComplete(part))) ||
 			(lastPart.kind === 'progressTask' && lastPart.deferred.isSettled) ||
-			lastPart.kind === 'prepareToolInvocation' || lastPart.kind === 'mcpServersStarting'
+			lastPart.kind === 'mcpServersStarting'


The removal of lastPart.kind === 'prepareToolInvocation' from this condition creates inconsistency. The earlier check at line 774 prevents showing "working" when prepareToolInvocation is present, but this line determines when to show "working" based on the last part. If prepareToolInvocation is the last part, the earlier check would prevent showing "working", making this change unnecessary. Consider documenting why prepareToolInvocation should not be treated as a completion state here.

src/vs/workbench/api/browser/mainThreadChatAgents2.ts

src/vs/workbench/contrib/chat/common/chatModel.ts

connor4312 · 2025-11-20T19:26:00Z

src/vscode-dts/vscode.proposed.chatParticipantAdditions.d.ts

+		/**
+		 * Raw argument payload, such as the streamed JSON fragment from the language model.
+		 */
+		readonly rawInput?: unknown;


I question whether this is useful/something we'd want to expose in the API like this. Tools are generic to any specific model. The tool doesn't know how the input is encoded. JSON is the default right now, but OAI's has `raw tool calls, and maybe in the future something like Toon gains traction.

We can also do some transforms on the call object after validation -- for example, Sonnet has a propensity to double JSON-encode nested objects and arrays and we can correct those.

That said I really like the idea of being able to show e.g. the files that are about to be edited while the call is streaming in... maybe we just

spec this is a fragment of a JSON-encoded string (and it's up to the implementation to convert that if needed)

explicitly say the input might be transformed before the actual tool call

(of course this is all proposed API right now, but for the general shape of things)

I think it's safe to say handleToolStream is just a string that will be partial data from whatever the stream returns. Then when prepareInvocation is called we guarnatee it's in some format the tool expects. My guess is the tool will have to know how to handle various formats.

The benefit of exposing this is if you have a durable JSON parser that can parse incomplete data you could likely render that file foo.js is being created and it has x lines at the moment

I agree, but I'm suggesting we have a way to identify that. Maybe there's a mimeType on the stream so the producer/consumer can handle the data correctly? OAI "raw" tools would emit text/plain as their mimetype. (We don't have any of these yet, but they have said that raw works a lot better for apply_patch and it's on our todo to adopt that)

roblourens

Can I see the extension side of it too?

src/vs/workbench/api/browser/mainThreadChatAgents2.ts

src/vscode-dts/vscode.proposed.chatParticipantAdditions.d.ts

lramos15 · 2025-11-24T13:57:39Z

Can I see the extension side of it too?

Please see microsoft/vscode-copilot-chat#2165

connor4312 · 2025-11-24T22:26:35Z

src/vs/workbench/api/browser/mainThreadLanguageModelTools.ts

 					}
 				},
 				prepareToolInvocation: (context, token) => this._proxy.$prepareToolInvocation(id, context, token),
+				handleToolStream: (context, token) => this._proxy.$handleToolStream(id, context, token),


Should this only be defined if the tool in the ext host side has this method defined?

How would we know if the tool in the ext host has this defined? Isn't this the same as prepareToolInvocation?

connor4312 · 2025-11-24T22:33:31Z

src/vs/workbench/contrib/chat/browser/chatContentParts/chatPrepareToolInvocationPart.ts

+		// Hide when following content contains parts that are not prepareToolInvocation or progressMessage
+		// This is similar to how progressMessage parts hide when other content arrives
+		const followingContent = context.content.slice(context.contentIndex + 1);
+		this.isHiddenByFollowingContent = followingContent.some(part => part.kind !== 'prepareToolInvocation' && part.kind !== 'progressMessage');


I'm not a huge fan of of this approach of rendering a new part for each piece of the stream. If we imagine a pedantic case where I have a tool call emitting a word at a time generating an enormous file this could get expensive even for relatively small files of a few hundred KB.

How I would imagine it should work is the stream gets passed to invokeTool() in the LanguageModelToolsService, which internally could create a content part that includes an IObservable with whatever information should get shown in the UI

We already do this approach for progress which is why I followed it

vscode/src/vs/workbench/contrib/chat/browser/chatContentParts/chatProgressContentPart.ts

Lines 51 to 53 in 95bfedc

const followingContent = context.content.slice(context.contentIndex + 1);

this.showSpinner = forceShowSpinner ?? shouldShowSpinner(followingContent, context.element);

this.isHidden = forceShowMessage !== true && followingContent.some(part => part.kind !== 'progressMessage');

since effectively this is how we want this to behave as well. But I guess with progress that is a part that gets emitted a lot less

connor4312 · 2025-11-24T22:34:56Z

src/vs/workbench/contrib/chat/browser/chatContentParts/chatPrepareToolInvocationPart.ts

+		};
+
+		try {
+			const streamResult = await this.languageModelToolsService.handleToolStream(toolData.id, streamContext, CancellationToken.None);


We cannot do side-effects like this from the chat widgets, as this will not work for background sessions.

One could say that partial results don't matter if the user isn't actively looking at the window, but we then might get into weird states if the user opens or closes the session mid-stream. Better imo to keep the view and the data model separate.

roblourens · 2025-11-26T00:17:31Z

src/vs/workbench/contrib/chat/browser/chatContentParts/chatPrepareToolInvocationPart.ts

+
+		// If following content contains parts that are not prepareToolInvocation or progressMessage,
+		// we need to re-render to hide this part (similar to progressMessage behavior)
+		const shouldBeHidden = followingContent.some(part => part.kind !== 'prepareToolInvocation' && part.kind !== 'progressMessage');


I don't think we should follow this pattern of hiding things. Now we have a toolcallid so we can associate the incomplete call with the real call, it seems to me like the incoplete call should just sort of resolve into the real call widget.

lramos15 added 9 commits November 20, 2025 10:51

Start work on handling tool stream

4384189

Start working on rendering tool progress

8faa408

Handle complete better

f40703c

Inovcation message

f502d22

Revert "Inovcation message"

8556686

This reverts commit f502d22.

Reapply "Inovcation message"

4c4db3a

This reverts commit 8556686.

Revert "Reapply "Inovcation message""

bb9ca62

This reverts commit 4c4db3a.

Handle updating progress

a4c4ce7

Better messages

5c8e7d9

Copilot AI review requested due to automatic review settings November 20, 2025 19:03

lramos15 enabled auto-merge (squash) November 20, 2025 19:03

lramos15 self-assigned this Nov 20, 2025

lramos15 requested review from connor4312, justschen, roblourens and sbatten November 20, 2025 19:04

lramos15 added this to the November 2025 milestone Nov 20, 2025

Copilot AI reviewed Nov 20, 2025

View reviewed changes

connor4312 reviewed Nov 20, 2025

View reviewed changes

lramos15 requested review from justschen and removed request for justschen November 20, 2025 19:54

Have progress re-render if content changes

3494a28

roblourens reviewed Nov 21, 2025

View reviewed changes

src/vs/workbench/api/browser/mainThreadChatAgents2.ts Outdated Show resolved Hide resolved

src/vscode-dts/vscode.proposed.chatParticipantAdditions.d.ts Show resolved Hide resolved

lramos15 mentioned this pull request Nov 24, 2025

Handle partial tool streaming microsoft/vscode-copilot-chat#2165

Draft

lramos15 and others added 4 commits November 24, 2025 10:42

Merge branch 'main' into lramos15/fuzzy-marten

edde8ce

Fix import

79f690c

Move prepare to view layer

157d91b

Clean up diff

7ed00f6

Pass tool call id through

e143288

lramos15 disabled auto-merge November 24, 2025 21:41

Pin it

95bfedc

lramos15 requested a review from roblourens November 24, 2025 22:00

connor4312 requested changes Nov 24, 2025

View reviewed changes

lramos15 added 2 commits November 25, 2025 10:07

Merge branch 'main' into lramos15/fuzzy-marten

4e96567

Merge branch 'main' into lramos15/fuzzy-marten

d700026

roblourens reviewed Nov 26, 2025

View reviewed changes

	const followingContent = context.content.slice(context.contentIndex + 1);
	this.showSpinner = forceShowSpinner ?? shouldShowSpinner(followingContent, context.element);
	this.isHidden = forceShowMessage !== true && followingContent.some(part => part.kind !== 'progressMessage');

Support streaming of partial tool data #278640

Are you sure you want to change the base?

Support streaming of partial tool data #278640

Conversation

lramos15 commented Nov 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

vs-code-engineering bot commented Nov 20, 2025

📬 CODENOTIFY

@roblourens

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Copilot AI Nov 20, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

connor4312 Nov 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lramos15 Nov 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

connor4312 Nov 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

roblourens left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

lramos15 commented Nov 24, 2025

Uh oh!

connor4312 Nov 24, 2025

Choose a reason for hiding this comment

Uh oh!

lramos15 Nov 25, 2025

Choose a reason for hiding this comment

Uh oh!

connor4312 Nov 24, 2025

Choose a reason for hiding this comment

Uh oh!

lramos15 Nov 25, 2025

Choose a reason for hiding this comment

Uh oh!

connor4312 Nov 24, 2025

Choose a reason for hiding this comment

Uh oh!

roblourens Nov 26, 2025

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

lramos15 commented Nov 20, 2025 •

edited

Loading

connor4312 Nov 20, 2025 •

edited

Loading

lramos15 Nov 20, 2025 •

edited

Loading

connor4312 Nov 20, 2025 •

edited

Loading