fix: bracket stripping in gemini responses #8280

uinstinct · 2025-10-16T05:38:56Z

Description

Use streamSse instead of processGeminiResponse when receiving responses from Gemini streamChat .

resolves CON-4266
closes #2651

AI Code Review

Team members only: AI review runs automatically when PR is opened or marked ready for review
Team members can also trigger a review by commenting @continue-review

Checklist

[] I've read the contributing guide
[] The relevant docs, if any, have been updated or created
[] The relevant tests, if any, have been updated or created

Screen recording or screenshot

before.mp4

after.mp4

Tests

[ What tests were added or updated to ensure the changes work as expected? ]

Summary by cubic

Switched Gemini streamChat to use streamSse for proper SSE handling and direct chunk streaming, fixing broken/laggy responses and simplifying the streaming path.

Bug Fixes
- Iterate streamSse(response) and yield chunks directly.
- Remove processGeminiResponse in the Gemini streaming flow.

cubic-dev-ai

1 issue found across 1 file

Prompt for AI agents (all 1 issues)


Understand the root cause of the following 1 issues and fix them.


<file name="core/llm/llms/Gemini.ts">

<violation number="1" location="core/llm/llms/Gemini.ts:418">
streamChatGemini now yields raw SSE payloads instead of ChatMessage objects, so downstream rendering breaks when accessing message.role/content.</violation>
</file>

_{React with 👍 or 👎 to teach cubic. Mention @cubic-dev-ai to give feedback, ask questions, or re-run the review.}

core/llm/llms/Gemini.ts

RomneyDa

screen recording seems promising, nice!

Wondering how this is working when it removes the translation of gemini chunks into chat chunks?
Let's keep processGeminiResponse/implement the fix for processGeminiResponse since it is used by VertexAI
Let's also implement for openai adapters!

uinstinct · 2025-10-16T06:18:26Z

Wondering how this is working when it removes the translation of gemini chunks into chat chunks?

Ah, missed it. It works for normal chat messages but not for tool calls. Fixed that!

Let's keep processGeminiResponse/implement the fix for processGeminiResponse since it is used by VertexAI
Let's also implement for openai adapters!

implemented!

RomneyDa

Something still feels off here, I think streamSse already does the buffer splitting which makes e.g. this line redundant:
const parts = buffer.split("\n,");

Thinking you should be able to remove streamResponse entirely from openai adapter and just use streamSse?

I think Vertex LLM class usage of processGeminiResponse would also require using streamSse. Alternatively, you could change processGeminiResponse to take response directly and use streamSse on it

cubic-dev-ai

Reviewed changes from recent commits (found 1 issue).

1 issue found across 2 files

Prompt for AI agents (all 1 issues)


Understand the root cause of the following 1 issues and fix them.


<file name="core/llm/llms/Gemini.ts">

<violation number="1" location="core/llm/llms/Gemini.ts:409">
`streamSse(response)` yields parsed SSE objects, not raw string chunks, so feeding it into `processGeminiResponse` (which expects strings and re-parses JSON) will immediately corrupt the buffer (`[object Object]`) and break Gemini streaming. Please pass the original response stream or update `processGeminiResponse` to accept objects.</violation>
</file>

_{React with 👍 or 👎 to teach cubic. Mention @cubic-dev-ai to give feedback, ask questions, or re-run the review.}

cubic-dev-ai · 2025-10-16T06:33:49Z

core/llm/llms/Gemini.ts

-    )) {
-      yield message;
+
+    for await (const chunk of this.processGeminiResponse(streamSse(response))) {


streamSse(response) yields parsed SSE objects, not raw string chunks, so feeding it into processGeminiResponse (which expects strings and re-parses JSON) will immediately corrupt the buffer ([object Object]) and break Gemini streaming. Please pass the original response stream or update processGeminiResponse to accept objects.

Prompt for AI agents

Address the following comment on core/llm/llms/Gemini.ts at line 409: <comment>`streamSse(response)` yields parsed SSE objects, not raw string chunks, so feeding it into `processGeminiResponse` (which expects strings and re-parses JSON) will immediately corrupt the buffer (`[object Object]`) and break Gemini streaming. Please pass the original response stream or update `processGeminiResponse` to accept objects.</comment> <file context> @@ -415,7 +406,7 @@ class Gemini extends BaseLLM { }); - for await (const chunk of streamSse(response)) { + for await (const chunk of this.processGeminiResponse(streamSse(response))) { yield chunk; } </file context>

uinstinct · 2025-10-16T06:58:07Z

Something still feels off here, I think streamSse already does the buffer splitting which makes e.g. this line redundant:
const parts = buffer.split("\n,");

it is needed for parsing the gemini chat response

Thinking you should be able to remove streamResponse entirely from openai adapter and just use streamSse?

I think Vertex LLM class usage of processGeminiResponse would also require using streamSse. Alternatively, you could change processGeminiResponse to take response directly and use streamSse on it

implemented!

cubic-dev-ai

Reviewed changes from recent commits (found 1 issue).

1 issue found across 2 files

Prompt for AI agents (all 1 issues)


Understand the root cause of the following 1 issues and fix them.


<file name="core/llm/llms/VertexAI.ts">

<violation number="1" location="core/llm/llms/VertexAI.ts:290">
processGeminiResponse expects raw string chunks, but streamSse yields parsed objects. Passing streamSse here causes the handler to concatenate &quot;[object Object]&quot; and fail parsing, breaking Gemini streaming.</violation>
</file>

_{React with 👍 or 👎 to teach cubic. Mention @cubic-dev-ai to give feedback, ask questions, or re-run the review.}

cubic-dev-ai · 2025-10-16T07:23:20Z

core/llm/llms/VertexAI.ts

      signal,
    });
-    yield* this.geminiInstance.processGeminiResponse(streamResponse(response));
+    yield* this.geminiInstance.processGeminiResponse(streamSse(response));


processGeminiResponse expects raw string chunks, but streamSse yields parsed objects. Passing streamSse here causes the handler to concatenate "[object Object]" and fail parsing, breaking Gemini streaming.

Prompt for AI agents

Address the following comment on core/llm/llms/VertexAI.ts at line 290: <comment>processGeminiResponse expects raw string chunks, but streamSse yields parsed objects. Passing streamSse here causes the handler to concatenate "[object Object]" and fail parsing, breaking Gemini streaming.</comment> <file context> @@ -287,7 +287,7 @@ class VertexAI extends BaseLLM { signal, }); - yield* this.geminiInstance.processGeminiResponse(streamResponse(response)); + yield* this.geminiInstance.processGeminiResponse(streamSse(response)); } </file context>

RomneyDa · 2025-10-16T15:44:16Z

Something still feels off here, I think streamSse already does the buffer splitting which makes e.g. this line redundant:
const parts = buffer.split("\n,");

it is needed for parsing the gemini chat response

What I meant is I think streamSse already splits chunks by new lines so the split at "\n," is extraneous

continue/packages/fetch/src/stream.ts

Lines 124 to 150 in 579bcbf

    
           export async function* streamSse(response: Response): AsyncGenerator<any> { 
        
             let buffer = ""; 
        
             for await (const value of streamResponse(response)) { 
        
               buffer += value; 
        
               let position: number; 
        
               while ((position = buffer.indexOf("\n")) >= 0) { 
        
                 const line = buffer.slice(0, position); 
        
                 buffer = buffer.slice(position + 1); 
        
                 const { done, data } = parseSseLine(line); 
        
                 if (done) { 
        
                   break; 
        
                 } 
        
                 if (data) { 
        
                   yield data; 
        
                 } 
        
               } 
        
             } 
        
             if (buffer.length > 0) { 
        
               const { done, data } = parseSseLine(buffer); 
        
               if (!done && data) { 
        
                 yield data; 
        
               } 
        
             } 
        
           }

RomneyDa · 2025-10-16T15:47:33Z

core/llm/llms/Gemini.ts

    stream: AsyncIterable<string>,
  ): AsyncGenerator<ChatMessage> {
    let buffer = "";
    for await (const chunk of stream) {


why don't we just streamSse(stream) directly in processGeminiReponse?

implemented!

RomneyDa

Nice!

sestinj · 2025-10-17T18:48:59Z

🎉 This PR is included in version 1.3.0 🎉

The release is available on:

Your semantic-release bot 📦🚀

sestinj · 2025-10-18T15:55:36Z

🎉 This PR is included in version 1.27.0 🎉

The release is available on:

Your semantic-release bot 📦🚀

sestinj · 2025-10-21T22:53:40Z

🎉 This PR is included in version 1.5.0 🎉

The release is available on:

Your semantic-release bot 📦🚀

sestinj · 2025-10-22T21:46:09Z

🎉 This PR is included in version 1.30.0 🎉

The release is available on:

Your semantic-release bot 📦🚀

fix: use streamsse for streaming chat in gemini

779b3e4

uinstinct requested a review from a team as a code owner October 16, 2025 05:38

uinstinct requested review from Patrick-Erichsen and removed request for a team October 16, 2025 05:38

github-project-automation bot added this to Issues and PRs Oct 16, 2025

github-project-automation bot moved this to Todo in Issues and PRs Oct 16, 2025

dosubot bot added the size:XS This PR changes 0-9 lines, ignoring generated files. label Oct 16, 2025

cubic-dev-ai bot reviewed Oct 16, 2025

View reviewed changes

core/llm/llms/Gemini.ts Outdated Show resolved Hide resolved

RomneyDa requested changes Oct 16, 2025

View reviewed changes

github-project-automation bot moved this from Todo to In Progress in Issues and PRs Oct 16, 2025

remove bracket stripping in process gemini response

8c52580

uinstinct changed the title ~~fix: use streamsse for streaming chat in gemini~~ fix: bracket stripping in gemini responses Oct 16, 2025

remove bracket stripping from gemini openai adapter

31c7383

dosubot bot added size:S This PR changes 10-29 lines, ignoring generated files. and removed size:XS This PR changes 0-9 lines, ignoring generated files. labels Oct 16, 2025

uinstinct requested a review from RomneyDa October 16, 2025 06:18

RomneyDa requested changes Oct 16, 2025

View reviewed changes

cubic-dev-ai bot reviewed Oct 16, 2025

View reviewed changes

use streamsse in vertexai

f72199d

dosubot bot added size:M This PR changes 30-99 lines, ignoring generated files. and removed size:S This PR changes 10-29 lines, ignoring generated files. labels Oct 16, 2025

uinstinct requested a review from RomneyDa October 16, 2025 06:58

cubic-dev-ai bot reviewed Oct 16, 2025

View reviewed changes

RomneyDa requested changes Oct 16, 2025

View reviewed changes

uinstinct added 2 commits October 17, 2025 07:49

Merge branch 'main' into gemini-streamsse

baad1e2

do not split lines again in processgeminiresponse

99d9cd0

dosubot bot removed the size:M This PR changes 30-99 lines, ignoring generated files. label Oct 17, 2025

dosubot bot added the size:L This PR changes 100-499 lines, ignoring generated files. label Oct 17, 2025

uinstinct requested a review from RomneyDa October 17, 2025 02:29

RomneyDa approved these changes Oct 17, 2025

View reviewed changes

dosubot bot added the lgtm This PR has been approved by a maintainer label Oct 17, 2025

RomneyDa merged commit 6d08fdc into continuedev:main Oct 17, 2025
53 of 55 checks passed

github-project-automation bot moved this from In Progress to Done in Issues and PRs Oct 17, 2025

github-actions bot locked and limited conversation to collaborators Oct 17, 2025

sestinj added the released label Oct 17, 2025

uinstinct deleted the gemini-streamsse branch October 20, 2025 05:16

fix: bracket stripping in gemini responses #8280

fix: bracket stripping in gemini responses #8280

Conversation

uinstinct commented Oct 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

AI Code Review

Checklist

Screen recording or screenshot

Tests

Summary by cubic

Uh oh!

cubic-dev-ai bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

RomneyDa left a comment

Choose a reason for hiding this comment

Uh oh!

uinstinct commented Oct 16, 2025

Uh oh!

RomneyDa left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cubic-dev-ai bot left a comment

Choose a reason for hiding this comment

Uh oh!

cubic-dev-ai bot Oct 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

uinstinct commented Oct 16, 2025

Uh oh!

cubic-dev-ai bot left a comment

Choose a reason for hiding this comment

Uh oh!

cubic-dev-ai bot Oct 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

RomneyDa commented Oct 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

RomneyDa Oct 16, 2025

Choose a reason for hiding this comment

Uh oh!

uinstinct Oct 17, 2025

Choose a reason for hiding this comment

Uh oh!

RomneyDa left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

sestinj commented Oct 17, 2025

Uh oh!

sestinj commented Oct 18, 2025

Uh oh!

sestinj commented Oct 21, 2025

Uh oh!

sestinj commented Oct 22, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

uinstinct commented Oct 16, 2025 •

edited

Loading

RomneyDa left a comment •

edited

Loading

cubic-dev-ai bot Oct 16, 2025 •

edited

Loading

cubic-dev-ai bot Oct 16, 2025 •

edited

Loading

RomneyDa commented Oct 16, 2025 •

edited

Loading