Re-enable Response Stream

[ollieanwyll]

Hey guys, I've noticed that the main branch has had the feature of streaming the response back into the FE in real-time removed.
Does any one know why this is? or is able to assist in taking main and re-enabling the feature?
Cheers.

[travisparkerm]

Just want to second this, I'm trying to figure it out for myself

[groovybits]

I do this in my fork still, it was removed it seems in this repo, you can stream it by tokens but that requires more work and it seems here it was removed perhaps because it is harder to deal with that case?

https://github.com/groovybits/gaib see in utils/makechain.ts and pages/api/chat.ts for how that all works / differences.

I have done a lot of stuff there for having more fine grained control of the OpenAI and Pinecone handing though, so not an easy comparison (it handles various things like history buffer overflows, retries, avoiding crashing when the prompt is too big...)

[travisparkerm]

Awesome, thanks for this! I dove in and its a bit above my skill level but I will keep working through it! :)

[jesse-stewart]

I've been trying to get streaming working on Vercel and I am not having any luck - My issues seems to be related to pinecone not working as an edge function - Do you mind sharing your hosting provider? I think I might try deploying it elsewhere and see if it works.

[groovybits]

Vercel's times out too fast, even the Pro plan I cannot get it to stream at all and times out / retries over and over again.

This is a problem I see right now with serverless, maybe needs an edge function on Vercels, I do not have this issue solved. I run on a basic box still :). GCP Cloudrun sounds best cost for functionality still working from my analysis and best compromise for now. Firebase doesn't seem to allow the complexity, I have a branch and got it "almost in there" but it doesn't work, technically runs locally even. Timeouts and permissions etc are all hard for this.

[Steve979311]

I noticed that in the chat.ts file there is the boolean isStreaming parameter, what is it for?