Streaming text responses

[sixlive]

Description

Enabling streaming of generated text, which will be particularly useful for real-time applications and improved user experiences.

[petervandijck]

I would like to second that this is a must-have for even starting to use the package 👍

[tognee]

Hi, is there a timeframe for this feature?
I could help to develop it if there is a clear definition on how it should be implemented.

The first thing that needs to be discussed is if the streaming option should be an extension of the Prism::text static function or a new Prism::streamingText function.

After that the implementation seems simple, there should be a generate function that calls the API endpoint with the streaming parameter and returns a generator. This generator should collect the data from the request and parses the SSE events coming from the provider, map each chunk to a new ChunkResponse object and yield the object one at a time.

This ChunkResponse should be following the OpenAI Chunk type or should it be something custom?

If we can get to a specification together I could start working on this feature

[tomtev]

Really need this also :)

[petervandijck]

Same here, waiting for it to start implementing prism

…

On Sat, Nov 23, 2024 at 9:30 AM Tommy Vedvik ***@***.***> wrote: Really need this also :) — Reply to this email directly, view it on GitHub <#27 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAD6PVG2R2A3VPQOPTWZ5C32CCGSFAVCNFSM6AAAAABQBQRNEGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDIOJVGQ4TONZUGY> . You are receiving this because you commented.Message ID: ***@***.***>

-- -- My new company: we do AI: https://www.getsimplyput.ai/ https://www.linkedin.com/in/petervandijck/ http://petervandijck.com/ Cell: +57 (300) 836-7690

[heychazza]

Hey @sixlive, thanks for creating this package! Do you have any idea when streaming will be a thing?

[mafrasil]

+1 @sixlive - thanks for the package, is awesome! also eagerly looking to have stream available

[vesper8]

I have achieved this functionality using a python library and websockets to stream to my frontend. That was ~2 years ago already.

If you're struggling to implement this fully, then as a first phase you could maybe just expose the text streaming and let us handle it as we wish, so if we want to broadcast to a websocket channel we can do that.

[tognee]

@vesper8 I don't think this library is trying to implement backend to frontend communication. It needs to recieve the API stream directly from the providers (that usually is always an SSE (Server-Sent Event) stream) and map it into a standard Structured ResponseChunks that can be used with any model and provider.
Frontend communication is the next step for developers that are using the library, not for the devs of this one.