Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Move streaming to StreamCallbacks.jl #65

Open
svilupp opened this issue Sep 26, 2024 · 3 comments
Open

Move streaming to StreamCallbacks.jl #65

svilupp opened this issue Sep 26, 2024 · 3 comments

Comments

@svilupp
Copy link
Contributor

svilupp commented Sep 26, 2024

I would like to propose moving streaming functionality to a separate package (made by me): https://github.com/svilupp/StreamCallbacks.jl

Why?

  • provides an implementation that can be shared across other providers (Anthropic, OpenAI, etc)
  • Implements more of the SSE "standard" (eg, includes event names, does proper splitting, catches message overflow between "masterchunks", etc)
  • Allows easy extensibility for users (to choose the sink, logging, etc.)

Downsides

  • Opinionated
  • Extra-dependency (lightweight though)
  • By default, it builds the standard Response afterwards (to allow standard downstream functionality as well), which might be wasteful for some users/workflows (but the cost and time are sooo tiny compared to the generation time)

I've been using it in PromptingTools.jl for a few releases already and it seems to work well.

@svilupp
Copy link
Contributor Author

svilupp commented Oct 23, 2024

Bump here.

Is there any interest in refactoring the functionality with a new dep? The current implementation is not correct and will lead to more errors.

@cpfiffer
Copy link
Collaborator

I would say it's probably better to separate them out -- I'd be happy to see StreamingCallbacks.jl implemented here. As to this point:

By default, it builds the standard Response afterwards (to allow standard downstream functionality as well), which might be wasteful for some users/workflows (but the cost and time are sooo tiny compared to the generation time)

Not a concern on my end, if anything that strikes me as pretty ergonomic.

@roryl23
Copy link
Collaborator

roryl23 commented Oct 29, 2024

If the current implementation is not correct and will lead to errors, I say full steam ahead.

devin-ai-integration bot added a commit to svilupp/OpenAI.jl that referenced this issue Nov 17, 2024
- Add StreamCallbacks.jl as a dependency
- Update request_body_live to use StreamCallbacks.jl
- Add support for custom stream parsing and sinks
- Update documentation with comprehensive streaming examples
- Maintain backward compatibility with existing streamcallback usage

Resolves JuliaML#65
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants