Allow to call a callback on streaming responses #310
Closed
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Title: Allow to call a callback on streaming responses
Description: (optional)
This PR allows to call a user-defined function (null-safe) with the stream content (list of chunks)
Motivation: (optional)
The motivation behind this is that i run a fork of the gateway (i needed to introduce a custom middleware) and to add tracing for the stream responses, i have a working solution (transform the stream without actually changing the format, only adding the same logic i added in this PR) but having this as a native feature of the gateway would improve the overall implementation.
The idea is actually similar to what is done with the
getFromCache
function.This is a first (working) proof-of-concept, if the idea of this feature is accepted, i'm more than ready to improve the implementation if needed.
Regarding the impact on performance, i think that if the function is not set the impact is non-existant, and if the function is set, then the performance has to be accepted by those implementing the function.
If accepted, it could also be a first step to have some level of callbacks/hooks for those running forks to extends the open-source gateway.