Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FEATURE_REQUEST]: KV store to share data between frames #71

Closed
miguelaeh opened this issue Nov 14, 2023 · 0 comments · Fixed by #78
Closed

[FEATURE_REQUEST]: KV store to share data between frames #71

miguelaeh opened this issue Nov 14, 2023 · 0 comments · Fixed by #78
Assignees

Comments

@miguelaeh
Copy link
Collaborator

miguelaeh commented Nov 14, 2023

The stage context is read-only by design, mainly because it depends on the language and can contain non-serializable values. This means it cannot be modified from hooks to share data with other frames. Plus, in a distributed setup, the stage context on each node will be independent of the stage context in other nodes.
We could have a global KV instance in the Pipeless node when running on device and use Redis when using distributed mode.

Sharing data between the processing of frames is sometimes useful, thus providing a KV store, that can also be distributed since will contain serializable data only, makes sense.
Also, it should be namespaced by stream id in order to differentiate data from different streams.

Finally, since frames are processed in parallel, it could be a good idea to provide some util functions with this KV store. For example, it could happen that the hook code tries to access data for the previous frame and it is not yet available. The library could provide wait functionality (which would add delay) or error bypass.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant