Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make streaming opt-in #3170

Open
ion-elgreco opened this issue Jan 28, 2025 · 2 comments
Open

Make streaming opt-in #3170

ion-elgreco opened this issue Jan 28, 2025 · 2 comments
Labels
binding/python Issues for the Python package enhancement New feature or request good first issue Good for newcomers

Comments

@ion-elgreco
Copy link
Collaborator

ion-elgreco commented Jan 28, 2025

Description

Use Case
Currently I've enabled streaming when we see a potentially streaming source, I think we should add streaming parameter in python to merge and let users configure it. This makes it then possible to disable it or set it for sources that are not streamed but might still benefit from the streamed execution.

We should also raise an experimental warning when it's true and say users should share positive or negative findings :)

Related Issue(s)

@ion-elgreco ion-elgreco added enhancement New feature or request good first issue Good for newcomers binding/python Issues for the Python package labels Jan 28, 2025
@rtyler
Copy link
Member

rtyler commented Jan 28, 2025

@ion-elgreco why not make it the default?

@ion-elgreco
Copy link
Collaborator Author

@rtyler non-streamed might be better in some cases since you can use min max statistics from the incoming source data to prune the target table.

Haven't found a way to do this during streaming

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
binding/python Issues for the Python package enhancement New feature or request good first issue Good for newcomers
Projects
None yet
Development

No branches or pull requests

2 participants