Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add ProgressiveEval operator for optimize SortPreservingMerge #10488

Closed
Tracked by #10313
NGA-TRAN opened this issue May 13, 2024 · 3 comments
Closed
Tracked by #10313

Add ProgressiveEval operator for optimize SortPreservingMerge #10488

NGA-TRAN opened this issue May 13, 2024 · 3 comments
Labels
enhancement New feature or request

Comments

@NGA-TRAN
Copy link
Contributor

NGA-TRAN commented May 13, 2024

Is your feature request related to a problem or challenge?

Yes, this is a task of #10316. We will use the porting ProgressiveEval to optimize SortPreservingMerge

Describe the solution you'd like

In InfluxDB IOx, when the inputs of SortPreservingMerge are all sorted on the sort key and their data do not overlap, we replace SortPreservingMerge with ProgressiveEval which:

  1. Avoids starting all input streams at once
  2. Avoids having to compare any keys (doesn't actually do a merge)

We wrote about using this operator here: https://www.influxdata.com/blog/making-recent-value-queries-hundreds-times-faster/

This ticket is to port ProgressiveEval from InfluxDB to DataFusion

Describe alternatives you've considered

No response

Additional context

No response

@NGA-TRAN
Copy link
Contributor Author

I start working on this

@alamb alamb changed the title Porting ProgressiveEval from InfluxDB IOx to optimize SortPreservingMerge Add ProgressiveEval for optimize SortPreservingMerge May 15, 2024
@alamb
Copy link
Contributor

alamb commented May 15, 2024

I updated this ticket's description with some additional background and linked: https://www.influxdata.com/blog/making-recent-value-queries-hundreds-times-faster/

There is a PR for adding this operator here: #10490

@alamb alamb changed the title Add ProgressiveEval for optimize SortPreservingMerge Add ProgressiveEval operator for optimize SortPreservingMerge May 15, 2024
@alamb
Copy link
Contributor

alamb commented May 15, 2024

I think this ticket largely duplicates #10316 so let's use that instead

@alamb alamb closed this as completed May 15, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants