Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add partial Sort Operator #7456

Closed
mustafasrepo opened this issue Aug 31, 2023 · 0 comments · Fixed by #9125
Closed

Add partial Sort Operator #7456

mustafasrepo opened this issue Aug 31, 2023 · 0 comments · Fixed by #9125
Labels
enhancement New feature or request

Comments

@mustafasrepo
Copy link
Contributor

mustafasrepo commented Aug 31, 2023

Is your feature request related to a problem or challenge?

Consider a use case where required ordering is (a ASC,b ASC), and existing ordering is (a ASC).
As an example input is like following

a b
1 2
1 3
1 1
2 2
2 3
2 1

expected output is like following

a b
1 1
1 2
1 3
2 1
2 2
2 3

If we were to use information about existing ordering. We could buffer up a values until it changes like below

a b
1 2
1 3
1 1

when 2 is received for the value of a. We could then sort subtable according to desired ordering (b ASC), then emit following result

a b
1 1
1 2
1 3

I think this operator

  • Enable us to use SortExec without breaking pipeline for some use cases (for this behavior we can write a new operator also).
  • Decrease the memory usage of the SortExec when input ordering satisfy a prefix of the desired ordering.

Describe the solution you'd like

No response

Describe alternatives you've considered

This should be a new operator or current SortExec can be extended to behave this way.
However, I do think that extending current SortExec to behave this way is better option because:

  • Existing plans will immediately benefit from this change (otherwise we need to write a rule to choose between SortExec and new PartialSortExec)
  • less change will be introduced to code base. Since I presume, the two operators will have lots of code common anyway.

Additional context

See discussion for more background.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant