Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Re-implement DistinctCombineOperator to utilize more threads to merge #14714

Open
wants to merge 2 commits into
base: master
Choose a base branch
from

Conversation

Jackie-Jiang
Copy link
Contributor

On top of #14701

The combine algorithm is as following:

Use an atomic reference as the swap space to merge the results blocks.

  • If the swap space is null, put the new results block in it.
  • If the swap space is not null, take the results block from the swap space, and merge it with the new results block.

Repeat this process until successfully put the new results block in the swap space.
After all threads are done, the results block in the swap space is the final merged results block.

@codecov-commenter
Copy link

codecov-commenter commented Dec 25, 2024

Codecov Report

Attention: Patch coverage is 70.40720% with 625 lines in your changes missing coverage. Please review.

Project coverage is 63.82%. Comparing base (59551e4) to head (c4fe425).
Report is 1509 commits behind head on master.

Files with missing lines Patch % Lines
.../core/query/distinct/table/BytesDistinctTable.java 51.92% 48 Missing and 27 partials ⚠️
...t/core/query/distinct/table/LongDistinctTable.java 59.25% 47 Missing and 19 partials ⚠️
.../query/distinct/table/BigDecimalDistinctTable.java 62.82% 32 Missing and 26 partials ⚠️
...core/query/distinct/table/StringDistinctTable.java 62.82% 40 Missing and 18 partials ⚠️
...perator/query/DictionaryBasedDistinctOperator.java 76.47% 35 Missing and 9 partials ⚠️
...y/distinct/raw/RawMultiColumnDistinctExecutor.java 49.41% 34 Missing and 9 partials ⚠️
.../core/query/distinct/table/FloatDistinctTable.java 72.07% 23 Missing and 20 partials ⚠️
...core/query/distinct/table/DoubleDistinctTable.java 75.48% 22 Missing and 16 partials ⚠️
...ot/core/query/distinct/table/IntDistinctTable.java 77.16% 19 Missing and 18 partials ⚠️
...y/DictionaryBasedSingleColumnDistinctExecutor.java 73.75% 15 Missing and 6 partials ⚠️
... and 18 more
Additional details and impacted files
@@             Coverage Diff              @@
##             master   #14714      +/-   ##
============================================
+ Coverage     61.75%   63.82%   +2.07%     
- Complexity      207     1607    +1400     
============================================
  Files          2436     2702     +266     
  Lines        133233   150647   +17414     
  Branches      20636    23272    +2636     
============================================
+ Hits          82274    96147   +13873     
- Misses        44911    47314    +2403     
- Partials       6048     7186    +1138     
Flag Coverage Δ
custom-integration1 100.00% <ø> (+99.99%) ⬆️
integration 100.00% <ø> (+99.99%) ⬆️
integration1 100.00% <ø> (+99.99%) ⬆️
integration2 0.00% <ø> (ø)
java-11 63.80% <70.40%> (+2.09%) ⬆️
java-21 63.70% <70.40%> (+2.08%) ⬆️
skip-bytebuffers-false 63.81% <70.40%> (+2.07%) ⬆️
skip-bytebuffers-true 63.69% <70.40%> (+35.96%) ⬆️
temurin 63.82% <70.40%> (+2.07%) ⬆️
unittests 63.81% <70.40%> (+2.07%) ⬆️
unittests1 56.25% <70.40%> (+9.36%) ⬆️
unittests2 34.16% <0.00%> (+6.43%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants