Skip to content

improve scan read for aggregations (use commands in step before group… #6756

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
ivanmorozov333 merged 4 commits into from
Jul 17, 2024
Merged

improve scan read for aggregations (use commands in step before group… #6756

ivanmorozov333 merged 4 commits into from
Jul 17, 2024

Conversation

@ivanmorozov333
Copy link
Collaborator

@ivanmorozov333 ivanmorozov333 commented Jul 16, 2024

…ing)

  • Performance improvement

@github-actions
Copy link

github-actions bot commented Jul 16, 2024 •
edited
Loading

⚪ 2024-07-16 18:17:04 UTC Pre-commit check for c25bcb5 has started.
⚪ 2024-07-16 18:19:46 UTC Build+Tests linux-x86_64-release-clang14 is running...
🟢 2024-07-16 18:25:51 UTC Build successful.

@github-actions
Copy link

github-actions bot commented Jul 16, 2024 •
edited
Loading

⚪ 2024-07-16 18:17:27 UTC Pre-commit check for c25bcb5 has started.
⚪ 2024-07-16 18:20:07 UTC Build+Tests linux-x86_64-release-asan is running...
🔴 2024-07-16 18:28:20 UTC Test run completed, no test results found for commit db17b65.
⚫ 2024-07-16 18:29:31 UTC Check cancelled

@github-actions
Copy link

github-actions bot commented Jul 16, 2024 •
edited
Loading

⚪ 2024-07-16 18:19:12 UTC Pre-commit check for c25bcb5 has started.
⚪ 2024-07-16 18:21:53 UTC Build+Tests linux-x86_64-relwithdebinfo is running...
🔴 2024-07-16 18:28:29 UTC Test run completed, no test results found for commit db17b65.
⚫ 2024-07-16 18:28:50 UTC Check cancelled

@github-actions
Copy link

github-actions bot commented Jul 16, 2024 •
edited
Loading

⚪ 2024-07-16 18:30:56 UTC Pre-commit check for 67204c8 has started.
⚪ 2024-07-16 18:34:22 UTC Build+Tests linux-x86_64-release-asan is running...
🟢 2024-07-16 20:40:44 UTC Build successful.
🔴 2024-07-16 20:43:45 UTC Some tests failed, follow the links below.

Test history | Test log

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
9755 9177 0 80 323 175

🟢 2024-07-16 20:44:31 UTC ydbd size 5.2 GiB changed* by -8.1 KiB, which is <= 0 Bytes vs main: OK

ydbd size dash main: 5191be5 merge: 67204c8 diff diff %
ydbd size 5 549 217 464 Bytes 5 549 209 120 Bytes -8.1 KiB -0.000%
ydbd stripped size 1 195 003 888 Bytes 1 195 002 160 Bytes -1.7 KiB -0.000%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

@github-actions
Copy link

github-actions bot commented Jul 16, 2024 •
edited
Loading

⚪ 2024-07-16 18:33:11 UTC Pre-commit check for 67204c8 has started.
⚪ 2024-07-16 18:35:58 UTC Build+Tests linux-x86_64-release-clang14 is running...
🟢 2024-07-16 18:41:26 UTC Build successful.

@github-actions
Copy link

github-actions bot commented Jul 16, 2024 •
edited
Loading

⚪ 2024-07-16 18:33:11 UTC Pre-commit check for 67204c8 has started.
⚪ 2024-07-16 18:35:54 UTC Build+Tests linux-x86_64-relwithdebinfo is running...
🟢 2024-07-16 20:29:44 UTC Build successful.
🔴 2024-07-16 20:37:10 UTC Some tests failed, follow the links below.

Test history | Test log

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
38260 33017 0 5 5221 17

🟢 2024-07-16 20:37:53 UTC ydbd size 8.0 GiB changed* by -3.6 KiB, which is <= 0 Bytes vs main: OK

ydbd size dash main: 5191be5 merge: 67204c8 diff diff %
ydbd size 8 643 166 608 Bytes 8 643 162 920 Bytes -3.6 KiB -0.000%
ydbd stripped size 470 547 296 Bytes 470 546 976 Bytes -320 Bytes -0.000%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

@github-actions
Copy link

github-actions bot commented Jul 17, 2024 •
edited
Loading

⚪ 2024-07-17 10:23:28 UTC Pre-commit check for 55ee2c3 has started.
⚪ 2024-07-17 10:26:09 UTC Build+Tests linux-x86_64-release-clang14 is running...
🔴 2024-07-17 10:29:29 UTC Build failed. see the logs.

@github-actions
Copy link

github-actions bot commented Jul 17, 2024 •
edited
Loading

⚪ 2024-07-17 10:23:39 UTC Pre-commit check for 55ee2c3 has started.
⚪ 2024-07-17 10:26:19 UTC Build+Tests linux-x86_64-release-asan is running...
🔴 2024-07-17 10:31:48 UTC Test run completed, no test results found for commit 0ea0231.
⚫ 2024-07-17 10:32:01 UTC Check cancelled

@github-actions
Copy link

github-actions bot commented Jul 17, 2024 •
edited
Loading

⚪ 2024-07-17 10:23:48 UTC Pre-commit check for 55ee2c3 has started.
⚪ 2024-07-17 10:26:29 UTC Build+Tests linux-x86_64-relwithdebinfo is running...
🔴 2024-07-17 10:31:47 UTC Test run completed, no test results found for commit 0ea0231.
⚫ 2024-07-17 10:32:00 UTC Check cancelled

@github-actions
Copy link

github-actions bot commented Jul 17, 2024 •
edited
Loading

⚪ 2024-07-17 10:33:58 UTC Pre-commit check for 50c42b6 has started.
⚪ 2024-07-17 10:36:41 UTC Build+Tests linux-x86_64-release-asan is running...
🟢 2024-07-17 12:45:55 UTC Build successful.
🔴 2024-07-17 12:49:33 UTC Some tests failed, follow the links below.

Test history | Test log

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
10237 9648 0 85 324 180

🟢 2024-07-17 12:50:24 UTC ydbd size 5.2 GiB changed* by -8.2 KiB, which is <= 0 Bytes vs main: OK

ydbd size dash main: a703d87 merge: 50c42b6 diff diff %
ydbd size 5 545 766 768 Bytes 5 545 758 392 Bytes -8.2 KiB -0.000%
ydbd stripped size 1 194 925 328 Bytes 1 194 923 536 Bytes -1.8 KiB -0.000%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

@github-actions
Copy link

github-actions bot commented Jul 17, 2024 •
edited
Loading

⚪ 2024-07-17 10:34:13 UTC Pre-commit check for 50c42b6 has started.
⚪ 2024-07-17 10:36:56 UTC Build+Tests linux-x86_64-release-clang14 is running...
🟢 2024-07-17 10:42:51 UTC Build successful.

@github-actions
Copy link

github-actions bot commented Jul 17, 2024 •
edited
Loading

⚪ 2024-07-17 10:35:47 UTC Pre-commit check for 50c42b6 has started.
⚪ 2024-07-17 10:38:36 UTC Build+Tests linux-x86_64-relwithdebinfo is running...
🟢 2024-07-17 12:35:46 UTC Build successful.
🔴 2024-07-17 12:43:05 UTC Some tests failed, follow the links below.

Test history | Test log

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
38745 33499 0 5 5224 17

🟢 2024-07-17 12:43:44 UTC ydbd size 8.0 GiB changed* by -3.1 KiB, which is <= 0 Bytes vs main: OK

ydbd size dash main: a703d87 merge: 50c42b6 diff diff %
ydbd size 8 641 480 472 Bytes 8 641 477 312 Bytes -3.1 KiB -0.000%
ydbd stripped size 470 443 008 Bytes 470 442 624 Bytes -384 Bytes -0.000%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

zverevgeny
zverevgeny approved these changes Jul 17, 2024
View reviewed changes
ydb/core/formats/arrow/program.cpp

const std::set<ui32>& TProgramStep::GetFilterOriginalColumnIds() const {
AFL_VERIFY(IsFilterOnly());
// AFL_VERIFY(IsFilterOnly());
Copy link
Collaborator

@zverevgeny zverevgeny Jul 17, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Зачем оставлять комментированные строчки?

@ivanmorozov333 ivanmorozov333 merged commit 90eea8d into ydb-platform:main Jul 17, 2024
@ivanmorozov333 ivanmorozov333 deleted the improve_scan_read_for_aggregations branch July 17, 2024 14:39
This was referenced Jul 19, 2024
Merged
Merged
Merged
Merged
Merged
zverevgeny pushed a commit to zverevgeny/ydb that referenced this pull request Jul 23, 2024
@ivanmorozov333 @zverevgeny
improve scan read for aggregations (use commands in step before group… (
9be9dea 
ydb-platform#6756)
@GrigoriyPA GrigoriyPA mentioned this pull request Jul 24, 2024
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@zverevgeny zverevgeny zverevgeny approved these changes

Assignees

No one assigned

Labels

performance

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@ivanmorozov333 @zverevgeny