[Fix](status)Fix leaky abstraction and shield the status code `END_OF_FILE` from upper layers #24165

bobhan1 · 2023-09-11T02:59:41Z

Proposed changes

In #23855 and #24139, the low level status code END_OF_FILE is leaked to upper layers and causes some problems. This PR shields the status code END_OF_FILE from upper layers and transforms it to the status code with appropriate semantics.

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

bobhan1 · 2023-09-11T02:59:48Z

run buildall

github-actions · 2023-09-11T03:06:10Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2023-09-11T03:25:26Z

TeamCity be ut coverage result:
Function Coverage: 36.92% (7917/21441)
Line Coverage: 28.95% (63612/219695)
Region Coverage: 27.88% (32997/118345)
Branch Coverage: 24.46% (16945/69276)
Coverage Report: http://coverage.selectdb-in.cc/coverage/0e73444c52f5166079ae2e6314d408ff62790502_0e73444c52f5166079ae2e6314d408ff62790502/report/index.html

doris-robot · 2023-09-11T03:36:54Z

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 49.08 seconds
stream load tsv: 582 seconds loaded 74807831229 Bytes, about 122 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 30 seconds loaded 861443392 Bytes, about 27 MB/s
insert into select: 29.2 seconds inserted 10000000 Rows, about 342K ops/s
storage size: 17162198511 Bytes

bobhan1 · 2023-09-11T04:14:46Z

run buildall

doris-robot · 2023-09-11T04:34:02Z

TeamCity be ut coverage result:
Function Coverage: 36.93% (7919/21441)
Line Coverage: 28.96% (63629/219698)
Region Coverage: 27.90% (33014/118347)
Branch Coverage: 24.47% (16950/69278)
Coverage Report: http://coverage.selectdb-in.cc/coverage/0e73444c52f5166079ae2e6314d408ff62790502_0e73444c52f5166079ae2e6314d408ff62790502/report/index.html

doris-robot · 2023-09-11T04:49:47Z

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 46.18 seconds
stream load tsv: 581 seconds loaded 74807831229 Bytes, about 122 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 64 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
insert into select: 29.1 seconds inserted 10000000 Rows, about 343K ops/s
storage size: 17162112939 Bytes

github-actions · 2023-09-11T07:06:56Z

clang-tidy review says "All clean, LGTM! 👍"

bobhan1 · 2023-09-11T07:11:07Z

run buildall

github-actions · 2023-09-11T07:14:01Z

clang-tidy review says "All clean, LGTM! 👍"

bobhan1 · 2023-09-11T07:44:03Z

run buildall

github-actions · 2023-09-11T07:48:53Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2023-09-11T07:52:12Z

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 46.44 seconds
stream load tsv: 602 seconds loaded 74807831229 Bytes, about 118 MB/s
stream load json: 21 seconds loaded 2358488459 Bytes, about 107 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
insert into select: 28.7 seconds inserted 10000000 Rows, about 348K ops/s
storage size: 17162524201 Bytes

doris-robot · 2023-09-11T08:00:20Z

TeamCity be ut coverage result:
Function Coverage: 36.93% (7919/21443)
Line Coverage: 28.96% (63634/219710)
Region Coverage: 27.89% (33005/118351)
Branch Coverage: 24.46% (16947/69276)
Coverage Report: http://coverage.selectdb-in.cc/coverage/7c46ef71d6527d0196cd2836a7a82e337e2a9fb8_7c46ef71d6527d0196cd2836a7a82e337e2a9fb8/report/index.html

doris-robot · 2023-09-11T08:22:01Z

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 46.93 seconds
stream load tsv: 606 seconds loaded 74807831229 Bytes, about 117 MB/s
stream load json: 24 seconds loaded 2358488459 Bytes, about 93 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
insert into select: 28.8 seconds inserted 10000000 Rows, about 347K ops/s
storage size: 17162543980 Bytes

bobhan1 · 2023-09-11T10:40:51Z

run buildall

bobhan1 · 2023-09-11T10:42:40Z

run buildall

github-actions · 2023-09-11T10:48:47Z

clang-tidy review says "All clean, LGTM! 👍"

github-actions · 2023-09-11T10:53:54Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2023-09-11T11:18:48Z

TeamCity be ut coverage result:
Function Coverage: 36.93% (7920/21444)
Line Coverage: 28.97% (63647/219726)
Region Coverage: 27.90% (33024/118367)
Branch Coverage: 24.47% (16956/69288)
Coverage Report: http://coverage.selectdb-in.cc/coverage/746dd12d4ededf80d2e2bf4b7501521d3fd0a66d_746dd12d4ededf80d2e2bf4b7501521d3fd0a66d/report/index.html

doris-robot · 2023-09-11T11:26:24Z

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 46.32 seconds
stream load tsv: 583 seconds loaded 74807831229 Bytes, about 122 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 64 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
insert into select: 28.9 seconds inserted 10000000 Rows, about 346K ops/s
storage size: 17162213040 Bytes

zhannngchen · 2023-09-11T12:05:35Z

be/src/olap/rowset/segment_v2/binary_prefix_page.cpp

+        if (!st.ok()) {
+            return st;
+        }
    }


should return a Status::OK() at the end of this method?

zhannngchen

LGTM

github-actions · 2023-09-11T12:07:31Z

PR approved by at least one committer and no changes requested.

github-actions · 2023-09-11T12:07:34Z

PR approved by anyone and no changes requested.

liaoxin01 · 2023-09-11T13:49:16Z

be/src/olap/rowset/segment_v2/binary_prefix_page.cpp

        _cur_pos++;
-        RETURN_IF_ERROR(_read_next_value());
+        auto st = _read_next_value();
+        if (st.is<ErrorCode::END_OF_FILE>()) {


Does it have an impact on other users of this code？

only IndexedColumnIterator::seek_at_or_after may call BinaryPrefixPageDecoder::seek_at_or_after_value

liaoxin01

LGTM

…_FILE` from upper layers (#24165)

bobhan1 force-pushed the fix-status branch from 0e73444 to 3501aa4 Compare September 11, 2023 07:00

bobhan1 changed the title ~~[Fix](status)FIx status~~ [Fix](status)Shield the status code END_OF_FILE from upper layers Sep 11, 2023

bobhan1 changed the title ~~[Fix](status)Shield the status code END_OF_FILE from upper layers~~ [Fix](status)Fix leaky abstraction and shield the status code END_OF_FILE from upper layers Sep 11, 2023

zhannngchen added the dev/2.0.2 label Sep 11, 2023

bobhan1 added 4 commits September 11, 2023 18:42

fix status

6ec8cdc

update

a79251a

update

1ce260c

update

fda5776

update

746dd12

bobhan1 force-pushed the fix-status branch from 2129c5a to 746dd12 Compare September 11, 2023 10:45

zhannngchen reviewed Sep 11, 2023

View reviewed changes

zhannngchen approved these changes Sep 11, 2023

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Sep 11, 2023

github-actions bot added the reviewed label Sep 11, 2023

liaoxin01 reviewed Sep 11, 2023

View reviewed changes

liaoxin01 approved these changes Sep 12, 2023

View reviewed changes

xiaokang merged commit bdacefa into apache:master Sep 12, 2023

xiaokang added p0_c dev/2.0.2-merged and removed dev/2.0.2 labels Sep 12, 2023

xiaokang pushed a commit that referenced this pull request Sep 12, 2023

[Fix](status)Fix leaky abstraction and shield the status code `END_OF…

7e37817

…_FILE` from upper layers (#24165)

xiaokang pushed a commit that referenced this pull request Sep 12, 2023

[Fix](status)Fix leaky abstraction and shield the status code `END_OF…

653d375

…_FILE` from upper layers (#24165)

xiaokang mentioned this pull request Sep 28, 2023

Release Note 2.0.2 #25011

Closed

[Fix](status)Fix leaky abstraction and shield the status code END_OF_FILE from upper layers #24165

[Fix](status)Fix leaky abstraction and shield the status code END_OF_FILE from upper layers #24165

Uh oh!

Conversation

bobhan1 commented Sep 11, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed changes

Further comments

Uh oh!

bobhan1 commented Sep 11, 2023

Uh oh!

github-actions bot commented Sep 11, 2023

Uh oh!

doris-robot commented Sep 11, 2023

Uh oh!

doris-robot commented Sep 11, 2023

Uh oh!

bobhan1 commented Sep 11, 2023

Uh oh!

doris-robot commented Sep 11, 2023

Uh oh!

doris-robot commented Sep 11, 2023

Uh oh!

github-actions bot commented Sep 11, 2023

Uh oh!

bobhan1 commented Sep 11, 2023

Uh oh!

github-actions bot commented Sep 11, 2023

Uh oh!

bobhan1 commented Sep 11, 2023

Uh oh!

github-actions bot commented Sep 11, 2023

Uh oh!

doris-robot commented Sep 11, 2023

Uh oh!

doris-robot commented Sep 11, 2023

Uh oh!

doris-robot commented Sep 11, 2023

Uh oh!

bobhan1 commented Sep 11, 2023

Uh oh!

bobhan1 commented Sep 11, 2023

Uh oh!

github-actions bot commented Sep 11, 2023

Uh oh!

github-actions bot commented Sep 11, 2023

Uh oh!

doris-robot commented Sep 11, 2023

Uh oh!

doris-robot commented Sep 11, 2023

Uh oh!

zhannngchen Sep 11, 2023

Choose a reason for hiding this comment

Uh oh!

zhannngchen left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Sep 11, 2023

Uh oh!

github-actions bot commented Sep 11, 2023

Uh oh!

liaoxin01 Sep 11, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bobhan1 Sep 11, 2023

Choose a reason for hiding this comment

Uh oh!

liaoxin01 left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

[Fix](status)Fix leaky abstraction and shield the status code `END_OF_FILE` from upper layers #24165

[Fix](status)Fix leaky abstraction and shield the status code `END_OF_FILE` from upper layers #24165

bobhan1 commented Sep 11, 2023 •

edited

Loading

liaoxin01 Sep 11, 2023 •

edited

Loading