[fix](inveted index) fix variant index #36163
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Thank you for your contribution to Apache Doris. Since 2024-03-18, the Document has been moved to doris-website. |
csun5285
changed the title
|
clang-tidy review says "All clean, LGTM! 👍" |
1 similar comment
|
clang-tidy review says "All clean, LGTM! 👍" |
csun5285
force-pushed
the
test_variant_v1
branch
from
ad3cd43
to
1dde935
Compare
|
run buildall |
|
clang-tidy review says "All clean, LGTM! 👍" |
1 similar comment
|
clang-tidy review says "All clean, LGTM! 👍" |
|
TeamCity be ut coverage result: |
TPC-H: Total hot run time: 42346 ms |
TPC-DS: Total hot run time: 172357 ms |
ClickBench: Total hot run time: 30.55 s |
|
clang-tidy review says "All clean, LGTM! 👍" |
|
run buildall |
|
TeamCity be ut coverage result: |
TPC-H: Total hot run time: 39433 ms |
TPC-DS: Total hot run time: 173001 ms |
ClickBench: Total hot run time: 30.82 s |
csun5285
changed the title
eldenmoon
reviewed
Jun 12, 2024
| @@ -389,7 +389,7 @@ void inherit_column_attributes(const TabletColumn& source, TabletColumn& target, | |||
| // add index meta | |||
| TabletIndex index_info = *source_index_meta; | |||
| index_info.set_escaped_escaped_index_suffix_path(target.path_info_ptr()->get_path()); | |||
| const auto* target_index_meta = target_schema->get_inverted_index(target); | |||
| const auto* target_index_meta = target_schema->get_inverted_index(target, false); | |||
There was a problem hiding this comment.
why set to false, and default true, add comment to explain?
There was a problem hiding this comment.
True: Check if the columns from the variant support inverted index
False: No need to check, just inherit directly
There was a problem hiding this comment.
Why not always check it?
- When the variant column is created, it has a TabletIndex. The extracted column from the variant will inherit this TabletIndex.
- The variant column itself is JSONB and does not support index, no need to check.
|
run buildall |
|
clang-tidy review says "All clean, LGTM! 👍" |
TPC-H: Total hot run time: 40122 ms |
|
TeamCity be ut coverage result: |
TPC-DS: Total hot run time: 170087 ms |
ClickBench: Total hot run time: 31.08 s |
|
run buildall |
|
clang-tidy review says "All clean, LGTM! 👍" |
|
TeamCity be ut coverage result: |
TPC-H: Total hot run time: 40095 ms |
csun5285
force-pushed
the
test_variant_v1
branch
from
85c525b
to
0c0a2b0
Compare
|
run buildall |
|
clang-tidy review says "All clean, LGTM! 👍" |
|
TeamCity be ut coverage result: |
TPC-H: Total hot run time: 40168 ms |
TPC-DS: Total hot run time: 173994 ms |
ClickBench: Total hot run time: 30.83 s |
csun5285
force-pushed
the
test_variant_v1
branch
from
0c0a2b0
to
c3c85de
Compare
|
run buildall |
|
clang-tidy review says "All clean, LGTM! 👍" |
TPC-H: Total hot run time: 39975 ms |
|
TeamCity be ut coverage result: |
TPC-DS: Total hot run time: 171343 ms |
ClickBench: Total hot run time: 30.93 s |
iszhangpch
pushed a commit
to iszhangpch/doris-p
that referenced
this pull request
Jun 21, 2024
Some columns from the variant do not support indexing, but they are listed in TabletIndex. If such a column can obtain the TabletIndex, the corresponding index file will not be found in copying files, creating snapshots, and uploading files to S3. When the variant column is created, it has a TabletIndex. The extracted column from the variant will inherit this TabletIndex. If the column extracted from the variant does not support index, it should not get TabletIndex.
dataroaring
pushed a commit
that referenced
this pull request
Jun 21, 2024
Some columns from the variant do not support indexing, but they are listed in TabletIndex. If such a column can obtain the TabletIndex, the corresponding index file will not be found in copying files, creating snapshots, and uploading files to S3. When the variant column is created, it has a TabletIndex. The extracted column from the variant will inherit this TabletIndex. If the column extracted from the variant does not support index, it should not get TabletIndex.
eldenmoon
added a commit
that referenced
this pull request
Aug 29, 2024
When reading from segment, the schema type is variant, if we check type valid in `get_inverted_index`, the result should always return nullptr(since variant type it self does not support inverted index), but the actual storage could be `string` or etc.So we should ignore the type check and return the correct inverted index iterators introduced by #36163
eldenmoon
added a commit
to eldenmoon/incubator-doris
that referenced
this pull request
Sep 3, 2024
When reading from segment, the schema type is variant, if we check type valid in `get_inverted_index`, the result should always return nullptr(since variant type it self does not support inverted index), but the actual storage could be `string` or etc.So we should ignore the type check and return the correct inverted index iterators introduced by apache#36163
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Proposed changes
Some columns from the variant do not support indexing, but they are listed in TabletIndex. If such a column can obtain the TabletIndex, the corresponding index file will not be found in copying files, creating snapshots, and uploading files to S3.
When the variant column is created, it has a TabletIndex. The extracted column from the variant will inherit this TabletIndex.
If the column extracted from the variant does not support index, it should not get TabletIndex.
Issue Number: close #xxx