-
Notifications
You must be signed in to change notification settings - Fork 3.7k
[enhancement](maxcompute)support maxcompute timestamp column type. #48768
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
|
run buildall |
TPC-H: Total hot run time: 32802 ms |
TPC-DS: Total hot run time: 185488 ms |
ClickBench: Total hot run time: 31.29 s |
|
run buildall |
TPC-H: Total hot run time: 32230 ms |
TPC-DS: Total hot run time: 192222 ms |
ClickBench: Total hot run time: 30.65 s |
morningman
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
|
PR approved by at least one committer and no changes requested. |
|
PR approved by anyone and no changes requested. |
…48768) ### What problem does this PR solve? Problem Summary: 1. support maxcompute timestamp column type. 2. Add the parameter `mc.datetime_predicate_push_down` to disable predicate pushdown for odps catalog datetime type, because the timestamp precision of odps is 9, while the mapping precision of Doris is 6. If we insert `2023-02-02 00:00:00.123456789` into odps, doris will read it as `2023-02-02 00:00:00.123456`. Due to the lack of "789", we cannot push it down correctly. If you don't need such a high precision (greater than 6) on odps, it will not affect your normal use.
…48768) ### What problem does this PR solve? Problem Summary: 1. support maxcompute timestamp column type. 2. Add the parameter `mc.datetime_predicate_push_down` to disable predicate pushdown for odps catalog datetime type, because the timestamp precision of odps is 9, while the mapping precision of Doris is 6. If we insert `2023-02-02 00:00:00.123456789` into odps, doris will read it as `2023-02-02 00:00:00.123456`. Due to the lack of "789", we cannot push it down correctly. If you don't need such a high precision (greater than 6) on odps, it will not affect your normal use.
…pache#48768) Problem Summary: 1. support maxcompute timestamp column type. 2. Add the parameter `mc.datetime_predicate_push_down` to disable predicate pushdown for odps catalog datetime type, because the timestamp precision of odps is 9, while the mapping precision of Doris is 6. If we insert `2023-02-02 00:00:00.123456789` into odps, doris will read it as `2023-02-02 00:00:00.123456`. Due to the lack of "789", we cannot push it down correctly. If you don't need such a high precision (greater than 6) on odps, it will not affect your normal use. (cherry picked from commit 2988dc3)
…pache#48768) Problem Summary: 1. support maxcompute timestamp column type. 2. Add the parameter `mc.datetime_predicate_push_down` to disable predicate pushdown for odps catalog datetime type, because the timestamp precision of odps is 9, while the mapping precision of Doris is 6. If we insert `2023-02-02 00:00:00.123456789` into odps, doris will read it as `2023-02-02 00:00:00.123456`. Due to the lack of "789", we cannot push it down correctly. If you don't need such a high precision (greater than 6) on odps, it will not affect your normal use. (cherry picked from commit 2988dc3)
…pache#48768) ### What problem does this PR solve? Problem Summary: 1. support maxcompute timestamp column type. 2. Add the parameter `mc.datetime_predicate_push_down` to disable predicate pushdown for odps catalog datetime type, because the timestamp precision of odps is 9, while the mapping precision of Doris is 6. If we insert `2023-02-02 00:00:00.123456789` into odps, doris will read it as `2023-02-02 00:00:00.123456`. Due to the lack of "789", we cannot push it down correctly. If you don't need such a high precision (greater than 6) on odps, it will not affect your normal use.
…pache#48768) Problem Summary: 1. support maxcompute timestamp column type. 2. Add the parameter `mc.datetime_predicate_push_down` to disable predicate pushdown for odps catalog datetime type, because the timestamp precision of odps is 9, while the mapping precision of Doris is 6. If we insert `2023-02-02 00:00:00.123456789` into odps, doris will read it as `2023-02-02 00:00:00.123456`. Due to the lack of "789", we cannot push it down correctly. If you don't need such a high precision (greater than 6) on odps, it will not affect your normal use. (cherry picked from commit 2988dc3)
…n prune && timstamp type(apache#44508, apache#48768, apache#48325) (apache#3805) pick apache#44508 apache#48768 apache#48325 --------- Co-authored-by: zhangdong <zhangdong@selectdb.com>
…pache#48768) Problem Summary: 1. support maxcompute timestamp column type. 2. Add the parameter `mc.datetime_predicate_push_down` to disable predicate pushdown for odps catalog datetime type, because the timestamp precision of odps is 9, while the mapping precision of Doris is 6. If we insert `2023-02-02 00:00:00.123456789` into odps, doris will read it as `2023-02-02 00:00:00.123456`. Due to the lack of "789", we cannot push it down correctly. If you don't need such a high precision (greater than 6) on odps, it will not affect your normal use.
…f the timestamp data (#49600) ### What problem does this PR solve? Related pr : #48768 Problem Summary: When using the maxcompute storage api to read the mc timestamp column, if the timestamp precision is required to be at the nanosecond level, only data in the range of 1970-01-01 to 2038-01-19 can be read. Otherwise, an error will be reported: `ErrorCode=ODPS-0422214, ErrorMessage=ODPS-0422214: Read data failed - RuntimeTimestamp data overflow`. This pr changes the precision to milliseconds to solve this problem.
…f the timestamp data (#49600) ### What problem does this PR solve? Related pr : #48768 Problem Summary: When using the maxcompute storage api to read the mc timestamp column, if the timestamp precision is required to be at the nanosecond level, only data in the range of 1970-01-01 to 2038-01-19 can be read. Otherwise, an error will be reported: `ErrorCode=ODPS-0422214, ErrorMessage=ODPS-0422214: Read data failed - RuntimeTimestamp data overflow`. This pr changes the precision to milliseconds to solve this problem.
…f the timestamp data (#49600) ### What problem does this PR solve? Related pr : #48768 Problem Summary: When using the maxcompute storage api to read the mc timestamp column, if the timestamp precision is required to be at the nanosecond level, only data in the range of 1970-01-01 to 2038-01-19 can be read. Otherwise, an error will be reported: `ErrorCode=ODPS-0422214, ErrorMessage=ODPS-0422214: Read data failed - RuntimeTimestamp data overflow`. This pr changes the precision to milliseconds to solve this problem.
…pache#48768) ### What problem does this PR solve? Problem Summary: 1. support maxcompute timestamp column type. 2. Add the parameter `mc.datetime_predicate_push_down` to disable predicate pushdown for odps catalog datetime type, because the timestamp precision of odps is 9, while the mapping precision of Doris is 6. If we insert `2023-02-02 00:00:00.123456789` into odps, doris will read it as `2023-02-02 00:00:00.123456`. Due to the lack of "789", we cannot push it down correctly. If you don't need such a high precision (greater than 6) on odps, it will not affect your normal use.
…f the timestamp data (apache#49600) ### What problem does this PR solve? Related pr : apache#48768 Problem Summary: When using the maxcompute storage api to read the mc timestamp column, if the timestamp precision is required to be at the nanosecond level, only data in the range of 1970-01-01 to 2038-01-19 can be read. Otherwise, an error will be reported: `ErrorCode=ODPS-0422214, ErrorMessage=ODPS-0422214: Read data failed - RuntimeTimestamp data overflow`. This pr changes the precision to milliseconds to solve this problem.
What problem does this PR solve?
Problem Summary:
mc.datetime_predicate_push_downto disable predicate pushdown for odps catalog datetime type, because the timestamp precision of odps is 9, while the mapping precision of Doris is 6. If we insert2023-02-02 00:00:00.123456789into odps, doris will read it as2023-02-02 00:00:00.123456. Due to the lack of "789", we cannot push it down correctly. If you don't need such a high precision (greater than 6) on odps, it will not affect your normal use.Release note
support maxcompute
timestampcolumn type.Check List (For Author)
Test
Behavior changed:
Does this need documentation?
Check List (For Reviewer who merge this PR)