[feature](mtmv)Support iceberg partition refresh. #44726

Jibing-Li · 2024-11-28T07:03:29Z

What problem does this PR solve?

Previously, when using Iceberg to create MTMV, it was not possible to perceive changes in partition lists and data, so only refresh materialized view mv1 complete could be used to force full refresh.

This PR obtains the partition list of Iceberg, the last update time of the partition, and the latest snapshotId of the table.

Therefore, MTMV can be partition based on Iceberg tables and perceive changes in data, automatically refreshing partitions

For now, we only support single partition column table and the partition transform must one of hour, day, month or year.
Will support Identity transform soon.

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Release note

None

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)
- No need to test or manual test. Explain why:
  - This is a refactor/code format and no logic has been changed.
  - Previous test can cover this change.
  - No code files have been changed.
  - Other reason
Behavior changed:
- No.
- Yes.
Does this need documentation?
- No.
- Yes.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

doris-robot · 2024-11-28T07:03:34Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

Jibing-Li · 2024-11-28T09:18:26Z

run buildall

fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/IcebergExternalTable.java

regression-test/suites/mtmv_p0/test_iceberg_mtmv.groovy

Jibing-Li · 2024-12-06T06:23:08Z

run buildall

fe/fe-core/src/main/java/org/apache/doris/mtmv/MTMVRelatedTableIf.java

fe/fe-core/src/main/java/org/apache/doris/job/extensions/mtmv/MTMVTask.java

fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/IcebergPartitionInfo.java

fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/IcebergExternalTable.java

Jibing-Li · 2024-12-10T08:31:09Z

run buildall

zddr

LGTM

github-actions · 2024-12-10T08:37:07Z

PR approved by anyone and no changes requested.

Jibing-Li · 2024-12-10T13:47:17Z

run p0

fe/fe-core/src/main/java/org/apache/doris/job/extensions/mtmv/MTMVTask.java

fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/IcebergExternalTable.java

Jibing-Li · 2024-12-16T03:44:03Z

run buildall

github-actions · 2024-12-16T03:49:58Z

PR approved by at least one committer and no changes requested.

### What problem does this PR solve? Previously, when using Iceberg to create MTMV, it was not possible to perceive changes in partition lists and data, so only ```refresh materialized view mv1 complete ```could be used to force full refresh. This PR obtains the partition list of Iceberg, the last update time of the partition, and the latest snapshotId of the table. Therefore, MTMV can be partition based on Iceberg tables and perceive changes in data, automatically refreshing partitions For now, we only support single partition column table and the partition transform must one of hour, day, month or year. Will support Identity transform soon. Issue Number: close #xxx Related PR: #xxx Problem Summary: ### Release note None

Previously, when using Iceberg to create MTMV, it was not possible to perceive changes in partition lists and data, so only ```refresh materialized view mv1 complete ```could be used to force full refresh. This PR obtains the partition list of Iceberg, the last update time of the partition, and the latest snapshotId of the table. Therefore, MTMV can be partition based on Iceberg tables and perceive changes in data, automatically refreshing partitions For now, we only support single partition column table and the partition transform must one of hour, day, month or year. Will support Identity transform soon. Issue Number: close #xxx Related PR: #xxx Problem Summary: None

Cherry-pick from #43959 #44419 #44415 #44567 #44673 #44998 #45273 #44911 #44726 #45652 #45659 #46257 #46641 #47026 #47166 #48172 #49956 #50979 --------- Co-authored-by: James <lijibing@selectdb.com> Co-authored-by: Tiewei Fang <fangtiewei@selectdb.com>

Jibing-Li force-pushed the iceberg branch 3 times, most recently from 870c5ab to ea79a6a Compare November 28, 2024 09:16

Jibing-Li marked this pull request as ready for review November 28, 2024 09:17

morrySnow reviewed Nov 29, 2024

View reviewed changes

zddr reviewed Nov 29, 2024

View reviewed changes

fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/IcebergExternalTable.java Show resolved Hide resolved

regression-test/suites/mtmv_p0/test_iceberg_mtmv.groovy Outdated Show resolved Hide resolved

regression-test/suites/mtmv_p0/test_iceberg_mtmv.groovy Show resolved Hide resolved

Jibing-Li force-pushed the iceberg branch 10 times, most recently from 235a77f to 7ef6b3a Compare December 6, 2024 06:08

zddr reviewed Dec 10, 2024

View reviewed changes

Jibing-Li force-pushed the iceberg branch 4 times, most recently from 708edf0 to 1df4979 Compare December 10, 2024 08:28

zddr approved these changes Dec 10, 2024

View reviewed changes

github-actions bot added the reviewed label Dec 10, 2024

morrySnow reviewed Dec 13, 2024

View reviewed changes

fe/fe-core/src/main/java/org/apache/doris/job/extensions/mtmv/MTMVTask.java Outdated Show resolved Hide resolved

fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/IcebergExternalTable.java Outdated Show resolved Hide resolved

Jibing-Li force-pushed the iceberg branch from 1df4979 to e7471a3 Compare December 16, 2024 03:24

Support get iceberg partition table info.

fdd1cb0

Jibing-Li force-pushed the iceberg branch from e7471a3 to fdd1cb0 Compare December 16, 2024 03:36

morrySnow approved these changes Dec 16, 2024

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Dec 16, 2024

Jibing-Li merged commit f49e609 into apache:master Dec 16, 2024
32 of 35 checks passed

Jibing-Li added dev/2.1.x dev/3.0.x labels Dec 16, 2024

github-actions bot mentioned this pull request Dec 16, 2024

branch-3.0: [feature](mtmv)Support iceberg partition refresh. #44726 #45462

Closed

github-actions bot mentioned this pull request Dec 16, 2024

branch-2.1: [feature](mtmv)Support iceberg partition refresh. #44726 #45463

Closed

Jibing-Li removed dev/2.1.x dev/3.0.x labels Dec 16, 2024

Jibing-Li deleted the iceberg branch December 16, 2024 08:31

zddr mentioned this pull request Jun 20, 2025

branch-3.1:[enhance](mtmv)MTMV support iceberg/hudi/paimon #52040

Merged

morrySnow added the dev/3.1.0-merged label Jun 23, 2025

[feature](mtmv)Support iceberg partition refresh. #44726

[feature](mtmv)Support iceberg partition refresh. #44726

Uh oh!

Conversation

Jibing-Li commented Nov 28, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

Release note

Check List (For Author)

Check List (For Reviewer who merge this PR)

Uh oh!

doris-robot commented Nov 28, 2024

Uh oh!

Jibing-Li commented Nov 28, 2024

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Jibing-Li commented Dec 6, 2024

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Jibing-Li commented Dec 10, 2024

Uh oh!

zddr left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Dec 10, 2024

Uh oh!

Jibing-Li commented Dec 10, 2024

Uh oh!

Uh oh!

Uh oh!

Jibing-Li commented Dec 16, 2024

Uh oh!

github-actions bot commented Dec 16, 2024

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Jibing-Li commented Nov 28, 2024 •

edited

Loading