Skip to content

Conversation

@BePPPower
Copy link
Contributor

@BePPPower BePPPower commented Mar 26, 2025

It's because when Hudi performs transparent rewriting, the timestamp of the base table obtained by loadSnapshot is inconsistent with the timestamp stored after partition refresh, which causes the comparison of tableSnapshot to fail, resulting in the materialized view not being hit.

Currently, the logic of loadSnapshot to obtain the timestamp of the base table is a bit strange and doesn't quite meet expectations. Further in - depth research is needed on how to modify it.

For now, in the getTableSnapshot function, simply return 0L constantly, indicating that the tableSnapshot is always synchronized, to bypass this problem. This modification is consistent with the original expectation of manual refresh. We'll deal with this issue later.

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@Thearas
Copy link
Contributor

Thearas commented Mar 26, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@morningman morningman merged commit 961e1b8 into apache:branch-hudi-mtmv Mar 27, 2025
7 checks passed
morningman pushed a commit that referenced this pull request Apr 2, 2025
…nvolved in transparent rewriting (#49513)

It's because when Hudi performs transparent rewriting, the timestamp of
the base table obtained by `loadSnapshot` is inconsistent with the
timestamp stored after partition refresh, which causes the comparison of
`tableSnapshot` to fail, resulting in the materialized view not being
hit.

Currently, the logic of `loadSnapshot` to obtain the timestamp of the
base table is a bit strange and doesn't quite meet expectations. Further
in - depth research is needed on how to modify it.

For now, in the `getTableSnapshot` function, simply return `0L`
constantly, indicating that the `tableSnapshot` is always synchronized,
to bypass this problem. This modification is consistent with the
original expectation of manual refresh. We'll deal with this issue
later.
BePPPower added a commit to BePPPower/doris that referenced this pull request Apr 10, 2025
…nvolved in transparent rewriting (apache#49513)

It's because when Hudi performs transparent rewriting, the timestamp of
the base table obtained by `loadSnapshot` is inconsistent with the
timestamp stored after partition refresh, which causes the comparison of
`tableSnapshot` to fail, resulting in the materialized view not being
hit.

Currently, the logic of `loadSnapshot` to obtain the timestamp of the
base table is a bit strange and doesn't quite meet expectations. Further
in - depth research is needed on how to modify it.

For now, in the `getTableSnapshot` function, simply return `0L`
constantly, indicating that the `tableSnapshot` is always synchronized,
to bypass this problem. This modification is consistent with the
original expectation of manual refresh. We'll deal with this issue
later.
BePPPower added a commit to BePPPower/doris that referenced this pull request Apr 21, 2025
…nvolved in transparent rewriting (apache#49513)

It's because when Hudi performs transparent rewriting, the timestamp of
the base table obtained by `loadSnapshot` is inconsistent with the
timestamp stored after partition refresh, which causes the comparison of
`tableSnapshot` to fail, resulting in the materialized view not being
hit.

Currently, the logic of `loadSnapshot` to obtain the timestamp of the
base table is a bit strange and doesn't quite meet expectations. Further
in - depth research is needed on how to modify it.

For now, in the `getTableSnapshot` function, simply return `0L`
constantly, indicating that the `tableSnapshot` is always synchronized,
to bypass this problem. This modification is consistent with the
original expectation of manual refresh. We'll deal with this issue
later.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants