Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

WIP: Support Snapshot Copy-On-Write Hudi Table to Iceberg Table #6642

Closed
wants to merge 20 commits into from
Closed

WIP: Support Snapshot Copy-On-Write Hudi Table to Iceberg Table #6642

wants to merge 20 commits into from

Commits on Jan 17, 2023

  1. add test base for hudi

    @JonasJ-ap
    JonasJ-ap committed Jan 17, 2023
    Configuration menu
    Copy the full SHA
    91ee4ac View commit details
    Browse the repository at this point in the history

Commits on Jan 18, 2023

  1. add write data to partitioned hudi table

    @JonasJ-ap
    JonasJ-ap committed Jan 18, 2023
    Configuration menu
    Copy the full SHA
    57f93f1 View commit details
    Browse the repository at this point in the history

  2. test fail

    @JonasJ-ap
    JonasJ-ap committed Jan 18, 2023
    Configuration menu
    Copy the full SHA
    c091988 View commit details
    Browse the repository at this point in the history

  3. test work

    @JonasJ-ap
    JonasJ-ap committed Jan 18, 2023
    Configuration menu
    Copy the full SHA
    dda6a1f View commit details
    Browse the repository at this point in the history

Commits on Jan 19, 2023

  1. work out schema conversion

    @JonasJ-ap
    JonasJ-ap committed Jan 19, 2023
    Configuration menu
    Copy the full SHA
    0ef3583 View commit details
    Browse the repository at this point in the history

Commits on Jan 21, 2023

  1. rename some methods

    @JonasJ-ap
    JonasJ-ap committed Jan 21, 2023
    Configuration menu
    Copy the full SHA
    ced38a1 View commit details
    Browse the repository at this point in the history

  2. COW first draft, but currently cannot get file groups

    @JonasJ-ap
    JonasJ-ap committed Jan 21, 2023
    Configuration menu
    Copy the full SHA
    7286cab View commit details
    Browse the repository at this point in the history

Commits on Jan 22, 2023

  1. prepare for draft PR discussion

    @JonasJ-ap
    JonasJ-ap committed Jan 22, 2023
    Configuration menu
    Copy the full SHA
    bbb5c36 View commit details
    Browse the repository at this point in the history

Commits on Jan 25, 2023

  1. fix get all file groups

    @JonasJ-ap
    JonasJ-ap committed Jan 25, 2023
    Configuration menu
    Copy the full SHA
    44f7f69 View commit details
    Browse the repository at this point in the history

  2. successfully snapshot first hoodie table

    @JonasJ-ap
    JonasJ-ap committed Jan 25, 2023
    Configuration menu
    Copy the full SHA
    43e30de View commit details
    Browse the repository at this point in the history

Commits on Jan 30, 2023

  1. pass test for all primitive types and partition table

    @JonasJ-ap
    JonasJ-ap committed Jan 30, 2023
    Configuration menu
    Copy the full SHA
    cb382db View commit details
    Browse the repository at this point in the history

Commits on Jan 31, 2023

  1. find bugs when arrayType presents

    @JonasJ-ap
    JonasJ-ap committed Jan 31, 2023
    Configuration menu
    Copy the full SHA
    62ef777 View commit details
    Browse the repository at this point in the history

  2. remove the need of hadoop-mr

    @JonasJ-ap
    JonasJ-ap committed Jan 31, 2023
    Configuration menu
    Copy the full SHA
    8fef8c9 View commit details
    Browse the repository at this point in the history

Commits on Feb 1, 2023

  1. verify multiple commits

    @JonasJ-ap
    JonasJ-ap committed Feb 1, 2023
    Configuration menu
    Copy the full SHA
    9db1ead View commit details
    Browse the repository at this point in the history

Commits on Feb 2, 2023

  1. make arrayType possible by enforcing the new list type in parquet

    @JonasJ-ap
    JonasJ-ap committed Feb 2, 2023
    Configuration menu
    Copy the full SHA
    94dddef View commit details
    Browse the repository at this point in the history

  2. add tests refactor the base action implementation and add ci

    @JonasJ-ap
    JonasJ-ap committed Feb 2, 2023
    Configuration menu
    Copy the full SHA
    cc0a9bb View commit details
    Browse the repository at this point in the history

  3. resolve dependency issue (kind of...)

    @JonasJ-ap
    JonasJ-ap committed Feb 2, 2023
    Configuration menu
    Copy the full SHA
    3ee5a6b View commit details
    Browse the repository at this point in the history

Commits on Feb 8, 2023

  1. handle multiple partition columns

    @JonasJ-ap
    JonasJ-ap committed Feb 8, 2023
    Configuration menu
    Copy the full SHA
    fb479a6 View commit details
    Browse the repository at this point in the history

  2. fix bug for unpartitioned table

    @JonasJ-ap
    JonasJ-ap committed Feb 8, 2023
    Configuration menu
    Copy the full SHA
    42f9709 View commit details
    Browse the repository at this point in the history

  3. checked multi partitions

    @JonasJ-ap
    JonasJ-ap committed Feb 8, 2023
    Configuration menu
    Copy the full SHA
    dde7fcd View commit details
    Browse the repository at this point in the history