-
Notifications
You must be signed in to change notification settings - Fork 2.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Delta: Support Snapshot Delta Lake Table to Iceberg Table #6449
Commits on Dec 18, 2022
-
Configuration menu - View commit details
-
Copy full SHA for 73e38e5 - Browse repository at this point
Copy the full SHA 73e38e5View commit details -
Configuration menu - View commit details
-
Copy full SHA for 5544f45 - Browse repository at this point
Copy the full SHA 5544f45View commit details -
Configuration menu - View commit details
-
Copy full SHA for b8b6119 - Browse repository at this point
Copy the full SHA b8b6119View commit details -
Configuration menu - View commit details
-
Copy full SHA for 274560c - Browse repository at this point
Copy the full SHA 274560cView commit details
Commits on Dec 19, 2022
-
implement direct schema transformation and stop using spark context t…
…o do the migration
Configuration menu - View commit details
-
Copy full SHA for 39e3541 - Browse repository at this point
Copy the full SHA 39e3541View commit details -
Configuration menu - View commit details
-
Copy full SHA for 92f962c - Browse repository at this point
Copy the full SHA 92f962cView commit details -
Configuration menu - View commit details
-
Copy full SHA for 033c997 - Browse repository at this point
Copy the full SHA 033c997View commit details -
Configuration menu - View commit details
-
Copy full SHA for 681a32f - Browse repository at this point
Copy the full SHA 681a32fView commit details -
Configuration menu - View commit details
-
Copy full SHA for 77bdb27 - Browse repository at this point
Copy the full SHA 77bdb27View commit details -
Configuration menu - View commit details
-
Copy full SHA for 3dd540a - Browse repository at this point
Copy the full SHA 3dd540aView commit details -
Configuration menu - View commit details
-
Copy full SHA for 27ece93 - Browse repository at this point
Copy the full SHA 27ece93View commit details
Commits on Dec 21, 2022
-
Configuration menu - View commit details
-
Copy full SHA for a9faabf - Browse repository at this point
Copy the full SHA a9faabfView commit details -
Configuration menu - View commit details
-
Copy full SHA for 3982711 - Browse repository at this point
Copy the full SHA 3982711View commit details -
Configuration menu - View commit details
-
Copy full SHA for 9a7c443 - Browse repository at this point
Copy the full SHA 9a7c443View commit details -
Configuration menu - View commit details
-
Copy full SHA for 173534e - Browse repository at this point
Copy the full SHA 173534eView commit details -
Configuration menu - View commit details
-
Copy full SHA for bdd1ccf - Browse repository at this point
Copy the full SHA bdd1ccfView commit details -
Configuration menu - View commit details
-
Copy full SHA for 85abac2 - Browse repository at this point
Copy the full SHA 85abac2View commit details
Commits on Dec 24, 2022
-
Migrate delta to iceberg round 1 (#29)
* remove redundant todo * move everything to iceberg-delta-lake and make the action a mixin called SupportMigrateDeltaLake * make constant string final and static * use filesToAdd/Remove to determine transaction directly * refactor and delete withRecordNumber from DataFiles.builder * refactor get partitionValues and use FileIO to get size when necessary * Add javadoc * refactor exceptions to be ValidationException * fix validationException format issuse * create new test base for spark delta test
Configuration menu - View commit details
-
Copy full SHA for 32e1af8 - Browse repository at this point
Copy the full SHA 32e1af8View commit details
Commits on Dec 25, 2022
-
Migrate delta to iceberg util refactor (#30)
* refactor the structure of the package * copy-pase the util methods from TableMigrationUtil
Configuration menu - View commit details
-
Copy full SHA for ac1141d - Browse repository at this point
Copy the full SHA ac1141dView commit details
Commits on Dec 28, 2022
-
Migrate delta to iceberg refactor 1.5 (#31)
* move getFileMetrics to FileMetricsReader * add unit tests for schema conversion
Configuration menu - View commit details
-
Copy full SHA for 8e9b3fc - Browse repository at this point
Copy the full SHA 8e9b3fcView commit details -
Configuration menu - View commit details
-
Copy full SHA for 12b60ca - Browse repository at this point
Copy the full SHA 12b60caView commit details
Commits on Dec 29, 2022
-
use transaction, refactor structure, add optional newTableLocation, a…
…dd tableProperty (#32) * use transaction to commit all changes once * add optional newTableLocation * simplify the datafile build process, remove FileMetricsReader * refactor package structure * remove unnecessary types * fix format issue * add tableProperty method
Configuration menu - View commit details
-
Copy full SHA for 8a8adef - Browse repository at this point
Copy the full SHA 8a8adefView commit details -
Configuration menu - View commit details
-
Copy full SHA for 6fbf740 - Browse repository at this point
Copy the full SHA 6fbf740View commit details
Commits on Dec 30, 2022
-
refactor getFullPath with unit tests, use newCreateTableTransaction, …
…remove unnecessary parameters and try-catch (#33) * refactor getFullFilePath to be static and add test * refactor the interface name * use newCreateTableTransaction, remove redundant parameters in helper methods * remove unnecessary try, catch
Configuration menu - View commit details
-
Copy full SHA for 69671b9 - Browse repository at this point
Copy the full SHA 69671b9View commit details
Commits on Jan 3, 2023
-
allow user to specify a custom location for migrated table, fix load …
…error of icebergCatalog (#34) * modify build.gradle to remove unnecessary dependency * fix nit problem * pass real env test in a questionable manner * allow user to specify a custom location for migrated table * remove unnecessary logger * restore build.gradle for spark
Configuration menu - View commit details
-
Copy full SHA for e3138a6 - Browse repository at this point
Copy the full SHA e3138a6View commit details
Commits on Jan 6, 2023
-
Fix nit problems and optimize some implementation (#38)
* refactor mixin order * fix nit problems * add null check to the constructor * let copyFromDeltaLakeToIceberg return the number of totalDataFiles directly * use ImmutableMap.Builder * fix the problem in getFullFilePath * use hadoopFileIO to read dataFile * make type conversion util package-private * fix format
Configuration menu - View commit details
-
Copy full SHA for 2e8dfd0 - Browse repository at this point
Copy the full SHA 2e8dfd0View commit details -
Configuration menu - View commit details
-
Copy full SHA for f4589e8 - Browse repository at this point
Copy the full SHA f4589e8View commit details
Commits on Jan 7, 2023
-
move everthing to iceberg-delta-lake, build demo integration test (#35)
* refactor mixin order * fix nit problems * add null check to the constructor * let copyFromDeltaLakeToIceberg return the number of totalDataFiles directly * use ImmutableMap.Builder * fix the problem in getFullFilePath * use hadoopFileIO to read dataFile * make type conversion util package-private * fix format issue * move everything to iceberg-delta-lake. Remove all changes made to iceberg-spark * fix test delta_core dependency * fix format * conditionally build the test * refactor to integrationTest * suppress warnings * test delta core 2.2.0
Configuration menu - View commit details
-
Copy full SHA for 59c96cb - Browse repository at this point
Copy the full SHA 59c96cbView commit details
Commits on Jan 9, 2023
-
optimize api structure, refactor the integration test, add more tests(#…
…39) * use validation exception for unsupported types * check result file count in integration test * fix format * add tableLocation api and remove constructors * add javadoc for constructor * remove unnecessary test * use UUID to generate table records * resolve format issue * rename everything from migrate to snapshot * simplify test configuration round 1 * refactor test spark integration * refactor correctness check to helper function * add test for table location and table properties
Configuration menu - View commit details
-
Copy full SHA for afd783b - Browse repository at this point
Copy the full SHA afd783bView commit details
Commits on Jan 10, 2023
-
refactor the interfaces, add new tests to integration tests, add new …
…unit tests (#40) * rename the interface * add new APIs and add unit test for precondition checks * refactor interface and precondition check * remove redundant private method and refactor javadoc * add test logic for table contains external data files * test the inclusion of delta lake table properties
Configuration menu - View commit details
-
Copy full SHA for 5b95925 - Browse repository at this point
Copy the full SHA 5b95925View commit details -
fix error messages and add default implementation for actionProvider (#…
…41) * fix error messages * add getDefault implementation to the action provider
Configuration menu - View commit details
-
Copy full SHA for f43c325 - Browse repository at this point
Copy the full SHA f43c325View commit details -
refactor the default implementation and javadoc (#43)
* rename default implementation and make it an instance * optimize javadoc * make base classes package-private * refactor javadoc in the interface
Configuration menu - View commit details
-
Copy full SHA for b2a8bfe - Browse repository at this point
Copy the full SHA b2a8bfeView commit details
Commits on Jan 12, 2023
-
fix error when migrating table with nested fields, add CI, upgrade te…
…st (#44) * fix parquet import error for nested schema * add delta conversion CI * upgrade the test
Configuration menu - View commit details
-
Copy full SHA for 450a08c - Browse repository at this point
Copy the full SHA 450a08cView commit details -
Configuration menu - View commit details
-
Copy full SHA for 300d39b - Browse repository at this point
Copy the full SHA 300d39bView commit details -
Configuration menu - View commit details
-
Copy full SHA for a285c4a - Browse repository at this point
Copy the full SHA a285c4aView commit details -
Configuration menu - View commit details
-
Copy full SHA for 5760a83 - Browse repository at this point
Copy the full SHA 5760a83View commit details -
Configuration menu - View commit details
-
Copy full SHA for e41c787 - Browse repository at this point
Copy the full SHA e41c787View commit details
Commits on Jan 13, 2023
-
Configuration menu - View commit details
-
Copy full SHA for 7a16809 - Browse repository at this point
Copy the full SHA 7a16809View commit details -
Configuration menu - View commit details
-
Copy full SHA for 7072612 - Browse repository at this point
Copy the full SHA 7072612View commit details -
remove unnecessary fields and class and let integrationTest collected…
… by CI (#45) * remove unnecessary fields and class * make integration test collected by check
Configuration menu - View commit details
-
Copy full SHA for c2293c9 - Browse repository at this point
Copy the full SHA c2293c9View commit details -
Configuration menu - View commit details
-
Copy full SHA for f38d7b1 - Browse repository at this point
Copy the full SHA f38d7b1View commit details -
Configuration menu - View commit details
-
Copy full SHA for 10163f8 - Browse repository at this point
Copy the full SHA 10163f8View commit details
Commits on Jan 15, 2023
-
Merge remote-tracking branch 'origin/master' into migrate_delta_to_ic…
…eberg # Conflicts: # settings.gradle # versions.props
Configuration menu - View commit details
-
Copy full SHA for 99dbba8 - Browse repository at this point
Copy the full SHA 99dbba8View commit details
Commits on Jan 17, 2023
-
* remove unnecessary namespace creation * move namespace creation to TestSnapshotDeltaLakeTable.java
Configuration menu - View commit details
-
Copy full SHA for a7c3de1 - Browse repository at this point
Copy the full SHA a7c3de1View commit details
Commits on Jan 20, 2023
-
Configuration menu - View commit details
-
Copy full SHA for 6c4ab2c - Browse repository at this point
Copy the full SHA 6c4ab2cView commit details
Commits on Jan 21, 2023
-
add null check for table.currentSnapshot() when querying the total nu…
…mber of data files migrated
Configuration menu - View commit details
-
Copy full SHA for dadd76a - Browse repository at this point
Copy the full SHA dadd76aView commit details
Commits on Jan 24, 2023
-
Refactor iceberg-delta's integration test(#48)
* use assertj for all tests * add null check for the spark integration method * use a method to generate the hardcode dataframe * drop iceberg table afterwards * add typetest table * test all delta lake types * test conversion of NullType * fix format issue * add a second dataframe * refactor the integration test * correctly decoded delta's path * fix wrong decoding * fix wrong decoding 2
Configuration menu - View commit details
-
Copy full SHA for 1cd36b9 - Browse repository at this point
Copy the full SHA 1cd36b9View commit details
Commits on Jan 25, 2023
-
Adapt for delta.logRetentionDuration (#49)
* remove a redundant map collector in commitDeltaVersionLogToIcebergTransaction * get the earliest possible version rather than hard code from 0 * add unit test to check if table exists * refactor action extracted from the versionlog * fix format issue * move non-share table write operation to the test itself, instead of in before() * fix type
Configuration menu - View commit details
-
Copy full SHA for 4463f30 - Browse repository at this point
Copy the full SHA 4463f30View commit details
Commits on Jan 26, 2023
-
Configuration menu - View commit details
-
Copy full SHA for d3ccc86 - Browse repository at this point
Copy the full SHA d3ccc86View commit details
Commits on Feb 6, 2023
-
Configuration menu - View commit details
-
Copy full SHA for 1affcb3 - Browse repository at this point
Copy the full SHA 1affcb3View commit details -
Configuration menu - View commit details
-
Copy full SHA for 098a3a2 - Browse repository at this point
Copy the full SHA 098a3a2View commit details -
Merge remote-tracking branch 'origin/master' into migrate_delta_to_ic…
…eberg # Conflicts: # versions.props
Configuration menu - View commit details
-
Copy full SHA for f0d1536 - Browse repository at this point
Copy the full SHA f0d1536View commit details
Commits on Feb 7, 2023
-
Configuration menu - View commit details
-
Copy full SHA for a98461a - Browse repository at this point
Copy the full SHA a98461aView commit details -
Configuration menu - View commit details
-
Copy full SHA for fe6da17 - Browse repository at this point
Copy the full SHA fe6da17View commit details -
Configuration menu - View commit details
-
Copy full SHA for 8e9a3e2 - Browse repository at this point
Copy the full SHA 8e9a3e2View commit details -
Configuration menu - View commit details
-
Copy full SHA for 24405e0 - Browse repository at this point
Copy the full SHA 24405e0View commit details -
Configuration menu - View commit details
-
Copy full SHA for c5a6186 - Browse repository at this point
Copy the full SHA c5a6186View commit details