[AMS-Refactor] fix optimizing integration test #1487
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
github-actions
bot
added
module:core
zhoujinsong
pushed a commit
that referenced
this pull request
May 31, 2023
* fix optimizing integration test * fix optimizing integration test * fix optimizing integration test * fix optimizing integration test and fix runtime bug * fix optimizing integration test
zhoujinsong
added a commit
that referenced
this pull request
Jun 1, 2023
* replace ams module with refactored codes * remove optimizer module * fix compile errors * Refactor core and hive * fix terminal catalog connector check * check terminal session NPE before release. * modify ams-server and dashboard module and package names * [Arctic-1372] Resolve compile error (#1417) * resolve compile error. * [ARCTIC-1062][AMS]Terminal support config spark properties in the local model (#1094) * [ARCTIC-1090][AMS]Terminal support add hadoop conf into sparkConf (#1099) * add some PRs since v0.4 * [AMS-Refactor] fix runtime bug (#1421) fix runtime bug * return null when got thrift MISSING_RESULT error * [ARCTIC][AMS]:display login error msg (#1424) feat: display login error msg Co-authored-by: zhubeibei <zhubeibei@corp.netease.com> * Commit the DeleteFile delete operation separately in Iceberg Format (#1425) * add 0.5.0-init.sql * rename optimize_group to optimizer_groups in config.xml * remove useless configs * Reuse PropertyNames property definetion in OptimizerConfig * remove old optimizer module files * fix some compile error in test codes * [AMS-Refactor] fix runtime bug (#1430) * fix runtime bug * fix runtime bug * add unit tests for optimizer * rename util class name from utils to util * remove resourceId from optimizerRegisterInfo * support derby database * format sql in mappers * fix some warnings * change toString implementation by guava * [AMS-Refactor] fix runtime bug (#1433) * fix runtime bug * fix runtime bug * fix runtime bug * Mix-format committer * Fix compile error * fix optimizer executor retry bug * [Arctic-1372][Refactor-AMS] Add unit test for executors (#1434) * resolve compile error. * resolve compile error. * add Executor unit test * [Arctic-1372][Refactor-AMS] Refactor package name (#1435) * resolve compile error. * resolve compile error. * add Executor unit test * refactor package level * [AMS-refactor]: add committing status (#1439) * feat: display login error msg * feat: optimizers tables add committing status --------- Co-authored-by: zhubeibei <zhubeibei@corp.netease.com> * support ha mode for ams * add fromSequence toSequence to TableOptimizingProcess * fix unit test errors in module core * fix unit tests in module hive * add default external container * adapt ams ha thrift url for optimizer container * [AMS-Refactor] fix runtime bug (#1436) * fix runtime bug * fix runtime bug * fix runtime bug * fix runtime bug * [AMS-Refactor] Abstracted test code (#1441) * resolve compile error. * resolve compile error. * add Executor unit test * refactor package level * abstracted test code to reduce duplication of code * [AMS-Refactor] merge `blocker` to refactor branch (#1427) * introduce blocker to 0.5.0 * add exception compatative * fix get optimizer groups bug for optimizer controller * [AMS-Refactor]Use ArcticSparkSessionCatalog for terminal (#1442) * use ArcticSparkSessionCatalog in terminal * remove useless import --------- Co-authored-by: jinsilei <jinsilei@corp.netease.com> * add TestCatalogService * mixed iceberg plan * support mixed hive plan * refactor plan scan files * refactor AbstractPartitionPlan * refactor FileTree * replace OptimizingTaskProperties with OptimizingInputProperties * refactor MixedFormatTableFileScanHelper to UnkeyedTableFileScanHelper and KeyedTableFileScanHelper * refactor unkeyed table split task * add partition plan simple test case * refactor DefaultPartitionEvaluator * union addFile of AbstractPartitionPlan * add is FullNecessary * fix compile error * split deleteFiles in RewriteFilesInput into readOnlyDeleteFiles and rewriteDeleteFiles * add BasicMixedHivePartitionPlan * refactor test * add assert for unit test * fix conflict * [WIP][AMS-Refactor] merge allocating transaction id from AMS (#1428) * support new transactionId in ams * remove table_transaction_meta from debry * not support allocate TransactionId from AMS in 0.5.0 * fix checkstyle * remove currentTxId * add max-txId to hidden properties * refactor test case AbstractMixedTablePartitionPlan * add TestTableService * fix conflicts * add TestKeyedTableFileScanHelper * fix checkstyle * fix scan for empty table * Fix some error * Fix some error * Fix some error * Fix some error * Format code style * Format code style * refactor test case for plan and scan * fix checkstyle * Add only delete * Add cdc * DataTestHelpers support writeBaseStorePosDelete * add plan test case for segment files and pos delete files * convert Map<String, Long> partitionSequence to StructLikeMap<Long> * fix trhift service proxy exception handle bug * fix plan for only one file * [AMS-Refactor] fix runtime bug and add unit test (#1446) 1.fix runtime bug 2.add ams env and optimize result checker unit test * persist from and to sequence for optimizing process * add server catalog for mixed-hive format tables * remove duplicate codes in mixed hive catalog * optimizing adapt to blocker with snapshot id * add TestKeyedOptimizingPlanner TestKeyedOptimizingEvaluator * fix test * add test case for TestKeyedOptimizingEvaluator * expose hive client pool in mixed hive server catalog * fix OptimizingPlanner isNecessary * planer * fix errors in AMSTableTestBase * Optimizer Unit test * refactor some service code * add file statistics for PartitionEvaluator * mock TableRuntime * check style * Format code style * Add Iceberg write data * add test case for plan testWithDeleteFiles * add plan test case testChangeFilesWithDelete * add test case for plan evaluator for unkeyed table * add test case for full optimizing * refactor some service code1 * merge * fix compile error * modify blocker * add TestIcebergTableFileScanHelper * add TestIcebergPartitionPlan * check style * fix keyedTable data losing * add test case for check from to sequence * refactor to MixedIcebergPartition MixedHivePartitionPlan * move TableSnapshot to package com.netease.arctic.server.table * support set minorLeastInterval to -1 * fix optimize type of OptimizingPlanner * fix evaluator cost * fix different hive locations * fix hive subdirectory * fix new hive output dir * ADD StatedPersistentBase and unit test * Make TaskRuntime and TableRuntime extended from StatedPersistentBase * fix conflicts * fix AMS start up error * fix NestSqlSession close() bug * add MixedHivePartitionEvaluator MixedIcebergPartitionEvaluator * fix full optimize continus execute * fix getBaseSplitCount * fix init BasicPartitionEvaluator * rename BasicPartitionEvaluator to CommonPartitionEvaluator * fix ams-mysql-init.sql * remove repair.sh * [AMS-Refactor] fix runtime bug and add unit test (#1459) * fix runtime bug * fix runtime bug * fix runtime bug * fix runtime bug * 1.fix runtime bug 2.add ams env and optimize result checker unit test * 1.fix runtime bug 2.controller adapt mixed-format * 1.fix runtime bug 2.add mix-format table optimize unit test * resolve comment * resolve comment * resolve comment * Fix some bug * Add Optimizing UT * fix table not exist * fix get operations of keyed table * fix transaction detail of keyed table * add target_change_snapshot_id and last_optimized_change_snapshotId to fix keyed table optimizing * fix select/update/insert properties of task_runtime * remove useless import * fix check style * close table trash * Add Schedule policy * load to sequence and from sequence * fix not update currentStatusStartTime * fix some bug * fix output file * Format code style * not show begin transaction snapshot * fix create catalog rollback * remove useless optimizing properties * [Refactor-AMS] Fix some bugs and polish UT test (#1466) * resolve compile error. * resolve compile error. * add Executor unit test * refactor package level * abstracted test code to reduce duplication of code * fix some bugs and polish ut test * fix ut case * fix some unit test erros in ams server * code coverage support for asm server * add unit test for blocker * add unit test for BlockerExpiringExecutor * move unit test for Executor to server.table.exector * addTaskQuota before retry * add more test case for TestOptimizingQueue * fix load optimize input * add test case for TestOptimizingQueue * remove useless dependencies * reverse operations * fix Transactions order by commit time * fix checkstyle * TableRuntimeRefreshExecutor always execute periodically * [AMS-Refactor] Solve some restart problems (#1472) * resolve compile error. * resolve compile error. * add Executor unit test * refactor package level * abstracted test code to reduce duplication of code * fix some bugs and polish ut test * fix ut case * fix some bugs * fix some bugs * fix some bugs * fix some bugs * fix some bugs * fix some bugs * [ARCTIC][AMS]: update container settings info (#1473) feat: container settings update Co-authored-by: zhubeibei <zhubeibei@corp.netease.com> * Fix UnKeyed commit error * make ArcticRuntimeException methods more clear * [WIP]Upgrade SQL script (#1450) * add sql file * add sql file * update sql file * fix * init table_runtime * init table_runtime * add column * add column --------- Co-authored-by: jinsilei <jinsilei@corp.netease.com> * change readme * update uprade sql script * fix optimizer execute error message is null * [AMS-Refactor] add scan table file list (#1468) * add scan table file list * add scan table file list * add scan table file list * add scan table file list * resolve comment * fix hive table upgrade logic * resolve comment * update docs to adapt new ams configurations * fix conflicts * [AMS-Refactor] Optimizing trigger support hive/base max delay for each partition (#1480) * refactor PartitionTransactionOperation: apply() return partition properties * core support table partition property 'base-op-time' * optimizing support base max delay and hive max delay * add partition evaluator weight and add unit test case * add TestHiveOptimizingEvaluator * add result cache for CommonPartitionEvaluator * remove useless code * modify property name * change property to base.hive.refresh-interval and base.refresh-interval * remove annotations.NotNull * change to filesNotInHiveLocation * fix configuration file values * fix pom dependency erros * [Arctic-1160][Spark]: Unit test refactor for spark module (#1366) * unit test framework for spark create table test * refactor create table tests * stash * test framework * test framework * create table test * create table test assert hive schema * test create table like * refactor test code * create table like * TestTable * Test Tables * junit5 tests * junit5 tests * junit5 tests framework * junit5 tests framework * junit5 tests framework * junit5 tests framework * add test * add alter table test * add update test * CTAS * some comments * some comments * add jacoco exclude packages * insert into test * optimize alter table test * optimize truncate test * some insert overwrite tests * some insert overwrite tests * optimize write for overwrite * add desc test and optimize code * add dataFarame api test * merge into tests * MultiDelegateSessionCatalog test case * fix npe in unit tests * fix resource clean-up in test * remove some codes * remove some codes * new antlr for extend sql * new antlr for extend sql * remove used codes * remove un-used junit4 tests * add ut for ast builder * ctas test for query * ctas test for query * ctas test for query * ctas test for query * write avro files * merge from master * simplify command parser * spark 3.2 new unit tests * spark 3.2 new unit tests * fix truncate problem * fix merge into 3.1 * unit test case for spark 3.3 * fix merge into for 3.2 & 3,3 * fix merge into for 3.2 & 3,3 * remove useless comment * code check style * fix 3.3 check condition * fix alter table add column * fix alter table add column * session catalog tests * spark 3.3 sql extend parser * spark 3.3 sql command parser * enable jacoco in github ci * add junit vintage engines * flink 1.4 test * add junit vintage engines * disable jacoco code cover report for flink module * fix ams compile error * spark surefire plugin remove suite entry * class load for spark 3.2 * remove useless code for optimzie * fix test impala in spark * optimize drop partition test * Add log when create HMS thread * increment HMS pool size * change log level for spark test logger * test code style for spark/v3.1 * test code style for spark/v3.2 * test code style for spark/v3.3 * fix unit tests * fix review comment * remove CollectionUtil.asMap * remove CollectionUtil.zip --------- Co-authored-by: jinsilei <jinsilei@corp.netease.com> * change insert table runtime table sql * fix a ut bug in optimizer * Add UnKeyed table flow UT * Delete useless code * fix spark compile error * fix checkstyle * remove useless summary in table detail * [AMS-Refactor] fix update catalog properties (#1486) * add onMetadataUpdate for ServerCatalog * use refreshCatalogMeta() * useless change * simplify updateMetadata * fix update optimize group * if there is no delete file and duplicate.ratio = 0, it should be minor optimizing * [AMS-Refactor] fix optimizing integration test (#1487) * fix optimizing integration test * fix optimizing integration test * fix optimizing integration test * fix optimizing integration test and fix runtime bug * fix optimizing integration test * change flink group property names * fix local optimizer memory error * [Refactor-AMS] Fix TableRuntime recreate (#1488) * resolve compile error. * resolve compile error. * add Executor unit test * refactor package level * abstracted test code to reduce duplication of code * fix some bugs and polish ut test * fix ut case * fix some bugs * fix some bugs * fix some bugs * fix some bugs * fix some bugs * fix some bugs * modify upgrade sql * fix table runtime recreate * add unique to derby init * resolve conflict * resolve conflict * resolve conflict * resolve conflict * resolve conflict * modify log * modify ServerTableIdentifier toString() * rename taskMap of OptimizingQueue to executingTaskMap(SCHEDULED,ACKED) * fix checkstyle * fix losing some task's metrics * should not beginCommitting serveral times * to avoid concurrent optimizing commit for a table * fix repeat commit * make shouldRewritePosForSegmentFile more readable * [AMS-Refactor] make `TableRuntime`'s fileds as `StateField` (#1495) * add StatedPersistentBase.StateField for fields of TableRuntime and tableHandler should not throw exception * replace @StatedPersistentBase.StateField with @StateField * [AMS-Refactor] persistence optimizing process failedReason (#1496) persistence optimizing process failedReason * add logs and comments * [AMS-Refactor] fix optimizing integration test (#1497) * persistence optimizing process failedReason * fix optimizing integration test * fix unit test for native iceberg plan --------- Co-authored-by: shidayang <530847445@qq.com> Co-authored-by: BaiyangTX <xiangnebula@163.com> Co-authored-by: majin1102 <majin1102@163.com> Co-authored-by: HuangFru <68625618+HuangFru@users.noreply.github.com> Co-authored-by: wangzeyu <hameizi369@gmail.com> Co-authored-by: huiyuan_ <819747197@qq.com> Co-authored-by: zhubeibei <zhubeibei@corp.netease.com> Co-authored-by: wangzeyu <1249369293@qq.com> Co-authored-by: wangtao <wangtao3@corp.netease.com> Co-authored-by: wangtaohz <103108928+wangtaohz@users.noreply.github.com> Co-authored-by: PlanetWalker <52364847+hellojinsilei@users.noreply.github.com> Co-authored-by: jinsilei <jinsilei@corp.netease.com>
zhoujinsong
pushed a commit
that referenced
this pull request
Jun 1, 2023
* replace ams module with refactored codes * remove optimizer module * fix compile errors * Refactor core and hive * fix terminal catalog connector check * check terminal session NPE before release. * modify ams-server and dashboard module and package names * [Arctic-1372] Resolve compile error (#1417) * resolve compile error. * [ARCTIC-1062][AMS]Terminal support config spark properties in the local model (#1094) * [ARCTIC-1090][AMS]Terminal support add hadoop conf into sparkConf (#1099) * add some PRs since v0.4 * [AMS-Refactor] fix runtime bug (#1421) fix runtime bug * return null when got thrift MISSING_RESULT error * [ARCTIC][AMS]:display login error msg (#1424) feat: display login error msg Co-authored-by: zhubeibei <zhubeibei@corp.netease.com> * Commit the DeleteFile delete operation separately in Iceberg Format (#1425) * add 0.5.0-init.sql * rename optimize_group to optimizer_groups in config.xml * remove useless configs * Reuse PropertyNames property definetion in OptimizerConfig * remove old optimizer module files * fix some compile error in test codes * [AMS-Refactor] fix runtime bug (#1430) * fix runtime bug * fix runtime bug * add unit tests for optimizer * rename util class name from utils to util * remove resourceId from optimizerRegisterInfo * support derby database * format sql in mappers * fix some warnings * change toString implementation by guava * [AMS-Refactor] fix runtime bug (#1433) * fix runtime bug * fix runtime bug * fix runtime bug * Mix-format committer * Fix compile error * fix optimizer executor retry bug * [Arctic-1372][Refactor-AMS] Add unit test for executors (#1434) * resolve compile error. * resolve compile error. * add Executor unit test * [Arctic-1372][Refactor-AMS] Refactor package name (#1435) * resolve compile error. * resolve compile error. * add Executor unit test * refactor package level * [AMS-refactor]: add committing status (#1439) * feat: display login error msg * feat: optimizers tables add committing status --------- Co-authored-by: zhubeibei <zhubeibei@corp.netease.com> * support ha mode for ams * add fromSequence toSequence to TableOptimizingProcess * fix unit test errors in module core * fix unit tests in module hive * add default external container * adapt ams ha thrift url for optimizer container * [AMS-Refactor] fix runtime bug (#1436) * fix runtime bug * fix runtime bug * fix runtime bug * fix runtime bug * [AMS-Refactor] Abstracted test code (#1441) * resolve compile error. * resolve compile error. * add Executor unit test * refactor package level * abstracted test code to reduce duplication of code * [AMS-Refactor] merge `blocker` to refactor branch (#1427) * introduce blocker to 0.5.0 * add exception compatative * fix get optimizer groups bug for optimizer controller * [AMS-Refactor]Use ArcticSparkSessionCatalog for terminal (#1442) * use ArcticSparkSessionCatalog in terminal * remove useless import --------- Co-authored-by: jinsilei <jinsilei@corp.netease.com> * add TestCatalogService * mixed iceberg plan * support mixed hive plan * refactor plan scan files * refactor AbstractPartitionPlan * refactor FileTree * replace OptimizingTaskProperties with OptimizingInputProperties * refactor MixedFormatTableFileScanHelper to UnkeyedTableFileScanHelper and KeyedTableFileScanHelper * refactor unkeyed table split task * add partition plan simple test case * refactor DefaultPartitionEvaluator * union addFile of AbstractPartitionPlan * add is FullNecessary * fix compile error * split deleteFiles in RewriteFilesInput into readOnlyDeleteFiles and rewriteDeleteFiles * add BasicMixedHivePartitionPlan * refactor test * add assert for unit test * fix conflict * [WIP][AMS-Refactor] merge allocating transaction id from AMS (#1428) * support new transactionId in ams * remove table_transaction_meta from debry * not support allocate TransactionId from AMS in 0.5.0 * fix checkstyle * remove currentTxId * add max-txId to hidden properties * refactor test case AbstractMixedTablePartitionPlan * add TestTableService * fix conflicts * add TestKeyedTableFileScanHelper * fix checkstyle * fix scan for empty table * Fix some error * Fix some error * Fix some error * Fix some error * Format code style * Format code style * refactor test case for plan and scan * fix checkstyle * Add only delete * Add cdc * DataTestHelpers support writeBaseStorePosDelete * add plan test case for segment files and pos delete files * convert Map<String, Long> partitionSequence to StructLikeMap<Long> * fix trhift service proxy exception handle bug * fix plan for only one file * [AMS-Refactor] fix runtime bug and add unit test (#1446) 1.fix runtime bug 2.add ams env and optimize result checker unit test * persist from and to sequence for optimizing process * add server catalog for mixed-hive format tables * remove duplicate codes in mixed hive catalog * optimizing adapt to blocker with snapshot id * add TestKeyedOptimizingPlanner TestKeyedOptimizingEvaluator * fix test * add test case for TestKeyedOptimizingEvaluator * expose hive client pool in mixed hive server catalog * fix OptimizingPlanner isNecessary * planer * fix errors in AMSTableTestBase * Optimizer Unit test * refactor some service code * add file statistics for PartitionEvaluator * mock TableRuntime * check style * Format code style * Add Iceberg write data * add test case for plan testWithDeleteFiles * add plan test case testChangeFilesWithDelete * add test case for plan evaluator for unkeyed table * add test case for full optimizing * refactor some service code1 * merge * fix compile error * modify blocker * add TestIcebergTableFileScanHelper * add TestIcebergPartitionPlan * check style * fix keyedTable data losing * add test case for check from to sequence * refactor to MixedIcebergPartition MixedHivePartitionPlan * move TableSnapshot to package com.netease.arctic.server.table * support set minorLeastInterval to -1 * fix optimize type of OptimizingPlanner * fix evaluator cost * fix different hive locations * fix hive subdirectory * fix new hive output dir * ADD StatedPersistentBase and unit test * Make TaskRuntime and TableRuntime extended from StatedPersistentBase * fix conflicts * fix AMS start up error * fix NestSqlSession close() bug * add MixedHivePartitionEvaluator MixedIcebergPartitionEvaluator * fix full optimize continus execute * fix getBaseSplitCount * fix init BasicPartitionEvaluator * rename BasicPartitionEvaluator to CommonPartitionEvaluator * fix ams-mysql-init.sql * remove repair.sh * [AMS-Refactor] fix runtime bug and add unit test (#1459) * fix runtime bug * fix runtime bug * fix runtime bug * fix runtime bug * 1.fix runtime bug 2.add ams env and optimize result checker unit test * 1.fix runtime bug 2.controller adapt mixed-format * 1.fix runtime bug 2.add mix-format table optimize unit test * resolve comment * resolve comment * resolve comment * Fix some bug * Add Optimizing UT * fix table not exist * fix get operations of keyed table * fix transaction detail of keyed table * add target_change_snapshot_id and last_optimized_change_snapshotId to fix keyed table optimizing * fix select/update/insert properties of task_runtime * remove useless import * fix check style * close table trash * Add Schedule policy * load to sequence and from sequence * fix not update currentStatusStartTime * fix some bug * fix output file * Format code style * not show begin transaction snapshot * fix create catalog rollback * remove useless optimizing properties * [Refactor-AMS] Fix some bugs and polish UT test (#1466) * resolve compile error. * resolve compile error. * add Executor unit test * refactor package level * abstracted test code to reduce duplication of code * fix some bugs and polish ut test * fix ut case * fix some unit test erros in ams server * code coverage support for asm server * add unit test for blocker * add unit test for BlockerExpiringExecutor * move unit test for Executor to server.table.exector * addTaskQuota before retry * add more test case for TestOptimizingQueue * fix load optimize input * add test case for TestOptimizingQueue * remove useless dependencies * reverse operations * fix Transactions order by commit time * fix checkstyle * TableRuntimeRefreshExecutor always execute periodically * [AMS-Refactor] Solve some restart problems (#1472) * resolve compile error. * resolve compile error. * add Executor unit test * refactor package level * abstracted test code to reduce duplication of code * fix some bugs and polish ut test * fix ut case * fix some bugs * fix some bugs * fix some bugs * fix some bugs * fix some bugs * fix some bugs * [ARCTIC][AMS]: update container settings info (#1473) feat: container settings update Co-authored-by: zhubeibei <zhubeibei@corp.netease.com> * Fix UnKeyed commit error * make ArcticRuntimeException methods more clear * [WIP]Upgrade SQL script (#1450) * add sql file * add sql file * update sql file * fix * init table_runtime * init table_runtime * add column * add column --------- Co-authored-by: jinsilei <jinsilei@corp.netease.com> * change readme * update uprade sql script * fix optimizer execute error message is null * [AMS-Refactor] add scan table file list (#1468) * add scan table file list * add scan table file list * add scan table file list * add scan table file list * resolve comment * fix hive table upgrade logic * resolve comment * update docs to adapt new ams configurations * fix conflicts * [AMS-Refactor] Optimizing trigger support hive/base max delay for each partition (#1480) * refactor PartitionTransactionOperation: apply() return partition properties * core support table partition property 'base-op-time' * optimizing support base max delay and hive max delay * add partition evaluator weight and add unit test case * add TestHiveOptimizingEvaluator * add result cache for CommonPartitionEvaluator * remove useless code * modify property name * change property to base.hive.refresh-interval and base.refresh-interval * remove annotations.NotNull * change to filesNotInHiveLocation * fix configuration file values * fix pom dependency erros * [Arctic-1160][Spark]: Unit test refactor for spark module (#1366) * unit test framework for spark create table test * refactor create table tests * stash * test framework * test framework * create table test * create table test assert hive schema * test create table like * refactor test code * create table like * TestTable * Test Tables * junit5 tests * junit5 tests * junit5 tests framework * junit5 tests framework * junit5 tests framework * junit5 tests framework * add test * add alter table test * add update test * CTAS * some comments * some comments * add jacoco exclude packages * insert into test * optimize alter table test * optimize truncate test * some insert overwrite tests * some insert overwrite tests * optimize write for overwrite * add desc test and optimize code * add dataFarame api test * merge into tests * MultiDelegateSessionCatalog test case * fix npe in unit tests * fix resource clean-up in test * remove some codes * remove some codes * new antlr for extend sql * new antlr for extend sql * remove used codes * remove un-used junit4 tests * add ut for ast builder * ctas test for query * ctas test for query * ctas test for query * ctas test for query * write avro files * merge from master * simplify command parser * spark 3.2 new unit tests * spark 3.2 new unit tests * fix truncate problem * fix merge into 3.1 * unit test case for spark 3.3 * fix merge into for 3.2 & 3,3 * fix merge into for 3.2 & 3,3 * remove useless comment * code check style * fix 3.3 check condition * fix alter table add column * fix alter table add column * session catalog tests * spark 3.3 sql extend parser * spark 3.3 sql command parser * enable jacoco in github ci * add junit vintage engines * flink 1.4 test * add junit vintage engines * disable jacoco code cover report for flink module * fix ams compile error * spark surefire plugin remove suite entry * class load for spark 3.2 * remove useless code for optimzie * fix test impala in spark * optimize drop partition test * Add log when create HMS thread * increment HMS pool size * change log level for spark test logger * test code style for spark/v3.1 * test code style for spark/v3.2 * test code style for spark/v3.3 * fix unit tests * fix review comment * remove CollectionUtil.asMap * remove CollectionUtil.zip --------- Co-authored-by: jinsilei <jinsilei@corp.netease.com> * change insert table runtime table sql * fix a ut bug in optimizer * Add UnKeyed table flow UT * Delete useless code * fix spark compile error * fix checkstyle * remove useless summary in table detail * [AMS-Refactor] fix update catalog properties (#1486) * add onMetadataUpdate for ServerCatalog * use refreshCatalogMeta() * useless change * simplify updateMetadata * fix update optimize group * if there is no delete file and duplicate.ratio = 0, it should be minor optimizing * [AMS-Refactor] fix optimizing integration test (#1487) * fix optimizing integration test * fix optimizing integration test * fix optimizing integration test * fix optimizing integration test and fix runtime bug * fix optimizing integration test * change flink group property names * fix local optimizer memory error * [Refactor-AMS] Fix TableRuntime recreate (#1488) * resolve compile error. * resolve compile error. * add Executor unit test * refactor package level * abstracted test code to reduce duplication of code * fix some bugs and polish ut test * fix ut case * fix some bugs * fix some bugs * fix some bugs * fix some bugs * fix some bugs * fix some bugs * modify upgrade sql * fix table runtime recreate * add unique to derby init * resolve conflict * resolve conflict * resolve conflict * resolve conflict * resolve conflict * modify log * modify ServerTableIdentifier toString() * rename taskMap of OptimizingQueue to executingTaskMap(SCHEDULED,ACKED) * fix checkstyle * fix losing some task's metrics * should not beginCommitting serveral times * to avoid concurrent optimizing commit for a table * fix repeat commit * make shouldRewritePosForSegmentFile more readable * [AMS-Refactor] make `TableRuntime`'s fileds as `StateField` (#1495) * add StatedPersistentBase.StateField for fields of TableRuntime and tableHandler should not throw exception * replace @StatedPersistentBase.StateField with @StateField * [AMS-Refactor] persistence optimizing process failedReason (#1496) persistence optimizing process failedReason * add logs and comments * [AMS-Refactor] fix optimizing integration test (#1497) * persistence optimizing process failedReason * fix optimizing integration test * fix unit test for native iceberg plan --------- Co-authored-by: shidayang <530847445@qq.com> Co-authored-by: BaiyangTX <xiangnebula@163.com> Co-authored-by: majin1102 <majin1102@163.com> Co-authored-by: HuangFru <68625618+HuangFru@users.noreply.github.com> Co-authored-by: wangzeyu <hameizi369@gmail.com> Co-authored-by: huiyuan_ <819747197@qq.com> Co-authored-by: zhubeibei <zhubeibei@corp.netease.com> Co-authored-by: wangzeyu <1249369293@qq.com> Co-authored-by: wangtao <wangtao3@corp.netease.com> Co-authored-by: wangtaohz <103108928+wangtaohz@users.noreply.github.com> Co-authored-by: PlanetWalker <52364847+hellojinsilei@users.noreply.github.com> Co-authored-by: jinsilei <jinsilei@corp.netease.com>
zhoujinsong
added a commit
that referenced
this pull request
Jun 7, 2023
* add some PRs since v0.4 * [AMS-Refactor] fix runtime bug (#1421) fix runtime bug * return null when got thrift MISSING_RESULT error * [ARCTIC][AMS]:display login error msg (#1424) feat: display login error msg Co-authored-by: zhubeibei <zhubeibei@corp.netease.com> * Commit the DeleteFile delete operation separately in Iceberg Format (#1425) * add 0.5.0-init.sql * rename optimize_group to optimizer_groups in config.xml * remove useless configs * Reuse PropertyNames property definetion in OptimizerConfig * remove old optimizer module files * fix some compile error in test codes * [AMS-Refactor] fix runtime bug (#1430) * fix runtime bug * fix runtime bug * add unit tests for optimizer * rename util class name from utils to util * remove resourceId from optimizerRegisterInfo * support derby database * format sql in mappers * fix some warnings * change toString implementation by guava * [AMS-Refactor] fix runtime bug (#1433) * fix runtime bug * fix runtime bug * fix runtime bug * Mix-format committer * Fix compile error * fix optimizer executor retry bug * [Arctic-1372][Refactor-AMS] Add unit test for executors (#1434) * resolve compile error. * resolve compile error. * add Executor unit test * [Arctic-1372][Refactor-AMS] Refactor package name (#1435) * resolve compile error. * resolve compile error. * add Executor unit test * refactor package level * [AMS-refactor]: add committing status (#1439) * feat: display login error msg * feat: optimizers tables add committing status --------- Co-authored-by: zhubeibei <zhubeibei@corp.netease.com> * support ha mode for ams * add fromSequence toSequence to TableOptimizingProcess * fix unit test errors in module core * fix unit tests in module hive * add default external container * adapt ams ha thrift url for optimizer container * [AMS-Refactor] fix runtime bug (#1436) * fix runtime bug * fix runtime bug * fix runtime bug * fix runtime bug * [AMS-Refactor] Abstracted test code (#1441) * resolve compile error. * resolve compile error. * add Executor unit test * refactor package level * abstracted test code to reduce duplication of code * [AMS-Refactor] merge `blocker` to refactor branch (#1427) * introduce blocker to 0.5.0 * add exception compatative * fix get optimizer groups bug for optimizer controller * [AMS-Refactor]Use ArcticSparkSessionCatalog for terminal (#1442) * use ArcticSparkSessionCatalog in terminal * remove useless import --------- Co-authored-by: jinsilei <jinsilei@corp.netease.com> * add TestCatalogService * mixed iceberg plan * support mixed hive plan * refactor plan scan files * refactor AbstractPartitionPlan * refactor FileTree * replace OptimizingTaskProperties with OptimizingInputProperties * refactor MixedFormatTableFileScanHelper to UnkeyedTableFileScanHelper and KeyedTableFileScanHelper * refactor unkeyed table split task * add partition plan simple test case * refactor DefaultPartitionEvaluator * union addFile of AbstractPartitionPlan * add is FullNecessary * fix compile error * split deleteFiles in RewriteFilesInput into readOnlyDeleteFiles and rewriteDeleteFiles * add BasicMixedHivePartitionPlan * refactor test * add assert for unit test * fix conflict * [WIP][AMS-Refactor] merge allocating transaction id from AMS (#1428) * support new transactionId in ams * remove table_transaction_meta from debry * not support allocate TransactionId from AMS in 0.5.0 * fix checkstyle * remove currentTxId * add max-txId to hidden properties * refactor test case AbstractMixedTablePartitionPlan * add TestTableService * fix conflicts * add TestKeyedTableFileScanHelper * fix checkstyle * fix scan for empty table * Fix some error * Fix some error * Fix some error * Fix some error * Format code style * Format code style * refactor test case for plan and scan * fix checkstyle * Add only delete * Add cdc * DataTestHelpers support writeBaseStorePosDelete * add plan test case for segment files and pos delete files * convert Map<String, Long> partitionSequence to StructLikeMap<Long> * fix trhift service proxy exception handle bug * fix plan for only one file * [AMS-Refactor] fix runtime bug and add unit test (#1446) 1.fix runtime bug 2.add ams env and optimize result checker unit test * persist from and to sequence for optimizing process * add server catalog for mixed-hive format tables * remove duplicate codes in mixed hive catalog * optimizing adapt to blocker with snapshot id * add TestKeyedOptimizingPlanner TestKeyedOptimizingEvaluator * fix test * add test case for TestKeyedOptimizingEvaluator * expose hive client pool in mixed hive server catalog * fix OptimizingPlanner isNecessary * planer * fix errors in AMSTableTestBase * Optimizer Unit test * refactor some service code * add file statistics for PartitionEvaluator * mock TableRuntime * check style * Format code style * Add Iceberg write data * add test case for plan testWithDeleteFiles * add plan test case testChangeFilesWithDelete * add test case for plan evaluator for unkeyed table * add test case for full optimizing * refactor some service code1 * merge * fix compile error * modify blocker * add TestIcebergTableFileScanHelper * add TestIcebergPartitionPlan * check style * fix keyedTable data losing * add test case for check from to sequence * refactor to MixedIcebergPartition MixedHivePartitionPlan * move TableSnapshot to package com.netease.arctic.server.table * support set minorLeastInterval to -1 * fix optimize type of OptimizingPlanner * fix evaluator cost * fix different hive locations * fix hive subdirectory * fix new hive output dir * ADD StatedPersistentBase and unit test * Make TaskRuntime and TableRuntime extended from StatedPersistentBase * fix conflicts * fix AMS start up error * fix NestSqlSession close() bug * add MixedHivePartitionEvaluator MixedIcebergPartitionEvaluator * fix full optimize continus execute * fix getBaseSplitCount * fix init BasicPartitionEvaluator * rename BasicPartitionEvaluator to CommonPartitionEvaluator * fix ams-mysql-init.sql * remove repair.sh * [AMS-Refactor] fix runtime bug and add unit test (#1459) * fix runtime bug * fix runtime bug * fix runtime bug * fix runtime bug * 1.fix runtime bug 2.add ams env and optimize result checker unit test * 1.fix runtime bug 2.controller adapt mixed-format * 1.fix runtime bug 2.add mix-format table optimize unit test * resolve comment * resolve comment * resolve comment * Fix some bug * Add Optimizing UT * fix table not exist * fix get operations of keyed table * fix transaction detail of keyed table * add target_change_snapshot_id and last_optimized_change_snapshotId to fix keyed table optimizing * fix select/update/insert properties of task_runtime * remove useless import * fix check style * close table trash * Add Schedule policy * load to sequence and from sequence * fix not update currentStatusStartTime * fix some bug * fix output file * Format code style * not show begin transaction snapshot * fix create catalog rollback * remove useless optimizing properties * [Refactor-AMS] Fix some bugs and polish UT test (#1466) * resolve compile error. * resolve compile error. * add Executor unit test * refactor package level * abstracted test code to reduce duplication of code * fix some bugs and polish ut test * fix ut case * fix some unit test erros in ams server * code coverage support for asm server * add unit test for blocker * add unit test for BlockerExpiringExecutor * move unit test for Executor to server.table.exector * addTaskQuota before retry * add more test case for TestOptimizingQueue * fix load optimize input * add test case for TestOptimizingQueue * remove useless dependencies * reverse operations * fix Transactions order by commit time * fix checkstyle * TableRuntimeRefreshExecutor always execute periodically * [AMS-Refactor] Solve some restart problems (#1472) * resolve compile error. * resolve compile error. * add Executor unit test * refactor package level * abstracted test code to reduce duplication of code * fix some bugs and polish ut test * fix ut case * fix some bugs * fix some bugs * fix some bugs * fix some bugs * fix some bugs * fix some bugs * [ARCTIC][AMS]: update container settings info (#1473) feat: container settings update Co-authored-by: zhubeibei <zhubeibei@corp.netease.com> * Fix UnKeyed commit error * make ArcticRuntimeException methods more clear * [WIP]Upgrade SQL script (#1450) * add sql file * add sql file * update sql file * fix * init table_runtime * init table_runtime * add column * add column --------- Co-authored-by: jinsilei <jinsilei@corp.netease.com> * change readme * update uprade sql script * fix optimizer execute error message is null * [AMS-Refactor] add scan table file list (#1468) * add scan table file list * add scan table file list * add scan table file list * add scan table file list * resolve comment * fix hive table upgrade logic * resolve comment * update docs to adapt new ams configurations * fix conflicts * [AMS-Refactor] Optimizing trigger support hive/base max delay for each partition (#1480) * refactor PartitionTransactionOperation: apply() return partition properties * core support table partition property 'base-op-time' * optimizing support base max delay and hive max delay * add partition evaluator weight and add unit test case * add TestHiveOptimizingEvaluator * add result cache for CommonPartitionEvaluator * remove useless code * modify property name * change property to base.hive.refresh-interval and base.refresh-interval * remove annotations.NotNull * change to filesNotInHiveLocation * fix configuration file values * fix pom dependency erros * [Arctic-1160][Spark]: Unit test refactor for spark module (#1366) * unit test framework for spark create table test * refactor create table tests * stash * test framework * test framework * create table test * create table test assert hive schema * test create table like * refactor test code * create table like * TestTable * Test Tables * junit5 tests * junit5 tests * junit5 tests framework * junit5 tests framework * junit5 tests framework * junit5 tests framework * add test * add alter table test * add update test * CTAS * some comments * some comments * add jacoco exclude packages * insert into test * optimize alter table test * optimize truncate test * some insert overwrite tests * some insert overwrite tests * optimize write for overwrite * add desc test and optimize code * add dataFarame api test * merge into tests * MultiDelegateSessionCatalog test case * fix npe in unit tests * fix resource clean-up in test * remove some codes * remove some codes * new antlr for extend sql * new antlr for extend sql * remove used codes * remove un-used junit4 tests * add ut for ast builder * ctas test for query * ctas test for query * ctas test for query * ctas test for query * write avro files * merge from master * simplify command parser * spark 3.2 new unit tests * spark 3.2 new unit tests * fix truncate problem * fix merge into 3.1 * unit test case for spark 3.3 * fix merge into for 3.2 & 3,3 * fix merge into for 3.2 & 3,3 * remove useless comment * code check style * fix 3.3 check condition * fix alter table add column * fix alter table add column * session catalog tests * spark 3.3 sql extend parser * spark 3.3 sql command parser * enable jacoco in github ci * add junit vintage engines * flink 1.4 test * add junit vintage engines * disable jacoco code cover report for flink module * fix ams compile error * spark surefire plugin remove suite entry * class load for spark 3.2 * remove useless code for optimzie * fix test impala in spark * optimize drop partition test * Add log when create HMS thread * increment HMS pool size * change log level for spark test logger * test code style for spark/v3.1 * test code style for spark/v3.2 * test code style for spark/v3.3 * fix unit tests * fix review comment * remove CollectionUtil.asMap * remove CollectionUtil.zip --------- Co-authored-by: jinsilei <jinsilei@corp.netease.com> * change insert table runtime table sql * fix a ut bug in optimizer * Add UnKeyed table flow UT * Delete useless code * fix spark compile error * fix checkstyle * remove useless summary in table detail * [AMS-Refactor] fix update catalog properties (#1486) * add onMetadataUpdate for ServerCatalog * use refreshCatalogMeta() * useless change * simplify updateMetadata * fix update optimize group * if there is no delete file and duplicate.ratio = 0, it should be minor optimizing * [AMS-Refactor] fix optimizing integration test (#1487) * fix optimizing integration test * fix optimizing integration test * fix optimizing integration test * fix optimizing integration test and fix runtime bug * fix optimizing integration test * change flink group property names * fix local optimizer memory error * [Refactor-AMS] Fix TableRuntime recreate (#1488) * resolve compile error. * resolve compile error. * add Executor unit test * refactor package level * abstracted test code to reduce duplication of code * fix some bugs and polish ut test * fix ut case * fix some bugs * fix some bugs * fix some bugs * fix some bugs * fix some bugs * fix some bugs * modify upgrade sql * fix table runtime recreate * add unique to derby init * resolve conflict * resolve conflict * resolve conflict * resolve conflict * resolve conflict * modify log * modify ServerTableIdentifier toString() * rename taskMap of OptimizingQueue to executingTaskMap(SCHEDULED,ACKED) * fix checkstyle * fix losing some task's metrics * should not beginCommitting serveral times * to avoid concurrent optimizing commit for a table * fix repeat commit * make shouldRewritePosForSegmentFile more readable * [AMS-Refactor] make `TableRuntime`'s fileds as `StateField` (#1495) * add StatedPersistentBase.StateField for fields of TableRuntime and tableHandler should not throw exception * replace @StatedPersistentBase.StateField with @StateField * [AMS-Refactor] persistence optimizing process failedReason (#1496) persistence optimizing process failedReason * add logs and comments * [AMS-Refactor] fix optimizing integration test (#1497) * persistence optimizing process failedReason * fix optimizing integration test * fix unit test for native iceberg plan * Fix Trino ci * fix init sql for table resource * modify log touching to DEBUG * remove thread_id/optimize_token from task_runtime of upgrade-0.4.1-to-0.5.0.sql * fix flink 1.12 test log error * fix flink 1.12 unit test compile error * fix flink 1.12 unit test log4j2 config loading error * polish code * fix loading log4j configuration file in flink 1.14/1.15 * fix pusalr-client-all dependency error * [AMS-Refactor] Remove optimizer table and controller switch to use global exception handle (#1502) controller switch to using global exception handling * add unit test case for TestOptimizingQueue * fix checkstyle * change flink unit test log configuration * fix flink 1.12 unit test dependency error * fix mock ams serve ut bug * ignore blocked ut case in flink module * disable TestAutomaticLogWriter in flink 1.12 * change Mock AMS Thrift server implementation * fix optimizer unit test error * fix mock ams complete task handle error * [Refactor-AMS] Fix quota calculation logic (#1520) * Fix quota calculation logic * [Refactor-AMS] Recover task through persistent optimizer information (#1519) Recover task through persistent optimizer information * Format code style * support config optimizer&task timeout --------- Co-authored-by: majin1102 <majin1102@163.com> Co-authored-by: wangzeyu <hameizi369@gmail.com> Co-authored-by: zhoujinsong <463763777@qq.com> Co-authored-by: huiyuan_ <819747197@qq.com> Co-authored-by: zhubeibei <zhubeibei@corp.netease.com> Co-authored-by: shidayang <530847445@qq.com> Co-authored-by: wangzeyu <1249369293@qq.com> Co-authored-by: HuangFru <68625618+HuangFru@users.noreply.github.com> Co-authored-by: wangtao <wangtao3@corp.netease.com> Co-authored-by: wangtaohz <103108928+wangtaohz@users.noreply.github.com> Co-authored-by: PlanetWalker <52364847+hellojinsilei@users.noreply.github.com> Co-authored-by: jinsilei <jinsilei@corp.netease.com> Co-authored-by: baiyangtx <xiangnebula@163.com> Co-authored-by: ZhouJinsong <zhoujinsong0505@163.com>
ShawHee
pushed a commit
to ShawHee/arctic
that referenced
this pull request
Dec 29, 2023
* add some PRs since v0.4 * [AMS-Refactor] fix runtime bug (apache#1421) fix runtime bug * return null when got thrift MISSING_RESULT error * [ARCTIC][AMS]:display login error msg (apache#1424) feat: display login error msg Co-authored-by: zhubeibei <zhubeibei@corp.netease.com> * Commit the DeleteFile delete operation separately in Iceberg Format (apache#1425) * add 0.5.0-init.sql * rename optimize_group to optimizer_groups in config.xml * remove useless configs * Reuse PropertyNames property definetion in OptimizerConfig * remove old optimizer module files * fix some compile error in test codes * [AMS-Refactor] fix runtime bug (apache#1430) * fix runtime bug * fix runtime bug * add unit tests for optimizer * rename util class name from utils to util * remove resourceId from optimizerRegisterInfo * support derby database * format sql in mappers * fix some warnings * change toString implementation by guava * [AMS-Refactor] fix runtime bug (apache#1433) * fix runtime bug * fix runtime bug * fix runtime bug * Mix-format committer * Fix compile error * fix optimizer executor retry bug * [Arctic-1372][Refactor-AMS] Add unit test for executors (apache#1434) * resolve compile error. * resolve compile error. * add Executor unit test * [Arctic-1372][Refactor-AMS] Refactor package name (apache#1435) * resolve compile error. * resolve compile error. * add Executor unit test * refactor package level * [AMS-refactor]: add committing status (apache#1439) * feat: display login error msg * feat: optimizers tables add committing status --------- Co-authored-by: zhubeibei <zhubeibei@corp.netease.com> * support ha mode for ams * add fromSequence toSequence to TableOptimizingProcess * fix unit test errors in module core * fix unit tests in module hive * add default external container * adapt ams ha thrift url for optimizer container * [AMS-Refactor] fix runtime bug (apache#1436) * fix runtime bug * fix runtime bug * fix runtime bug * fix runtime bug * [AMS-Refactor] Abstracted test code (apache#1441) * resolve compile error. * resolve compile error. * add Executor unit test * refactor package level * abstracted test code to reduce duplication of code * [AMS-Refactor] merge `blocker` to refactor branch (apache#1427) * introduce blocker to 0.5.0 * add exception compatative * fix get optimizer groups bug for optimizer controller * [AMS-Refactor]Use ArcticSparkSessionCatalog for terminal (apache#1442) * use ArcticSparkSessionCatalog in terminal * remove useless import --------- Co-authored-by: jinsilei <jinsilei@corp.netease.com> * add TestCatalogService * mixed iceberg plan * support mixed hive plan * refactor plan scan files * refactor AbstractPartitionPlan * refactor FileTree * replace OptimizingTaskProperties with OptimizingInputProperties * refactor MixedFormatTableFileScanHelper to UnkeyedTableFileScanHelper and KeyedTableFileScanHelper * refactor unkeyed table split task * add partition plan simple test case * refactor DefaultPartitionEvaluator * union addFile of AbstractPartitionPlan * add is FullNecessary * fix compile error * split deleteFiles in RewriteFilesInput into readOnlyDeleteFiles and rewriteDeleteFiles * add BasicMixedHivePartitionPlan * refactor test * add assert for unit test * fix conflict * [WIP][AMS-Refactor] merge allocating transaction id from AMS (apache#1428) * support new transactionId in ams * remove table_transaction_meta from debry * not support allocate TransactionId from AMS in 0.5.0 * fix checkstyle * remove currentTxId * add max-txId to hidden properties * refactor test case AbstractMixedTablePartitionPlan * add TestTableService * fix conflicts * add TestKeyedTableFileScanHelper * fix checkstyle * fix scan for empty table * Fix some error * Fix some error * Fix some error * Fix some error * Format code style * Format code style * refactor test case for plan and scan * fix checkstyle * Add only delete * Add cdc * DataTestHelpers support writeBaseStorePosDelete * add plan test case for segment files and pos delete files * convert Map<String, Long> partitionSequence to StructLikeMap<Long> * fix trhift service proxy exception handle bug * fix plan for only one file * [AMS-Refactor] fix runtime bug and add unit test (apache#1446) 1.fix runtime bug 2.add ams env and optimize result checker unit test * persist from and to sequence for optimizing process * add server catalog for mixed-hive format tables * remove duplicate codes in mixed hive catalog * optimizing adapt to blocker with snapshot id * add TestKeyedOptimizingPlanner TestKeyedOptimizingEvaluator * fix test * add test case for TestKeyedOptimizingEvaluator * expose hive client pool in mixed hive server catalog * fix OptimizingPlanner isNecessary * planer * fix errors in AMSTableTestBase * Optimizer Unit test * refactor some service code * add file statistics for PartitionEvaluator * mock TableRuntime * check style * Format code style * Add Iceberg write data * add test case for plan testWithDeleteFiles * add plan test case testChangeFilesWithDelete * add test case for plan evaluator for unkeyed table * add test case for full optimizing * refactor some service code1 * merge * fix compile error * modify blocker * add TestIcebergTableFileScanHelper * add TestIcebergPartitionPlan * check style * fix keyedTable data losing * add test case for check from to sequence * refactor to MixedIcebergPartition MixedHivePartitionPlan * move TableSnapshot to package com.netease.arctic.server.table * support set minorLeastInterval to -1 * fix optimize type of OptimizingPlanner * fix evaluator cost * fix different hive locations * fix hive subdirectory * fix new hive output dir * ADD StatedPersistentBase and unit test * Make TaskRuntime and TableRuntime extended from StatedPersistentBase * fix conflicts * fix AMS start up error * fix NestSqlSession close() bug * add MixedHivePartitionEvaluator MixedIcebergPartitionEvaluator * fix full optimize continus execute * fix getBaseSplitCount * fix init BasicPartitionEvaluator * rename BasicPartitionEvaluator to CommonPartitionEvaluator * fix ams-mysql-init.sql * remove repair.sh * [AMS-Refactor] fix runtime bug and add unit test (apache#1459) * fix runtime bug * fix runtime bug * fix runtime bug * fix runtime bug * 1.fix runtime bug 2.add ams env and optimize result checker unit test * 1.fix runtime bug 2.controller adapt mixed-format * 1.fix runtime bug 2.add mix-format table optimize unit test * resolve comment * resolve comment * resolve comment * Fix some bug * Add Optimizing UT * fix table not exist * fix get operations of keyed table * fix transaction detail of keyed table * add target_change_snapshot_id and last_optimized_change_snapshotId to fix keyed table optimizing * fix select/update/insert properties of task_runtime * remove useless import * fix check style * close table trash * Add Schedule policy * load to sequence and from sequence * fix not update currentStatusStartTime * fix some bug * fix output file * Format code style * not show begin transaction snapshot * fix create catalog rollback * remove useless optimizing properties * [Refactor-AMS] Fix some bugs and polish UT test (apache#1466) * resolve compile error. * resolve compile error. * add Executor unit test * refactor package level * abstracted test code to reduce duplication of code * fix some bugs and polish ut test * fix ut case * fix some unit test erros in ams server * code coverage support for asm server * add unit test for blocker * add unit test for BlockerExpiringExecutor * move unit test for Executor to server.table.exector * addTaskQuota before retry * add more test case for TestOptimizingQueue * fix load optimize input * add test case for TestOptimizingQueue * remove useless dependencies * reverse operations * fix Transactions order by commit time * fix checkstyle * TableRuntimeRefreshExecutor always execute periodically * [AMS-Refactor] Solve some restart problems (apache#1472) * resolve compile error. * resolve compile error. * add Executor unit test * refactor package level * abstracted test code to reduce duplication of code * fix some bugs and polish ut test * fix ut case * fix some bugs * fix some bugs * fix some bugs * fix some bugs * fix some bugs * fix some bugs * [ARCTIC][AMS]: update container settings info (apache#1473) feat: container settings update Co-authored-by: zhubeibei <zhubeibei@corp.netease.com> * Fix UnKeyed commit error * make ArcticRuntimeException methods more clear * [WIP]Upgrade SQL script (apache#1450) * add sql file * add sql file * update sql file * fix * init table_runtime * init table_runtime * add column * add column --------- Co-authored-by: jinsilei <jinsilei@corp.netease.com> * change readme * update uprade sql script * fix optimizer execute error message is null * [AMS-Refactor] add scan table file list (apache#1468) * add scan table file list * add scan table file list * add scan table file list * add scan table file list * resolve comment * fix hive table upgrade logic * resolve comment * update docs to adapt new ams configurations * fix conflicts * [AMS-Refactor] Optimizing trigger support hive/base max delay for each partition (apache#1480) * refactor PartitionTransactionOperation: apply() return partition properties * core support table partition property 'base-op-time' * optimizing support base max delay and hive max delay * add partition evaluator weight and add unit test case * add TestHiveOptimizingEvaluator * add result cache for CommonPartitionEvaluator * remove useless code * modify property name * change property to base.hive.refresh-interval and base.refresh-interval * remove annotations.NotNull * change to filesNotInHiveLocation * fix configuration file values * fix pom dependency erros * [Arctic-1160][Spark]: Unit test refactor for spark module (apache#1366) * unit test framework for spark create table test * refactor create table tests * stash * test framework * test framework * create table test * create table test assert hive schema * test create table like * refactor test code * create table like * TestTable * Test Tables * junit5 tests * junit5 tests * junit5 tests framework * junit5 tests framework * junit5 tests framework * junit5 tests framework * add test * add alter table test * add update test * CTAS * some comments * some comments * add jacoco exclude packages * insert into test * optimize alter table test * optimize truncate test * some insert overwrite tests * some insert overwrite tests * optimize write for overwrite * add desc test and optimize code * add dataFarame api test * merge into tests * MultiDelegateSessionCatalog test case * fix npe in unit tests * fix resource clean-up in test * remove some codes * remove some codes * new antlr for extend sql * new antlr for extend sql * remove used codes * remove un-used junit4 tests * add ut for ast builder * ctas test for query * ctas test for query * ctas test for query * ctas test for query * write avro files * merge from master * simplify command parser * spark 3.2 new unit tests * spark 3.2 new unit tests * fix truncate problem * fix merge into 3.1 * unit test case for spark 3.3 * fix merge into for 3.2 & 3,3 * fix merge into for 3.2 & 3,3 * remove useless comment * code check style * fix 3.3 check condition * fix alter table add column * fix alter table add column * session catalog tests * spark 3.3 sql extend parser * spark 3.3 sql command parser * enable jacoco in github ci * add junit vintage engines * flink 1.4 test * add junit vintage engines * disable jacoco code cover report for flink module * fix ams compile error * spark surefire plugin remove suite entry * class load for spark 3.2 * remove useless code for optimzie * fix test impala in spark * optimize drop partition test * Add log when create HMS thread * increment HMS pool size * change log level for spark test logger * test code style for spark/v3.1 * test code style for spark/v3.2 * test code style for spark/v3.3 * fix unit tests * fix review comment * remove CollectionUtil.asMap * remove CollectionUtil.zip --------- Co-authored-by: jinsilei <jinsilei@corp.netease.com> * change insert table runtime table sql * fix a ut bug in optimizer * Add UnKeyed table flow UT * Delete useless code * fix spark compile error * fix checkstyle * remove useless summary in table detail * [AMS-Refactor] fix update catalog properties (apache#1486) * add onMetadataUpdate for ServerCatalog * use refreshCatalogMeta() * useless change * simplify updateMetadata * fix update optimize group * if there is no delete file and duplicate.ratio = 0, it should be minor optimizing * [AMS-Refactor] fix optimizing integration test (apache#1487) * fix optimizing integration test * fix optimizing integration test * fix optimizing integration test * fix optimizing integration test and fix runtime bug * fix optimizing integration test * change flink group property names * fix local optimizer memory error * [Refactor-AMS] Fix TableRuntime recreate (apache#1488) * resolve compile error. * resolve compile error. * add Executor unit test * refactor package level * abstracted test code to reduce duplication of code * fix some bugs and polish ut test * fix ut case * fix some bugs * fix some bugs * fix some bugs * fix some bugs * fix some bugs * fix some bugs * modify upgrade sql * fix table runtime recreate * add unique to derby init * resolve conflict * resolve conflict * resolve conflict * resolve conflict * resolve conflict * modify log * modify ServerTableIdentifier toString() * rename taskMap of OptimizingQueue to executingTaskMap(SCHEDULED,ACKED) * fix checkstyle * fix losing some task's metrics * should not beginCommitting serveral times * to avoid concurrent optimizing commit for a table * fix repeat commit * make shouldRewritePosForSegmentFile more readable * [AMS-Refactor] make `TableRuntime`'s fileds as `StateField` (apache#1495) * add StatedPersistentBase.StateField for fields of TableRuntime and tableHandler should not throw exception * replace @StatedPersistentBase.StateField with @StateField * [AMS-Refactor] persistence optimizing process failedReason (apache#1496) persistence optimizing process failedReason * add logs and comments * [AMS-Refactor] fix optimizing integration test (apache#1497) * persistence optimizing process failedReason * fix optimizing integration test * fix unit test for native iceberg plan * Fix Trino ci * fix init sql for table resource * modify log touching to DEBUG * remove thread_id/optimize_token from task_runtime of upgrade-0.4.1-to-0.5.0.sql * fix flink 1.12 test log error * fix flink 1.12 unit test compile error * fix flink 1.12 unit test log4j2 config loading error * polish code * fix loading log4j configuration file in flink 1.14/1.15 * fix pusalr-client-all dependency error * [AMS-Refactor] Remove optimizer table and controller switch to use global exception handle (apache#1502) controller switch to using global exception handling * add unit test case for TestOptimizingQueue * fix checkstyle * change flink unit test log configuration * fix flink 1.12 unit test dependency error * fix mock ams serve ut bug * ignore blocked ut case in flink module * disable TestAutomaticLogWriter in flink 1.12 * change Mock AMS Thrift server implementation * fix optimizer unit test error * fix mock ams complete task handle error * [Refactor-AMS] Fix quota calculation logic (apache#1520) * Fix quota calculation logic * [Refactor-AMS] Recover task through persistent optimizer information (apache#1519) Recover task through persistent optimizer information * Format code style * support config optimizer&task timeout --------- Co-authored-by: majin1102 <majin1102@163.com> Co-authored-by: wangzeyu <hameizi369@gmail.com> Co-authored-by: zhoujinsong <463763777@qq.com> Co-authored-by: huiyuan_ <819747197@qq.com> Co-authored-by: zhubeibei <zhubeibei@corp.netease.com> Co-authored-by: shidayang <530847445@qq.com> Co-authored-by: wangzeyu <1249369293@qq.com> Co-authored-by: HuangFru <68625618+HuangFru@users.noreply.github.com> Co-authored-by: wangtao <wangtao3@corp.netease.com> Co-authored-by: wangtaohz <103108928+wangtaohz@users.noreply.github.com> Co-authored-by: PlanetWalker <52364847+hellojinsilei@users.noreply.github.com> Co-authored-by: jinsilei <jinsilei@corp.netease.com> Co-authored-by: baiyangtx <xiangnebula@163.com> Co-authored-by: ZhouJinsong <zhoujinsong0505@163.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Why are the changes needed?
fix optimizing integration test
Brief change log
fix optimizing integration test
How was this patch tested?
Add some test cases that check the changes thoroughly including negative and positive cases if possible
Add screenshots for manual tests if appropriate
Run test locally before making a pull request
Documentation