Release-0.5.1 roadmap

[wangtaohz]

This issue will track the roadmap of Amoro 0.5.1

Feature Development

• [Feature]: Show summary details of snapshots in the table transaction tab #1827
• [Subtask]: Implement displaying metrics on the front end of AMS #1771
• support running ams in jdk17 environment. #1896
• [Feature]: Support table filters to make AMS ignore tables that are not needed #1870
• [Feature]: Support Kubernetes deployment for AMS #1335
• [Feature]: Support MOR with Flink Engine in SQL/Table API #1422
• [Feature]: Add PostgresSQL RDS as system database #1966
• [Subtask]: Support creating mixed-iceberg format tables in any catalog which support iceberg. #1336

Improvement

• [Improvement]: modify the parameter transfer method for flink optimize #1789
• [Subtask]: Remove IcebergContentFile and related codes #1839
• [Improvement]: Extract Spark Common Module to reuse code between different spark versions. #1449
• [Improvement]: Show correct resource occupation for external optimizer #1845
• [Improvement] Polish optimizing executor code #1895
• [Improvement]: Remove AmsClient in UnkeyedTable and KeyedTable  #1904
• [Improvement]: add mysql schema init #1899
• [Improvement]: Perform file filtering as early as possible when during optimizing plan process #1883
• [Improvement] Add bounds of timestamp type for mixed-hive table #1920
• [Improvement]: Flink Logstore supports 'group-offsets' and 'specific-offsets' Startup mode. #1931
• [Improvement]: Using the parameters in TableConfiguration instead of from Table properties #1943
• [Improvement]: Improve the triggering conditions for segment file rewriting #1953
• [Improvement]: Helm Chart Support Overwrite JVM Optional #1962
• [Subtask]: Do not cache DataFile in memory when evaluating tables. #1840
• [Improvement]: Introduce a table property to set the task splitting size of self-optimizing #1935
• doc: update dashboard readme #1984
• [Improvement]: Flink optimizer configure managed memory explicitly in both the doc and the startup script #1907
• [Improvement]: CI run check style first #1293
• [Improvement]: Add basic configration of spotless #1926
• [Improvement]: The version of the Amoro in the Docker startup script should be the latest by default #2010

Bug Tracking

• [Bug]: the historical metadata file has not been deleted #1864
• [Bug]: Segment file first binpack strategy leads to inability to optimize SplitTasks #1866
• [Bug]: The water mark is not displayed properly for tables without primary keys #1603
• [Bug]: Fix unit conversion for parameters -msz to spillmap configuration #1893
• [Bug]: Snapshot file not found after Flink ingestion job has been running for a while #1882
• [Bug]: BasicUnkeyedTable does not Override the name method #1914
• [Bug]: Can not set configuration when setting UGI #1912
• [Bug]: If the optimizing of a Mixed Hive table fails to alter the Hive location, the files will be moved to the old Hive location. #1898
• [Bug]: MAJOR Optimizing is running repeatedly #1924
• [Bug]: The optimizing task has been in a suspended state all along. #1939
• [Bug]: Failed to persist optimizing process records #1869
• [Bug]: Mix-iceberg format with hive metastore not working on spark #1963
• [Bug]: Mix-iceberg format with hive metastore not working on spark session catalog #1964
• [Bug]:When using an HDFS cluster protected by Kerberos, the fs.hdfs.impl.disable.cache=false configuration is invalid, FileSystem is unable to hit the cache  #1857
• [Bug]: When using s3a schema, orphan files could not be deleted. #1981
• [Bug]: Using mix iceberg in spark has some docs problems #1908
• [Bug]: Avoid NPE during disposing ArcticServiceContainer #1985
• [Bug]: Orphan files clean mistakenly deleted the Iceberg statistic file #1959
• [Bug]: If the iceberg table has a primary key of date type, in upsert mode, after Amoro merge, the primary key duplicate data can be found #1979
• [Bug]: If the projected schema of the Lookup Table Source doesn't include all primary keys and requires automatic addition #1891
• [Bug]: ams-optimizer FLINK job expired oom due to frequent requests for Kerberos. #1980
• [Bug]: Terminal is unable to create the iceberg table, when Kerberos authentication is enabled #1996
• [Bug] Trino failed to query Amoro Mixed Format Table after restarting AMS #1977
• [Bug]: The startup script has a judgment error #2006
• [Bug]: Spark SQL using amoro spark runtime error class cannot find ResolveProcedures #2032
• [Bug]: Dashboard can't show files page with kerberos and mixed-iceberg format in external catalog #2029
• [Bug]: When a time value that is too large or too small is written, it will encounter a 'long overflow' exception. #2044
• [Bug]: Ams Server always executes initSqlScript when using mysql8 #2049

Document Improvement