[AMORO-1723] Support auto create tag daily for Iceberg Format #2263

wangtaohz · 2023-11-08T07:19:03Z

Why are the changes needed?

Close [Subtask]: Automatically create tags on snapshots daily for Iceberg Format Table #1723.

Brief change log

add a new TagsCheckingExecutor to AMS
add a new interface method autoCreateTags to TableMaintainer
add AutoCreateIcebergTagAction to create daily tags for the Iceberg table

How was this patch tested?

Add some test cases that check the changes thoroughly including negative and positive cases if possible
Add screenshots for manual tests if appropriate
Run test locally before making a pull request

Documentation

Does this pull request introduce a new feature? (yes / no)
If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented)
https://docs.google.com/document/d/1_56YHZO7XSkkZV7bTFQ7yWNVfiV0TknwOIMt0j8Me6Y/edit#heading=h.m88xjoz356hr

huyuanfeng2018

LGTM.

baiyangtx · 2023-11-20T02:29:09Z

core/src/main/java/com/netease/arctic/table/TableProperties.java

+  public static final String AUTO_CREATE_TAG_FORMAT = "tag.auto-create.daily.tag-format";
+  public static final String AUTO_CREATE_TAG_FORMAT_DEFAULT = "'tag-day-'yyyyMMdd";
+


tag-20220901 is fine. 20220901 has already mark the tag as a daily tag. so tag-day is not necessary

docs/user-guides/configurations.md

zhoujinsong

@wangtaohz I left some comments, please task another look.

Besides, I am confused about the usage of property tag.auto-create.trigger.max-delay.minutes. I have reread the design document, but I still couldn't find the use case for this property. Would you mind making some additions in the document to clarify it?

core/src/main/java/com/netease/arctic/table/TagTriggerPeriod.java

core/src/test/java/com/netease/arctic/scan/TestKeyedTableScan.java

...rc/main/java/com/netease/arctic/server/optimizing/maintainer/AutoCreateIcebergTagAction.java

ams/server/src/main/java/com/netease/arctic/server/table/executor/TagsAutoCreatingExecutor.java

CLAassistant · 2023-11-22T08:42:21Z

All committers have signed the CLA.

CLAassistant · 2023-11-22T08:42:28Z

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.
0 out of 3 committers have signed the CLA.

❌ zhoujinsong
❌ wangtaohz
❌ baiyangtx
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

wangtaohz · 2023-11-24T01:42:03Z

Besides, I am confused about the usage of property tag.auto-create.trigger.max-delay.minutes. I have reread the design document, but I still couldn't find the use case for this property. Would you mind making some additions in the document to clarify it?

I added some explanations in the document.
https://docs.google.com/document/d/1_56YHZO7XSkkZV7bTFQ7yWNVfiV0TknwOIMt0j8Me6Y/edit#heading=h.m88xjoz356hr

zhoujinsong

Hi, l left some small suggestion.

...rc/main/java/com/netease/arctic/server/optimizing/maintainer/AutoCreateIcebergTagAction.java

ams/server/src/main/java/com/netease/arctic/server/table/TableConfiguration.java

zhoujinsong

LGTM.

…#2263) * upgrade iceberg to 1.3.0 * fix flink * fix flink * remove useless * change version from 0.5.0-SNAPSHOT to 0.5.1-SNAPSHOT * update iceberg version 1.3.x in Flink 1.12 module * update iceberg version 1.3.x in Flink 1.12 module * ArcticUpdate support toBranch * fix ci error * update iceberg version 1.3.x in Flink 1.14 module * update iceberg version 1.3.x in Flink 1.15 module * add PuffinUtil and unit test * remove legacyPartitionMaxTransactionId before 0.4.1 * store optimized sequence to puffin * support overwrite puffin when retry * add table property table.version * calculate available core * change max input size per thread from 5GB to 500MB * add auto create tag properties * add PuffinUtil and unit test * remove legacyPartitionMaxTransactionId before 0.4.1 * store optimized sequence to puffin * support overwrite puffin when retry * add table property table.version * implement TagsCheckingExecutor and add unit test * fix unit test * fix unit test for hive * change version from 0.5.1-SNAPSHOT to 0.5.0-SNAPSHOT * change version from 0.5.0-SNAPSHOT to 0.5.1-SNAPSHOT * support keyed table scan use ref(tag/branch) * no need to get optimized sequence from KeyedTableSnapshot * change version back to 0.5.0-SNAPSHOT * fix get null sequence * fix sequence number = -1 * fix optimizing integration test for hive * create puffin for each snapshot * for compatibility, if puffin not exist, using table properties * remove useless table version * add readWithCompatibility * add some comments for PuffinUtil * expire statistics files * fix check style * fix compile error * remove useless deprecate * fix unit test * fix compile error and unit test * refactor PuffinUtil * fix compile error in ams server * spotless:apply core * fix unit test * support generic type for PartitionDataSerializer * rename PuffinUtil to StatisticsFileUtil * rename method to readFromStatisticsFile * spotless: apply * spotless: apply * spotless: apply * 1.fix comment 2.add writerBuilder for StatisticsFileUtil.Writer 3.store puffin files in the location of data/puffin/ 4.search snapshot based on a mark in snapsnot summary * fix snapshot expring unit test error * fix compile error * fix TableConfiguration equals hashcode * refactor code and only support create tag now * add unit test for TestAutoCreateIcebergTagAction * spotless:apply * remove the support for mixed format * Update ams/server/src/main/java/com/netease/arctic/server/optimizing/maintainer/AutoCreateIcebergTagAction.java Co-authored-by: baiyangtx <xiangnebula@163.com> * Update ams/server/src/main/java/com/netease/arctic/server/optimizing/maintainer/AutoCreateIcebergTagAction.java Co-authored-by: baiyangtx <xiangnebula@163.com> * revert useRef when scan * fix compile error * Update ams/server/src/main/java/com/netease/arctic/server/table/executor/TagsCheckingExecutor.java Co-authored-by: baiyangtx <xiangnebula@163.com> * fix comment * support more auto create configs * Update core/src/main/java/com/netease/arctic/table/TagTriggerPeriod.java Co-authored-by: big face cat <731030576@qq.com> * fix compile error * fix unit test * add auto create tag max delay * implement auto creating tag max delay * add docs * change daily tag format to 'tag-'yyyyMMdd * change docs * add docs for TagTriggerPeriod * revert test assert * refactor TagConfiguration * add docs * spotless:apply * add docs * improve logs and remove autoCreateTagEnabled from TableConfiguration * improve logs --------- Co-authored-by: lklhdu <lekeleihz@163.com> Co-authored-by: ZhouJinsong <zhoujinsong0505@163.com> Co-authored-by: baiyangtx <xiangnebula@163.com> Co-authored-by: big face cat <731030576@qq.com>

wangtaohz and others added 30 commits July 20, 2023 19:03

upgrade iceberg to 1.3.0

c05681c

fix flink

4549bee

fix flink

0deb5b0

remove useless

c2dce07

change version from 0.5.0-SNAPSHOT to 0.5.1-SNAPSHOT

2e531ab

Merge branch 'master' into fix-1668

00ee683

update iceberg version 1.3.x in Flink 1.12 module

d1c480f

update iceberg version 1.3.x in Flink 1.12 module

5f49e66

ArcticUpdate support toBranch

588909f

fix ci error

96a2194

update iceberg version 1.3.x in Flink 1.14 module

4ad8e2a

update iceberg version 1.3.x in Flink 1.15 module

58887a3

Merge branch 'master' into fix-1668

52d77b5

add PuffinUtil and unit test

c61cefb

remove legacyPartitionMaxTransactionId before 0.4.1

784c137

Merge branch 'master' into fix-1668

e64e5c9

store optimized sequence to puffin

6b9f696

support overwrite puffin when retry

5970a42

add table property table.version

dde1bd4

Merge branch 'master' into fix-1668

5f7cfd7

calculate available core

283078a

change max input size per thread from 5GB to 500MB

5a0e79b

Merge branch 'master' into fix-1668

ae72c95

add auto create tag properties

03bb592

Merge branch 'master' into fix-1668

bb2bc83

add PuffinUtil and unit test

bddf29b

remove legacyPartitionMaxTransactionId before 0.4.1

c5f0ede

store optimized sequence to puffin

6f4f2bc

support overwrite puffin when retry

1ef344d

add table property table.version

bb8ac9d

add docs

f27b849

github-actions bot added the type:docs Improvements or additions to documentation label Nov 13, 2023

wangtaohz requested review from zhoujinsong and huyuanfeng2018 November 13, 2023 07:06

huyuanfeng2018 approved these changes Nov 13, 2023

View reviewed changes

wangtaohz added 2 commits November 13, 2023 19:00

Merge branch 'master' into fix-1723

43c0f94

Merge branch 'master' into fix-1723

3d7318f

baiyangtx reviewed Nov 20, 2023

View reviewed changes

wangtaohz added 3 commits November 20, 2023 18:17

Merge branch 'master' into fix-1723

ffdec24

change daily tag format to 'tag-'yyyyMMdd

63d5df7

change docs

e61e411

zhoujinsong reviewed Nov 20, 2023

View reviewed changes

Merge branch 'master' into fix-1723

4d348da

wangtaohz added 6 commits November 22, 2023 16:47

add docs for TagTriggerPeriod

8d75419

revert test assert

83a2c37

refactor TagConfiguration

ec333ed

add docs

ea9ca7d

spotless:apply

8d2e8df

add docs

b278625

zhoujinsong reviewed Nov 27, 2023

View reviewed changes

wangtaohz and others added 4 commits November 27, 2023 11:26

Merge branch 'master' into fix-1723

e7321bf

improve logs and remove autoCreateTagEnabled from TableConfiguration

fd56f6b

Merge branch 'master' into fix-1723

5dbd449

improve logs

e66fac0

zhoujinsong approved these changes Nov 27, 2023

View reviewed changes

zhoujinsong merged commit a0b98b7 into apache:master Nov 27, 2023
7 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[AMORO-1723] Support auto create tag daily for Iceberg Format #2263

[AMORO-1723] Support auto create tag daily for Iceberg Format #2263

wangtaohz commented Nov 8, 2023 •

edited

Loading

huyuanfeng2018 left a comment

baiyangtx Nov 20, 2023

zhoujinsong left a comment

CLAassistant commented Nov 22, 2023 •

edited

Loading

CLAassistant commented Nov 22, 2023

wangtaohz commented Nov 24, 2023

zhoujinsong left a comment

zhoujinsong left a comment

		public static final String AUTO_CREATE_TAG_FORMAT = "tag.auto-create.daily.tag-format";
		public static final String AUTO_CREATE_TAG_FORMAT_DEFAULT = "'tag-day-'yyyyMMdd";

[AMORO-1723] Support auto create tag daily for Iceberg Format #2263

[AMORO-1723] Support auto create tag daily for Iceberg Format #2263

Conversation

wangtaohz commented Nov 8, 2023 • edited Loading

Why are the changes needed?

Brief change log

How was this patch tested?

Documentation

huyuanfeng2018 left a comment

Choose a reason for hiding this comment

baiyangtx Nov 20, 2023

Choose a reason for hiding this comment

zhoujinsong left a comment

Choose a reason for hiding this comment

CLAassistant commented Nov 22, 2023 • edited Loading

CLAassistant commented Nov 22, 2023

wangtaohz commented Nov 24, 2023

zhoujinsong left a comment

Choose a reason for hiding this comment

zhoujinsong left a comment

Choose a reason for hiding this comment

wangtaohz commented Nov 8, 2023 •

edited

Loading

CLAassistant commented Nov 22, 2023 •

edited

Loading