Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Bug]: Full compaction completed, cannot commit #1095

Closed
1 task done
zhongqishang opened this issue Feb 10, 2023 · 0 comments · Fixed by #1101
Closed
1 task done

[Bug]: Full compaction completed, cannot commit #1095

zhongqishang opened this issue Feb 10, 2023 · 0 comments · Fixed by #1101
Labels
type:bug Something isn't working

Comments

@zhongqishang
Copy link
Contributor

zhongqishang commented Feb 10, 2023 •
edited
Loading

What happened?

Native iceberg format table
Set configration write.metadata.metrics.default=full

Affects Versions

master

What engines are you seeing the problem on?

AMS

How to reproduce

No response

Relevant log output

2023-02-10 16:12:56,679 ERROR [Metastore Scheduled Commit Worker 4] [com.netease.arctic.table.TableMetaStore] [] - run with ugi request failed.
org.apache.iceberg.exceptions.ValidationException: Cannot commit, found new delete for replaced data file: com.netease.arctic.data.file.DataFileWithSequence@4af4abb4
        at org.apache.iceberg.exceptions.ValidationException.check(ValidationException.java:46) ~[iceberg-api-0.13.2.jar:?]
        at org.apache.iceberg.MergingSnapshotProducer.validateNoNewDeletesForDataFiles(MergingSnapshotProducer.java:371) ~[iceberg-core-0.13.2.jar:?]
        at org.apache.iceberg.MergingSnapshotProducer.validateNoNewDeletesForDataFiles(MergingSnapshotProducer.java:316) ~[iceberg-core-0.13.2.jar:?]
        at org.apache.iceberg.BaseRewriteFiles.validate(BaseRewriteFiles.java:108) ~[iceberg-core-0.13.2.jar:?]
        at org.apache.iceberg.SnapshotProducer.apply(SnapshotProducer.java:162) ~[iceberg-core-0.13.2.jar:?]
        at org.apache.iceberg.SnapshotProducer.lambda$commit$2(SnapshotProducer.java:283) ~[iceberg-core-0.13.2.jar:?]
        at org.apache.iceberg.util.Tasks$Builder.runTaskWithRetry(Tasks.java:404) ~[iceberg-core-0.13.2.jar:?]
        at org.apache.iceberg.util.Tasks$Builder.runSingleThreaded(Tasks.java:214) ~[iceberg-core-0.13.2.jar:?]
        at org.apache.iceberg.util.Tasks$Builder.run(Tasks.java:198) ~[iceberg-core-0.13.2.jar:?]
        at org.apache.iceberg.util.Tasks$Builder.run(Tasks.java:190) ~[iceberg-core-0.13.2.jar:?]
        at org.apache.iceberg.SnapshotProducer.commit(SnapshotProducer.java:282) ~[iceberg-core-0.13.2.jar:?]
        at com.netease.arctic.ams.server.optimize.IcebergOptimizeCommit.lambda$majorCommit$8(IcebergOptimizeCommit.java:236) ~[arctic-ams-server-0.5.0-SNAPSHOT.jar:?]
        at com.netease.arctic.table.TableMetaStore.doAsUgi(TableMetaStore.java:355) ~[arctic-core-0.5.0-SNAPSHOT.jar:?]
        at com.netease.arctic.table.TableMetaStore.lambda$doAs$0(TableMetaStore.java:335) ~[arctic-core-0.5.0-SNAPSHOT.jar:?]
        at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_141]
        at javax.security.auth.Subject.doAs(Subject.java:360) ~[?:1.8.0_141]
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1873) ~[hadoop-common-2.9.2.jar:?]
        at com.netease.arctic.table.TableMetaStore.doAs(TableMetaStore.java:335) ~[arctic-core-0.5.0-SNAPSHOT.jar:?]
        at com.netease.arctic.io.ArcticHadoopFileIO.doAs(ArcticHadoopFileIO.java:177) ~[arctic-core-0.5.0-SNAPSHOT.jar:?]
        at com.netease.arctic.ams.server.optimize.IcebergOptimizeCommit.majorCommit(IcebergOptimizeCommit.java:235) ~[arctic-ams-server-0.5.0-SNAPSHOT.jar:?]
        at com.netease.arctic.ams.server.optimize.IcebergOptimizeCommit.commit(IcebergOptimizeCommit.java:100) ~[arctic-ams-server-0.5.0-SNAPSHOT.jar:?]
        at com.netease.arctic.ams.server.optimize.TableOptimizeItem.commitOptimizeTasks(TableOptimizeItem.java:938) ~[arctic-ams-server-0.5.0-SNAPSHOT.jar:?]
        at com.netease.arctic.ams.server.optimize.OptimizeCommitWorker.run(OptimizeCommitWorker.java:49) ~[arctic-ams-server-0.5.0-SNAPSHOT.jar:?]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[?:1.8.0_141]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_141]
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) ~[?:1.8.0_141]
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) ~[?:1.8.0_141]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_141]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_141]
        at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_141]

Anything else

No response

Code of Conduct

  • I agree to follow this project's Code of Conduct
@zhongqishang zhongqishang added the type:bug Something isn't working label Feb 10, 2023
hzluting pushed a commit to hzluting/arctic that referenced this issue Feb 13, 2023
fix apache#1095
0387b34 
Adding the sequence number in the plan when the major commit for the native iceberg table
@hzluting hzluting mentioned this issue Feb 13, 2023
Merged
3 tasks
zhoujinsong pushed a commit that referenced this issue Feb 14, 2023
@hzluting
[ARCTIC-1095][AMS] Add the sequence number for the native iceberg tab…
7c520e0 
…le when the major optimizing commit (#1101)

* fix #1095
Adding the sequence number in the plan when the major commit for the native iceberg table

* fix code review

* add comment

---------

Co-authored-by: luting <dylzlt93299@gmail.com>
wangtaohz added a commit that referenced this issue Feb 15, 2023
@wangtaohz @hellojinsilei
@hameizi @shendanfengg @hzluting
[Hotfix] merge master into 0.4.x (#1116)
b2c401a 
* [ARCTIC-1062][AMS]Terminal support config spark properties in the local model (#1094)

* terminal support config spark properties in the local model

---------

Co-authored-by: jinsilei <jinsilei@corp.netease.com>

* [AMS][Improvement]: Support set login user and login password in config yaml file (#1086)

* Support set login user and login password in config yaml file

* [ARCTIC-1091] Browser tab does not display Arctic's icon (#1092)

fix-1091

Co-authored-by: shendanfeng01 <shendanfeng01@corp.netease.com>

* [ARCTIC-1090][AMS]:Terminal support add hadoop conf when use native iceberg (#1099)

* terminal supoort hadoop conf

---------

Co-authored-by: jinsilei <jinsilei@corp.netease.com>

* [ARCTIC-1093] Self-Optimizing scan files from metadata instead of from file info cache (#1100)

* fix-1093 optimize use TableScan

* modify OptimizeIntegrationTest for TestHiveSupport Table

* 1.remove checking any tasks running during the plan
2.remove checking table changed during the plan and check it before scan files
3.AbstractIcebergOptimizePlan use the correct currentSnapshot
4.import OptimizePlanResult to to encapsulate the plan result

* [hotfix] Lower the log level in ShuffleSplitAssigner (#1106)

* [ARCTIC-1095][AMS] Add the sequence number for the native iceberg table when the major optimizing commit (#1101)

* fix #1095
Adding the sequence number in the plan when the major commit for the native iceberg table


---------

Co-authored-by: luting <dylzlt93299@gmail.com>

* [ARCTIC-924][Hive] When AMS runs for a period of time and then cannot connect to HMS (#1054)

---------

Co-authored-by: shendanfeng01 <shendanfeng01@corp.netease.com>

---------

Co-authored-by: PlanetWalker <52364847+hellojinsilei@users.noreply.github.com>
Co-authored-by: jinsilei <jinsilei@corp.netease.com>
Co-authored-by: wangzeyu <hameizi369@gmail.com>
Co-authored-by: shendanfengg <109209550+shendanfengg@users.noreply.github.com>
Co-authored-by: shendanfeng01 <shendanfeng01@corp.netease.com>
Co-authored-by: Xianxun Ye <yxx_cmhd@163.com>
Co-authored-by: luting <1004611953@qq.com>
Co-authored-by: luting <dylzlt93299@gmail.com>
XBaith pushed a commit to XBaith/amoro that referenced this issue Feb 17, 2023
@hzluting @XBaith
[ARCTIC-1095][AMS] Add the sequence number for the native iceberg tab…
c99218f 
…le when the major optimizing commit (apache#1101)

* fix apache#1095
Adding the sequence number in the plan when the major commit for the native iceberg table

* fix code review

* add comment

---------

Co-authored-by: luting <dylzlt93299@gmail.com>
(cherry picked from commit 7c520e0)
zhoujinsong pushed a commit that referenced this issue May 31, 2023
@hzluting
[ARCTIC-1095][AMS] Add the sequence number for the native iceberg tab…
85429d1 
…le when the major optimizing commit (#1101)

* fix #1095
Adding the sequence number in the plan when the major commit for the native iceberg table

* fix code review

* add comment

---------

Co-authored-by: luting <dylzlt93299@gmail.com>
ShawHee pushed a commit to ShawHee/arctic that referenced this issue Dec 29, 2023
@hzluting @ShawHee
[ARCTIC-1095][AMS] Add the sequence number for the native iceberg tab…
1b6530b 
…le when the major optimizing commit (apache#1101)

* fix apache#1095
Adding the sequence number in the plan when the major commit for the native iceberg table

* fix code review

* add comment

---------

Co-authored-by: luting <dylzlt93299@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
type:bug Something isn't working
Projects
None yet
Milestone
No milestone
1 participant
@zhongqishang