Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Bug]: Snapshot file not found after Flink ingestion job has been running for a while #1882

Closed
1 task done
Tracked by #1930
nicochen opened this issue Aug 23, 2023 · 2 comments · Fixed by #1915
Closed
1 task done
Tracked by #1930

[Bug]: Snapshot file not found after Flink ingestion job has been running for a while #1882

nicochen opened this issue Aug 23, 2023 · 2 comments · Fixed by #1915
Assignees
@YesOrNo828
Labels
type:bug Something isn't working

Comments

@nicochen
Copy link
Contributor

What happened?

Occasional null pointer and snapshot not found exceptions after Flink ingestion for a period of time.
Details of the exception can be found in the following log file.

Affects Versions

0.4.x

What engines are you seeing the problem on?

Core, Flink

How to reproduce

It's not clear how it will be reproduced

Relevant log output

2023-08-17 08:45:25,780 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - FilesCommitter -> Sink: ArcticSink xxxxxx (1/1) (83c0deda9cd3443a7212537f521536f3) switched from RUNNING to FAILED on container_e06_1690009450264_0175_01_000003 @ xxxxxx (dataPort=45415).
java.lang.reflect.InvocationTargetException: null
	at sun.reflect.GeneratedMethodAccessor52.invoke(Unknown Source) ~[?:?]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_181]
	at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_181]
	at com.netease.arctic.flink.interceptor.KerberosInvocationHandler.lambda$invoke$0(KerberosInvocationHandler.java:58) ~[plugin_ne-flink-1.12.4-1.1.5-xxxxxx-SNAPSHOT_scala2.12_hive2.1.1-release-3.8.3-1.3.4-SNAPSHOT_0713_1.0.0.jar:]
	at com.netease.arctic.table.TableMetaStore.doAsUgi(TableMetaStore.java:365) ~[plugin_ne-flink-1.12.4-1.1.5-xxxxxx-SNAPSHOT_scala2.12_hive2.1.1-release-3.8.3-1.3.4-SNAPSHOT_0713_1.0.0.jar:]
	at com.netease.arctic.table.TableMetaStore.lambda$doAs$0(TableMetaStore.java:345) ~[plugin_ne-flink-1.12.4-1.1.5-xxxxxx-SNAPSHOT_scala2.12_hive2.1.1-release-3.8.3-1.3.4-SNAPSHOT_0713_1.0.0.jar:]
	at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_181]
	at javax.security.auth.Subject.doAs(Subject.java:360) ~[?:1.8.0_181]
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1734) ~[flink-shaded-hadoop-2-uber-2.7.5-10.0.jar:2.7.5-10.0]
	at com.netease.arctic.table.TableMetaStore.doAs(TableMetaStore.java:345) ~[plugin_ne-flink-1.12.4-1.1.5-xxxxxx-SNAPSHOT_scala2.12_hive2.1.1-release-3.8.3-1.3.4-SNAPSHOT_0713_1.0.0.jar:]
	at com.netease.arctic.io.ArcticHadoopFileIO.doAs(ArcticHadoopFileIO.java:184) ~[plugin_ne-flink-1.12.4-1.1.5-xxxxxx-SNAPSHOT_scala2.12_hive2.1.1-release-3.8.3-1.3.4-SNAPSHOT_0713_1.0.0.jar:]
	at com.netease.arctic.flink.interceptor.KerberosInvocationHandler.invoke(KerberosInvocationHandler.java:55) ~[plugin_ne-flink-1.12.4-1.1.5-xxxxxx-SNAPSHOT_scala2.12_hive2.1.1-release-3.8.3-1.3.4-SNAPSHOT_0713_1.0.0.jar:]
	at com.sun.proxy.$Proxy39.notifyCheckpointComplete(Unknown Source) ~[?:?]
	at org.apache.flink.streaming.runtime.tasks.StreamOperatorWrapper.notifyCheckpointComplete(StreamOperatorWrapper.java:99) ~[plugin_ne-flink-1.12.4-1.1.5-xxxxxx-SNAPSHOT_scala2.12_hive2.1.1-release-3.8.3-1.3.4-SNAPSHOT_0713_1.0.0.jar:]
	at org.apache.flink.streaming.runtime.tasks.SubtaskCheckpointCoordinatorImpl.notifyCheckpointComplete(SubtaskCheckpointCoordinatorImpl.java:319) ~[plugin_ne-flink-1.12.4-1.1.5-xxxxxx-SNAPSHOT_scala2.12_hive2.1.1-release-3.8.3-1.3.4-SNAPSHOT_0713_1.0.0.jar:]
	at org.apache.flink.streaming.runtime.tasks.StreamTask.notifyCheckpointComplete(StreamTask.java:1089) ~[plugin_ne-flink-1.12.4-1.1.5-xxxxxx-SNAPSHOT_scala2.12_hive2.1.1-release-3.8.3-1.3.4-SNAPSHOT_0713_1.0.0.jar:]
	at org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$notifyCheckpointCompleteAsync$11(StreamTask.java:1054) ~[plugin_ne-flink-1.12.4-1.1.5-xxxxxx-SNAPSHOT_scala2.12_hive2.1.1-release-3.8.3-1.3.4-SNAPSHOT_0713_1.0.0.jar:]
	at org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$notifyCheckpointOperation$13(StreamTask.java:1077) ~[plugin_ne-flink-1.12.4-1.1.5-xxxxxx-SNAPSHOT_scala2.12_hive2.1.1-release-3.8.3-1.3.4-SNAPSHOT_0713_1.0.0.jar:]
	at org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$1.runThrowing(StreamTaskActionExecutor.java:50) ~[plugin_ne-flink-1.12.4-1.1.5-xxxxxx-SNAPSHOT_scala2.12_hive2.1.1-release-3.8.3-1.3.4-SNAPSHOT_0713_1.0.0.jar:]
	at org.apache.flink.streaming.runtime.tasks.mailbox.Mail.run(Mail.java:90) ~[plugin_ne-flink-1.12.4-1.1.5-xxxxxx-SNAPSHOT_scala2.12_hive2.1.1-release-3.8.3-1.3.4-SNAPSHOT_0713_1.0.0.jar:]
	at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.processMail(MailboxProcessor.java:317) ~[plugin_ne-flink-1.12.4-1.1.5-xxxxxx-SNAPSHOT_scala2.12_hive2.1.1-release-3.8.3-1.3.4-SNAPSHOT_0713_1.0.0.jar:]
	at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:189) ~[plugin_ne-flink-1.12.4-1.1.5-xxxxxx-SNAPSHOT_scala2.12_hive2.1.1-release-3.8.3-1.3.4-SNAPSHOT_0713_1.0.0.jar:]
	at org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:619) ~[plugin_ne-flink-1.12.4-1.1.5-xxxxxx-SNAPSHOT_scala2.12_hive2.1.1-release-3.8.3-1.3.4-SNAPSHOT_0713_1.0.0.jar:]
	at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:583) ~[plugin_ne-flink-1.12.4-1.1.5-xxxxxx-SNAPSHOT_scala2.12_hive2.1.1-release-3.8.3-1.3.4-SNAPSHOT_0713_1.0.0.jar:]
	at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:758) ~[plugin_ne-flink-1.12.4-1.1.5-xxxxxx-SNAPSHOT_scala2.12_hive2.1.1-release-3.8.3-1.3.4-SNAPSHOT_0713_1.0.0.jar:]
	at org.apache.flink.runtime.taskmanager.Task.run(Task.java:573) ~[plugin_ne-flink-1.12.4-1.1.5-xxxxxx-SNAPSHOT_scala2.12_hive2.1.1-release-3.8.3-1.3.4-SNAPSHOT_0713_1.0.0.jar:]
	at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_181]
Caused by: java.lang.reflect.InvocationTargetException
	at sun.reflect.GeneratedMethodAccessor52.invoke(Unknown Source) ~[?:?]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_181]
	at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_181]
	at com.netease.arctic.flink.interceptor.KerberosInvocationHandler.lambda$invoke$0(KerberosInvocationHandler.java:58) ~[plugin_ne-flink-1.12.4-1.1.5-xxxxxx-SNAPSHOT_scala2.12_hive2.1.1-release-3.8.3-1.3.4-SNAPSHOT_0713_1.0.0.jar:]
	at com.netease.arctic.table.TableMetaStore.doAsUgi(TableMetaStore.java:365) ~[plugin_ne-flink-1.12.4-1.1.5-xxxxxx-SNAPSHOT_scala2.12_hive2.1.1-release-3.8.3-1.3.4-SNAPSHOT_0713_1.0.0.jar:]
	at com.netease.arctic.table.TableMetaStore.lambda$doAs$0(TableMetaStore.java:345) ~[plugin_ne-flink-1.12.4-1.1.5-xxxxxx-SNAPSHOT_scala2.12_hive2.1.1-release-3.8.3-1.3.4-SNAPSHOT_0713_1.0.0.jar:]
	at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_181]
	at javax.security.auth.Subject.doAs(Subject.java:360) ~[?:1.8.0_181]
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1734) ~[flink-shaded-hadoop-2-uber-2.7.5-10.0.jar:2.7.5-10.0]
	at com.netease.arctic.table.TableMetaStore.doAs(TableMetaStore.java:345) ~[plugin_ne-flink-1.12.4-1.1.5-xxxxxx-SNAPSHOT_scala2.12_hive2.1.1-release-3.8.3-1.3.4-SNAPSHOT_0713_1.0.0.jar:]
	at com.netease.arctic.io.ArcticHadoopFileIO.doAs(ArcticHadoopFileIO.java:184) ~[plugin_ne-flink-1.12.4-1.1.5-xxxxxx-SNAPSHOT_scala2.12_hive2.1.1-release-3.8.3-1.3.4-SNAPSHOT_0713_1.0.0.jar:]
	at com.netease.arctic.flink.interceptor.KerberosInvocationHandler.invoke(KerberosInvocationHandler.java:55) ~[plugin_ne-flink-1.12.4-1.1.5-xxxxxx-SNAPSHOT_scala2.12_hive2.1.1-release-3.8.3-1.3.4-SNAPSHOT_0713_1.0.0.jar:]
	at com.sun.proxy.$Proxy39.notifyCheckpointComplete(Unknown Source) ~[?:?]
	... 27 more
Caused by: java.lang.NullPointerException
	at com.netease.arctic.shaded.org.apache.iceberg.util.LockManagers$InMemoryLockManager.release(LockManagers.java:225) ~[plugin_ne-flink-1.12.4-1.1.5-xxxxxx-SNAPSHOT_scala2.12_hive2.1.1-release-3.8.3-1.3.4-SNAPSHOT_0713_1.0.0.jar:]
	at com.netease.arctic.shaded.org.apache.iceberg.hadoop.HadoopTableOperations.renameToFinal(HadoopTableOperations.java:374) ~[plugin_ne-flink-1.12.4-1.1.5-xxxxxx-SNAPSHOT_scala2.12_hive2.1.1-release-3.8.3-1.3.4-SNAPSHOT_0713_1.0.0.jar:]
	at com.netease.arctic.shaded.org.apache.iceberg.hadoop.HadoopTableOperations.commit(HadoopTableOperations.java:158) ~[plugin_ne-flink-1.12.4-1.1.5-xxxxxx-SNAPSHOT_scala2.12_hive2.1.1-release-3.8.3-1.3.4-SNAPSHOT_0713_1.0.0.jar:]
	at com.netease.arctic.op.ArcticHadoopTableOperations.lambda$commit$1(ArcticHadoopTableOperations.java:48) ~[plugin_ne-flink-1.12.4-1.1.5-xxxxxx-SNAPSHOT_scala2.12_hive2.1.1-release-3.8.3-1.3.4-SNAPSHOT_0713_1.0.0.jar:]
	at com.netease.arctic.table.TableMetaStore.doAsUgi(TableMetaStore.java:365) ~[plugin_ne-flink-1.12.4-1.1.5-xxxxxx-SNAPSHOT_scala2.12_hive2.1.1-release-3.8.3-1.3.4-SNAPSHOT_0713_1.0.0.jar:]
	at com.netease.arctic.table.TableMetaStore.lambda$doAs$0(TableMetaStore.java:345) ~[plugin_ne-flink-1.12.4-1.1.5-xxxxxx-SNAPSHOT_scala2.12_hive2.1.1-release-3.8.3-1.3.4-SNAPSHOT_0713_1.0.0.jar:]
	at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_181]
	at javax.security.auth.Subject.doAs(Subject.java:360) ~[?:1.8.0_181]
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1734) ~[flink-shaded-hadoop-2-uber-2.7.5-10.0.jar:2.7.5-10.0]
	at com.netease.arctic.table.TableMetaStore.doAs(TableMetaStore.java:345) ~[plugin_ne-flink-1.12.4-1.1.5-xxxxxx-SNAPSHOT_scala2.12_hive2.1.1-release-3.8.3-1.3.4-SNAPSHOT_0713_1.0.0.jar:]
	at com.netease.arctic.io.ArcticHadoopFileIO.doAs(ArcticHadoopFileIO.java:184) ~[plugin_ne-flink-1.12.4-1.1.5-xxxxxx-SNAPSHOT_scala2.12_hive2.1.1-release-3.8.3-1.3.4-SNAPSHOT_0713_1.0.0.jar:]
	at com.netease.arctic.op.ArcticHadoopTableOperations.commit(ArcticHadoopTableOperations.java:46) ~[plugin_ne-flink-1.12.4-1.1.5-xxxxxx-SNAPSHOT_scala2.12_hive2.1.1-release-3.8.3-1.3.4-SNAPSHOT_0713_1.0.0.jar:]
	at com.netease.arctic.shaded.org.apache.iceberg.BaseTransaction.lambda$commitSimpleTransaction$5(BaseTransaction.java:356) ~[plugin_ne-flink-1.12.4-1.1.5-xxxxxx-SNAPSHOT_scala2.12_hive2.1.1-release-3.8.3-1.3.4-SNAPSHOT_0713_1.0.0.jar:]
	at com.netease.arctic.shaded.org.apache.iceberg.util.Tasks$Builder.runTaskWithRetry(Tasks.java:404) ~[plugin_ne-flink-1.12.4-1.1.5-xxxxxx-SNAPSHOT_scala2.12_hive2.1.1-release-3.8.3-1.3.4-SNAPSHOT_0713_1.0.0.jar:]
	at com.netease.arctic.shaded.org.apache.iceberg.util.Tasks$Builder.runSingleThreaded(Tasks.java:214) ~[plugin_ne-flink-1.12.4-1.1.5-xxxxxx-SNAPSHOT_scala2.12_hive2.1.1-release-3.8.3-1.3.4-SNAPSHOT_0713_1.0.0.jar:]
	at com.netease.arctic.shaded.org.apache.iceberg.util.Tasks$Builder.run(Tasks.java:198) ~[plugin_ne-flink-1.12.4-1.1.5-xxxxxx-SNAPSHOT_scala2.12_hive2.1.1-release-3.8.3-1.3.4-SNAPSHOT_0713_1.0.0.jar:]
	at com.netease.arctic.shaded.org.apache.iceberg.util.Tasks$Builder.run(Tasks.java:190) ~[plugin_ne-flink-1.12.4-1.1.5-xxxxxx-SNAPSHOT_scala2.12_hive2.1.1-release-3.8.3-1.3.4-SNAPSHOT_0713_1.0.0.jar:]
	at com.netease.arctic.shaded.org.apache.iceberg.BaseTransaction.commitSimpleTransaction(BaseTransaction.java:348) ~[plugin_ne-flink-1.12.4-1.1.5-xxxxxx-SNAPSHOT_scala2.12_hive2.1.1-release-3.8.3-1.3.4-SNAPSHOT_0713_1.0.0.jar:]
	at com.netease.arctic.shaded.org.apache.iceberg.BaseTransaction.commitTransaction(BaseTransaction.java:238) ~[plugin_ne-flink-1.12.4-1.1.5-xxxxxx-SNAPSHOT_scala2.12_hive2.1.1-release-3.8.3-1.3.4-SNAPSHOT_0713_1.0.0.jar:]
	at com.netease.arctic.op.ArcticUpdate.commit(ArcticUpdate.java:137) ~[plugin_ne-flink-1.12.4-1.1.5-xxxxxx-SNAPSHOT_scala2.12_hive2.1.1-release-3.8.3-1.3.4-SNAPSHOT_0713_1.0.0.jar:]
	at com.netease.arctic.shaded.org.apache.iceberg.flink.sink.IcebergFilesCommitter.commitOperation(IcebergFilesCommitter.java:312) ~[plugin_ne-flink-1.12.4-1.1.5-xxxxxx-SNAPSHOT_scala2.12_hive2.1.1-release-3.8.3-1.3.4-SNAPSHOT_0713_1.0.0.jar:]
	at com.netease.arctic.shaded.org.apache.iceberg.flink.sink.IcebergFilesCommitter.commitDeltaTxn(IcebergFilesCommitter.java:276) ~[plugin_ne-flink-1.12.4-1.1.5-xxxxxx-SNAPSHOT_scala2.12_hive2.1.1-release-3.8.3-1.3.4-SNAPSHOT_0713_1.0.0.jar:]
	at com.netease.arctic.shaded.org.apache.iceberg.flink.sink.IcebergFilesCommitter.commitUpToCheckpoint(IcebergFilesCommitter.java:218) ~[plugin_ne-flink-1.12.4-1.1.5-xxxxxx-SNAPSHOT_scala2.12_hive2.1.1-release-3.8.3-1.3.4-SNAPSHOT_0713_1.0.0.jar:]
	at com.netease.arctic.shaded.org.apache.iceberg.flink.sink.IcebergFilesCommitter.notifyCheckpointComplete(IcebergFilesCommitter.java:188) ~[plugin_ne-flink-1.12.4-1.1.5-xxxxxx-SNAPSHOT_scala2.12_hive2.1.1-release-3.8.3-1.3.4-SNAPSHOT_0713_1.0.0.jar:]
	at sun.reflect.GeneratedMethodAccessor52.invoke(Unknown Source) ~[?:?]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_181]
	at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_181]
	at com.netease.arctic.flink.interceptor.KerberosInvocationHandler.lambda$invoke$0(KerberosInvocationHandler.java:58) ~[plugin_ne-flink-1.12.4-1.1.5-xxxxxx-SNAPSHOT_scala2.12_hive2.1.1-release-3.8.3-1.3.4-SNAPSHOT_0713_1.0.0.jar:]
	at com.netease.arctic.table.TableMetaStore.doAsUgi(TableMetaStore.java:365) ~[plugin_ne-flink-1.12.4-1.1.5-xxxxxx-SNAPSHOT_scala2.12_hive2.1.1-release-3.8.3-1.3.4-SNAPSHOT_0713_1.0.0.jar:]
	at com.netease.arctic.table.TableMetaStore.lambda$doAs$0(TableMetaStore.java:345) ~[plugin_ne-flink-1.12.4-1.1.5-xxxxxx-SNAPSHOT_scala2.12_hive2.1.1-release-3.8.3-1.3.4-SNAPSHOT_0713_1.0.0.jar:]
	at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_181]
	at javax.security.auth.Subject.doAs(Subject.java:360) ~[?:1.8.0_181]
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1734) ~[flink-shaded-hadoop-2-uber-2.7.5-10.0.jar:2.7.5-10.0]
	at com.netease.arctic.table.TableMetaStore.doAs(TableMetaStore.java:345) ~[plugin_ne-flink-1.12.4-1.1.5-xxxxxx-SNAPSHOT_scala2.12_hive2.1.1-release-3.8.3-1.3.4-SNAPSHOT_0713_1.0.0.jar:]
	at com.netease.arctic.io.ArcticHadoopFileIO.doAs(ArcticHadoopFileIO.java:184) ~[plugin_ne-flink-1.12.4-1.1.5-xxxxxx-SNAPSHOT_scala2.12_hive2.1.1-release-3.8.3-1.3.4-SNAPSHOT_0713_1.0.0.jar:]
	at com.netease.arctic.flink.interceptor.KerberosInvocationHandler.invoke(KerberosInvocationHandler.java:55) ~[plugin_ne-flink-1.12.4-1.1.5-xxxxxx-SNAPSHOT_scala2.12_hive2.1.1-release-3.8.3-1.3.4-SNAPSHOT_0713_1.0.0.jar:]
	at com.sun.proxy.$Proxy39.notifyCheckpointComplete(Unknown Source) ~[?:?]
	... 27 more
2023-08-17 08:50:08,340 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - FilesCommitter -> Sink: ArcticSink xxxxxx (1/1) (0652a55221f24af52a92767f557d5abe) switched from RUNNING to FAILED on container_e06_1690009450264_0175_01_000006 @ xxxxxx (dataPort=46629).
java.lang.reflect.InvocationTargetException: null
	at sun.reflect.GeneratedMethodAccessor51.invoke(Unknown Source) ~[?:?]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_181]
	at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_181]
	at com.netease.arctic.flink.interceptor.KerberosInvocationHandler.lambda$invoke$0(KerberosInvocationHandler.java:58) ~[plugin_ne-flink-1.12.4-1.1.5-xxxxxx-SNAPSHOT_scala2.12_hive2.1.1-release-3.8.3-1.3.4-SNAPSHOT_0713_1.0.0.jar:]
	at com.netease.arctic.table.TableMetaStore.doAsUgi(TableMetaStore.java:365) ~[plugin_ne-flink-1.12.4-1.1.5-xxxxxx-SNAPSHOT_scala2.12_hive2.1.1-release-3.8.3-1.3.4-SNAPSHOT_0713_1.0.0.jar:]
	at com.netease.arctic.table.TableMetaStore.lambda$doAs$0(TableMetaStore.java:345) ~[plugin_ne-flink-1.12.4-1.1.5-xxxxxx-SNAPSHOT_scala2.12_hive2.1.1-release-3.8.3-1.3.4-SNAPSHOT_0713_1.0.0.jar:]
	at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_181]
	at javax.security.auth.Subject.doAs(Subject.java:360) ~[?:1.8.0_181]
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1734) ~[flink-shaded-hadoop-2-uber-2.7.5-10.0.jar:2.7.5-10.0]
	at com.netease.arctic.table.TableMetaStore.doAs(TableMetaStore.java:345) ~[plugin_ne-flink-1.12.4-1.1.5-xxxxxx-SNAPSHOT_scala2.12_hive2.1.1-release-3.8.3-1.3.4-SNAPSHOT_0713_1.0.0.jar:]
	at com.netease.arctic.io.ArcticHadoopFileIO.doAs(ArcticHadoopFileIO.java:184) ~[plugin_ne-flink-1.12.4-1.1.5-xxxxxx-SNAPSHOT_scala2.12_hive2.1.1-release-3.8.3-1.3.4-SNAPSHOT_0713_1.0.0.jar:]
	at com.netease.arctic.flink.interceptor.KerberosInvocationHandler.invoke(KerberosInvocationHandler.java:55) ~[plugin_ne-flink-1.12.4-1.1.5-xxxxxx-SNAPSHOT_scala2.12_hive2.1.1-release-3.8.3-1.3.4-SNAPSHOT_0713_1.0.0.jar:]
	at com.sun.proxy.$Proxy39.notifyCheckpointComplete(Unknown Source) ~[?:?]
	at org.apache.flink.streaming.runtime.tasks.StreamOperatorWrapper.notifyCheckpointComplete(StreamOperatorWrapper.java:99) ~[plugin_ne-flink-1.12.4-1.1.5-xxxxxx-SNAPSHOT_scala2.12_hive2.1.1-release-3.8.3-1.3.4-SNAPSHOT_0713_1.0.0.jar:]
	at org.apache.flink.streaming.runtime.tasks.SubtaskCheckpointCoordinatorImpl.notifyCheckpointComplete(SubtaskCheckpointCoordinatorImpl.java:319) ~[plugin_ne-flink-1.12.4-1.1.5-xxxxxx-SNAPSHOT_scala2.12_hive2.1.1-release-3.8.3-1.3.4-SNAPSHOT_0713_1.0.0.jar:]
	at org.apache.flink.streaming.runtime.tasks.StreamTask.notifyCheckpointComplete(StreamTask.java:1089) ~[plugin_ne-flink-1.12.4-1.1.5-xxxxxx-SNAPSHOT_scala2.12_hive2.1.1-release-3.8.3-1.3.4-SNAPSHOT_0713_1.0.0.jar:]
	at org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$notifyCheckpointCompleteAsync$11(StreamTask.java:1054) ~[plugin_ne-flink-1.12.4-1.1.5-xxxxxx-SNAPSHOT_scala2.12_hive2.1.1-release-3.8.3-1.3.4-SNAPSHOT_0713_1.0.0.jar:]
	at org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$notifyCheckpointOperation$13(StreamTask.java:1077) ~[plugin_ne-flink-1.12.4-1.1.5-xxxxxx-SNAPSHOT_scala2.12_hive2.1.1-release-3.8.3-1.3.4-SNAPSHOT_0713_1.0.0.jar:]
	at org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$1.runThrowing(StreamTaskActionExecutor.java:50) ~[plugin_ne-flink-1.12.4-1.1.5-xxxxxx-SNAPSHOT_scala2.12_hive2.1.1-release-3.8.3-1.3.4-SNAPSHOT_0713_1.0.0.jar:]
	at org.apache.flink.streaming.runtime.tasks.mailbox.Mail.run(Mail.java:90) ~[plugin_ne-flink-1.12.4-1.1.5-xxxxxx-SNAPSHOT_scala2.12_hive2.1.1-release-3.8.3-1.3.4-SNAPSHOT_0713_1.0.0.jar:]
	at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.processMail(MailboxProcessor.java:317) ~[plugin_ne-flink-1.12.4-1.1.5-xxxxxx-SNAPSHOT_scala2.12_hive2.1.1-release-3.8.3-1.3.4-SNAPSHOT_0713_1.0.0.jar:]
	at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:189) ~[plugin_ne-flink-1.12.4-1.1.5-xxxxxx-SNAPSHOT_scala2.12_hive2.1.1-release-3.8.3-1.3.4-SNAPSHOT_0713_1.0.0.jar:]
	at org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:619) ~[plugin_ne-flink-1.12.4-1.1.5-xxxxxx-SNAPSHOT_scala2.12_hive2.1.1-release-3.8.3-1.3.4-SNAPSHOT_0713_1.0.0.jar:]
	at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:583) ~[plugin_ne-flink-1.12.4-1.1.5-xxxxxx-SNAPSHOT_scala2.12_hive2.1.1-release-3.8.3-1.3.4-SNAPSHOT_0713_1.0.0.jar:]
	at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:758) ~[plugin_ne-flink-1.12.4-1.1.5-xxxxxx-SNAPSHOT_scala2.12_hive2.1.1-release-3.8.3-1.3.4-SNAPSHOT_0713_1.0.0.jar:]
	at org.apache.flink.runtime.taskmanager.Task.run(Task.java:573) ~[plugin_ne-flink-1.12.4-1.1.5-xxxxxx-SNAPSHOT_scala2.12_hive2.1.1-release-3.8.3-1.3.4-SNAPSHOT_0713_1.0.0.jar:]
	at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_181]
Caused by: java.lang.reflect.InvocationTargetException
	at sun.reflect.GeneratedMethodAccessor51.invoke(Unknown Source) ~[?:?]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_181]
	at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_181]
	at com.netease.arctic.flink.interceptor.KerberosInvocationHandler.lambda$invoke$0(KerberosInvocationHandler.java:58) ~[plugin_ne-flink-1.12.4-1.1.5-xxxxxx-SNAPSHOT_scala2.12_hive2.1.1-release-3.8.3-1.3.4-SNAPSHOT_0713_1.0.0.jar:]
	at com.netease.arctic.table.TableMetaStore.doAsUgi(TableMetaStore.java:365) ~[plugin_ne-flink-1.12.4-1.1.5-xxxxxx-SNAPSHOT_scala2.12_hive2.1.1-release-3.8.3-1.3.4-SNAPSHOT_0713_1.0.0.jar:]
	at com.netease.arctic.table.TableMetaStore.lambda$doAs$0(TableMetaStore.java:345) ~[plugin_ne-flink-1.12.4-1.1.5-xxxxxx-SNAPSHOT_scala2.12_hive2.1.1-release-3.8.3-1.3.4-SNAPSHOT_0713_1.0.0.jar:]
	at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_181]
	at javax.security.auth.Subject.doAs(Subject.java:360) ~[?:1.8.0_181]
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1734) ~[flink-shaded-hadoop-2-uber-2.7.5-10.0.jar:2.7.5-10.0]
	at com.netease.arctic.table.TableMetaStore.doAs(TableMetaStore.java:345) ~[plugin_ne-flink-1.12.4-1.1.5-xxxxxx-SNAPSHOT_scala2.12_hive2.1.1-release-3.8.3-1.3.4-SNAPSHOT_0713_1.0.0.jar:]
	at com.netease.arctic.io.ArcticHadoopFileIO.doAs(ArcticHadoopFileIO.java:184) ~[plugin_ne-flink-1.12.4-1.1.5-xxxxxx-SNAPSHOT_scala2.12_hive2.1.1-release-3.8.3-1.3.4-SNAPSHOT_0713_1.0.0.jar:]
	at com.netease.arctic.flink.interceptor.KerberosInvocationHandler.invoke(KerberosInvocationHandler.java:55) ~[plugin_ne-flink-1.12.4-1.1.5-xxxxxx-SNAPSHOT_scala2.12_hive2.1.1-release-3.8.3-1.3.4-SNAPSHOT_0713_1.0.0.jar:]
	at com.sun.proxy.$Proxy39.notifyCheckpointComplete(Unknown Source) ~[?:?]
	... 27 more
Caused by: com.netease.arctic.shaded.org.apache.iceberg.exceptions.NotFoundException: Failed to open input stream for file: hdfs://xxxxx/user/hive/warehouse/xxxxxx/xxxxxx/base/metadata/snap-401738113162842981-1-664733ee-11b5-4930-b21b-2d299873dc36.avro
	at com.netease.arctic.shaded.org.apache.iceberg.hadoop.HadoopInputFile.newStream(HadoopInputFile.java:177) ~[plugin_ne-flink-1.12.4-1.1.5-xxxxxx-SNAPSHOT_scala2.12_hive2.1.1-release-3.8.3-1.3.4-SNAPSHOT_0713_1.0.0.jar:]
	at com.netease.arctic.shaded.org.apache.iceberg.avro.AvroIterable.newFileReader(AvroIterable.java:101) ~[plugin_ne-flink-1.12.4-1.1.5-xxxxxx-SNAPSHOT_scala2.12_hive2.1.1-release-3.8.3-1.3.4-SNAPSHOT_0713_1.0.0.jar:]
	at com.netease.arctic.shaded.org.apache.iceberg.avro.AvroIterable.iterator(AvroIterable.java:77) ~[plugin_ne-flink-1.12.4-1.1.5-xxxxxx-SNAPSHOT_scala2.12_hive2.1.1-release-3.8.3-1.3.4-SNAPSHOT_0713_1.0.0.jar:]
	at com.netease.arctic.shaded.org.apache.iceberg.avro.AvroIterable.iterator(AvroIterable.java:37) ~[plugin_ne-flink-1.12.4-1.1.5-xxxxxx-SNAPSHOT_scala2.12_hive2.1.1-release-3.8.3-1.3.4-SNAPSHOT_0713_1.0.0.jar:]
	at com.netease.arctic.shaded.org.apache.iceberg.relocated.com.google.common.collect.Iterables.addAll(Iterables.java:320) ~[plugin_ne-flink-1.12.4-1.1.5-xxxxxx-SNAPSHOT_scala2.12_hive2.1.1-release-3.8.3-1.3.4-SNAPSHOT_0713_1.0.0.jar:]
	at com.netease.arctic.shaded.org.apache.iceberg.relocated.com.google.common.collect.Lists.newLinkedList(Lists.java:237) ~[plugin_ne-flink-1.12.4-1.1.5-xxxxxx-SNAPSHOT_scala2.12_hive2.1.1-release-3.8.3-1.3.4-SNAPSHOT_0713_1.0.0.jar:]
	at com.netease.arctic.shaded.org.apache.iceberg.ManifestLists.read(ManifestLists.java:46) ~[plugin_ne-flink-1.12.4-1.1.5-xxxxxx-SNAPSHOT_scala2.12_hive2.1.1-release-3.8.3-1.3.4-SNAPSHOT_0713_1.0.0.jar:]
	at com.netease.arctic.shaded.org.apache.iceberg.BaseSnapshot.cacheManifests(BaseSnapshot.java:141) ~[plugin_ne-flink-1.12.4-1.1.5-xxxxxx-SNAPSHOT_scala2.12_hive2.1.1-release-3.8.3-1.3.4-SNAPSHOT_0713_1.0.0.jar:]
	at com.netease.arctic.shaded.org.apache.iceberg.BaseSnapshot.dataManifests(BaseSnapshot.java:163) ~[plugin_ne-flink-1.12.4-1.1.5-xxxxxx-SNAPSHOT_scala2.12_hive2.1.1-release-3.8.3-1.3.4-SNAPSHOT_0713_1.0.0.jar:]
	at com.netease.arctic.shaded.org.apache.iceberg.MergingSnapshotProducer.apply(MergingSnapshotProducer.java:518) ~[plugin_ne-flink-1.12.4-1.1.5-xxxxxx-SNAPSHOT_scala2.12_hive2.1.1-release-3.8.3-1.3.4-SNAPSHOT_0713_1.0.0.jar:]
	at com.netease.arctic.shaded.org.apache.iceberg.SnapshotProducer.apply(SnapshotProducer.java:164) ~[plugin_ne-flink-1.12.4-1.1.5-xxxxxx-SNAPSHOT_scala2.12_hive2.1.1-release-3.8.3-1.3.4-SNAPSHOT_0713_1.0.0.jar:]
	at com.netease.arctic.shaded.org.apache.iceberg.SnapshotProducer.lambda$commit$2(SnapshotProducer.java:283) ~[plugin_ne-flink-1.12.4-1.1.5-xxxxxx-SNAPSHOT_scala2.12_hive2.1.1-release-3.8.3-1.3.4-SNAPSHOT_0713_1.0.0.jar:]
	at com.netease.arctic.shaded.org.apache.iceberg.util.Tasks$Builder.runTaskWithRetry(Tasks.java:404) ~[plugin_ne-flink-1.12.4-1.1.5-xxxxxx-SNAPSHOT_scala2.12_hive2.1.1-release-3.8.3-1.3.4-SNAPSHOT_0713_1.0.0.jar:]
	at com.netease.arctic.shaded.org.apache.iceberg.util.Tasks$Builder.runSingleThreaded(Tasks.java:214) ~[plugin_ne-flink-1.12.4-1.1.5-xxxxxx-SNAPSHOT_scala2.12_hive2.1.1-release-3.8.3-1.3.4-SNAPSHOT_0713_1.0.0.jar:]
	at com.netease.arctic.shaded.org.apache.iceberg.util.Tasks$Builder.run(Tasks.java:198) ~[plugin_ne-flink-1.12.4-1.1.5-xxxxxx-SNAPSHOT_scala2.12_hive2.1.1-release-3.8.3-1.3.4-SNAPSHOT_0713_1.0.0.jar:]
	at com.netease.arctic.shaded.org.apache.iceberg.util.Tasks$Builder.run(Tasks.java:190) ~[plugin_ne-flink-1.12.4-1.1.5-xxxxxx-SNAPSHOT_scala2.12_hive2.1.1-release-3.8.3-1.3.4-SNAPSHOT_0713_1.0.0.jar:]
	at com.netease.arctic.shaded.org.apache.iceberg.SnapshotProducer.commit(SnapshotProducer.java:282) ~[plugin_ne-flink-1.12.4-1.1.5-xxxxxx-SNAPSHOT_scala2.12_hive2.1.1-release-3.8.3-1.3.4-SNAPSHOT_0713_1.0.0.jar:]
	at com.netease.arctic.trace.ArcticAppendFiles.doCommit(ArcticAppendFiles.java:96) ~[plugin_ne-flink-1.12.4-1.1.5-xxxxxx-SNAPSHOT_scala2.12_hive2.1.1-release-3.8.3-1.3.4-SNAPSHOT_0713_1.0.0.jar:]
	at com.netease.arctic.op.ArcticUpdate.commit(ArcticUpdate.java:128) ~[plugin_ne-flink-1.12.4-1.1.5-xxxxxx-SNAPSHOT_scala2.12_hive2.1.1-release-3.8.3-1.3.4-SNAPSHOT_0713_1.0.0.jar:]
	at com.netease.arctic.shaded.org.apache.iceberg.flink.sink.IcebergFilesCommitter.commitOperation(IcebergFilesCommitter.java:312) ~[plugin_ne-flink-1.12.4-1.1.5-xxxxxx-SNAPSHOT_scala2.12_hive2.1.1-release-3.8.3-1.3.4-SNAPSHOT_0713_1.0.0.jar:]
	at com.netease.arctic.shaded.org.apache.iceberg.flink.sink.IcebergFilesCommitter.commitDeltaTxn(IcebergFilesCommitter.java:276) ~[plugin_ne-flink-1.12.4-1.1.5-xxxxxx-SNAPSHOT_scala2.12_hive2.1.1-release-3.8.3-1.3.4-SNAPSHOT_0713_1.0.0.jar:]
	at com.netease.arctic.shaded.org.apache.iceberg.flink.sink.IcebergFilesCommitter.commitUpToCheckpoint(IcebergFilesCommitter.java:218) ~[plugin_ne-flink-1.12.4-1.1.5-xxxxxx-SNAPSHOT_scala2.12_hive2.1.1-release-3.8.3-1.3.4-SNAPSHOT_0713_1.0.0.jar:]
	at com.netease.arctic.shaded.org.apache.iceberg.flink.sink.IcebergFilesCommitter.notifyCheckpointComplete(IcebergFilesCommitter.java:188) ~[plugin_ne-flink-1.12.4-1.1.5-xxxxxx-SNAPSHOT_scala2.12_hive2.1.1-release-3.8.3-1.3.4-SNAPSHOT_0713_1.0.0.jar:]
	at sun.reflect.GeneratedMethodAccessor51.invoke(Unknown Source) ~[?:?]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_181]
	at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_181]
	at com.netease.arctic.flink.interceptor.KerberosInvocationHandler.lambda$invoke$0(KerberosInvocationHandler.java:58) ~[plugin_ne-flink-1.12.4-1.1.5-xxxxxx-SNAPSHOT_scala2.12_hive2.1.1-release-3.8.3-1.3.4-SNAPSHOT_0713_1.0.0.jar:]
	at com.netease.arctic.table.TableMetaStore.doAsUgi(TableMetaStore.java:365) ~[plugin_ne-flink-1.12.4-1.1.5-xxxxxx-SNAPSHOT_scala2.12_hive2.1.1-release-3.8.3-1.3.4-SNAPSHOT_0713_1.0.0.jar:]
	at com.netease.arctic.table.TableMetaStore.lambda$doAs$0(TableMetaStore.java:345) ~[plugin_ne-flink-1.12.4-1.1.5-xxxxxx-SNAPSHOT_scala2.12_hive2.1.1-release-3.8.3-1.3.4-SNAPSHOT_0713_1.0.0.jar:]
	at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_181]
	at javax.security.auth.Subject.doAs(Subject.java:360) ~[?:1.8.0_181]
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1734) ~[flink-shaded-hadoop-2-uber-2.7.5-10.0.jar:2.7.5-10.0]
	at com.netease.arctic.table.TableMetaStore.doAs(TableMetaStore.java:345) ~[plugin_ne-flink-1.12.4-1.1.5-xxxxxx-SNAPSHOT_scala2.12_hive2.1.1-release-3.8.3-1.3.4-SNAPSHOT_0713_1.0.0.jar:]
	at com.netease.arctic.io.ArcticHadoopFileIO.doAs(ArcticHadoopFileIO.java:184) ~[plugin_ne-flink-1.12.4-1.1.5-xxxxxx-SNAPSHOT_scala2.12_hive2.1.1-release-3.8.3-1.3.4-SNAPSHOT_0713_1.0.0.jar:]
	at com.netease.arctic.flink.interceptor.KerberosInvocationHandler.invoke(KerberosInvocationHandler.java:55) ~[plugin_ne-flink-1.12.4-1.1.5-xxxxxx-SNAPSHOT_scala2.12_hive2.1.1-release-3.8.3-1.3.4-SNAPSHOT_0713_1.0.0.jar:]
	at com.sun.proxy.$Proxy39.notifyCheckpointComplete(Unknown Source) ~[?:?]
	... 27 more
Caused by: java.io.FileNotFoundException: File does not exist: /user/hive/warehouse/xxxxxx/xxxxxx/base/metadata/snap-401738113162842981-1-664733ee-11b5-4930-b21b-2d299873dc36.avro
	at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:85)
	at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:75)
	at org.apache.hadoop.hdfs.server.namenode.FSDirStatAndListingOp.getBlockLocations(FSDirStatAndListingOp.java:152)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1909)
	at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:735)
	at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:415)
	at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:523)
	at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:991)
	at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:870)
	at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:816)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1875)
	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2680)

	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:1.8.0_181]
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:1.8.0_181]
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:1.8.0_181]
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_181]
	at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) ~[flink-shaded-hadoop-2-uber-2.7.5-10.0.jar:2.7.5-10.0]
	at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:73) ~[flink-shaded-hadoop-2-uber-2.7.5-10.0.jar:2.7.5-10.0]
	at org.apache.hadoop.hdfs.DFSClient.callGetBlockLocations(DFSClient.java:1237) ~[flink-shaded-hadoop-2-uber-2.7.5-10.0.jar:2.7.5-10.0]
	at org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1222) ~[flink-shaded-hadoop-2-uber-2.7.5-10.0.jar:2.7.5-10.0]
	at org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1210) ~[flink-shaded-hadoop-2-uber-2.7.5-10.0.jar:2.7.5-10.0]
	at org.apache.hadoop.hdfs.DFSInputStream.fetchLocatedBlocksAndGetLastBlockLength(DFSInputStream.java:306) ~[flink-shaded-hadoop-2-uber-2.7.5-10.0.jar:2.7.5-10.0]
	at org.apache.hadoop.hdfs.DFSInputStream.openInfo(DFSInputStream.java:272) ~[flink-shaded-hadoop-2-uber-2.7.5-10.0.jar:2.7.5-10.0]
	at org.apache.hadoop.hdfs.DFSInputStream.<init>(DFSInputStream.java:264) ~[flink-shaded-hadoop-2-uber-2.7.5-10.0.jar:2.7.5-10.0]
	at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:1535) ~[flink-shaded-hadoop-2-uber-2.7.5-10.0.jar:2.7.5-10.0]
	at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:304) ~[flink-shaded-hadoop-2-uber-2.7.5-10.0.jar:2.7.5-10.0]
	at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:299) ~[flink-shaded-hadoop-2-uber-2.7.5-10.0.jar:2.7.5-10.0]
	at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) ~[flink-shaded-hadoop-2-uber-2.7.5-10.0.jar:2.7.5-10.0]
	at org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:312) ~[flink-shaded-hadoop-2-uber-2.7.5-10.0.jar:2.7.5-10.0]
	at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:768) ~[flink-shaded-hadoop-2-uber-2.7.5-10.0.jar:2.7.5-10.0]
	at com.netease.arctic.shaded.org.apache.iceberg.hadoop.HadoopInputFile.newStream(HadoopInputFile.java:175) ~[plugin_ne-flink-1.12.4-1.1.5-xxxxxx-SNAPSHOT_scala2.12_hive2.1.1-release-3.8.3-1.3.4-SNAPSHOT_0713_1.0.0.jar:]
	at com.netease.arctic.shaded.org.apache.iceberg.avro.AvroIterable.newFileReader(AvroIterable.java:101) ~[plugin_ne-flink-1.12.4-1.1.5-xxxxxx-SNAPSHOT_scala2.12_hive2.1.1-release-3.8.3-1.3.4-SNAPSHOT_0713_1.0.0.jar:]
	at com.netease.arctic.shaded.org.apache.iceberg.avro.AvroIterable.iterator(AvroIterable.java:77) ~[plugin_ne-flink-1.12.4-1.1.5-xxxxxx-SNAPSHOT_scala2.12_hive2.1.1-release-3.8.3-1.3.4-SNAPSHOT_0713_1.0.0.jar:]
	at com.netease.arctic.shaded.org.apache.iceberg.avro.AvroIterable.iterator(AvroIterable.java:37) ~[plugin_ne-flink-1.12.4-1.1.5-xxxxxx-SNAPSHOT_scala2.12_hive2.1.1-release-3.8.3-1.3.4-SNAPSHOT_0713_1.0.0.jar:]
	at com.netease.arctic.shaded.org.apache.iceberg.relocated.com.google.common.collect.Iterables.addAll(Iterables.java:320) ~[plugin_ne-flink-1.12.4-1.1.5-xxxxxx-SNAPSHOT_scala2.12_hive2.1.1-release-3.8.3-1.3.4-SNAPSHOT_0713_1.0.0.jar:]
	at com.netease.arctic.shaded.org.apache.iceberg.relocated.com.google.common.collect.Lists.newLinkedList(Lists.java:237) ~[plugin_ne-flink-1.12.4-1.1.5-xxxxxx-SNAPSHOT_scala2.12_hive2.1.1-release-3.8.3-1.3.4-SNAPSHOT_0713_1.0.0.jar:]
	at com.netease.arctic.shaded.org.apache.iceberg.ManifestLists.read(ManifestLists.java:46) ~[plugin_ne-flink-1.12.4-1.1.5-xxxxxx-SNAPSHOT_scala2.12_hive2.1.1-release-3.8.3-1.3.4-SNAPSHOT_0713_1.0.0.jar:]
	at com.netease.arctic.shaded.org.apache.iceberg.BaseSnapshot.cacheManifests(BaseSnapshot.java:141) ~[plugin_ne-flink-1.12.4-1.1.5-xxxxxx-SNAPSHOT_scala2.12_hive2.1.1-release-3.8.3-1.3.4-SNAPSHOT_0713_1.0.0.jar:]
	at com.netease.arctic.shaded.org.apache.iceberg.BaseSnapshot.dataManifests(BaseSnapshot.java:163) ~[plugin_ne-flink-1.12.4-1.1.5-xxxxxx-SNAPSHOT_scala2.12_hive2.1.1-release-3.8.3-1.3.4-SNAPSHOT_0713_1.0.0.jar:]
	at com.netease.arctic.shaded.org.apache.iceberg.MergingSnapshotProducer.apply(MergingSnapshotProducer.java:518) ~[plugin_ne-flink-1.12.4-1.1.5-xxxxxx-SNAPSHOT_scala2.12_hive2.1.1-release-3.8.3-1.3.4-SNAPSHOT_0713_1.0.0.jar:]
	at com.netease.arctic.shaded.org.apache.iceberg.SnapshotProducer.apply(SnapshotProducer.java:164) ~[plugin_ne-flink-1.12.4-1.1.5-xxxxxx-SNAPSHOT_scala2.12_hive2.1.1-release-3.8.3-1.3.4-SNAPSHOT_0713_1.0.0.jar:]
	at com.netease.arctic.shaded.org.apache.iceberg.SnapshotProducer.lambda$commit$2(SnapshotProducer.java:283) ~[plugin_ne-flink-1.12.4-1.1.5-xxxxxx-SNAPSHOT_scala2.12_hive2.1.1-release-3.8.3-1.3.4-SNAPSHOT_0713_1.0.0.jar:]
	at com.netease.arctic.shaded.org.apache.iceberg.util.Tasks$Builder.runTaskWithRetry(Tasks.java:404) ~[plugin_ne-flink-1.12.4-1.1.5-xxxxxx-SNAPSHOT_scala2.12_hive2.1.1-release-3.8.3-1.3.4-SNAPSHOT_0713_1.0.0.jar:]
	at com.netease.arctic.shaded.org.apache.iceberg.util.Tasks$Builder.runSingleThreaded(Tasks.java:214) ~[plugin_ne-flink-1.12.4-1.1.5-xxxxxx-SNAPSHOT_scala2.12_hive2.1.1-release-3.8.3-1.3.4-SNAPSHOT_0713_1.0.0.jar:]
	at com.netease.arctic.shaded.org.apache.iceberg.util.Tasks$Builder.run(Tasks.java:198) ~[plugin_ne-flink-1.12.4-1.1.5-xxxxxx-SNAPSHOT_scala2.12_hive2.1.1-release-3.8.3-1.3.4-SNAPSHOT_0713_1.0.0.jar:]
	at com.netease.arctic.shaded.org.apache.iceberg.util.Tasks$Builder.run(Tasks.java:190) ~[plugin_ne-flink-1.12.4-1.1.5-xxxxxx-SNAPSHOT_scala2.12_hive2.1.1-release-3.8.3-1.3.4-SNAPSHOT_0713_1.0.0.jar:]
	at com.netease.arctic.shaded.org.apache.iceberg.SnapshotProducer.commit(SnapshotProducer.java:282) ~[plugin_ne-flink-1.12.4-1.1.5-xxxxxx-SNAPSHOT_scala2.12_hive2.1.1-release-3.8.3-1.3.4-SNAPSHOT_0713_1.0.0.jar:]
	at com.netease.arctic.trace.ArcticAppendFiles.doCommit(ArcticAppendFiles.java:96) ~[plugin_ne-flink-1.12.4-1.1.5-xxxxxx-SNAPSHOT_scala2.12_hive2.1.1-release-3.8.3-1.3.4-SNAPSHOT_0713_1.0.0.jar:]
	at com.netease.arctic.op.ArcticUpdate.commit(ArcticUpdate.java:128) ~[plugin_ne-flink-1.12.4-1.1.5-xxxxxx-SNAPSHOT_scala2.12_hive2.1.1-release-3.8.3-1.3.4-SNAPSHOT_0713_1.0.0.jar:]
	at com.netease.arctic.shaded.org.apache.iceberg.flink.sink.IcebergFilesCommitter.commitOperation(IcebergFilesCommitter.java:312) ~[plugin_ne-flink-1.12.4-1.1.5-xxxxxx-SNAPSHOT_scala2.12_hive2.1.1-release-3.8.3-1.3.4-SNAPSHOT_0713_1.0.0.jar:]
	at com.netease.arctic.shaded.org.apache.iceberg.flink.sink.IcebergFilesCommitter.commitDeltaTxn(IcebergFilesCommitter.java:276) ~[plugin_ne-flink-1.12.4-1.1.5-xxxxxx-SNAPSHOT_scala2.12_hive2.1.1-release-3.8.3-1.3.4-SNAPSHOT_0713_1.0.0.jar:]
	at com.netease.arctic.shaded.org.apache.iceberg.flink.sink.IcebergFilesCommitter.commitUpToCheckpoint(IcebergFilesCommitter.java:218) ~[plugin_ne-flink-1.12.4-1.1.5-xxxxxx-SNAPSHOT_scala2.12_hive2.1.1-release-3.8.3-1.3.4-SNAPSHOT_0713_1.0.0.jar:]
	at com.netease.arctic.shaded.org.apache.iceberg.flink.sink.IcebergFilesCommitter.notifyCheckpointComplete(IcebergFilesCommitter.java:188) ~[plugin_ne-flink-1.12.4-1.1.5-xxxxxx-SNAPSHOT_scala2.12_hive2.1.1-release-3.8.3-1.3.4-SNAPSHOT_0713_1.0.0.jar:]
	at sun.reflect.GeneratedMethodAccessor51.invoke(Unknown Source) ~[?:?]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_181]
	at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_181]
	at com.netease.arctic.flink.interceptor.KerberosInvocationHandler.lambda$invoke$0(KerberosInvocationHandler.java:58) ~[plugin_ne-flink-1.12.4-1.1.5-xxxxxx-SNAPSHOT_scala2.12_hive2.1.1-release-3.8.3-1.3.4-SNAPSHOT_0713_1.0.0.jar:]
	at com.netease.arctic.table.TableMetaStore.doAsUgi(TableMetaStore.java:365) ~[plugin_ne-flink-1.12.4-1.1.5-xxxxxx-SNAPSHOT_scala2.12_hive2.1.1-release-3.8.3-1.3.4-SNAPSHOT_0713_1.0.0.jar:]
	at com.netease.arctic.table.TableMetaStore.lambda$doAs$0(TableMetaStore.java:345) ~[plugin_ne-flink-1.12.4-1.1.5-xxxxxx-SNAPSHOT_scala2.12_hive2.1.1-release-3.8.3-1.3.4-SNAPSHOT_0713_1.0.0.jar:]
	at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_181]
	at javax.security.auth.Subject.doAs(Subject.java:360) ~[?:1.8.0_181]
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1734) ~[flink-shaded-hadoop-2-uber-2.7.5-10.0.jar:2.7.5-10.0]
	at com.netease.arctic.table.TableMetaStore.doAs(TableMetaStore.java:345) ~[plugin_ne-flink-1.12.4-1.1.5-xxxxxx-SNAPSHOT_scala2.12_hive2.1.1-release-3.8.3-1.3.4-SNAPSHOT_0713_1.0.0.jar:]
	at com.netease.arctic.io.ArcticHadoopFileIO.doAs(ArcticHadoopFileIO.java:184) ~[plugin_ne-flink-1.12.4-1.1.5-xxxxxx-SNAPSHOT_scala2.12_hive2.1.1-release-3.8.3-1.3.4-SNAPSHOT_0713_1.0.0.jar:]
	at com.netease.arctic.flink.interceptor.KerberosInvocationHandler.invoke(KerberosInvocationHandler.java:55) ~[plugin_ne-flink-1.12.4-1.1.5-xxxxxx-SNAPSHOT_scala2.12_hive2.1.1-release-3.8.3-1.3.4-SNAPSHOT_0713_1.0.0.jar:]
	at com.sun.proxy.$Proxy39.notifyCheckpointComplete(Unknown Source) ~[?:?]
	... 27 more
Caused by: org.apache.hadoop.ipc.RemoteException: File does not exist: /user/hive/warehouse/xxxxxx/xxxxxx/base/metadata/snap-401738113162842981-1-664733ee-11b5-4930-b21b-2d299873dc36.avro
	at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:85)
	at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:75)
	at org.apache.hadoop.hdfs.server.namenode.FSDirStatAndListingOp.getBlockLocations(FSDirStatAndListingOp.java:152)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1909)
	at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:735)
	at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:415)
	at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:523)
	at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:991)
	at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:870)
	at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:816)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1875)
	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2680)

	at org.apache.hadoop.ipc.Client.call(Client.java:1476) ~[flink-shaded-hadoop-2-uber-2.7.5-10.0.jar:2.7.5-10.0]
	at org.apache.hadoop.ipc.Client.call(Client.java:1413) ~[flink-shaded-hadoop-2-uber-2.7.5-10.0.jar:2.7.5-10.0]
	at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229) ~[flink-shaded-hadoop-2-uber-2.7.5-10.0.jar:2.7.5-10.0]
	at com.sun.proxy.$Proxy27.getBlockLocations(Unknown Source) ~[?:?]
	at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getBlockLocations(ClientNamenodeProtocolTranslatorPB.java:255) ~[flink-shaded-hadoop-2-uber-2.7.5-10.0.jar:2.7.5-10.0]
	at sun.reflect.GeneratedMethodAccessor39.invoke(Unknown Source) ~[?:?]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_181]
	at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_181]
	at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:191) ~[flink-shaded-hadoop-2-uber-2.7.5-10.0.jar:2.7.5-10.0]
	at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) ~[flink-shaded-hadoop-2-uber-2.7.5-10.0.jar:2.7.5-10.0]
	at com.sun.proxy.$Proxy28.getBlockLocations(Unknown Source) ~[?:?]
	at org.apache.hadoop.hdfs.DFSClient.callGetBlockLocations(DFSClient.java:1235) ~[flink-shaded-hadoop-2-uber-2.7.5-10.0.jar:2.7.5-10.0]
	at org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1222) ~[flink-shaded-hadoop-2-uber-2.7.5-10.0.jar:2.7.5-10.0]
	at org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1210) ~[flink-shaded-hadoop-2-uber-2.7.5-10.0.jar:2.7.5-10.0]
	at org.apache.hadoop.hdfs.DFSInputStream.fetchLocatedBlocksAndGetLastBlockLength(DFSInputStream.java:306) ~[flink-shaded-hadoop-2-uber-2.7.5-10.0.jar:2.7.5-10.0]
	at org.apache.hadoop.hdfs.DFSInputStream.openInfo(DFSInputStream.java:272) ~[flink-shaded-hadoop-2-uber-2.7.5-10.0.jar:2.7.5-10.0]
	at org.apache.hadoop.hdfs.DFSInputStream.<init>(DFSInputStream.java:264) ~[flink-shaded-hadoop-2-uber-2.7.5-10.0.jar:2.7.5-10.0]
	at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:1535) ~[flink-shaded-hadoop-2-uber-2.7.5-10.0.jar:2.7.5-10.0]
	at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:304) ~[flink-shaded-hadoop-2-uber-2.7.5-10.0.jar:2.7.5-10.0]
	at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:299) ~[flink-shaded-hadoop-2-uber-2.7.5-10.0.jar:2.7.5-10.0]
	at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) ~[flink-shaded-hadoop-2-uber-2.7.5-10.0.jar:2.7.5-10.0]
	at org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:312) ~[flink-shaded-hadoop-2-uber-2.7.5-10.0.jar:2.7.5-10.0]
	at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:768) ~[flink-shaded-hadoop-2-uber-2.7.5-10.0.jar:2.7.5-10.0]
	at com.netease.arctic.shaded.org.apache.iceberg.hadoop.HadoopInputFile.newStream(HadoopInputFile.java:175) ~[plugin_ne-flink-1.12.4-1.1.5-xxxxxx-SNAPSHOT_scala2.12_hive2.1.1-release-3.8.3-1.3.4-SNAPSHOT_0713_1.0.0.jar:]
	at com.netease.arctic.shaded.org.apache.iceberg.avro.AvroIterable.newFileReader(AvroIterable.java:101) ~[plugin_ne-flink-1.12.4-1.1.5-xxxxxx-SNAPSHOT_scala2.12_hive2.1.1-release-3.8.3-1.3.4-SNAPSHOT_0713_1.0.0.jar:]
	at com.netease.arctic.shaded.org.apache.iceberg.avro.AvroIterable.iterator(AvroIterable.java:77) ~[plugin_ne-flink-1.12.4-1.1.5-xxxxxx-SNAPSHOT_scala2.12_hive2.1.1-release-3.8.3-1.3.4-SNAPSHOT_0713_1.0.0.jar:]
	at com.netease.arctic.shaded.org.apache.iceberg.avro.AvroIterable.iterator(AvroIterable.java:37) ~[plugin_ne-flink-1.12.4-1.1.5-xxxxxx-SNAPSHOT_scala2.12_hive2.1.1-release-3.8.3-1.3.4-SNAPSHOT_0713_1.0.0.jar:]
	at com.netease.arctic.shaded.org.apache.iceberg.relocated.com.google.common.collect.Iterables.addAll(Iterables.java:320) ~[plugin_ne-flink-1.12.4-1.1.5-xxxxxx-SNAPSHOT_scala2.12_hive2.1.1-release-3.8.3-1.3.4-SNAPSHOT_0713_1.0.0.jar:]
	at com.netease.arctic.shaded.org.apache.iceberg.relocated.com.google.common.collect.Lists.newLinkedList(Lists.java:237) ~[plugin_ne-flink-1.12.4-1.1.5-xxxxxx-SNAPSHOT_scala2.12_hive2.1.1-release-3.8.3-1.3.4-SNAPSHOT_0713_1.0.0.jar:]
	at com.netease.arctic.shaded.org.apache.iceberg.ManifestLists.read(ManifestLists.java:46) ~[plugin_ne-flink-1.12.4-1.1.5-xxxxxx-SNAPSHOT_scala2.12_hive2.1.1-release-3.8.3-1.3.4-SNAPSHOT_0713_1.0.0.jar:]
	at com.netease.arctic.shaded.org.apache.iceberg.BaseSnapshot.cacheManifests(BaseSnapshot.java:141) ~[plugin_ne-flink-1.12.4-1.1.5-xxxxxx-SNAPSHOT_scala2.12_hive2.1.1-release-3.8.3-1.3.4-SNAPSHOT_0713_1.0.0.jar:]
	at com.netease.arctic.shaded.org.apache.iceberg.BaseSnapshot.dataManifests(BaseSnapshot.java:163) ~[plugin_ne-flink-1.12.4-1.1.5-xxxxxx-SNAPSHOT_scala2.12_hive2.1.1-release-3.8.3-1.3.4-SNAPSHOT_0713_1.0.0.jar:]
	at com.netease.arctic.shaded.org.apache.iceberg.MergingSnapshotProducer.apply(MergingSnapshotProducer.java:518) ~[plugin_ne-flink-1.12.4-1.1.5-xxxxxx-SNAPSHOT_scala2.12_hive2.1.1-release-3.8.3-1.3.4-SNAPSHOT_0713_1.0.0.jar:]
	at com.netease.arctic.shaded.org.apache.iceberg.SnapshotProducer.apply(SnapshotProducer.java:164) ~[plugin_ne-flink-1.12.4-1.1.5-xxxxxx-SNAPSHOT_scala2.12_hive2.1.1-release-3.8.3-1.3.4-SNAPSHOT_0713_1.0.0.jar:]
	at com.netease.arctic.shaded.org.apache.iceberg.SnapshotProducer.lambda$commit$2(SnapshotProducer.java:283) ~[plugin_ne-flink-1.12.4-1.1.5-xxxxxx-SNAPSHOT_scala2.12_hive2.1.1-release-3.8.3-1.3.4-SNAPSHOT_0713_1.0.0.jar:]
	at com.netease.arctic.shaded.org.apache.iceberg.util.Tasks$Builder.runTaskWithRetry(Tasks.java:404) ~[plugin_ne-flink-1.12.4-1.1.5-xxxxxx-SNAPSHOT_scala2.12_hive2.1.1-release-3.8.3-1.3.4-SNAPSHOT_0713_1.0.0.jar:]
	at com.netease.arctic.shaded.org.apache.iceberg.util.Tasks$Builder.runSingleThreaded(Tasks.java:214) ~[plugin_ne-flink-1.12.4-1.1.5-xxxxxx-SNAPSHOT_scala2.12_hive2.1.1-release-3.8.3-1.3.4-SNAPSHOT_0713_1.0.0.jar:]
	at com.netease.arctic.shaded.org.apache.iceberg.util.Tasks$Builder.run(Tasks.java:198) ~[plugin_ne-flink-1.12.4-1.1.5-xxxxxx-SNAPSHOT_scala2.12_hive2.1.1-release-3.8.3-1.3.4-SNAPSHOT_0713_1.0.0.jar:]
	at com.netease.arctic.shaded.org.apache.iceberg.util.Tasks$Builder.run(Tasks.java:190) ~[plugin_ne-flink-1.12.4-1.1.5-xxxxxx-SNAPSHOT_scala2.12_hive2.1.1-release-3.8.3-1.3.4-SNAPSHOT_0713_1.0.0.jar:]
	at com.netease.arctic.shaded.org.apache.iceberg.SnapshotProducer.commit(SnapshotProducer.java:282) ~[plugin_ne-flink-1.12.4-1.1.5-xxxxxx-SNAPSHOT_scala2.12_hive2.1.1-release-3.8.3-1.3.4-SNAPSHOT_0713_1.0.0.jar:]
	at com.netease.arctic.trace.ArcticAppendFiles.doCommit(ArcticAppendFiles.java:96) ~[plugin_ne-flink-1.12.4-1.1.5-xxxxxx-SNAPSHOT_scala2.12_hive2.1.1-release-3.8.3-1.3.4-SNAPSHOT_0713_1.0.0.jar:]
	at com.netease.arctic.op.ArcticUpdate.commit(ArcticUpdate.java:128) ~[plugin_ne-flink-1.12.4-1.1.5-xxxxxx-SNAPSHOT_scala2.12_hive2.1.1-release-3.8.3-1.3.4-SNAPSHOT_0713_1.0.0.jar:]
	at com.netease.arctic.shaded.org.apache.iceberg.flink.sink.IcebergFilesCommitter.commitOperation(IcebergFilesCommitter.java:312) ~[plugin_ne-flink-1.12.4-1.1.5-xxxxxx-SNAPSHOT_scala2.12_hive2.1.1-release-3.8.3-1.3.4-SNAPSHOT_0713_1.0.0.jar:]
	at com.netease.arctic.shaded.org.apache.iceberg.flink.sink.IcebergFilesCommitter.commitDeltaTxn(IcebergFilesCommitter.java:276) ~[plugin_ne-flink-1.12.4-1.1.5-xxxxxx-SNAPSHOT_scala2.12_hive2.1.1-release-3.8.3-1.3.4-SNAPSHOT_0713_1.0.0.jar:]
	at com.netease.arctic.shaded.org.apache.iceberg.flink.sink.IcebergFilesCommitter.commitUpToCheckpoint(IcebergFilesCommitter.java:218) ~[plugin_ne-flink-1.12.4-1.1.5-xxxxxx-SNAPSHOT_scala2.12_hive2.1.1-release-3.8.3-1.3.4-SNAPSHOT_0713_1.0.0.jar:]
	at com.netease.arctic.shaded.org.apache.iceberg.flink.sink.IcebergFilesCommitter.notifyCheckpointComplete(IcebergFilesCommitter.java:188) ~[plugin_ne-flink-1.12.4-1.1.5-xxxxxx-SNAPSHOT_scala2.12_hive2.1.1-release-3.8.3-1.3.4-SNAPSHOT_0713_1.0.0.jar:]
	at sun.reflect.GeneratedMethodAccessor51.invoke(Unknown Source) ~[?:?]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_181]
	at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_181]
	at com.netease.arctic.flink.interceptor.KerberosInvocationHandler.lambda$invoke$0(KerberosInvocationHandler.java:58) ~[plugin_ne-flink-1.12.4-1.1.5-xxxxxx-SNAPSHOT_scala2.12_hive2.1.1-release-3.8.3-1.3.4-SNAPSHOT_0713_1.0.0.jar:]
	at com.netease.arctic.table.TableMetaStore.doAsUgi(TableMetaStore.java:365) ~[plugin_ne-flink-1.12.4-1.1.5-xxxxxx-SNAPSHOT_scala2.12_hive2.1.1-release-3.8.3-1.3.4-SNAPSHOT_0713_1.0.0.jar:]
	at com.netease.arctic.table.TableMetaStore.lambda$doAs$0(TableMetaStore.java:345) ~[plugin_ne-flink-1.12.4-1.1.5-xxxxxx-SNAPSHOT_scala2.12_hive2.1.1-release-3.8.3-1.3.4-SNAPSHOT_0713_1.0.0.jar:]
	at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_181]
	at javax.security.auth.Subject.doAs(Subject.java:360) ~[?:1.8.0_181]
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1734) ~[flink-shaded-hadoop-2-uber-2.7.5-10.0.jar:2.7.5-10.0]
	at com.netease.arctic.table.TableMetaStore.doAs(TableMetaStore.java:345) ~[plugin_ne-flink-1.12.4-1.1.5-xxxxxx-SNAPSHOT_scala2.12_hive2.1.1-release-3.8.3-1.3.4-SNAPSHOT_0713_1.0.0.jar:]
	at com.netease.arctic.io.ArcticHadoopFileIO.doAs(ArcticHadoopFileIO.java:184) ~[plugin_ne-flink-1.12.4-1.1.5-xxxxxx-SNAPSHOT_scala2.12_hive2.1.1-release-3.8.3-1.3.4-SNAPSHOT_0713_1.0.0.jar:]
	at com.netease.arctic.flink.interceptor.KerberosInvocationHandler.invoke(KerberosInvocationHandler.java:55) ~[plugin_ne-flink-1.12.4-1.1.5-xxxxxx-SNAPSHOT_scala2.12_hive2.1.1-release-3.8.3-1.3.4-SNAPSHOT_0713_1.0.0.jar:]
	at com.sun.proxy.$Proxy39.notifyCheckpointComplete(Unknown Source) ~[?:?]
	... 27 more

Anything else

No response

Code of Conduct

  • I agree to follow this project's Code of Conduct
@nicochen nicochen added the type:bug Something isn't working label Aug 23, 2023
@wangtaohz
Copy link
Contributor

AFAIK, the latest Amoro 0.4.x uses Iceberg version 0.13.2, and we should check if the same issue exists in higher versions of Iceberg.

@wangtaohz wangtaohz mentioned this issue Sep 4, 2023
Merged
3 tasks
@wangtaohz
Copy link
Contributor

AFAIK, the latest Amoro 0.4.x uses Iceberg version 0.13.2, and we should check if the same issue exists in higher versions of Iceberg.

The problem still exists in the latest version(1.3.1) of Iceberg.

@wangtaohz wangtaohz mentioned this issue Sep 7, 2023
Closed
56 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@YesOrNo828 YesOrNo828

Labels
type:bug Something isn't working
Projects
None yet
Milestone
No milestone
3 participants
@nicochen @YesOrNo828 @wangtaohz