Flink will throw MetaException when hive table property value is larger than 4000

[IceMimosa]

Apache Iceberg version

1.0.0

Query engine

Flink

Please describe the bug 🐞

Hive property key is current-schema, flink will throw MetaException when doing commit:

• flink version: 1.13.6

org.apache.iceberg.exceptions.CommitStateUnknownException: MetaException(message:Put request failed : UPDATE `TABLE_PARAMS` SET `PARAM_VALUE` = ? WHERE `TBL_ID`=? AND `PARAM_KEY`=?)
Cannot determine whether the commit was successful or not, the underlying data files may or may not be needed. Manual intervention via the Remove Orphan Files Action can remove these files when a connection to the Catalog can be re-established if the commit was actually unsuccessful.
Please check to see whether or not your commit was successful before retrying this commit. Retrying an already successful operation will result in duplicate records or unintentional modifications.
At this time no files will be deleted including possibly unused manifest lists.
	at org.apache.iceberg.hive.HiveTableOperations.doCommit(HiveTableOperations.java:329)
	at org.apache.iceberg.BaseMetastoreTableOperations.commit(BaseMetastoreTableOperations.java:133)
	at org.apache.iceberg.SnapshotProducer.lambda$commit$2(SnapshotProducer.java:357)
	at org.apache.iceberg.util.Tasks$Builder.runTaskWithRetry(Tasks.java:402)
	at org.apache.iceberg.util.Tasks$Builder.runSingleThreaded(Tasks.java:212)
	at org.apache.iceberg.util.Tasks$Builder.run(Tasks.java:196)
	at org.apache.iceberg.util.Tasks$Builder.run(Tasks.java:189)
	at org.apache.iceberg.SnapshotProducer.commit(SnapshotProducer.java:331)
	at org.apache.iceberg.flink.sink.IcebergFilesCommitter.commitOperation(IcebergFilesCommitter.java:363)
	at org.apache.iceberg.flink.sink.IcebergFilesCommitter.commitDeltaTxn(IcebergFilesCommitter.java:310)
	at org.apache.iceberg.flink.sink.IcebergFilesCommitter.commitUpToCheckpoint(IcebergFilesCommitter.java:243)
	at org.apache.iceberg.flink.sink.IcebergFilesCommitter.notifyCheckpointComplete(IcebergFilesCommitter.java:209)
	at org.apache.flink.streaming.runtime.tasks.StreamOperatorWrapper.notifyCheckpointComplete(StreamOperatorWrapper.java:99)
	at org.apache.flink.streaming.runtime.tasks.SubtaskCheckpointCoordinatorImpl.notifyCheckpointComplete(SubtaskCheckpointCoordinatorImpl.java:334)
	at org.apache.flink.streaming.runtime.tasks.StreamTask.notifyCheckpointComplete(StreamTask.java:1171)
	at org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$notifyCheckpointCompleteAsync$10(StreamTask.java:1136)
	at org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$notifyCheckpointOperation$12(StreamTask.java:1159)
	at org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$1.runThrowing(StreamTaskActionExecutor.java:50)
	at org.apache.flink.streaming.runtime.tasks.mailbox.Mail.run(Mail.java:90)
	at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.processMailsWhenDefaultActionUnavailable(MailboxProcessor.java:344)
	at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.processMail(MailboxProcessor.java:330)
	at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:202)
	at org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:684)
	at org.apache.flink.streaming.runtime.tasks.StreamTask.executeInvoke(StreamTask.java:639)
	at org.apache.flink.streaming.runtime.tasks.StreamTask.runWithCleanUpOnFail(StreamTask.java:650)
	at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:623)
	at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:779)
	at org.apache.flink.runtime.taskmanager.Task.run(Task.java:566)
	at java.lang.Thread.run(Thread.java:750)
Caused by: java.lang.RuntimeException: MetaException(message:Put request failed : UPDATE `TABLE_PARAMS` SET `PARAM_VALUE` = ? WHERE `TBL_ID`=? AND `PARAM_KEY`=?)
	at org.apache.iceberg.relocated.com.google.common.base.Throwables.propagate(Throwables.java:243)
	at org.apache.iceberg.common.DynMethods$UnboundMethod.invoke(DynMethods.java:75)
	at org.apache.iceberg.hive.MetastoreUtil.alterTable(MetastoreUtil.java:72)
	at org.apache.iceberg.hive.HiveTableOperations.lambda$persistTable$3(HiveTableOperations.java:354)
	at org.apache.iceberg.ClientPoolImpl.run(ClientPoolImpl.java:58)
	at org.apache.iceberg.ClientPoolImpl.run(ClientPoolImpl.java:51)
	at org.apache.iceberg.hive.CachedClientPool.run(CachedClientPool.java:82)
	at org.apache.iceberg.hive.HiveTableOperations.persistTable(HiveTableOperations.java:352)
	at org.apache.iceberg.hive.HiveTableOperations.doCommit(HiveTableOperations.java:299)
	... 28 more
Caused by: MetaException(message:Put request failed : UPDATE `TABLE_PARAMS` SET `PARAM_VALUE` = ? WHERE `TBL_ID`=? AND `PARAM_KEY`=?)
	at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$alter_table_with_environment_context_result$alter_table_with_environment_context_resultStandardScheme.read(ThriftHiveMetastore.java)
	at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$alter_table_with_environment_context_result$alter_table_with_environment_context_resultStandardScheme.read(ThriftHiveMetastore.java)
	at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$alter_table_with_environment_context_result.read(ThriftHiveMetastore.java)
	at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:86)
	at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_alter_table_with_environment_context(ThriftHiveMetastore.java:2270)
	at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.alter_table_with_environment_context(ThriftHiveMetastore.java:2254)
	at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.alter_table_with_environmentContext(HiveMetaStoreClient.java:405)
	at sun.reflect.GeneratedMethodAccessor105.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:208)
	at com.sun.proxy.$Proxy34.alter_table_with_environmentContext(Unknown Source)
	at sun.reflect.GeneratedMethodAccessor105.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.iceberg.common.DynMethods$UnboundMethod.invokeChecked(DynMethods.java:60)
	at org.apache.iceberg.common.DynMethods$UnboundMethod.invoke(DynMethods.java:72)
	... 35 more

[zinking]

there are very wide tables in production systems that span hundreds of complex columns, the schema alone for this kind of table would be 10s of MBs.

so it is neither desirable nor possible to store this kind of info within 'property'.

[chenjunjiedada]

How about adding a property to allow users to decide whether to store the current-schema?

[chenjunjiedada]

@flyrain @szehon-ho Do you have any option on this?

[flyrain]

I'd suggest to upgrade the HMS if you want to store the schema as a table property in catalog. Seems like your HMS is pretty old according to https://issues.apache.org/jira/browse/HIVE-12274. Otherwise, you can truncated it if that is acceptable.

Here is a related comments, #4456 (comment).

[szehon-ho]

Yea, the flag provided there: HIVE_TABLE_PROPERTY_MAX_SIZE can be set to 0 to not store the property at all

[flyrain]

Yup, but I'd assume he needs the schema to be stored at the catalog. We checked the size for schema as well, here is the code,

iceberg/hive-metastore/src/main/java/org/apache/iceberg/hive/HiveTableOperations.java

Line 504 in e2bb9ad

setField(parameters, TableProperties.CURRENT_SCHEMA, schema);

[github-actions]

This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. To permanently prevent this issue from being considered stale, add the label 'not-stale', but commenting on the issue is preferred when possible.

[github-actions]

This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale'