You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Since v10.0.0 kafka-connect-hdfs has been using kafka-connect-storage-common:10.x.x that in turn uses hive:2.3.7 which contains this breaking change. I suspect the same is true for v6.x.x tags.
Is this expected and >v10.0.0 should not be used against older hive metastore versions or should this be considered a bug?
The error given by kafka-connect-hdfs when testing against older metastore is: ERROR Altering Hive schema threw unexpected error (io.confluent.connect.hdfs.TopicPartitionWriter:923) io.confluent.connect.storage.errors.HiveMetaStoreException: Exception communicating with the Hive MetaStore at io.confluent.connect.storage.hive.HiveMetaStore.doAction(HiveMetaStore.java:101) at io.confluent.connect.storage.hive.HiveMetaStore.getTable(HiveMetaStore.java:303) at io.confluent.connect.hdfs.parquet.ParquetHiveUtil.alterSchema(ParquetHiveUtil.java:57) at io.confluent.connect.hdfs.DataWriter$2.alterSchema(DataWriter.java:294) at io.confluent.connect.hdfs.TopicPartitionWriter.lambda$alterHiveSchema$1(TopicPartitionWriter.java:921) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.thrift.TApplicationException: Invalid method name: 'get_table_req' at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:79) at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_table_req(ThriftHiveMetastore.java:1567) at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_table_req(ThriftHiveMetastore.java:1554) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getTable(HiveMetaStoreClient.java:1350) at sun.reflect.GeneratedMethodAccessor179.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:169) at com.sun.proxy.$Proxy47.getTable(Unknown Source) at io.confluent.connect.storage.hive.HiveMetaStore$9.call(HiveMetaStore.java:294) at io.confluent.connect.storage.hive.HiveMetaStore$9.call(HiveMetaStore.java:290) at io.confluent.connect.storage.hive.HiveMetaStore.doAction(HiveMetaStore.java:97)
I was unsure if I should report this here as the problem is seen with kafka-connect-hdfs or to kafka-connect-storage-common as the fix should probably be applied there.
The text was updated successfully, but these errors were encountered:
There is a breaking change in hive 2.3.0 in hiveMetastoreClient that causes an error when trying to get table from metastore with a lower version. The change is made in this commit: apache/hive@fb5bf0c#diff-b65524ea2624ea6627d6197ffc16d1af1d1c53935b278f40720efefde77692e6R1287
Since v10.0.0 kafka-connect-hdfs has been using kafka-connect-storage-common:10.x.x that in turn uses hive:2.3.7 which contains this breaking change. I suspect the same is true for v6.x.x tags.
Is this expected and >v10.0.0 should not be used against older hive metastore versions or should this be considered a bug?
The error given by kafka-connect-hdfs when testing against older metastore is:
ERROR Altering Hive schema threw unexpected error (io.confluent.connect.hdfs.TopicPartitionWriter:923) io.confluent.connect.storage.errors.HiveMetaStoreException: Exception communicating with the Hive MetaStore at io.confluent.connect.storage.hive.HiveMetaStore.doAction(HiveMetaStore.java:101) at io.confluent.connect.storage.hive.HiveMetaStore.getTable(HiveMetaStore.java:303) at io.confluent.connect.hdfs.parquet.ParquetHiveUtil.alterSchema(ParquetHiveUtil.java:57) at io.confluent.connect.hdfs.DataWriter$2.alterSchema(DataWriter.java:294) at io.confluent.connect.hdfs.TopicPartitionWriter.lambda$alterHiveSchema$1(TopicPartitionWriter.java:921) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.thrift.TApplicationException: Invalid method name: 'get_table_req' at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:79) at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_table_req(ThriftHiveMetastore.java:1567) at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_table_req(ThriftHiveMetastore.java:1554) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getTable(HiveMetaStoreClient.java:1350) at sun.reflect.GeneratedMethodAccessor179.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:169) at com.sun.proxy.$Proxy47.getTable(Unknown Source) at io.confluent.connect.storage.hive.HiveMetaStore$9.call(HiveMetaStore.java:294) at io.confluent.connect.storage.hive.HiveMetaStore$9.call(HiveMetaStore.java:290) at io.confluent.connect.storage.hive.HiveMetaStore.doAction(HiveMetaStore.java:97)
I was unsure if I should report this here as the problem is seen with kafka-connect-hdfs or to kafka-connect-storage-common as the fix should probably be applied there.
The text was updated successfully, but these errors were encountered: