[SPARK-45265][SQL] Support Hive 4.0 metastore #48823
Conversation
|
👍🏻 Thank you so much for taking over this, @yaooqinn . |
|
All GitHub actions are passing now. Please take a look. @LuciferYang @dongjoon-hyun @HyukjinKwon @cloud-fan |
dongjoon-hyun
changed the title
dongjoon-hyun
changed the title
There was a problem hiding this comment.
+1, LGTM. Thank you, @yaooqinn .
I believe we can merge this and test more before Apache Spark 4.0.0 release.
|
Thank you @dongjoon-hyun @LuciferYang |
|
Hi @yaooqinn , Thanks for bringing the HMS4 compatibility. I tried it and found one of the behaviour difference, reported in https://issues.apache.org/jira/browse/SPARK-50461. Could you please guide me, how I should be approaching this, to fix it? Thanks |
| state.err = new PrintStream(outputBuffer, true, UTF_8.name()) | ||
| val clz = state.getClass.getField("out").getType.asInstanceOf[Class[_ <: PrintStream]] | ||
| val ctor = clz.getConstructor(classOf[OutputStream], classOf[Boolean], classOf[String]) | ||
| state.getClass.getField("out").set(state, ctor.newInstance(outputBuffer, true, UTF_8.name())) |
There was a problem hiding this comment.
build fail ： ”the result type of an implicit conversion must be more specific than Object“
| @@ -1030,7 +1030,7 @@ private[spark] class HiveExternalCatalog(conf: SparkConf, hadoopConf: Configurat | |||
| } | |||
| val metaStoreParts = partsWithLocation | |||
| .map(p => p.copy(spec = toMetaStorePartitionSpec(p.spec))) | |||
| client.createPartitions(db, table, metaStoreParts, ignoreIfExists) | |||
| client.createPartitions(tableMeta, metaStoreParts, ignoreIfExists) | |||
What changes were proposed in this pull request?
This PR continues the work from #43064 and #45801 to support Hive Metastore Server 4.0. CHAR/VARCHAR type partition filter pushdown is not included in this PR, as it requires further investment.
Why are the changes needed?
Enhance the multiple hive metastore server support feature
Does this PR introduce any user-facing change?
no
How was this patch tested?
Passing HiveClient*Suites w/ 4.0
Was this patch authored or co-authored using generative AI tooling?
no