Support Hive3 when using Iceberg with Spark

[yabola]

Feature Request / Improvement

Spark uses an isolated classloader to load Hive3-related jars (specified via spark.sql.hive.metastore.jars). However, this mechanism only functions within HiveExternalCatalog and does not work for Iceberg. This is because Spark restores the classloader for the main thread after initializing the Hive client (as seen in this code snippet). As a result, the JVM in Spark always uses the Hive2 implementation class, making it impossible to leverage Hive3 features in Iceberg with Spark.

While the Hive2 client is generally compatible with Hive3, but some features are missing. For example, the ability to use catalog names to isolate databases and tables (e.g., the metastore.catalog.default property) is only supported in Hive3's metadata client.

To address this limitation, we need to introduce a mechanism similar to Spark's isolated classloader approach to load Hive3 jars specifically for Iceberg. This would enable Iceberg to fully utilize Hive3's advanced features when integrated with Spark.

Query engine

None

Willingness to contribute

• I can contribute this improvement/feature independently
• I would be willing to contribute this improvement/feature with guidance from the Iceberg community
• I cannot contribute this improvement/feature at this time

[yabola]

I have tried loading hiveCatalog using a separate classloader and it can work . I hope to get more opinions from the community, thanks!

PS: #1478 only fix the case when the JVM is Hive3 , but it does not work in the Spark.

[pvary]

Isn't this something which should be solved by Spark?

[yabola]

@pvary Yes, Spark needs to natively support Hive3(without isolate classloader). But upgrade spark to Hive3 needs huge efforts , spark community currently does not see the possibility of upgrading to Hive 3 in the long term (and Spark itself already provides classloader way).

[pvary]

How complicated would a classloader solution look like?

[yabola]

@pvary I can draft a PR for your review later. I test it locally, it would not be a big change, thanks to Iceberg's good encapsulation of the catalog implementation ( hive-metastore module). It can be done by passing the isolate classloader to HiveCatalog and load hive by this classloader.

[pvary]

It would be nice to see if this solution would work.
We also have issues with supporting multiple Hive versions, and if this could solve that issue as well, that would be great!

Thanks @yabola !