[SPARK-45265][SQL][WIP] Supporting Hive 4.0 metastore

[attilapiros]

What changes were proposed in this pull request?

Supporting Hive 4.0 metastore where partition filters even for CHAR and a VARCHAR types can be pushed down.

Hive 4.0 is still beta! This is why this is a work on progress PR.

Why are the changes needed?

Supporting more Hive versions (with extra performance improvement) is good for our users.

Does this PR introduce any user-facing change?

Yes. Regarding supporting Hive 4.0 metastore the documentation is updated accordingly.

How was this patch tested?

Manually

I used the docker image of apache/hive:4.0.0-beta-1 for starting a metastore and a hiveserver2 (along with a hadoop3 docker image).

Created a table:

CREATE EXTERNAL TABLE testTable1 ( 
  column1 String 
) PARTITIONED BY (partColumn1 CHAR(30), partColumn2 VARCHAR(30)) LOCATION 'hdfs://hadoop3:8020/tmp/hive_external/';

Inserted some values in beeline:

insert into table testtable1 values ("column1_v1", "partcolumn1_v1", "partcolumn2_v1"), ("column1_v2", "partcolumn1_v2", "partcolumn2_v2");

Started my spark in the hiveserver2 container as:

./bin/spark-shell --conf spark.sql.hive.metastore.version=4.0.0 --conf spark.sql.hive.metastore.jars="/opt/hive/lib/*"

Run the query as:

scala> sql("select * from testtable1 where partcolumn1 = 'partcolumn1_v1' and partcolumn2 = 'partcolumn2_v1'").show
Hive Session ID = 6846fe0e-968a-474d-afec-4f67b3a2a274
+----------+--------------------+--------------+
|   column1|         partcolumn1|   partcolumn2|
+----------+--------------------+--------------+
|column1_v1|partcolumn1_v1   ...|partcolumn2_v1|
+----------+--------------------+--------------+

And check the HMS calls in the metastore container in the file /tmp/hive/hive.log:

...
2023-09-22T21:06:34,293  INFO [Metastore-Handler-Pool: Thread-1356] HiveMetaStore.audit: ugi=hive       ip=172.30.0.5   cmd=source:172.30.0.5 get_partitions_by_filter : tbl=hive.default.testtable1
...

Which contains the expected get_partitions_by_filter.

Was this patch authored or co-authored using generative AI tooling?

No.

[dongjoon-hyun]

Nice. I have a few comments.

• Are you using the current beta-1?
• Is there a timeline for Hive 4.0 GA?
• Although I know that you filed this as Bug for some old releases, but I believe this PR should be a subtask for Apache Spark 4.0.0 because there is no existing Spark users with Apache Hive 4.0.0 Megastore.

[attilapiros]

@dongjoon-hyun

Thanks!

Are you using the current beta-1?

Yes.

Is there a timeline for Hive 4.0 GA?

I will ask around but as I know they still have some blockers.

Although I know that you filed this as Bug for some old releases, but I believe this PR should be a subtask for Apache Spark 4.0.0 because there is no existing Spark users with Apache Hive 4.0.0 Megastore.

Sorry that was a mistake of mine thanks for fixing that in Jira.

[dongjoon-hyun]

Thank you. And, if you are fine with Apache Spark 4.0, that's great! I was worried. 😄

[HyukjinKwon]

cc @wangyum too

[attilapiros]

@dongjoon-hyun
Regarding Hive 4.0 there is the Test with the TPC-DS benchmark to be done but when the release is out I will update this PR.

[dongjoon-hyun]

Thank you so much for keeping us up-to-date, @attilapiros !

Regarding Hive 4.0 there is the Test with the TPC-DS benchmark to be done but when the release is out I will update this PR.

[dongjoon-hyun]

Is there any update for Apache Hive 4.0, @attilapiros ?

[attilapiros]

Is there any update for Apache Hive 4.0, @attilapiros ?

@dongjoon-hyun they still having some more issues to solve (as I see some TPC-DS queries performance issues):
https://lists.apache.org/thread/3okjgw3y6tso7l2rg3hhy8lccp6d6mmy

[dongjoon-hyun]

Thank you for the updates and the link, @attilapiros .

[github-actions]

We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable.
If you'd like to revive this PR, please reopen it and ask a committer to remove the Stale tag!