[SPARK-51041][BUILD] Add `hive-llap-client` and `hive-llap-common` as test dependency of `hive-thriftserver` #49736

LuciferYang · 2025-01-30T16:16:00Z

What changes were proposed in this pull request?

This pr aims to add hive-llap-client and hive-llap-common as test dependency of hive-thriftserver

Why are the changes needed?

Fix maven test of hive-thriftserver. Due to the lack of these test dependencies, testing hive-thriftserver using Maven will hang.

Does this PR introduce any user-facing change?

No

How was this patch tested?

Pass GitHub Actions
Manually check: After adding the test dependencies, when testing the hive-thriftserver module using Maven, HiveThriftBinaryServerSuite will no longer hang.

build/mvn -Pyarn -Pkubernetes -Pvolcano -Phive -Phive-thriftserver -Phadoop-cloud -Pjvm-profiler -Pspark-ganglia-lgpl -Pkinesis-asl clean install
build/mvn -pl sql/hive-thriftserver -Pyarn -Pkubernetes -Pvolcano -Phive -Phive-thriftserver -Phadoop-cloud -Pjvm-profiler -Pspark-ganglia-lgpl -Pkinesis-asl clean install -fae

Was this patch authored or co-authored using generative AI tooling?

No

dongjoon-hyun · 2025-01-30T16:28:08Z

Thank you so much, @LuciferYang . I also looking at those failure.

LuciferYang · 2025-01-30T16:50:30Z

GenericUDTFGetSplits imports org.apache.hadoop.hive.llap.security.LlapSigner, which comes from llap-common.

https://github.com/apache/hive/blob/5160d3af392248255f68e41e1e0557eae4d95273/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDTFGetSplits.java#L44-L58

FunctionRegistry will load GenericUDTFGetSplits in a static block.

https://github.com/apache/hive/blob/5160d3af392248255f68e41e1e0557eae4d95273/ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java#L500

At the same time, I saw the following content in the log:

Exception in thread "HiveServer2-Handler-Pool: Thread-66" java.lang.RuntimeException: java.lang.NoClassDefFoundError: org/apache/hadoop/hive/llap/security/LlapSigner$Signable
	at org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:76)
	at org.apache.hive.service.cli.session.HiveSessionProxy.lambda$invoke$0(HiveSessionProxy.java:58)
	at java.base/java.security.AccessController.doPrivileged(AccessController.java:712)
	at java.base/javax.security.auth.Subject.doAs(Subject.java:439)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1953)
	at org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:58)
	at jdk.proxy2/jdk.proxy2.$Proxy46.close(Unknown Source)
	at org.apache.hive.service.cli.session.SessionManager.closeSession(SessionManager.java:304)
	at org.apache.spark.sql.hive.thriftserver.SparkSQLSessionManager.closeSession(SparkSQLSessionManager.scala:92)
	at org.apache.hive.service.cli.CLIService.closeSession(CLIService.java:244)
	at org.apache.hive.service.cli.thrift.ThriftCLIService$1.deleteContext(ThriftCLIService.java:120)
	at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:840)
Caused by: java.lang.NoClassDefFoundError: org/apache/hadoop/hive/llap/security/LlapSigner$Signable
	at java.base/java.lang.Class.getDeclaredConstructors0(Native Method)
	at java.base/java.lang.Class.privateGetDeclaredConstructors(Class.java:3373)
	at java.base/java.lang.Class.getConstructor0(Class.java:3578)
	at java.base/java.lang.Class.getDeclaredConstructor(Class.java:2754)
	at org.apache.hive.common.util.ReflectionUtil.newInstance(ReflectionUtil.java:79)
	at org.apache.hadoop.hive.ql.exec.Registry.registerGenericUDTF(Registry.java:208)
	at org.apache.hadoop.hive.ql.exec.Registry.registerGenericUDTF(Registry.java:201)
	at org.apache.hadoop.hive.ql.exec.FunctionRegistry.<clinit>(FunctionRegistry.java:500)
	at org.apache.hadoop.hive.ql.metadata.Hive.reloadFunctions(Hive.java:247)
	at org.apache.hadoop.hive.ql.metadata.Hive.registerAllFunctionsOnce(Hive.java:231)
	at org.apache.hadoop.hive.ql.metadata.Hive.<init>(Hive.java:395)
	at org.apache.hadoop.hive.ql.metadata.Hive.create(Hive.java:339)
	at org.apache.hadoop.hive.ql.metadata.Hive.getInternal(Hive.java:319)
	at org.apache.hadoop.hive.ql.metadata.Hive.get(Hive.java:288)
	at org.apache.hadoop.hive.ql.session.SessionState.unCacheDataNucleusClassLoaders(SessionState.java:1596)
	at org.apache.hadoop.hive.ql.session.SessionState.close(SessionState.java:1586)
	at org.apache.hive.service.cli.session.HiveSessionImpl.close(HiveSessionImpl.java:676)
	at org.apache.hive.service.cli.session.HiveSessionImplwithUGI.close(HiveSessionImplwithUGI.java:87)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:569)
	at org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:71)
	... 14 more
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.hive.llap.security.LlapSigner$Signable
	at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:641)
	at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:188)
	at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:525)
	... 37 more

It seems that this caused the FunctionRegistry initialization to fail, so it looks like we can't remove the hive-llap-common dependency from the compile scope, right? @dongjoon-hyun

dongjoon-hyun · 2025-01-30T16:59:38Z

I'll reply on the original PR, @LuciferYang ~

dongjoon-hyun · 2025-01-30T17:06:35Z

I verified this manually.

$ build/mvn -Phive-thriftserver install -DskipTests
$ build/mvn -pl sql/hive-thriftserver -Phive-thriftserver install -fae
...
HiveThriftBinaryServerSuite:
- SPARK-17819: Support default database in connection URIs
- GetInfo Thrift API
- SPARK-16563 ThriftCLIService FetchResults repeat fetching result
- Support beeline --hiveconf and --hivevar
- JDBC query execution
- Checks Hive version
- SPARK-3004 regression: result set containing NULL
- SPARK-4292 regression: result set iterator issue
- SPARK-4309 regression: Date type support
- SPARK-4407 regression: Complex type support
- SPARK-12143 regression: Binary type support
- test multiple session
- test jdbc cancel !!! IGNORED !!!
- test add jar
- Checks Hive version via SET -v
- Checks Hive version via SET
- SPARK-11595 ADD JAR with input path having URL scheme
- SPARK-11043 check operation log root directory
- SPARK-23547 Cleanup the .pipeout file when the Hive Session closed
- SPARK-24829 Checks cast as float
- SPARK-28463: Thriftserver throws BigDecimal incompatible with HiveDecimal
- Support interval type
- Query Intervals in VIEWs through thrift server
- ThriftCLIService FetchResults FETCH_FIRST, FETCH_NEXT, FETCH_PRIOR
- SPARK-29492: use add jar in sync mode
- SPARK-31859 Thriftserver works with spark.sql.datetime.java8API.enabled=true
- SPARK-31861 Thriftserver respects spark.sql.session.timeZone
- SPARK-31863 Session conf should persist between Thriftserver worker threads
- SPARK-30808: use Java 8 time API and Proleptic Gregorian calendar by default
- SPARK-26533: Support query auto timeout cancel on thriftserver - setQueryTimeout
- SPARK-26533: Support query auto timeout cancel on thriftserver - SQLConf
CliSuite:
- load warehouse dir from hive-site.xml
- ...

dongjoon-hyun

+1, LGTM. Thank you, @LuciferYang .

Let's merge this first. Although we change our hive-llap-common dependency later, this explicit depdency change will remain.

… test dependency of `hive-thriftserver` ### What changes were proposed in this pull request? This pr aims to add `hive-llap-client` and `hive-llap-common` as test dependency of `hive-thriftserver` ### Why are the changes needed? Fix maven test of `hive-thriftserver`. Due to the lack of these test dependencies, testing `hive-thriftserver` using Maven will hang. ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? - Pass GitHub Actions - Manually check: After adding the test dependencies, when testing the `hive-thriftserver` module using Maven, `HiveThriftBinaryServerSuite` will no longer hang. ``` build/mvn -Pyarn -Pkubernetes -Pvolcano -Phive -Phive-thriftserver -Phadoop-cloud -Pjvm-profiler -Pspark-ganglia-lgpl -Pkinesis-asl clean install build/mvn -pl sql/hive-thriftserver -Pyarn -Pkubernetes -Pvolcano -Phive -Phive-thriftserver -Phadoop-cloud -Pjvm-profiler -Pspark-ganglia-lgpl -Pkinesis-asl clean install -fae ``` ### Was this patch authored or co-authored using generative AI tooling? No Closes #49736 from LuciferYang/SPARK-51041. Authored-by: yangjie01 <yangjie01@baidu.com> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org> (cherry picked from commit ecf6851) Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>

LuciferYang · 2025-01-30T17:10:02Z

Thanks @dongjoon-hyun

… test dependency of `hive-thriftserver` ### What changes were proposed in this pull request? This pr aims to add `hive-llap-client` and `hive-llap-common` as test dependency of `hive-thriftserver` ### Why are the changes needed? Fix maven test of `hive-thriftserver`. Due to the lack of these test dependencies, testing `hive-thriftserver` using Maven will hang. ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? - Pass GitHub Actions - Manually check: After adding the test dependencies, when testing the `hive-thriftserver` module using Maven, `HiveThriftBinaryServerSuite` will no longer hang. ``` build/mvn -Pyarn -Pkubernetes -Pvolcano -Phive -Phive-thriftserver -Phadoop-cloud -Pjvm-profiler -Pspark-ganglia-lgpl -Pkinesis-asl clean install build/mvn -pl sql/hive-thriftserver -Pyarn -Pkubernetes -Pvolcano -Phive -Phive-thriftserver -Phadoop-cloud -Pjvm-profiler -Pspark-ganglia-lgpl -Pkinesis-asl clean install -fae ``` ### Was this patch authored or co-authored using generative AI tooling? No Closes apache#49736 from LuciferYang/SPARK-51041. Authored-by: yangjie01 <yangjie01@baidu.com> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org> (cherry picked from commit b87a6cc) Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>

init

55a4b28

LuciferYang marked this pull request as draft January 30, 2025 16:16

github-actions bot added SQL BUILD labels Jan 30, 2025

dongjoon-hyun marked this pull request as ready for review January 30, 2025 16:46

dongjoon-hyun marked this pull request as draft January 30, 2025 16:47

LuciferYang mentioned this pull request Jan 30, 2025

[SPARK-51029][BUILD] Remove hive-llap-common compile dependency #49725

Closed

LuciferYang marked this pull request as ready for review January 30, 2025 17:00

dongjoon-hyun approved these changes Jan 30, 2025

View reviewed changes

dongjoon-hyun closed this in ecf6851 Jan 30, 2025

LuciferYang mentioned this pull request Mar 11, 2025

[SPARK-51466][SQL][HIVE] Eliminate Hive built-in UDFs initialization on Hive UDF evaluation #50232

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[SPARK-51041][BUILD] Add `hive-llap-client` and `hive-llap-common` as test dependency of `hive-thriftserver` #49736

[SPARK-51041][BUILD] Add `hive-llap-client` and `hive-llap-common` as test dependency of `hive-thriftserver` #49736

Uh oh!

LuciferYang commented Jan 30, 2025

Uh oh!

dongjoon-hyun commented Jan 30, 2025

Uh oh!

LuciferYang commented Jan 30, 2025 •

edited

Loading

Uh oh!

dongjoon-hyun commented Jan 30, 2025

Uh oh!

dongjoon-hyun commented Jan 30, 2025

Uh oh!

dongjoon-hyun left a comment

Uh oh!

LuciferYang commented Jan 30, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

[SPARK-51041][BUILD] Add hive-llap-client and hive-llap-common as test dependency of hive-thriftserver #49736

[SPARK-51041][BUILD] Add hive-llap-client and hive-llap-common as test dependency of hive-thriftserver #49736

Uh oh!

Conversation

LuciferYang commented Jan 30, 2025

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Was this patch authored or co-authored using generative AI tooling?

Uh oh!

dongjoon-hyun commented Jan 30, 2025

Uh oh!

LuciferYang commented Jan 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dongjoon-hyun commented Jan 30, 2025

Uh oh!

dongjoon-hyun commented Jan 30, 2025

Uh oh!

dongjoon-hyun left a comment

Choose a reason for hiding this comment

Uh oh!

LuciferYang commented Jan 30, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

[SPARK-51041][BUILD] Add `hive-llap-client` and `hive-llap-common` as test dependency of `hive-thriftserver` #49736

[SPARK-51041][BUILD] Add `hive-llap-client` and `hive-llap-common` as test dependency of `hive-thriftserver` #49736

LuciferYang commented Jan 30, 2025 •

edited

Loading