Spark Driver/Executors dont start in K8S via Kyuubi #7096

justlucknb · 2025-06-11T12:26:46Z

justlucknb
Jun 11, 2025

Hello. I have a problem with start Kyuubi session in K8S..
I started session via beeline. But i has no effect in create spark driver/executors via spark-submit.

ServiceAccount for statefull set have a full permissions in Namespace.
Network access to k8s master via pod working.
Kyuubi deployed via official helm chart.

Logs:

kyuubi@kyuubi-0:/opt/kyuubi$ ./bin/beeline -u jdbc:hive2://localhost:10009/default
Warn: Not find kyuubi environment file /opt/kyuubi/conf/kyuubi-env.sh, using default ones...
Connecting to jdbc:hive2://localhost:10009/default
2025-06-11 12:17:59.379 INFO KyuubiSessionManager-exec-pool: Thread-68 org.apache.kyuubi.operation.LaunchEngine: Processing anonymous's query[48fe5799-f97e-4387-8253-a02f15cbf565]: PENDING_STATE -> RUNNING_STATE, statement:
LaunchEngine
2025-06-11 12:17:59.380 INFO KyuubiSessionManager-exec-pool: Thread-68 org.apache.kyuubi.shaded.curator.framework.imps.CuratorFrameworkImpl: Starting
2025-06-11 12:17:59.381 INFO KyuubiSessionManager-exec-pool: Thread-68 org.apache.kyuubi.shaded.zookeeper.ZooKeeper: Initiating client connection, connectString=100.74.1.85:2181 sessionTimeout=60000 watcher=org.apache.kyuubi.shaded.curator.ConnectionState@4ac4c6e0
2025-06-11 12:17:59.382 INFO KyuubiSessionManager-exec-pool: Thread-68-SendThread(kyuubi-0.kyuubi-headless.spark-test.svc.k8s-xx:2181) org.apache.kyuubi.shaded.zookeeper.ClientCnxn: Opening socket connection to server kyuubi-0.kyuubi-headless.spark-test.svc.xx/100.74.1.85:2181. Will not attempt to authenticate using SASL (unknown error)
2025-06-11 12:17:59.383 INFO KyuubiSessionManager-exec-pool: Thread-68-SendThread(kyuubi-0.kyuubi-headless.spark-test.svc.xx:2181) org.apache.kyuubi.shaded.zookeeper.ClientCnxn: Socket connection established to kyuubi-0.kyuubi-headless.spark-test.svc.xx/100.74.1.85:2181, initiating session
2025-06-11 12:17:59.384 INFO KyuubiSessionManager-exec-pool: Thread-68-SendThread(kyuubi-0.kyuubi-headless.spark-test.svc.xx:2181) org.apache.kyuubi.shaded.zookeeper.ClientCnxn: Session establishment complete on server kyuubi-0.kyuubi-headless.spark-test.svc.xx/100.74.1.85:2181, sessionid = 0x110988064e60002, negotiated timeout = 60000
2025-06-11 12:17:59.384 INFO KyuubiSessionManager-exec-pool: Thread-68-EventThread org.apache.kyuubi.shaded.curator.framework.state.ConnectionStateManager: State change: CONNECTED
2025-06-11 12:17:59.391 INFO KyuubiSessionManager-exec-pool: Thread-68 org.apache.kyuubi.engine.ProcBuilder: Logging to /opt/kyuubi/work/anonymous/kyuubi-spark-sql-engine.log.1
2025-06-11 12:17:59.392 INFO KyuubiSessionManager-exec-pool: Thread-68 org.apache.kyuubi.engine.EngineRef: Launching engine:
/opt/kyuubi/externals/spark-3.4.2-bin-hadoop3/bin/spark-submit \
        --class org.apache.kyuubi.engine.spark.SparkSQLEngine \
        --conf spark.hive.server2.thrift.resultset.default.fetch.size=1000 \
        --conf spark.kyuubi.client.ipAddress=xxx \
        --conf spark.kyuubi.client.version=1.8.3 \
        --conf spark.kyuubi.engine.engineLog.path=/opt/kyuubi/work/anonymous/kyuubi-spark-sql-engine.log.1 \
        --conf spark.kyuubi.engine.share.level=USER \
        --conf spark.kyuubi.engine.submit.time=1749644279388 \
        --conf spark.kyuubi.engine.type=SPARK_SQL \
        --conf spark.kyuubi.ha.addresses=xxx:2181 \
        --conf spark.kyuubi.ha.engine.ref.id=515cc8cd-18ef-48e2-a12e-69392c846e23 \
        --conf spark.kyuubi.ha.namespace=/spark-test_1.8.3_USER_SPARK_SQL/anonymous/default \
        --conf spark.kyuubi.ha.zookeeper.auth.type=NONE \
        --conf spark.kyuubi.ha.zookeeper.namespace=spark-test \
        --conf spark.kyuubi.kubernetes.namespace=spark-test \
        --conf spark.kyuubi.server.ipAddress=0.0.0.0 \
        --conf spark.kyuubi.session.connection.url=0.0.0.0:10009 \
        --conf spark.kyuubi.session.engine.initialize.timeout=100000 \
        --conf spark.kyuubi.session.engine.open.max.attempts=1 \
        --conf spark.kyuubi.session.engine.share.level=CONNECTION \
        --conf spark.kyuubi.session.real.user=anonymous \
        --conf spark.app.name=kyuubi_USER_SPARK_SQL_anonymous_default_515cc8cd-18ef-48e2-a12e-69392c846e23 \
        --conf spark.driver.extraJavaOptions=-Divy.cache.dir=/tmp -Divy.home=/tmp \
        --conf spark.driver.host=kyuubi-hs.spark-test.svc \
        --conf spark.driver.memory=2g \
        --conf spark.driver.port=44104 \
        --conf spark.dynamicAllocation.enabled=true \
        --conf spark.dynamicAllocation.executorAllocationRatio=1 \
        --conf spark.dynamicAllocation.initialExecutors=2 \
        --conf spark.dynamicAllocation.maxExecutors=20 \
        --conf spark.dynamicAllocation.minExecutors=2 \
        --conf spark.dynamicAllocation.shuffleTracking.enabled=true \
        --conf spark.executor.instances=2 \
        --conf spark.hadoop.fs.s3a.access.key=xxx \
        --conf spark.hadoop.fs.s3a.endpoint=http://minio-test.minio-common-test:9000 \
        --conf spark.hadoop.fs.s3a.fast.upload=true \
        --conf spark.hadoop.fs.s3a.impl=org.apache.hadoop.fs.s3a.S3AFileSystem \
        --conf spark.hadoop.fs.s3a.path.style.access=true \
        --conf spark.hadoop.fs.s3a.secret.key=xxx \
        --conf spark.jars.packages=org.apache.hadoop:hadoop-aws:3.3.4,com.amazonaws:aws-java-sdk-bundle:1.12.262 \
        --conf spark.kubernetes.authenticate.caCertFile=/var/run/secrets/kubernetes.io/serviceaccount/ca.crt \
        --conf spark.kubernetes.authenticate.driver.serviceAccountName=spark \
        --conf spark.kubernetes.authenticate.oauthTokenFile=/var/run/secrets/kubernetes.io/serviceaccount/token \
        --conf spark.kubernetes.authenticate.serviceAccountName=kyuubi \
        --conf spark.kubernetes.container.image=registry.xx/spark:3.4.0 \
        --conf spark.kubernetes.driver.label.kyuubi-unique-tag=515cc8cd-18ef-48e2-a12e-69392c846e23 \
        --conf spark.kubernetes.driver.pod.name=$HOSTNAME \
        --conf spark.kubernetes.driver.podNamePrefix=kyuubi \
        --conf spark.kubernetes.executor.podNamePrefix=kyuubi \
        --conf spark.kubernetes.file.upload.path=local:///opt/spark/tmp \
        --conf spark.kubernetes.namespace=spark-test \
        --conf spark.master=k8s://https://kubernetes.default.svc:443 \
        --conf spark.submit.deployMode=cluster \
        --conf spark.kubernetes.driverEnv.SPARK_USER_NAME=anonymous \
        --conf spark.executorEnv.SPARK_USER_NAME=anonymous \
        --proxy-user anonymous /opt/kyuubi/externals/engines/spark/kyuubi-spark-sql-engine_2.12-1.8.3.jar
25/06/11 12:18:01 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
:: loading settings :: url = jar:file:/opt/kyuubi/externals/spark-3.4.2-bin-hadoop3/jars/ivy-2.5.1.jar!/org/apache/ivy/core/settings/ivysettings.xml
Ivy Default Cache set to: /home/kyuubi/.ivy2/cache
The jars for the packages stored in: /home/kyuubi/.ivy2/jars
org.apache.hadoop#hadoop-aws added as a dependency
com.amazonaws#aws-java-sdk-bundle added as a dependency
:: resolving dependencies :: org.apache.spark#spark-submit-parent-8ca5fcf2-22dc-4c26-96a9-ab39e380d10f;1.0
        confs: [default]
2025-06-11 12:19:39.518 INFO KyuubiSessionManager-exec-pool: Thread-68 org.apache.kyuubi.Utils: Loading Kyuubi properties from /opt/spark/conf/spark-defaults.conf
2025-06-11 12:19:39.520 WARN KyuubiSessionManager-exec-pool: Thread-68 org.apache.kyuubi.engine.KubernetesApplicationOperation: No application info found, trying to delete pod with label: kyuubi-unique-tag=515cc8cd-18ef-48e2-a12e-69392c846e23
2025-06-11 12:19:40.915 INFO Curator-Framework-0 org.apache.kyuubi.shaded.curator.framework.imps.CuratorFrameworkImpl: backgroundOperationsLoop exiting
2025-06-11 12:19:40.917 INFO KyuubiSessionManager-exec-pool: Thread-68 org.apache.kyuubi.shaded.zookeeper.ZooKeeper: Session: 0x110988064e60002 closed
2025-06-11 12:19:40.917 INFO KyuubiSessionManager-exec-pool: Thread-68-EventThread org.apache.kyuubi.shaded.zookeeper.ClientCnxn: EventThread shut down for session: 0x110988064e60002
2025-06-11 12:19:40.919 INFO KyuubiSessionManager-exec-pool: Thread-68 org.apache.kyuubi.operation.LaunchEngine: Processing anonymous's query[48fe5799-f97e-4387-8253-a02f15cbf565]: RUNNING_STATE -> ERROR_STATE, time taken: 101.539 seconds
Error: org.apache.kyuubi.KyuubiSQLException: Timeout(100000 ms, you can modify kyuubi.session.engine.initialize.timeout to change it) to launched SPARK_SQL engine with /opt/kyuubi/externals/spark-3.4.2-bin-hadoop3/bin/spark-submit \
        --class org.apache.kyuubi.engine.spark.SparkSQLEngine \
        --conf spark.hive.server2.thrift.resultset.default.fetch.size=1000 \
        --conf spark.kyuubi.client.ipAddress=xx \
        --conf spark.kyuubi.client.version=1.8.3 \
        --conf spark.kyuubi.engine.engineLog.path=/opt/kyuubi/work/anonymous/kyuubi-spark-sql-engine.log.1 \
        --conf spark.kyuubi.engine.share.level=USER \
        --conf spark.kyuubi.engine.submit.time=1749644279388 \
        --conf spark.kyuubi.engine.type=SPARK_SQL \
        --conf spark.kyuubi.ha.addresses=xx:2181 \
        --conf spark.kyuubi.ha.engine.ref.id=515cc8cd-18ef-48e2-a12e-69392c846e23 \
        --conf spark.kyuubi.ha.namespace=/spark-test_1.8.3_USER_SPARK_SQL/anonymous/default \
        --conf spark.kyuubi.ha.zookeeper.auth.type=NONE \
        --conf spark.kyuubi.ha.zookeeper.namespace=spark-test \
        --conf spark.kyuubi.kubernetes.namespace=spark-test \
        --conf spark.kyuubi.server.ipAddress=0.0.0.0 \
        --conf spark.kyuubi.session.connection.url=0.0.0.0:10009 \
        --conf spark.kyuubi.session.engine.initialize.timeout=100000 \
        --conf spark.kyuubi.session.engine.open.max.attempts=1 \
        --conf spark.kyuubi.session.engine.share.level=CONNECTION \
        --conf spark.kyuubi.session.real.user=anonymous \
        --conf spark.app.name=kyuubi_USER_SPARK_SQL_anonymous_default_515cc8cd-18ef-48e2-a12e-69392c846e23 \
        --conf spark.driver.extraJavaOptions=-Divy.cache.dir=/tmp -Divy.home=/tmp \
        --conf spark.driver.host=kyuubi-hs.spark-test.svc \
        --conf spark.driver.memory=2g \
        --conf spark.driver.port=44104 \
        --conf spark.dynamicAllocation.enabled=true \
        --conf spark.dynamicAllocation.executorAllocationRatio=1 \
        --conf spark.dynamicAllocation.initialExecutors=2 \
        --conf spark.dynamicAllocation.maxExecutors=20 \
        --conf spark.dynamicAllocation.minExecutors=2 \
        --conf spark.dynamicAllocation.shuffleTracking.enabled=true \
        --conf spark.executor.instances=2 \
        --conf spark.hadoop.fs.s3a.access.key=xx \
        --conf spark.hadoop.fs.s3a.endpoint=http://minio-test.minio-common-test:9000 \
        --conf spark.hadoop.fs.s3a.fast.upload=true \
        --conf spark.hadoop.fs.s3a.impl=org.apache.hadoop.fs.s3a.S3AFileSystem \
        --conf spark.hadoop.fs.s3a.path.style.access=true \
        --conf spark.hadoop.fs.s3a.secret.key=xx \
        --conf spark.jars.packages=org.apache.hadoop:hadoop-aws:3.3.4,com.amazonaws:aws-java-sdk-bundle:1.12.262 \
        --conf spark.kubernetes.authenticate.caCertFile=/var/run/secrets/kubernetes.io/serviceaccount/ca.crt \
        --conf spark.kubernetes.authenticate.driver.serviceAccountName=spark \
        --conf spark.kubernetes.authenticate.oauthTokenFile=/var/run/secrets/kubernetes.io/serviceaccount/token \
        --conf spark.kubernetes.authenticate.serviceAccountName=kyuubi \
        --conf spark.kubernetes.container.image=registry.xx/spark:3.4.0 \
        --conf spark.kubernetes.driver.label.kyuubi-unique-tag=515cc8cd-18ef-48e2-a12e-69392c846e23 \
        --conf spark.kubernetes.driver.pod.name=$HOSTNAME \
        --conf spark.kubernetes.driver.podNamePrefix=kyuubi \
        --conf spark.kubernetes.executor.podNamePrefix=kyuubi \
        --conf spark.kubernetes.file.upload.path=local:///opt/spark/tmp \
        --conf spark.kubernetes.namespace=spark-test \
        --conf spark.master=k8s://https://kubernetes.default.svc:443 \
        --conf spark.submit.deployMode=cluster \
        --conf spark.kubernetes.driverEnv.SPARK_USER_NAME=anonymous \
        --conf spark.executorEnv.SPARK_USER_NAME=anonymous \
        --proxy-user anonymous /opt/kyuubi/externals/engines/spark/kyuubi-spark-sql-engine_2.12-1.8.3.jar. (false,[KubernetesInfo(None,Some(spark-test))] Operation of deleted pod with label: kyuubi-unique-tag=515cc8cd-18ef-48e2-a12e-69392c846e23 is completed)
        at org.apache.kyuubi.KyuubiSQLException$.apply(KyuubiSQLException.scala:70)
        at org.apache.kyuubi.engine.EngineRef.$anonfun$create$1(EngineRef.scala:243)
        at org.apache.kyuubi.ha.client.zookeeper.ZookeeperDiscoveryClient.tryWithLock(ZookeeperDiscoveryClient.scala:166)
        at org.apache.kyuubi.engine.EngineRef.tryWithLock(EngineRef.scala:174)
        at org.apache.kyuubi.engine.EngineRef.create(EngineRef.scala:179)
        at org.apache.kyuubi.engine.EngineRef.$anonfun$getOrCreate$1(EngineRef.scala:309)
        at scala.Option.getOrElse(Option.scala:189)
        at org.apache.kyuubi.engine.EngineRef.getOrCreate(EngineRef.scala:309)
        at org.apache.kyuubi.session.KyuubiSessionImpl.$anonfun$openEngineSession$2(KyuubiSessionImpl.scala:150)
        at org.apache.kyuubi.session.KyuubiSessionImpl.$anonfun$openEngineSession$2$adapted(KyuubiSessionImpl.scala:124)
        at org.apache.kyuubi.ha.client.DiscoveryClientProvider$.withDiscoveryClient(DiscoveryClientProvider.scala:36)
        at org.apache.kyuubi.session.KyuubiSessionImpl.$anonfun$openEngineSession$1(KyuubiSessionImpl.scala:124)
        at org.apache.kyuubi.session.KyuubiSession.handleSessionException(KyuubiSession.scala:49)
        at org.apache.kyuubi.session.KyuubiSessionImpl.openEngineSession(KyuubiSessionImpl.scala:124)
        at org.apache.kyuubi.operation.LaunchEngine.$anonfun$runInternal$1(LaunchEngine.scala:60)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:750)
Caused by: org.apache.kyuubi.KyuubiSQLException: Failed to detect the root cause, please check /opt/kyuubi/work/anonymous/kyuubi-spark-sql-engine.log.1 at server side if necessary. The last 10 line(s) of log are:
25/06/11 12:18:01 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
:: loading settings :: url = jar:file:/opt/kyuubi/externals/spark-3.4.2-bin-hadoop3/jars/ivy-2.5.1.jar!/org/apache/ivy/core/settings/ivysettings.xml
Ivy Default Cache set to: /home/kyuubi/.ivy2/cache
The jars for the packages stored in: /home/kyuubi/.ivy2/jars
org.apache.hadoop#hadoop-aws added as a dependency
com.amazonaws#aws-java-sdk-bundle added as a dependency
:: resolving dependencies :: org.apache.spark#spark-submit-parent-8ca5fcf2-22dc-4c26-96a9-ab39e380d10f;1.0
        confs: [default]
        at org.apache.kyuubi.KyuubiSQLException$.apply(KyuubiSQLException.scala:70)
        at org.apache.kyuubi.engine.ProcBuilder.getError(ProcBuilder.scala:275)
        at org.apache.kyuubi.engine.ProcBuilder.getError$(ProcBuilder.scala:268)
        at org.apache.kyuubi.engine.spark.SparkProcessBuilder.getError(SparkProcessBuilder.scala:42)
        at org.apache.kyuubi.engine.EngineRef.$anonfun$create$1(EngineRef.scala:246)
        ... 18 more (state=,code=0)

Answered by pan3793

Jun 12, 2025

My k8s not have access to internet only LAN,

oh, then you can not use the following conf. you can download and put them under $SPARK_HOME/jars instead

--conf spark.jars.packages=org.apache.hadoop:hadoop-aws:3.3.4,com.amazonaws:aws-java-sdk-bundle:1.12.262

View full answer

pan3793 · 2025-06-11T12:47:48Z

pan3793
Jun 11, 2025
Collaborator

it says:

Failed to detect the root cause, please check /opt/kyuubi/work/anonymous/kyuubi-spark-sql-engine.log.1 at server side if necessary.

then what does /opt/kyuubi/work/anonymous/kyuubi-spark-sql-engine.log.1 say?

0 replies

justlucknb · 2025-06-12T07:37:21Z

justlucknb
Jun 12, 2025
Author

@pan3793 Hello!
the fact of the matter is that there is no useful information in this log, the last 10 lines in stdout are the entire log.

Before that, there was a problem with the list pod in namespace, but I fixed it.

:: loading settings :: url = jar:file:/opt/kyuubi/externals/spark-3.4.2-bin-hadoop3/jars/ivy-2.5.1.jar!/org/apache/ivy/core/settings/ivysettings.xml
Ivy Default Cache set to: /home/kyuubi/.ivy2/cache
The jars for the packages stored in: /home/kyuubi/.ivy2/jars
org.apache.hadoop#hadoop-aws added as a dependency
com.amazonaws#aws-java-sdk-bundle added as a dependency
:: resolving dependencies :: org.apache.spark#spark-submit-parent-8ca5fcf2-22dc-4c26-96a9-ab39e380d10f;1.0
        confs: [default]
        at org.apache.kyuubi.KyuubiSQLException$.apply(KyuubiSQLException.scala:70)
        at org.apache.kyuubi.engine.ProcBuilder.getError(ProcBuilder.scala:275)
        at org.apache.kyuubi.engine.ProcBuilder.getError$(ProcBuilder.scala:268)
        at org.apache.kyuubi.engine.spark.SparkProcessBuilder.getError(SparkProcessBuilder.scala:42)
        at org.apache.kyuubi.engine.EngineRef.$anonfun$create$1(EngineRef.scala:246)
        ... 18 more (state=,code=0)

8 replies

pan3793 Jun 12, 2025
Collaborator

yes, and use the same OS user that runs the Kyuubi process.

justlucknb Jun 12, 2025
Author

same, long waiting and timeout. My k8s not have access to internet only LAN, maybe for start stark-sumbit i need internet-access?

pan3793 Jun 12, 2025
Collaborator

Debug or even trace level logs might be helpful. K8s has super flexible infrastructures compared to Hadoop YARN, I suggest you follow the Spark's docs to run Spark Pi first.

pan3793 Jun 12, 2025
Collaborator

My k8s not have access to internet only LAN,

oh, then you can not use the following conf. you can download and put them under $SPARK_HOME/jars instead

--conf spark.jars.packages=org.apache.hadoop:hadoop-aws:3.3.4,com.amazonaws:aws-java-sdk-bundle:1.12.262

Answer selected by justlucknb

justlucknb Jun 16, 2025
Author

@pan3793 thanks a lot for help, i put jars into jars home and its works!

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Spark Driver/Executors dont start in K8S via Kyuubi #7096

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{editor}}'s edit

{{editor}}'s edit

Uh oh!

Replies: 2 comments 8 replies

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{editor}}'s edit

{{editor}}'s edit

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Select a reply

Uh oh!

Spark Driver/Executors dont start in K8S via Kyuubi #7096

Uh oh!

Uh oh!

justlucknb Jun 11, 2025

Replies: 2 comments · 8 replies

Uh oh!

pan3793 Jun 11, 2025 Collaborator

Uh oh!

justlucknb Jun 12, 2025 Author

Uh oh!

pan3793 Jun 12, 2025 Collaborator

Uh oh!

justlucknb Jun 12, 2025 Author

Uh oh!

Uh oh!

pan3793 Jun 12, 2025 Collaborator

Uh oh!

pan3793 Jun 12, 2025 Collaborator

Uh oh!

justlucknb Jun 16, 2025 Author

justlucknb
Jun 11, 2025

Replies: 2 comments 8 replies

pan3793
Jun 11, 2025
Collaborator

justlucknb
Jun 12, 2025
Author

pan3793 Jun 12, 2025
Collaborator

justlucknb Jun 12, 2025
Author

pan3793 Jun 12, 2025
Collaborator

pan3793 Jun 12, 2025
Collaborator

justlucknb Jun 16, 2025
Author