Background

This is simple project to verify SPARK-48547.

In local / K8s cluster mode, unlike YARN, the non-daemon thread blocks the driver JVM exit even if SparkContext is stopped. This is a challenge for user to migrate their Spark workloads from YARN to K8s, especially when non-daemon threads are created by third-party libraries.

Test

To verify the patch, build this project and submit it to Spark.

./mvnw clean package

When submit to Spark 4.0.0 in local mode

spark-4.0.0-bin-hadoop3/bin/spark-submit \
  --master local \
  --class io.github.pan3793.Main \
  target/SPARK-48547-1.0.jar --stop-spark-context-at-the-end-of-main true

Driver JVM will not exit even after SparkContext is stopped.

...
SparkSession created
Waiting 10s
Hello from non-daemon thread
Hello from non-daemon thread
Hello from non-daemon thread
Hello from non-daemon thread
25/08/21 18:35:43 INFO SparkContext: SparkContext is stopping with exitCode 0 from stop at Main.scala:37.
25/08/21 18:35:43 INFO SparkUI: Stopped Spark web UI at http://10.242.159.140:4040
25/08/21 18:35:43 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!
25/08/21 18:35:43 INFO MemoryStore: MemoryStore started with capacity 2.2 GiB
25/08/21 18:35:43 INFO MemoryStore: MemoryStore cleared
25/08/21 18:35:43 INFO BlockManager: BlockManager stopped
25/08/21 18:35:43 INFO BlockManagerMaster: BlockManagerMaster stopped
25/08/21 18:35:43 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!
25/08/21 18:35:43 INFO SparkContext: Successfully stopped SparkContext
SparkSession stopped
Main exited
Hello from non-daemon thread
Hello from non-daemon thread
Hello from non-daemon thread
Hello from non-daemon thread
Hello from non-daemon thread
Hello from non-daemon thread
Hello from non-daemon thread
Hello from non-daemon thread
<repeat forerver ...>

When submit to Spark with SPARK-48547 in local mode

spark-4.1.0-SNAPSHOT-bin-SPARK-48547/bin/spark-submit \
  --master local \
  --class io.github.pan3793.Main \
  --conf spark.submit.callSystemExitOnMainExit=true \
  target/SPARK-48547-1.0.jar --stop-spark-context-at-the-end-of-main true

After SparkContext is stopped, Driver JVM exits after the configured delay duration.

...
SparkSession created
Waiting 10s
Hello from non-daemon thread
Hello from non-daemon thread
Hello from non-daemon thread
Hello from non-daemon thread
25/08/21 18:34:58 INFO SparkContext: SparkContext is stopping with exitCode 0 from stop at Main.scala:37.
25/08/21 18:34:58 INFO SparkUI: Stopped Spark web UI at http://10.242.159.140:4040
25/08/21 18:34:58 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!
25/08/21 18:34:58 INFO MemoryStore: MemoryStore started with capacity 434.4 MiB
25/08/21 18:34:58 INFO MemoryStore: MemoryStore cleared
25/08/21 18:34:58 INFO BlockManager: BlockManager stopped
25/08/21 18:34:58 INFO BlockManagerMaster: BlockManagerMaster stopped
25/08/21 18:34:58 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!
25/08/21 18:34:58 INFO SparkContext: Successfully stopped SparkContext (Uptime: 11212 ms)
SparkSession stopped
Main exited
Calling System.exit() with exit code 0 because main spark.submit.callSystemExitOnMainExit=true
25/08/21 18:34:58 INFO ShutdownHookManager: Shutdown hook called
25/08/21 18:34:58 INFO ShutdownHookManager: Deleting directory /private/var/folders/kz/qv5g7w5s1rgb_9d4mpym2v1h0000gn/T/spark-61b0a852-bf6d-4c97-b33d-7c4e51b1a25e
25/08/21 18:34:58 INFO ShutdownHookManager: Deleting directory /private/var/folders/kz/qv5g7w5s1rgb_9d4mpym2v1h0000gn/T/spark-b0c27d29-e009-4c88-8421-2386f887a9f2
<JVM exited>

Same result can be reproduced on K8s cluster mode, while YARN cluster mode does not have such issue, driver JVM will exited immediately once SparkContext is stopped.