[SPARK-54087][CORE] Spark Executor launch task failed should return task killed message #52792

AngersZhuuuu · 2025-10-30T09:02:52Z

What changes were proposed in this pull request?

When not enable speculation ,executor launch task failed caused by OOM,

DAGScheduler won't know task failed and won't scheduler again, causing application stuck

Why are the changes needed?

Some case will causing application stuck missing SLA

Does this PR introduce any user-facing change?

No

How was this patch tested?

UT + MT

Test code

import org.apache.spark.sql.functions.udf
import java.util.concurrent.TimeUnit// 定义一个会sleep的UDF

val sleepUdf = udf((value: String) => {
  TimeUnit.SECONDS.sleep(10)
  value
})

val df = spark.range(0, 2500).repartition(25).withColumn("value", (col("id") % 10).cast("string")).withColumn("value", sleepUdf(col("value")))df.createOrReplaceTempView("test_table")

val result = spark.sql("SELECT value, COUNT(*) as cnt FROM test_table GROUP BY value ORDER BY cnt DESC")

result.show()

task failed and re-scheduled

Was this patch authored or co-authored using generative AI tooling?

No

…ed message

AngersZhuuuu · 2025-11-03T06:22:36Z

@holdenk @cloud-fan How about this issue?

holdenk · 2025-11-03T06:26:54Z

Try and make tests part of a test suite not just manual when you can.

cloud-fan · 2025-11-04T18:42:49Z

application stuck is a serious issue, can we constuct a test case?

cc @jiangxb1987 @Ngone51 .

Ngone51 · 2025-11-07T07:40:36Z

core/src/main/scala/org/apache/spark/executor/Executor.scala

+    try {
+      threadPool.execute(tr)
+    } catch {
+      case e: Throwable =>


In the case of the uncontrolled OutOfMemoryError, I don't think we can guarantee the status update always be sent successfully. I actually would prefer shutdown the executor itself with an explicitly error code, e.g., SparkExitCode.OOM in the case of OutOfMemoryError.

You mean catch the error, is it;s OutOfMemoryError then shutdown the executor itself with SparkExitCode.OOM error code, how about other error?

Yes. Other errors can be handled in the current way.

Or we can mimic SparkUncaughtExceptionHandler. In the case of OutOfMemoryError, we can try to sent the status update firstly. If OutOfMemoryError is thrown again, we fail the executor.

UncaughtExceptionHandler seems can't catch create thread failed error, it only can handle exception after thread running

I didn't mean to use UncaughtExceptionHandler instead. I mean we can follow the way how UncaughtExceptionHandler handles the OOM error here.

Like current?

You could do:

try { threadPool.execute(tr) } catch { case t: Throawble => try { log.error(s"Execute task ${taskDescription.taskId} failed", e.getCause) context.statusUpdate( taskDescription.taskId, TaskState.FAILED, env.closureSerializer.newInstance().serialize( new ExceptionFailure(e, Seq.empty))) } catch { case oom: OutOfMemoryError => logError("...") System.exit(SparkExitCode.OOM) case t: Throwable => logError("...") System.exit(-1) } }

Ngone51 · 2025-11-10T04:57:02Z

core/src/main/scala/org/apache/spark/executor/Executor.scala

+      case t: Throwable =>
+        try {
+          logError(log"Executor launch task ${MDC(TASK_NAME, taskDescription.name)} failed," +
+            log" reason: ${MDC(REASON, t.getCause)}")


getCause could return null? Is there a way to print full stacks of the error?

Ngone51 · 2025-11-10T04:57:09Z

core/src/main/scala/org/apache/spark/executor/Executor.scala

+            System.exit(SparkExitCode.OOM)
+          case t: Throwable =>
+            logError(log"Executor update launching task ${MDC(TASK_NAME, taskDescription.name)} " +
+              log"failed status failed, reason: ${MDC(REASON, t.getCause)}")


use getMessage？

AngersZhuuuu · 2025-11-11T02:17:20Z

@Ngone51 GA passed

Ngone51

LGTM

AngersZhuuuu · 2025-11-16T07:19:55Z

ping @jiangxb1987 @Ngone51 .

cloud-fan · 2025-11-17T03:33:04Z

core/src/main/scala/org/apache/spark/executor/Executor.scala

+          case oom: OutOfMemoryError =>
+            logError(log"Executor update launching task ${MDC(TASK_NAME, taskDescription.name)} " +
+              log"failed status failed, reason: ${MDC(REASON, oom.getMessage)}")
+            System.exit(SparkExitCode.OOM)


any other special exit code for certain exceptions?

We only meet the case is create thread failed, then throw OutOfMemoryError

cloud-fan · 2025-11-17T03:36:07Z

core/src/main/scala/org/apache/spark/executor/Executor.scala

+        try {
+          logError(log"Executor launch task ${MDC(TASK_NAME, taskDescription.name)} failed," +
+            log" reason: ${MDC(REASON, t.getMessage)}")
+          context.statusUpdate(


so any place that invokes context.statusUpdate needs to add try-catch to crash the executor?

@Ngone51 means that if the RPC requests also fail due to OutOfMemoryError (OOM), exiting the executor can trigger corresponding task rerun to avoid getting stuck.
In my PR's scenario, this might be necessary in a special case. However, if the task is already running, I think it's unnecessary.

cloud-fan · 2025-11-18T05:05:46Z

core/src/main/scala/org/apache/spark/executor/Executor.scala

+            TaskState.FAILED,
+            env.closureSerializer.newInstance().serialize(new ExceptionFailure(t, Seq.empty)))
+        } catch {
+          case oom: OutOfMemoryError =>


I'm always skeptical to special cases, why OOM error is special here?

Need @Ngone51 answer your question, there's no need to handle this separately for me.

why OOM error is special here?

It's very likely to hit the OOM error again given that we're already in the OOM situation. Therefore, in the case of OOM, we should not expect the following operation could be always succesful. The special catch for the OutOfMemoryError is just for logging an exact error code when the error raises. This follow the behavior of SparkUncaughtExceptionHandler.

ok, can we add a code comment to explain it?

are we expecting context.statusUpdate to throw OOM?

are we expecting context.statusUpdate to throw OOM?

When sending RPC, if client is not created, need to create new connect task, may need create a new thread, can throw OOM since can't create thread. but always this case won't happen since client with driver should always be already created

I agree with @cloud-fan - special handling, especially of OOM, is not very robust.
This is one of the reasons why OnOutOfMemoryError is set for YARN - better to fail the executor than get it into unpredictable states.

Need to make a followup pr? cc @cloud-fan

Yes please!

dongjoon-hyun

+1, LGTM. Thank you, @AngersZhuuuu , @Ngone51 , @cloud-fan , @holdenk .
Merged to master/4.1 for Apache Spark 4.1.0.

…ask killed message ### What changes were proposed in this pull request? <img width="1906" height="754" alt="image" src="https://github.com/user-attachments/assets/115cb0fc-0651-4fd7-8dea-6b5e571b926e" /> When not enable speculation ,executor launch task failed caused by OOM, <img width="1766" height="858" alt="image" src="https://github.com/user-attachments/assets/5b88e607-99a3-4276-82da-3549e4ca5002" /> DAGScheduler won't know task failed and won't scheduler again, causing application stuck <img width="1728" height="377" alt="image-2025-06-23-12-13-10-770" src="https://github.com/user-attachments/assets/e425a4ae-fe04-4032-94a9-8833d96b63f3" /> ### Why are the changes needed? Some case will causing application stuck missing SLA ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? UT + MT Test code <img width="3022" height="1596" alt="image" src="https://github.com/user-attachments/assets/c51dd814-948c-4211-96b6-dd6edfe1d201" /> ``` import org.apache.spark.sql.functions.udf import java.util.concurrent.TimeUnit// 定义一个会sleep的UDF val sleepUdf = udf((value: String) => { TimeUnit.SECONDS.sleep(10) value }) val df = spark.range(0, 2500).repartition(25).withColumn("value", (col("id") % 10).cast("string")).withColumn("value", sleepUdf(col("value")))df.createOrReplaceTempView("test_table") val result = spark.sql("SELECT value, COUNT(*) as cnt FROM test_table GROUP BY value ORDER BY cnt DESC") result.show() ``` <img width="1728" height="482" alt="image-2025-07-23-14-44-53-911" src="https://github.com/user-attachments/assets/10d66f51-b054-47b2-a3f3-f2215da80087" /> <img width="1728" height="203" alt="image-2025-07-23-14-45-12-522" src="https://github.com/user-attachments/assets/15d09973-cb15-40d2-9618-e7dc1ce88aa8" /> task failed and re-scheduled ### Was this patch authored or co-authored using generative AI tooling? No Closes #52792 from AngersZhuuuu/SPARK-54087. Authored-by: Angerszhuuuu <angers.zhu@gmail.com> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org> (cherry picked from commit 7c279e5) Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>

…ask killed message ### What changes were proposed in this pull request? <img width="1906" height="754" alt="image" src="https://github.com/user-attachments/assets/115cb0fc-0651-4fd7-8dea-6b5e571b926e" /> When not enable speculation ,executor launch task failed caused by OOM, <img width="1766" height="858" alt="image" src="https://github.com/user-attachments/assets/5b88e607-99a3-4276-82da-3549e4ca5002" /> DAGScheduler won't know task failed and won't scheduler again, causing application stuck <img width="1728" height="377" alt="image-2025-06-23-12-13-10-770" src="https://github.com/user-attachments/assets/e425a4ae-fe04-4032-94a9-8833d96b63f3" /> ### Why are the changes needed? Some case will causing application stuck missing SLA ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? UT + MT Test code <img width="3022" height="1596" alt="image" src="https://github.com/user-attachments/assets/c51dd814-948c-4211-96b6-dd6edfe1d201" /> ``` import org.apache.spark.sql.functions.udf import java.util.concurrent.TimeUnit// 定义一个会sleep的UDF val sleepUdf = udf((value: String) => { TimeUnit.SECONDS.sleep(10) value }) val df = spark.range(0, 2500).repartition(25).withColumn("value", (col("id") % 10).cast("string")).withColumn("value", sleepUdf(col("value")))df.createOrReplaceTempView("test_table") val result = spark.sql("SELECT value, COUNT(*) as cnt FROM test_table GROUP BY value ORDER BY cnt DESC") result.show() ``` <img width="1728" height="482" alt="image-2025-07-23-14-44-53-911" src="https://github.com/user-attachments/assets/10d66f51-b054-47b2-a3f3-f2215da80087" /> <img width="1728" height="203" alt="image-2025-07-23-14-45-12-522" src="https://github.com/user-attachments/assets/15d09973-cb15-40d2-9618-e7dc1ce88aa8" /> task failed and re-scheduled ### Was this patch authored or co-authored using generative AI tooling? No Closes apache#52792 from AngersZhuuuu/SPARK-54087. Authored-by: Angerszhuuuu <angers.zhu@gmail.com> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>

…n task killed message should not special handling OOM ### What changes were proposed in this pull request? Follow comment #52792 (comment) ### Why are the changes needed? Follow comment ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? No need ### Was this patch authored or co-authored using generative AI tooling? No Closes #53379 from AngersZhuuuu/SPARK-54087-FOLLOWUP. Authored-by: Angerszhuuuu <angers.zhu@gmail.com> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>

…n task killed message should not special handling OOM ### What changes were proposed in this pull request? Follow comment #52792 (comment) ### Why are the changes needed? Follow comment ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? No need ### Was this patch authored or co-authored using generative AI tooling? No Closes #53379 from AngersZhuuuu/SPARK-54087-FOLLOWUP. Authored-by: Angerszhuuuu <angers.zhu@gmail.com> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org> (cherry picked from commit 6a60616) Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>

…n task killed message should not special handling OOM ### What changes were proposed in this pull request? Follow comment apache#52792 (comment) ### Why are the changes needed? Follow comment ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? No need ### Was this patch authored or co-authored using generative AI tooling? No Closes apache#53379 from AngersZhuuuu/SPARK-54087-FOLLOWUP. Authored-by: Angerszhuuuu <angers.zhu@gmail.com> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>

SPARK-54087 Spark Executor launch task failed should return task kill…

725c549

…ed message

github-actions bot added the CORE label Oct 30, 2025

Update ExecutorSuite.scala

2fe5144

Ngone51 reviewed Nov 7, 2025

View reviewed changes

AngersZhuuuu added 4 commits November 10, 2025 10:35

Merge branch 'master' into SPARK-54087

2e58abb

Update Executor.scala

27a3a46

Update Executor.scala

776f58a

Update ExecutorSuite.scala

bbcbf19

Ngone51 reviewed Nov 10, 2025

View reviewed changes

AngersZhuuuu added 2 commits November 10, 2025 14:10

Update Executor.scala

87383d4

Update ExecutorSuite.scala

232c7eb

Ngone51 approved these changes Nov 12, 2025

View reviewed changes

cloud-fan reviewed Nov 17, 2025

View reviewed changes

cloud-fan reviewed Nov 18, 2025

View reviewed changes

holdenk approved these changes Nov 21, 2025

View reviewed changes

dongjoon-hyun approved these changes Nov 21, 2025

View reviewed changes

dongjoon-hyun closed this in 7c279e5 Nov 21, 2025

AngersZhuuuu mentioned this pull request Dec 8, 2025

[SPARK-54087][CORE][FOLLOWUP] Spark Executor launch task failed return task killed message should not special handling OOM #53379

Closed

[SPARK-54087][CORE] Spark Executor launch task failed should return task killed message #52792

[SPARK-54087][CORE] Spark Executor launch task failed should return task killed message #52792

Uh oh!

Conversation

AngersZhuuuu commented Oct 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Was this patch authored or co-authored using generative AI tooling?

Uh oh!

AngersZhuuuu commented Nov 3, 2025

Uh oh!

holdenk commented Nov 3, 2025

Uh oh!

cloud-fan commented Nov 4, 2025

Uh oh!

Ngone51 Nov 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Ngone51 Nov 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

AngersZhuuuu commented Nov 11, 2025

Uh oh!

Ngone51 left a comment

Choose a reason for hiding this comment

Uh oh!

AngersZhuuuu commented Nov 16, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mridulm Dec 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

AngersZhuuuu commented Oct 30, 2025 •

edited

Loading

Ngone51 Nov 7, 2025 •

edited

Loading

Ngone51 Nov 10, 2025 •

edited

Loading

mridulm Dec 6, 2025 •

edited

Loading