Skip to content

Commit 337eb85

Browse files
zhztheplayerdongjoon-hyun
authored andcommitted
[SPARK-53128][CORE] Include unmanaged memory bytes in the usage log before execution memory OOM
### What changes were proposed in this pull request? We have a log before OOM for off-heap memory allocation. Before the change, the log is: > 25/08/05 16:44:32 INFO TaskMemoryManager: 100 bytes of memory are used for execution and 100 bytes of memory are used for storage After: > 25/08/05 16:44:32 INFO TaskMemoryManager: 100 bytes of memory are used for execution and 100 bytes of memory are used for storage and 500 bytes of memory are used but unmanaged ### Why are the changes needed? Following #51708, to allow user to know the reason if the unmanaged memory causes OOM. ### Does this PR introduce _any_ user-facing change? Only changes a log message. ### How was this patch tested? Existing tests. ### Was this patch authored or co-authored using generative AI tooling? No. Closes #51848 from zhztheplayer/wip-53128. Authored-by: Hongze Zhang <hongze.zzz123@gmail.com> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org> (cherry picked from commit c4ad381) Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
1 parent 67f2ac5 commit 337eb85

File tree

2 files changed

+13
-2
lines changed

2 files changed

+13
-2
lines changed

core/src/main/java/org/apache/spark/memory/TaskMemoryManager.java

Lines changed: 5 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -339,9 +339,12 @@ public void showMemoryUsage() {
339339
MDC.of(LogKeys.MEMORY_SIZE, memoryNotAccountedFor),
340340
MDC.of(LogKeys.TASK_ATTEMPT_ID, taskAttemptId));
341341
logger.info(
342-
"{} bytes of memory are used for execution and {} bytes of memory are used for storage",
342+
"{} bytes of memory are used for execution " +
343+
"and {} bytes of memory are used for storage " +
344+
"and {} bytes of unmanaged memory are used",
343345
MDC.of(LogKeys.EXECUTION_MEMORY_SIZE, memoryManager.executionMemoryUsed()),
344-
MDC.of(LogKeys.STORAGE_MEMORY_SIZE, memoryManager.storageMemoryUsed()));
346+
MDC.of(LogKeys.STORAGE_MEMORY_SIZE, memoryManager.storageMemoryUsed()),
347+
MDC.of(LogKeys.MEMORY_SIZE, UnifiedMemoryManager$.MODULE$.getUnmanagedMemoryUsed()));
345348
}
346349
}
347350

core/src/main/scala/org/apache/spark/memory/UnifiedMemoryManager.scala

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -273,6 +273,14 @@ object UnifiedMemoryManager extends Logging {
273273
// Atomic flag to ensure polling is only started once per JVM
274274
private val pollingStarted = new AtomicBoolean(false)
275275

276+
/**
277+
* Returns the total unmanaged memory in bytes, including both
278+
* on-heap unmanaged memory and off-heap unmanaged memory.
279+
*/
280+
private[spark] def getUnmanagedMemoryUsed: Long = {
281+
UnifiedMemoryManager.unmanagedOnHeapUsed.get() + UnifiedMemoryManager.unmanagedOffHeapUsed.get()
282+
}
283+
276284
/**
277285
* Register an unmanaged memory consumer to track its memory usage.
278286
*

0 commit comments

Comments
 (0)