Commit 337eb85
[SPARK-53128][CORE] Include unmanaged memory bytes in the usage log before execution memory OOM
### What changes were proposed in this pull request?
We have a log before OOM for off-heap memory allocation.
Before the change, the log is:
> 25/08/05 16:44:32 INFO TaskMemoryManager: 100 bytes of memory are used for execution and 100 bytes of memory are used for storage
After:
> 25/08/05 16:44:32 INFO TaskMemoryManager: 100 bytes of memory are used for execution and 100 bytes of memory are used for storage and 500 bytes of memory are used but unmanaged
### Why are the changes needed?
Following #51708, to allow user to know the reason if the unmanaged memory causes OOM.
### Does this PR introduce _any_ user-facing change?
Only changes a log message.
### How was this patch tested?
Existing tests.
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes #51848 from zhztheplayer/wip-53128.
Authored-by: Hongze Zhang <hongze.zzz123@gmail.com>
Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
(cherry picked from commit c4ad381)
Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>1 parent 67f2ac5 commit 337eb85
File tree
2 files changed
+13
-2
lines changed- core/src/main
- java/org/apache/spark/memory
- scala/org/apache/spark/memory
2 files changed
+13
-2
lines changedLines changed: 5 additions & 2 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
339 | 339 | | |
340 | 340 | | |
341 | 341 | | |
342 | | - | |
| 342 | + | |
| 343 | + | |
| 344 | + | |
343 | 345 | | |
344 | | - | |
| 346 | + | |
| 347 | + | |
345 | 348 | | |
346 | 349 | | |
347 | 350 | | |
| |||
Lines changed: 8 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
273 | 273 | | |
274 | 274 | | |
275 | 275 | | |
| 276 | + | |
| 277 | + | |
| 278 | + | |
| 279 | + | |
| 280 | + | |
| 281 | + | |
| 282 | + | |
| 283 | + | |
276 | 284 | | |
277 | 285 | | |
278 | 286 | | |
| |||
0 commit comments