Skip to content

[SPARK-3000][CORE] drop old blocks to disk in parallel when free memory is not enough for caching new blocks #2134

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
liyezhang556520 wants to merge 9 commits into from
Closed

[SPARK-3000][CORE] drop old blocks to disk in parallel when free memory is not enough for caching new blocks #2134

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
da2afad
[SPARK-3000][CORE] drop old blocks to disk in parallel when memory is…
liyezhang556520 Aug 26, 2014
4f76f4c
add logInfo back when dropping a block from memory
liyezhang556520 Aug 27, 2014
cd0f547
indentation correction and some comments modified
liyezhang556520 Aug 29, 2014
5079e41
handle exception when dropping blocks accroding comments in [SPARK-18…
liyezhang556520 Sep 2, 2014
0fe07e6
refine code style
liyezhang556520 Sep 2, 2014
4390f15
make logic correctly when unrolling memory if encounter exceptions
liyezhang556520 Sep 3, 2014
ab1f9f8
rebased and isolate each threads operation, the freed memory by dropp…
liyezhang556520 Nov 6, 2014
3192a6d
rebase again, and add some changes back after rebase.
liyezhang556520 Apr 16, 2015
c248156
fix test code style
liyezhang556520 Apr 16, 2015
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions core/src/main/scala/org/apache/spark/executor/Executor.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -292,8 +292,8 @@ private[spark] class Executor(
} finally {
// Release memory used by this thread for shuffles
env.shuffleMemoryManager.releaseMemoryForThisThread()
// Release memory used by this thread for unrolling blocks
env.blockManager.memoryStore.releaseUnrollMemoryForThisThread()
// cleanup maintained memory infomation for this thread
env.blockManager.memoryStore.cleanupForThisThread()
// Release memory used by this thread for accumulators
Accumulators.clear()
runningTasks.remove(taskId)
Expand Down
Loading