Skip to content

SPARK-5841 [CORE] [HOTFIX 2] Memory leak in DiskBlockManager #4690

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
nishkamravi2 wants to merge 37 commits into from
Closed

SPARK-5841 [CORE] [HOTFIX 2] Memory leak in DiskBlockManager #4690

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
37 commits
Select commit Hold shift + click to select a range
681b36f
Fix for SPARK-1758: failing test org.apache.spark.JavaAPISuite.wholeT…
nishkamravi2 May 8, 2014
5108700
Fix in Spark for the Concurrent thread modification issue (SPARK-1097…
nishkamravi2 Jun 3, 2014
6b840f0
Undo the fix for SPARK-1758 (the problem is fixed)
nishkamravi2 Jun 3, 2014
df2aeb1
Improved fix for ConcurrentModificationIssue (Spark-1097, Hadoop-10456)
nishkamravi2 Jun 9, 2014
eb663ca
Merge branch 'master' of https://github.com/apache/spark
nishkamravi2 Jun 9, 2014
5423a03
Merge branch 'master' of https://github.com/apache/spark
nishkamravi2 Jun 10, 2014
3bf8fad
Merge branch 'master' of https://github.com/apache/spark
nishkamravi2 Jun 16, 2014
2b630f9
Accept memory input as "30g", "512M" instead of an int value, to be c…
nishkamravi2 Jun 16, 2014
efd688a
Merge branch 'master' of https://github.com/apache/spark
nishkamravi2 Jul 13, 2014
2e69f11
Merge branch 'master' of https://github.com/apache/spark into master_…
nishkamravi2 Sep 21, 2014
ebcde10
Modify default YARN memory_overhead-- from an additive constant to a …
nishkamravi2 Sep 22, 2014
1cf2d1e
Update YarnAllocator.scala
nishkamravi2 Sep 22, 2014
f00fa31
Improving logging for AM memoryOverhead
nishkamravi2 Sep 22, 2014
c726bd9
Merge branch 'master' of https://github.com/apache/spark into master_…
nishkamravi2 Sep 25, 2014
362da5e
Additional changes for yarn memory overhead
nishkamravi2 Sep 25, 2014
42c2c3d
Additional changes for yarn memory overhead issue
nishkamravi2 Sep 25, 2014
dac1047
Additional documentation for yarn memory overhead issue
nishkamravi2 Sep 25, 2014
5ac2ec1
Remove out
nishkamravi2 Sep 25, 2014
35daa64
Slight change in the doc for yarn memory overhead
nishkamravi2 Sep 25, 2014
8f76c8b
Doc change for yarn memory overhead
nishkamravi2 Sep 25, 2014
636a9ff
Update YarnAllocator.scala
nishkamravi2 Sep 30, 2014
5f8f9ed
Merge branch 'master' of https://github.com/apache/spark into master_…
nishkamravi2 Nov 19, 2014
9f6583e
Changes to maxResultSize code (improve error message and add conditio…
nishkamravi2 Nov 19, 2014
3e1b616
Modify test for maxResultSize
nishkamravi2 Nov 19, 2014
535295a
Update TaskSetManager.scala
nishkamravi2 Nov 19, 2014
5c9a4cb
Update TaskSetManagerSuite.scala
nishkamravi2 Nov 19, 2014
b446edc
Merge branch 'master' of https://github.com/apache/spark into master_…
nishkamravi2 Nov 19, 2014
79ea8b4
Merge branch 'master' of https://github.com/apache/spark into master_…
nishkamravi2 Feb 19, 2015
f0d12de
Workaround for IllegalStateException caused by recent changes to Bloc…
nishkamravi2 Feb 19, 2015
3c5ddba
Update DiskBlockManager.scala
nishkamravi2 Feb 19, 2015
494d8c0
Update DiskBlockManager.scala
nishkamravi2 Feb 19, 2015
71d0e17
Merge branch 'master' of https://github.com/apache/spark into master_…
nishkamravi2 Feb 20, 2015
ba1eb8b
Try-catch block around the two occurrences of removeShutDownHook. Del…
nishkamravi2 Feb 20, 2015
f7e33c2
Merge branch 'master_nravi' of https://github.com/nishkamravi2/spark …
nishkamravi2 Feb 20, 2015
0ce2c32
Update HadoopRDD.scala
nishkamravi2 Feb 20, 2015
6f41a1d
Update NewHadoopRDD.scala
nishkamravi2 Feb 20, 2015
d453197
Update NewHadoopRDD.scala
nishkamravi2 Feb 20, 2015
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 5 additions & 1 deletion core/src/main/scala/org/apache/spark/deploy/worker/ExecutorRunner.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -105,7 +105,11 @@ private[spark] class ExecutorRunner(
workerThread.interrupt()
workerThread = null
state = ExecutorState.KILLED
Runtime.getRuntime.removeShutdownHook(shutdownHook)
try {
Runtime.getRuntime.removeShutdownHook(shutdownHook)
} catch {
case e: IllegalStateException => None
}
}
}

Expand Down
6 changes: 5 additions & 1 deletion core/src/main/scala/org/apache/spark/storage/DiskBlockManager.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -148,7 +148,11 @@ private[spark] class DiskBlockManager(blockManager: BlockManager, conf: SparkCon
/** Cleanup local dirs and stop shuffle sender. */
private[spark] def stop() {
// Remove the shutdown hook. It causes memory leaks if we leave it around.
Runtime.getRuntime.removeShutdownHook(shutdownHook)
try {
Runtime.getRuntime.removeShutdownHook(shutdownHook)
} catch {
case e: IllegalStateException => None
}
doStop()
}

Expand Down