[SPARK-3090] [core] Stop SparkContext if user forgets to. #5696
Conversation
Set up a shutdown hook to try to stop the Spark context in case the user forgets to do it. The main effect is that any open logs files are flushed and closed, which is particularly interesting for event logs.
|
Note that as is this will cause an exception if the hook actually runs; #5672 has a change that fixes that. |
There was a problem hiding this comment.
On the grounds that the YARN AM already tries to do this, I think this is probably a good change to implement for all modes. I think it's all too common that user programs exit without calling stop() and this does affect things like finding logs.
Calling stop() is idempotent right? so the extra hook shouldn't hurt anything even for well-behaved programs.
You have some spurious white-space changes in this PR. Not really worth fixing but might tell your IDE not to change trailing whitespace.
There was a problem hiding this comment.
Calling stop() is idempotent right?
Correct.
You have some spurious white-space changes in this PR.
IIRC that's in the style guide (not adding trailing whitespace) and we're really, really bad at enforcing that.
|
Test build #30948 has finished for PR 5696 at commit
|
|
LGTM. I dislike the proliferation of shutdown hooks but we have a better centralized system for it now, and this one is fairly important. Let me leave it for comments for a couple days though. |
Set up a shutdown hook to try to stop the Spark context in case the user forgets to do it. The main effect is that any open logs files are flushed and closed, which is particularly interesting for event logs. Author: Marcelo Vanzin <vanzin@cloudera.com> Closes apache#5696 from vanzin/SPARK-3090 and squashes the following commits: 3b554b5 [Marcelo Vanzin] [SPARK-3090] [core] Stop SparkContext if user forgets to.
Set up a shutdown hook to try to stop the Spark context in case the user forgets to do it. The main effect is that any open logs files are flushed and closed, which is particularly interesting for event logs. Author: Marcelo Vanzin <vanzin@cloudera.com> Closes apache#5696 from vanzin/SPARK-3090 and squashes the following commits: 3b554b5 [Marcelo Vanzin] [SPARK-3090] [core] Stop SparkContext if user forgets to.
3 participants
Set up a shutdown hook to try to stop the Spark context in
case the user forgets to do it. The main effect is that any
open logs files are flushed and closed, which is particularly
interesting for event logs.