Fix memory leak of kernel Popen object #360
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
After analyzing various leaked items when running either Notebook or Jupyter Kernel Gateway, one item that recurred across each kernel startup and shutdown sequence was the Popen object stored in the kernel manager in `self.kernel`. The issue is that in normal circumstances, when a kernel's termination is successful via the ZMQ messaging, the process is never waited for (which, in this case, is probably unnecessary but advised) nor is the member variable set to None. In the failing case, where the message-based shutdown does not terminate the kernel process, the `_kill_kernel()` method is used, which performs the `wait()` and _nullifies_ the kernel member. This change ensures that sequence occurs in normal situations as well.
kevin-bates
added a commit
to kevin-bates/enterprise_gateway
that referenced
this pull request
Mar 14, 2018
PR jupyter-server#279 introduced some fixes to address file descriptor leaks - one of which was to shutdown the communication port. Since the kernel launcher listening on the other side may have already terminated, the shutdown method could throw an exception. This change catches, logs, then ignores such exceptions. While looking into other leaks (in this case memory), it was discovered that the process proxy instance was being leaked across kernel cycles. This change addresses that particular leak. Note: Other PRs have also been submitted to address leaks in `jupyter_client` and `notebook`. These are: [PR 360 - Fix memory leak of kernel Popen object](jupyter/jupyter_client#360) [PR 361 - Fix memory leak of IOLoopKernelManager object](jupyter/jupyter_client#361) [PR 3424 - Fix memory leak of iopub object in activity monitoring](jupyter/notebook#3424)
This was referenced Mar 14, 2018
lresende
pushed a commit
to jupyter-server/enterprise_gateway
that referenced
this pull request
Mar 19, 2018
While looking into other leaks (in this case memory), it was discovered that the process proxy instance was being leaked across kernel cycles. This change addresses that particular leak. Note: Other PRs have also been submitted to address leaks in `jupyter_client` and `notebook`. These are: [PR 360 - Fix memory leak of kernel Popen object](jupyter/jupyter_client#360) [PR 361 - Fix memory leak of IOLoopKernelManager object](jupyter/jupyter_client#361) [PR 3424 - Fix memory leak of iopub object in activity monitoring](jupyter/notebook#3424)
minrk
reviewed
Mar 22, 2018
| @@ -272,6 +272,10 @@ def finish_shutdown(self, waittime=None, pollinterval=0.1): | |||
| if self.is_alive(): | |||
| time.sleep(pollinterval) | |||
| else: | |||
| # If there's still a proc, wait and clear | |||
| if self.has_kernel: | |||
| self.kernel.wait() | |||
There was a problem hiding this comment.
Is there a reason to wait if is_alive() is False?
There was a problem hiding this comment.
Thanks @minrk . Since there's still a process object around, I consider it good practice and we're not seeing runtime issues in the sequence.
9 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
After analyzing various leaked items when running either Notebook or
Jupyter Kernel Gateway, one item that recurred across each kernel
startup and shutdown sequence was the Popen object stored in the kernel
manager in
self.kernel.The issue is that in normal circumstances, when a kernel's termination
is successful via the ZMQ messaging, the process is never waited for
(which, in this case, is probably unnecessary but advised) nor is the
member variable set to None. In the failing case, where the message-based
shutdown does not terminate the kernel process, the
_kill_kernel()method is used, which performs the
wait()and nullifies the kernelmember. This change ensures that sequence occurs in normal situations
as well.