Handle IOException in onOnline when failing to delete workspace, to prevent a node disconnection #142
Conversation
…revent a node disconnection
There was a problem hiding this comment.
@giacomolozito
Throwing an exception out of this method should not take the node offline:
If an exception here does take a node offline, we can patch that here but the that is also a bug in Jenkins core.
Have you tested this locally?
What version of Jenkins are you using?
|
@giacomolozito But this looks like a reasonable way to address the problem locally - giving meaningful/helpful output when this happens. Could you add a test for this to WorkspaceLocatorImplTest.java? Then we can merge. |
|
I'm going to go ahead and merge this for inclusion in the next release. It is a safe change. |
|
@giacomolozito |
Hello,
currently a failed workspace deletion attempt within onOnline will result in node disconnection, because the exception is not handled.
Unfortunately workspace deletion issues are getting more common as users run jobs inside docker containers that might run with a different user than jenkins, thus resulting in files left around in the workspace that jenkins can't delete. While this is a user problem and should be dealt with by ensuring jobs clean up their own files, what would normally be just an annoyance (the workspace files left around) can turn into an outage as the node fails to re-connect if it can't delete a workspace.
So this patch mitigates the issue by preventing onOnline from failing and by warning the operator that there are workspaces requiring manual cleanup (or jobs requiring a review).
Please note that https://issues.jenkins-ci.org/browse/JENKINS-55597 might be related to this or it might be another plugin - in my case it was branch-api not handling the exception so I patched it.
Kind regards