Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Job to clean up old processes #591

Closed
gibfahn opened this issue Jan 9, 2017 · 8 comments
Closed

Job to clean up old processes #591

gibfahn opened this issue Jan 9, 2017 · 8 comments

Comments

@gibfahn
Copy link
Member

gibfahn commented Jan 9, 2017

We seem to quite frequently have an issue where node/python processes are left behind after a test run, usually due to a failing test. In our (IBM) CI, we just have a job we run before each flow that cleans up any node processes that might be lying around (there's a Unix version and a Windows version).

I feel like this is something that would be useful in CI, if we ran it as a job then it shouldn't interfere with build flows (as each machine only has one executor).

@gibfahn
Copy link
Member Author

gibfahn commented Jan 9, 2017

cc/ @mhdawson

@jbergstroem
Copy link
Member

SGTM as long as we don't forget about identifying the cause as well.

@mhdawson
Copy link
Member

mhdawson commented Jan 9, 2017

This sounds good to me as well. We have seen a number of cases were somebody runs the job on a PR, it exposes problems with the test which are later fixed, but in the mean time we have processes hanging around.

When you say it would be run as a job, is the idea that it would run at a particular interval (ex hourly) or do you know of some way we can tie it to the execution of other jobs ?

The other nice thing about a separate job is that we could have it "fail" if there were processes that needed to be killed, include that info in the job output and also email interested parties so that we'd know when processes were being left around and be notified with at least some amount of info. In the case when when the same problem occurred repeatedly we would likely catch that from the the job failures/emails.

@joaocgreis
Copy link
Member

Windows jobs already kill processes as part of the script, before and after running, including run-tests.exe of libuv, that shares the same machines.

I would also like to put more emphasis on finding out why and fixing this. Perhaps find a way to mark the build as unstable (yellow) when processes are left behind?

@gibfahn
Copy link
Member Author

gibfahn commented Feb 7, 2017

I've been thinking about this some more (not least because we've seen quite a few problems due to this).

I agree with @joaocgreis that we should consider processes left behind as a separate test-case (unless we can work out which processes came from which tests). Processes left behind should be considered a test-case failure.

I think the best place to do it would be in tools/test.py rather than individual jobs. The python test runner should be able to work out which node processes are from the current test run and kill them.

@mhdawson
Copy link
Member

mhdawson commented Feb 7, 2017

@gibfahn sounds good, we should make sure the info about processes being killed is output to the job and the idea of it reporting as at least yellow would make sense to me so that we can identify tests that are leaving jobs behind and fix them.

@richardlau
Copy link
Member

Related: nodejs/node#11246

@Trott
Copy link
Member

Trott commented Mar 22, 2019

Closing due to long period of inactivity. Feel free to re-open if this is a thing. I'm just trying to close stuff that has been ignored for sufficiently long that it seems likely it's not something we're going to get to.

@Trott Trott closed this as completed Mar 22, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

7 participants