sequential.test-watch-mode-inspect seems flaky

[mhdawson]

Test

sequential.test-watch-mode-inspect

Platform

Linux ARMv7

Console output

Error Message
fail (-15)
Stacktrace
---
duration_ms: 360029.938
exitcode: -15
severity: fail
...

Build links

• https://ci.nodejs.org/job/node-test-binary-armv7l/7719/RUN_SUBSET=js,nodes=ubuntu2004-armv7l/
• https://ci.nodejs.org/job/node-test-binary-armv7l/7715/RUN_SUBSET=js,nodes=ubuntu2004-armv7l/

Additional information

No response

[mhdawson]

Failed on same machine, two builds in a row
test-equinix-ubuntu2004_container-armv7l-1

[mhdawson]

Passed when it ran on test-equinix-ubuntu2204-x64-2 so could be specific to that machine.

Need to track to see if it fails across the machines or not.

[mhdawson]

and again - https://ci.nodejs.org/job/node-test-binary-armv7l/7720/RUN_SUBSET=js,nodes=ubuntu2004-armv7l/console

[mhdawson]

Interestingly I see this in the output

not ok 3844 sequential/test-watch-mode-inspect # TODO : Fix flaky test

which seems to say it's a known issue even though I could not find a tracking issue.

[richardlau]

FWIW the test is already marked flaky

node/test/sequential/sequential.status

Lines 10 to 12 in 1b96975

	# https://github.com/nodejs/node/issues/44898
	test-watch-mode: PASS, FLAKY
	test-watch-mode-inspect: PASS, FLAKY

https://ci.nodejs.org/job/node-test-binary-armv7l/7720/RUN_SUBSET=js,nodes=ubuntu2004-armv7l/console has failed because of leftover Node.js processes (which are probably related to the test being timed out):

17:02:09 All tests passed.
17:02:09 ps awwx | grep Release/node | grep -v grep | cat
17:02:09 2212632 ?        Sl     0:00 /home/iojs/build/workspace/node-test-binary-armv7l/out/Release/node --inspect=0 --watch /home/iojs/build/workspace/node-test-binary-armv7l/test/fixtures/watch-mode/inspect.js
17:02:09 2212638 ?        Sl     0:00 /home/iojs/build/workspace/node-test-binary-armv7l/out/Release/node --inspect=0 /home/iojs/build/workspace/node-test-binary-armv7l/test/fixtures/watch-mode/inspect.js
17:02:09 make: *** [Makefile:538: test-ci-js] Error 1

[anonrig]

It failed on again on ubuntu 22.04 https://ci.nodejs.org/job/node-test-commit-osx-arm/13900/

[anonrig]

Seen on node-test-commit-arm-fanned https://ci.nodejs.org/job/node-test-binary-armv7l/8168/

[mhdawson]

I don't see if failing in https://ci.nodejs.org/job/node-test-commit-osx-arm/13900/, that seemed to be a different test test-watch-mode without the inspect part.

[mhdawson]

again - https://ci.nodejs.org/job/node-test-binary-armv7l/8829/

[mhdawson]

again - https://ci.nodejs.org/job/node-test-binary-armv7l/9064/

[mhdawson]

Not sure why I keep seeing these even though the test is marked as flaky. Looked at the CI jobs and I don't see anything obviously wrong.

@richardlau do you know of any reason off the top of your head a job might not be treating flaky tests as ok?

[richardlau]

@mhdawson https://ci.nodejs.org/job/node-test-binary-armv7l/9064/RUN_SUBSET=js,nodes=ubuntu2004-armv7l/console failed because:

18:27:41 All tests passed.
18:27:41 ps awwx | grep Release/node | grep -v grep | cat
18:27:41  601756 ?        Sl     0:00 /home/iojs/build/workspace/node-test-binary-armv7l/out/Release/node --inspect=0 --watch /home/iojs/build/workspace/node-test-binary-armv7l/test/fixtures/watch-mode/inspect.js
18:27:41  601762 ?        Sl     0:00 /home/iojs/build/workspace/node-test-binary-armv7l/out/Release/node --inspect=0 /home/iojs/build/workspace/node-test-binary-armv7l/test/fixtures/watch-mode/inspect.js
18:27:41 make: *** [Makefile:538: test-ci-js] Error 1

"All tests passed" means that the failure was treated as flaky. The problem is that it left behind child processes and the build will fail if any node process are still running after the test runner exits:

node/Makefile

Lines 554 to 559 in 342ddb0

	$(info Clean up any leftover processes, error if found.)
	ps awwx | grep Release/node | grep -v grep | cat
	@PS_OUT=`ps awwx | grep Release/node | grep -v grep | awk '{print $$1}'`; \
	if [ "$${PS_OUT}" ]; then \
	echo $${PS_OUT} | xargs kill -9; exit 1; \
	fi

[mhdawson]

@richardlau, if I get what you are saying is that even though the job out reports this test, that was not the problem, instead it was because child processes were left behind that may/likely have nothing to do with that flaky test right?

[richardlau]

@mhdawson Yes, although I suspect the leftover processes are from the flaky test which I believe has been timed out.

[mhdawson]

@richardlau ok, if we believe they are related to the test then SKIP might be more appropriate right? If so I'll submit a PR to do that.

[mhdawson]

Failed again - https://ci.nodejs.org/job/node-test-binary-armv7l/9322/

[mhdawson]

Faillures on ubuntu shared libs - https://ci.nodejs.org/job/node-test-commit-linux-containered/40987/nodes=ubuntu2204_sharedlibs_shared_x64/testReport/junit/(root)/sequential/test_watch_mode_inspect_/

[mhdawson]

Seems to fail on ubuntu as well - https://ci.nodejs.org/job/node-test-commit-linux-containered/nodes=ubuntu2204_sharedlibs_shared_x64/41479/

[jakecastelli]

Again - https://ci.nodejs.org/job/node-test-commit-smartos/nodes=smartos20-64/55713/

logs

22:09:53 not ok 4228 sequential/test-watch-mode-inspect # TODO : Fix flaky test
22:09:53   ---
22:09:53   duration_ms: 300129.29800
22:09:53   severity: fail
22:09:53   exitcode: -15
22:09:53   stack: |-
22:09:53     timeout
22:09:53     [test] Connecting to a child Node process
22:09:53     [test] Testing /json/list
22:09:53     TAP version 13
22:09:53     [err] Debugger listening on ws://127.0.0.1:61622/bec85c05-bcf7-4407-86de-720d30c10727
22:09:53     [err] For help, see: https://nodejs.org/en/docs/inspector
22:09:53     [err] 
22:09:53     [err] Debugger attached.
22:09:53     [err] 
22:09:53     [test] Connecting to a child Node process
22:09:53     [test] Testing /json/list
22:09:53     [err] Debugger ending on ws://127.0.0.1:61622/bec85c05-bcf7-4407-86de-720d30c10727
22:09:53     [err] For help, see: https://nodejs.org/en/docs/inspector
22:09:53     [err] 
22:09:53     [out] safe to debug now
22:09:53     [out] 
22:09:53   ...

This test seems to fail quite often lately

[targos]

https://ci.nodejs.org/job/node-test-commit-smartos/55913/nodes=smartos20-64/consoleFull

[aduh95]

Again https://ci.nodejs.org/job/node-test-commit-linux-containered/46199/