Intermittent failure in parallel/test-fs-watchfile on AIX

[mhdawson]

• Version: master
• Platform: AIX
• Subsystem: fs

I think I've seen this fail a few times recently:

https://ci.nodejs.org/job/node-test-commit-aix/6276/

not ok 433 parallel/test-fs-watchfile
  ---
  duration_ms: 60.193
  severity: fail
  stack: |-
    timeout
  ...

[mhdawson]

Another instance https://ci.nodejs.org/job/node-test-commit-aix/6280/nodes=aix61-ppc64/console

[mhdawson]

Looks like there was a recent change to the test: 3b12a8d

FYI - @Trott

[Trott]

Ack. Looking at it right now.

[Trott]

Replication on master using tools/test.py -j 64 --repeat 128 test/parallel/test-fs-watchfile.js: https://ci.nodejs.org/job/node-stress-single-test/1249/nodes=aix61-ppc64/console

So, still flaky under load. Will try to fix in place, but ultimately moving to sequential might help.

[refack]

From the logs it seems like something "snaps", 50-60 iterations work, then it just stops.
Maybe try more file names, or unlinking the file and rewriting it.

[Trott]

I'm able to get it up to over 100 iterations if I switch to a large enough timer that calls itself. I tried using a backoff algorithm but that didn't seem to help much. I wonder if what's really going on is that at some point, the operating system stops handing out working watchers.

I think the solution is to move it to sequential where it's not competing with other resources, at least for now. fs.watch() is quirky so there's no shame in that.

[Trott]

Removed the interval and put the test in sequential. Let's stress test it on AIX and macOS and see what happens.

AIX: https://ci.nodejs.org/job/node-stress-single-test/nodes=aix61-ppc64/1258/console

macOS: https://ci.nodejs.org/job/node-stress-single-test/1260/nodes=osx1010/console

EDIT: Typo'ed on the path names. Will try again...

[Trott]

Hopefully this time with the right parameters entered:

AIX: https://ci.nodejs.org/job/node-stress-single-test/1261/nodes=aix61-ppc64/console
macOS: https://ci.nodejs.org/job/node-stress-single-test/1262/nodes=osx1010/console

[Trott]

Failed with the interval missing. Put it back. That should fix it on macOS. Hopefully AIX too. Let's see.

AIX: https://ci.nodejs.org/job/node-stress-single-test/1263/nodes=aix61-ppc64/console
macOS: https://ci.nodejs.org/job/node-stress-single-test/nodes=osx1010/1265/console

[Trott]

Stress tests indicate this doesn't fix the flakiness on AIX. I'm starting to wonder if sometimes AIX hands us watchers that will simply never fire. The fs.watch() stuff is highly specific to the OS.

@nodejs/platform-aix

[gibfahn]

cc/ @gireeshpunathil

[gireeshpunathil]

I just commented in #13385 thus:

ref

Watch facility on folders are not fool-proof in AIX, and we have skipped those tests which do that. I guess the new changes in
test-fs-watchfile.js is not catering to that.

#13111 made that change, it's original intent was to make sure the filename argument appears in the callback, when it fires - so I requested to include AIX as well, as the filename argument is availabe in that platform. But then the change introduced folder watch and hence the flaky result.

Proposals - one of:

1. Amend the new change in test-fs-watchfile.js to skip AIX.
2. Amend the new change in test-fs-watchfile.js to file watch as opposed to folder watch

[refack]

Anybody on it?

[refack]

Ref: #13248
Ref: #13251

[Trott]

Anybody on it?

Yup. #13385

[refack]

Anybody on it?
Yup. #13385

I have a follow up to that in #13411