Implement test retry logic for flakiness

[marco-ippolito]

What is the problem this feature will solve?

Refs: nodejs/TSC#1614
Reduce flakiness in the CI

What is the feature you are proposing to solve the problem?

In today's TSC meeting nodejs/TSC#1784 it was discussed about flaky test detection and automatic re running flaky tests.
I noticed in https://github.com/nodejs/node/blob/main/.github/workflows/test-linux.yml#L73 there is a flag to instruct the test suite to re run flaky tests. I wonder if we can make this flag work in the whole CI (requires some work in the python runner I assume?) to help reduce flakiness

What alternatives have you considered?

No response

[aduh95]

Next steps were described in nodejs/build#3056 (comment), it looks like all it needs would be a champion with access to Jenkins web UI.

[joyeecheung]

For other platforms that uses Makefile we can just hard code that into the Makefile.

Note that despite having already configured this, the Windows CI is still the biggest source of flakes currently.

[targos]

It doesn't look like the Windows configuration is correct, or it's not doing what you expect.

From https://ci.nodejs.org/job/node-test-binary-windows-js-suites/36563/RUN_SUBSET=2,nodes=win2019-COMPILED_BY-vs2022_clang/console:

13:47:10  failed 1 out of 10
13:47:10 not ok 598 parallel/test-inspector-network-fetch
13:47:10   ---
13:47:10   duration_ms: 460.00800
13:47:10   severity: fail
13:47:10   exitcode: 1
13:47:10   stack: |-
13:47:10     Debugger listening on ws://127.0.0.1:61330/b38d81b1-e0f4-44cb-857c-1a2031a69a5e
13:47:10     For help, see: https://nodejs.org/en/docs/inspector
13:47:10     node:internal/deps/undici/undici:15445
13:47:10           Error.captureStackTrace(err);
13:47:10                 ^
13:47:10     
13:47:10     TypeError: fetch failed
13:47:10         at node:internal/deps/undici/undici:15445:13
13:47:10         at process.processTicksAndRejections (node:internal/process/task_queues:105:5)
13:47:10         at async testRequest (C:\workspace\node-test-binary-windows-js-suites\node\test\parallel\test-inspector-network-fetch.js:153:3)
13:47:10         at async testNetworkInspection (C:\workspace\node-test-binary-windows-js-suites\node\test\parallel\test-inspector-network-fetch.js:198:3)
13:47:10         at async Server.<anonymous> (C:\workspace\node-test-binary-windows-js-suites\node\test\parallel\test-inspector-network-fetch.js:212:7) {
13:47:10       [cause]: [Error: 0C1D0000:error:0A0000C6:SSL routines:tls_get_more_records:packet length too long:openssl\ssl\record\methods\tls_common.c:663:
13:47:10       ] {
13:47:10         library: 'SSL routines',
13:47:10         reason: 'packet length too long',
13:47:10         code: 'ERR_SSL_PACKET_LENGTH_TOO_LONG'
13:47:10       }
13:47:10     }
13:47:10     
13:47:10     Node.js v25.0.0-pre
13:47:10   ...

Flakyness was measured. It ran the test 10 times and it failed once. That single failure was enough to mark the run red.

[StefanStojanovic]

Maybe this is helpful #50051. My PR from 2 years ago, where I implemented a mechanism you're describing.

[aduh95]

It doesn't look like the Windows configuration is correct, or it's not doing what you expect.
[…]
Flakyness was measured. It ran the test 10 times and it failed once. That single failure was enough to mark the run red.

That sounds like correctly configured and doing what we expect, it was definitely how this feature was envisioned. It's not supposed to "make the flakes go away", it's meant to help detect when it's not a flake (if it had failed 9 times out of 10, don't bother re-kicking the job, the test is broken)

[targos]

The goal of this issue is to:

Reduce flakiness in the CI

And given this comment it seems like there is some misunderstanding.