test: make test-perf-hooks more robust and work with workers #49197

joyeecheung · 2023-08-16T11:46:44Z

Previously the test makes several assumptions about the absolute values of the nodeTiming fields, which can make the test flaky on slow machines. This patch rewrites the test to check the relative values instead. It also updates the test to make it work with workers instead of directly skipping in workers.

Refs: nodejs/reliability#638

Previously the test makes several assumptions about the absolute values of the nodeTiming fields, which can make the test flaky on slow machines. This patch rewrites the test to check the relative values instead. It also updates the test to make it work with workers instead of directly skipping in workers.

joyeecheung · 2023-08-16T11:48:12Z

From nodejs/reliability#638 this test has been failing 6 recent PRs, hopefully this makes the flake go away..

Reason	`sequential/test-perf-hooks`
Type	JS_TEST_FAILURE
Failed PR	6 (#49104, #49127, #49149, #49162, #49164, #46391)
Appeared	test-equinix-ubuntu1804_container-arm64-2, test-osuosl-ubuntu2004_sharedlibs_container-arm64-1, test-osuosl-ubuntu2004_container-armv7l-1, test-equinix-ubuntu2004_container-armv7l-2, test-azure_msft-win11_vs2022-arm64-4, test-equinix-ubuntu2004_container-arm64-2
First CI	https://ci.nodejs.org/job/node-test-pull-request/53267/
Last CI	https://ci.nodejs.org/job/node-test-pull-request/53336/

Example

not ok 3928 sequential/test-perf-hooks
  ---
  duration_ms: 426.79600
  severity: fail
  exitcode: 1
  stack: |-
    {
      name: 'node',
      entryType: 'node',
      startTime: 0,
      duration: { around: 285.6393041610718 },
      nodeStart: { around: 0 },
      v8Start: { around: 0 },
      bootstrapComplete: { around: 285.5992240905762, delay: 2500 },
      environment: { around: 0 },
      loopStart: -1,
      loopExit: -1
    }
    node:assert:399
        throw err;
        ^
    
    AssertionError [ERR_ASSERTION]: environment: 252.57520198822021 >= 250
        at checkNodeTiming (/home/iojs/build/workspace/node-test-commit-arm/test/sequential/test-perf-hooks.js:31:7)
        at Object.<anonymous> (/home/iojs/build/workspace/node-test-commit-arm/test/sequential/test-perf-hooks.js:43:1)
        at Module._compile (node:internal/modules/cjs/loader:1241:14)
        at Module._extensions..js (node:internal/modules/cjs/loader:1295:10)
        at Module.load (node:internal/modules/cjs/loader:1091:32)
   ...

nodejs-github-bot · 2023-08-16T11:50:48Z

CI: https://ci.nodejs.org/job/node-test-pull-request/53373/

test/sequential/test-perf-hooks.js

debadree25

Looks great!
Thank you!

debadree25 · 2023-08-16T17:29:02Z

i tried stress testing this too locally with python3 tools/test.py test/sequential/test-perf-hooks.js --repeat 50 -j10 it passed well! so maybe could graduate to parallel too?

joyeecheung · 2023-08-16T17:30:53Z

I am not sure why it was in sequential in the first place, but I think on the safe side we can just keep it there in this PR and if it looks well in the CI, move it to parallel afterwards (otherwise one doesn't know which is causing a regression if there is a regression).

debadree25 · 2023-08-16T17:31:54Z

Yeah makes sense!

aduh95 · 2023-08-16T18:41:52Z

test/sequential/test-perf-hooks.js

 // Use a fairly large epsilon value, since we can only guarantee that the node
 // process started up in 15 seconds.
-assert(Math.abs(performance.timeOrigin - Date.now()) < 15000);
+assert(testStartTime < 15000, `${testStartTime} >= 15000`);


nit

Suggested change

assert(testStartTime < 15000, `${testStartTime} >= 15000`);

assert(testStartTime < 15000, `${testStartTime} ≥ 15000`);

I think in general we use >= in this codebase instead?

joyeecheung · 2023-08-18T00:37:32Z

By the way the failure also shows up on v20.x-staging https://ci.nodejs.org/job/node-test-binary-armv7l/7027/RUN_SUBSET=js,nodes=ubuntu2004-armv7l/testReport/junit/(root)/sequential/test_perf_hooks/

nodejs-github-bot · 2023-08-18T11:49:58Z

Landed in ecde9d9

Previously the test makes several assumptions about the absolute values of the nodeTiming fields, which can make the test flaky on slow machines. This patch rewrites the test to check the relative values instead. It also updates the test to make it work with workers instead of directly skipping in workers. PR-URL: #49197 Refs: nodejs/reliability#638 Reviewed-By: Debadree Chatterjee <debadree333@gmail.com> Reviewed-By: Antoine du Hamel <duhamelantoine1995@gmail.com>

Previously the test makes several assumptions about the absolute values of the nodeTiming fields, which can make the test flaky on slow machines. This patch rewrites the test to check the relative values instead. It also updates the test to make it work with workers instead of directly skipping in workers. PR-URL: nodejs/node#49197 Refs: nodejs/reliability#638 Reviewed-By: Debadree Chatterjee <debadree333@gmail.com> Reviewed-By: Antoine du Hamel <duhamelantoine1995@gmail.com>

joyeecheung added the request-ci Add this label to start a Jenkins CI on a PR. label Aug 16, 2023

nodejs-github-bot added needs-ci PRs that need a full CI run. test Issues and PRs related to the tests. labels Aug 16, 2023

github-actions bot removed the request-ci Add this label to start a Jenkins CI on a PR. label Aug 16, 2023

debadree25 reviewed Aug 16, 2023

View reviewed changes

test/sequential/test-perf-hooks.js Show resolved Hide resolved

debadree25 approved these changes Aug 16, 2023

View reviewed changes

debadree25 added the author ready PRs that have at least one approval, no pending requests for changes, and a CI started. label Aug 16, 2023

aduh95 approved these changes Aug 16, 2023

View reviewed changes

joyeecheung added the commit-queue Add this label to land a pull request using GitHub Actions. label Aug 18, 2023

nodejs-github-bot removed the commit-queue Add this label to land a pull request using GitHub Actions. label Aug 18, 2023

nodejs-github-bot merged commit ecde9d9 into nodejs:main Aug 18, 2023
37 checks passed

UlisesGascon mentioned this pull request Sep 10, 2023

v20.7.0 proposal #49592

Merged

targos mentioned this pull request Nov 28, 2023

v18.19.0 release proposal #50953

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

test: make test-perf-hooks more robust and work with workers #49197

test: make test-perf-hooks more robust and work with workers #49197

joyeecheung commented Aug 16, 2023 •

edited

Loading

joyeecheung commented Aug 16, 2023

nodejs-github-bot commented Aug 16, 2023

debadree25 left a comment

debadree25 commented Aug 16, 2023

joyeecheung commented Aug 16, 2023 •

edited

Loading

debadree25 commented Aug 16, 2023

aduh95 Aug 16, 2023

joyeecheung Aug 17, 2023

joyeecheung commented Aug 18, 2023 •

edited

Loading

nodejs-github-bot commented Aug 18, 2023

	assert(testStartTime < 15000, `${testStartTime} >= 15000`);
	assert(testStartTime < 15000, `${testStartTime} ≥ 15000`);

test: make test-perf-hooks more robust and work with workers #49197

test: make test-perf-hooks more robust and work with workers #49197

Conversation

joyeecheung commented Aug 16, 2023 • edited Loading

joyeecheung commented Aug 16, 2023

nodejs-github-bot commented Aug 16, 2023

debadree25 left a comment

Choose a reason for hiding this comment

debadree25 commented Aug 16, 2023

joyeecheung commented Aug 16, 2023 • edited Loading

debadree25 commented Aug 16, 2023

aduh95 Aug 16, 2023

Choose a reason for hiding this comment

joyeecheung Aug 17, 2023

Choose a reason for hiding this comment

joyeecheung commented Aug 18, 2023 • edited Loading

nodejs-github-bot commented Aug 18, 2023

joyeecheung commented Aug 16, 2023 •

edited

Loading

joyeecheung commented Aug 16, 2023 •

edited

Loading

joyeecheung commented Aug 18, 2023 •

edited

Loading