CI is unusable #31955

tniessen · 2020-02-25T20:41:55Z

tniessen · 2020-02-25T20:43:12Z

cc @Trott @nodejs/build

rvagg · 2020-02-26T01:58:27Z

Browsing through some of these I'm not seeing a pattern that pertains to our build infra.

Unless you can hone in on a particular infra issue, it's on you all to give our infra better code so we don't have so many flaky failures.

Trott · 2020-02-26T03:58:49Z

@rvagg @tniessen One of the known_issues tests is failing (that is, passing) pretty much constantly on Raspberry Pi devices. I don't know why that started but it's very recent, probably in the last 24 hours I'd say.

gireeshpunathil · 2020-02-26T06:09:37Z

I am going through some of the failures and triaging those.

sam-github · 2020-02-26T15:03:47Z

@gireeshpunathil I would suggest not going through build failures in node-test-pull-request, failures are expected on PRs (code fails), so its hard to detangle expected vs unexpected failures.

I suggest you look at https://ci.nodejs.org/job/node-daily-master/ -- those builds all come from master, therefore every failure there is either a failure in infrastructure, or a flaky test.

nodejs/node-core-utils#379 can be helpful in analysis.

output

core/node-core-utils (add-daily-master $% u=) % ./bin/ncu-ci daily
✔  Done
--------------------------------------------------------------------------------
[1/8] Running health
--------------------------------------------------------------------------------
| UTC Time         | RUNNING | SUCCESS | UNSTABLE | ABORTED | FAILURE | Green Rate |
| ---------------- | ------- | ------- | -------- | ------- | ------- | ---------- |
| 2020-02-26 15:02 | 0       | 1       | 4        | 0       | 7       | 8.33%      |

--------------------------------------------------------------------------------
[2/8] Running https://ci.nodejs.org/job/node-daily-master/1847/
--------------------------------------------------------------------------------
✔  Build data downloaded
✔  Build data downloaded
✔  Data downloaded
----------------------------------- Summary ------------------------------------
Result     FAILURE
URL        https://ci.nodejs.org/job/node-test-commit/35963/
Source     https://api.github.com/repos/nodejs/node/git/refs/heads/master
Commit     [940325042bef] test: add secp224k1 check in crypto-dh-stateless
Date       2020-02-25 06:14:32 +0100
Author     Daniel Bevenius <daniel.bevenius@gmail.com>
------------------------ RUN_SUBSET=0,label=pi2-docker -------------------------
URL        https://ci.nodejs.org/job/node-test-binary-arm-12+/RUN_SUBSET=0,label=pi2-docker/4539/console
Type       JS_TEST_FAILURE
Built On   test-requireio_joeyvandijk-debian10-armv7l_pi2-1
Reason
  not ok 650 known_issues/test-vm-timeout-escape-queuemicrotask
    ---
    duration_ms: 1.243
    severity: fail
    stack: |-
    ...

------------------------ RUN_SUBSET=0,label=pi3-docker -------------------------
URL        https://ci.nodejs.org/job/node-test-binary-arm-12+/RUN_SUBSET=0,label=pi3-docker/4539/console
Type       JS_TEST_FAILURE
Built On   test-requireio_securogroup-debian10-arm64_pi3-2
Reason
  not ok 660 parallel/test-worker-stdio
    ---
    duration_ms: 251.448
    severity: fail
    exitcode: -15
    stack: |-
      timeout
    ...

--------------------------------- Other builds ---------------------------------
--------------------------------------------------------------------------------
[3/8] Running https://ci.nodejs.org/job/node-daily-master/1846/
--------------------------------------------------------------------------------
✔  Build data downloaded
✔  Build data downloaded
----------------------------------- Summary ------------------------------------
Result     UNSTABLE
URL        https://ci.nodejs.org/job/node-test-commit/35927/
Source     https://api.github.com/repos/nodejs/node/git/refs/heads/master
Commit     [9c702922cdcf] async_hooks: introduce async-context API
Date       2020-02-23 20:24:02 +0100
Author     Vladimir de Turckheim <vlad2t@hotmail.com>
--------------------------------- Other builds ---------------------------------
Unstable   https://ci.nodejs.org/job/node-test-commit-arm-fanned/13338/
--------------------------------------------------------------------------------
[4/8] Running https://ci.nodejs.org/job/node-daily-master/1844/
--------------------------------------------------------------------------------
✔  Build data downloaded
✔  Build data downloaded
✔  Data downloaded
----------------------------------- Summary ------------------------------------
Result     FAILURE
URL        https://ci.nodejs.org/job/node-test-commit/35887/
Source     https://api.github.com/repos/nodejs/node/git/refs/heads/master
Commit     [2f23918ca509] doc: update stream.pipeline() signature
Date       2020-02-22 08:18:43 +0100
Author     Luigi Pinca <luigipinca@gmail.com>
------------------------------- node-test-linter -------------------------------
URL        https://ci.nodejs.org/job/node-test-linter/32923/console
Type       BUILD_FAILURE
Built On   test-packetnet-ubuntu1604-x64-1
Reason
  Error: No module named enum
  Makefile:1347: recipe for target 'lint-py' failed
  make: *** [lint-py] Error 1
  + cat test-eslint.tap
  + grep -v '^ok\|^TAP version 13\|^1\.\.'
  + sed '/^\s*$/d'
--------------------------------- Other builds ---------------------------------
Unstable   https://ci.nodejs.org/job/node-test-commit-arm-fanned/13298/
--------------------------------------------------------------------------------
[5/8] Running https://ci.nodejs.org/job/node-daily-master/1841/
--------------------------------------------------------------------------------
✔  Build data downloaded
✔  Build data downloaded
----------------------------------- Summary ------------------------------------
Result     UNSTABLE
URL        https://ci.nodejs.org/job/node-test-commit/35839/
Source     https://api.github.com/repos/nodejs/node/git/refs/heads/master
Commit     [b8e41774d428] fs: add fs/promises alias module
Date       2020-02-18 22:15:50 -0800
Author     Gus Caplan <me@gus.host>
--------------------------------- Other builds ---------------------------------
Unstable   https://ci.nodejs.org/job/node-test-commit-arm-fanned/13255/
--------------------------------------------------------------------------------
[6/8] Running https://ci.nodejs.org/job/node-daily-master/1839/
--------------------------------------------------------------------------------
✔  Build data downloaded
✔  Build data downloaded
✔  Data downloaded
----------------------------------- Summary ------------------------------------
Result     FAILURE
URL        https://ci.nodejs.org/job/node-test-commit/35767/
Source     https://api.github.com/repos/nodejs/node/git/refs/heads/master
Commit     [4c746a6cfda9] doc: move gireeshpunathil to TSC emeritus
Date       2020-02-17 10:33:54 +0530
Author     Gireesh Punathil <gpunathi@in.ibm.com>
--------------- RUN_SUBSET=1,nodes=win2008r2-COMPILED_BY-vs2017 ----------------
URL        https://ci.nodejs.org/job/node-test-binary-windows-js-suites/RUN_SUBSET=1,nodes=win2008r2-COMPILED_BY-vs2017/1820/console
Type       JENKINS_FAILURE
Built On   test-rackspace-win2008r2_vs2017-x64-3
Reason
  Build timed out (after 10 minutes). Marking the build as failed.
------------------------------ centos7-arm64-gcc6 ------------------------------
URL        https://ci.nodejs.org/job/node-test-commit-arm/nodes=centos7-arm64-gcc6/29460/console
Type       JS_TEST_FAILURE
Built On   test-packetnet-centos7-arm64-2
Reason
  not ok 2899 sequential/test-worker-prof
    ---
    duration_ms: 0.814
    severity: fail
    exitcode: 1
    stack: |-
      assert.js:102
        throw new AssertionError(obj);
        ^

      AssertionError [ERR_ASSERTION]: child exited with signal: {
        status: null,
        signal: 'SIGSEGV',
        output: [ null, '', '' ],
        pid: 98106,
        stdout: '',
        stderr: ''
      }
          at Object.<anonymous> (/home/iojs/build/workspace/node-test-commit-arm/nodes/centos7-arm64-gcc6/test/sequential/test-worker-prof.js:57:10)
          at Module._compile (internal/modules/cjs/loader.js:1206:30)
          at Object.Module._extensions..js (internal/modules/cjs/loader.js:1226:10)
          at Module.load (internal/modules/cjs/loader.js:1055:32)
          at Function.Module._load (internal/modules/cjs/loader.js:950:14)
          at Function.executeUserEntryPoint [as runMain] (internal/modules/run_main.js:71:12)
          at internal/main/run_main_module.js:17:47 {
        generatedMessage: false,
        code: 'ERR_ASSERTION',
        actual: 'SIGSEGV',
        expected: null,
        operator: 'strictEqual'
      }
    ...

------------------------ RUN_SUBSET=1,label=pi3-docker -------------------------
URL        https://ci.nodejs.org/job/node-test-binary-arm-12+/RUN_SUBSET=1,label=pi3-docker/4399/console
Type       GIT_FAILURE
Built On   test-requireio_notthetup_sayanee-debian10-arm64_pi3-1
Reason
  warning: failed to remove out/Release/.nfs00000000005a46070000098f: Device or resource busy
------------------------ RUN_SUBSET=3,label=pi3-docker -------------------------
URL        https://ci.nodejs.org/job/node-test-binary-arm-12+/RUN_SUBSET=3,label=pi3-docker/4399/console
Type       GIT_FAILURE
Built On   test-requireio_notthetup_sayanee-debian10-arm64_pi3-1
Reason
  warning: failed to remove out/Release/.nfs00000000005a46070000098f: Device or resource busy
--------------------------------- Other builds ---------------------------------
--------------------------------------------------------------------------------
[7/8] Running https://ci.nodejs.org/job/node-daily-master/1838/
--------------------------------------------------------------------------------
✔  Build data downloaded
✔  Build data downloaded
✔  Data downloaded
----------------------------------- Summary ------------------------------------
Result     FAILURE
URL        https://ci.nodejs.org/job/node-test-commit/35752/
Source     https://api.github.com/repos/nodejs/node/git/refs/heads/master
Commit     [a7c523e26df4] src: prefer 3-argument Array::New()
Date       2020-02-15 21:39:32 +0100
Author     Anna Henningsen <anna@addaleax.net>
------------------------------ centos7-arm64-gcc6 ------------------------------
URL        https://ci.nodejs.org/job/node-test-commit-arm/nodes=centos7-arm64-gcc6/29448/console
Type       JS_TEST_FAILURE
Built On   test-packetnet-centos7-arm64-2
Reason
  not ok 2897 sequential/test-worker-prof
    ---
    duration_ms: 1.716
    severity: fail
    exitcode: 1
    stack: |-
      assert.js:102
        throw new AssertionError(obj);
        ^

      AssertionError [ERR_ASSERTION]: child exited with signal: {
        status: null,
        signal: 'SIGSEGV',
        output: [ null, '', '' ],
        pid: 91489,
        stdout: '',
        stderr: ''
      }
          at Object.<anonymous> (/home/iojs/build/workspace/node-test-commit-arm/nodes/centos7-arm64-gcc6/test/sequential/test-worker-prof.js:57:10)
          at Module._compile (internal/modules/cjs/loader.js:1206:30)
          at Object.Module._extensions..js (internal/modules/cjs/loader.js:1226:10)
          at Module.load (internal/modules/cjs/loader.js:1055:32)
          at Function.Module._load (internal/modules/cjs/loader.js:950:14)
          at Function.executeUserEntryPoint [as runMain] (internal/modules/run_main.js:71:12)
          at internal/main/run_main_module.js:17:47 {
        generatedMessage: false,
        code: 'ERR_ASSERTION',
        actual: 'SIGSEGV',
        expected: null,
        operator: 'strictEqual'
      }
    ...

------------------------ RUN_SUBSET=1,label=pi2-docker -------------------------
URL        https://ci.nodejs.org/job/node-test-binary-arm-12+/RUN_SUBSET=1,label=pi2-docker/4387/console
Type       JENKINS_FAILURE
Built On   test-requireio_louiscntr-debian10-armv7l_pi2-1
Reason
  java.io.IOException: Backing channel 'JNLP4-connect connection from 203.217.18.114/203.217.18.114:56254' is disconnected.
        at hudson.remoting.RemoteInvocationHandler.channelOrFail(RemoteInvocationHandler.java:214)
        at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:283)
        at com.sun.proxy.$Proxy90.isAlive(Unknown Source)
        at hudson.Launcher$RemoteLauncher$ProcImpl.isAlive(Launcher.java:1150)
        at hudson.Launcher$RemoteLauncher$ProcImpl.join(Launcher.java:1142)
------------------------ RUN_SUBSET=1,label=pi3-docker -------------------------
URL        https://ci.nodejs.org/job/node-test-binary-arm-12+/RUN_SUBSET=1,label=pi3-docker/4387/console
Type       GIT_FAILURE
Built On   test-requireio_notthetup_sayanee-debian10-arm64_pi3-1
Reason
  warning: failed to remove out/Release/.nfs00000000005a46070000098f: Device or resource busy
------------------------ RUN_SUBSET=3,label=pi3-docker -------------------------
URL        https://ci.nodejs.org/job/node-test-binary-arm-12+/RUN_SUBSET=3,label=pi3-docker/4387/console
Type       GIT_FAILURE
Built On   test-requireio_notthetup_sayanee-debian10-arm64_pi3-1
Reason
  warning: failed to remove out/Release/.nfs00000000005a46070000098f: Device or resource busy
--------------------------------- Other builds ---------------------------------
Unstable   https://ci.nodejs.org/job/node-test-commit-windows-fanned/33992/
--------------------------------------------------------------------------------
[8/8] Running https://ci.nodejs.org/job/node-daily-master/1837/
--------------------------------------------------------------------------------
✔  Build data downloaded
✔  Build data downloaded
✔  Data downloaded
----------------------------------- Summary ------------------------------------
Result     FAILURE
URL        https://ci.nodejs.org/job/node-test-commit/35734/
Source     https://api.github.com/repos/nodejs/node/git/refs/heads/master
Commit     [79296dc2d02c] doc: guide - using valgrind to debug memory leaks
Date       2020-02-14 18:19:16 -0500
Author     Michael Dawson <michael_dawson@ca.ibm.com>
------------------------ RUN_SUBSET=0,label=pi2-docker -------------------------
URL        https://ci.nodejs.org/job/node-test-binary-arm-12+/RUN_SUBSET=0,label=pi2-docker/4370/console
Type       GIT_FAILURE
Built On   test-requireio_rvagg-debian10-armv7l_pi2-1
Reason
  warning: failed to remove out/Release/.nfs0000000000763f2b000012bc: Device or resource busy
------------------------ RUN_SUBSET=0,label=pi3-docker -------------------------
URL        https://ci.nodejs.org/job/node-test-binary-arm-12+/RUN_SUBSET=0,label=pi3-docker/4370/console
Type       GIT_FAILURE
Built On   test-requireio_williamkapke-debian10-arm64_pi3-1
Reason
  warning: failed to remove out/Release/.nfs00000000002caa3800000faf: Device or resource busy
------------------------ RUN_SUBSET=1,label=pi2-docker -------------------------
URL        https://ci.nodejs.org/job/node-test-binary-arm-12+/RUN_SUBSET=1,label=pi2-docker/4370/console
Type       GIT_FAILURE
Built On   test-requireio_joeyvandijk-debian10-armv7l_pi2-2
Reason
  warning: failed to remove out/Release/.nfs00000000007c439200001175: Device or resource busy
------------------------ RUN_SUBSET=1,label=pi3-docker -------------------------
URL        https://ci.nodejs.org/job/node-test-binary-arm-12+/RUN_SUBSET=1,label=pi3-docker/4370/console
Type       GIT_FAILURE
Built On   test-requireio_notthetup_sayanee-debian10-arm64_pi3-1
Reason
  warning: failed to remove out/Release/.nfs00000000005a46070000098f: Device or resource busy
------------------------ RUN_SUBSET=2,label=pi2-docker -------------------------
URL        https://ci.nodejs.org/job/node-test-binary-arm-12+/RUN_SUBSET=2,label=pi2-docker/4370/console
Type       GIT_FAILURE
Built On   test-requireio_joeyvandijk-debian10-armv7l_pi2-1
Reason
  warning: failed to remove out/Release/.nfs00000000007c5ad000001385: Device or resource busy
------------------------ RUN_SUBSET=2,label=pi3-docker -------------------------
URL        https://ci.nodejs.org/job/node-test-binary-arm-12+/RUN_SUBSET=2,label=pi3-docker/4370/console
Type       GIT_FAILURE
Built On   test-requireio_pivotalagency-debian10-arm64_pi3-1
Reason
  warning: failed to remove out/Release/.nfs00000000005e146e000009f6: Device or resource busy
--------------------------------- Other builds ---------------------------------
Unstable   https://ci.nodejs.org/job/node-test-commit-windows-fanned/33978/

------------------------------------ Stats -------------------------------------

------------------------------- JENKINS_FAILURE --------------------------------
Reason     Backing channel 'JNLP4-connect connection from ... is disconnected.
Type       JENKINS_FAILURE
Failed PR  1 (https://api.github.com/repos/nodejs/node/git/refs/heads/master)
Appeared   test-requireio_louiscntr-debian10-armv7l_pi2-1
Last CI    https://ci.nodejs.org/job/node-test-commit/35752/

Example: https://ci.nodejs.org/job/node-test-binary-arm-12+/RUN_SUBSET=1,label=pi2-docker/4387/console

java.io.IOException: Backing channel 'JNLP4-connect connection from 203.217.18.114/203.217.18.114:56254' is disconnected.
        at hudson.remoting.RemoteInvocationHandler.channelOrFail(RemoteInvocationHandler.java:214)
        at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:283)
        at com.sun.proxy.$Proxy90.isAlive(Unknown Source)
        at hudson.Launcher$RemoteLauncher$ProcImpl.isAlive(Launcher.java:1150)
        at hudson.Launcher$RemoteLauncher$ProcImpl.join(Launcher.java:1142)
--------------------------------------------------------------------------------
Reason     Build timed out (after 10 minutes). Marking the build as failed.
Type       JENKINS_FAILURE
Failed PR  1 (https://api.github.com/repos/nodejs/node/git/refs/heads/master)
Appeared   test-rackspace-win2008r2_vs2017-x64-3
Last CI    https://ci.nodejs.org/job/node-test-commit/35767/

Example: https://ci.nodejs.org/job/node-test-binary-windows-js-suites/RUN_SUBSET=1,nodes=win2008r2-COMPILED_BY-vs2017/1820/console

Build timed out (after 10 minutes). Marking the build as failed.
--------------------------------------------------------------------------------
-------------------------------- BUILD_FAILURE ---------------------------------
Reason     Error: No module named enum
Type       BUILD_FAILURE
Failed PR  1 (https://api.github.com/repos/nodejs/node/git/refs/heads/master)
Appeared   test-packetnet-ubuntu1604-x64-1
Last CI    https://ci.nodejs.org/job/node-test-commit/35887/

Example: https://ci.nodejs.org/job/node-test-linter/32923/console

Error: No module named enum
Makefile:1347: recipe for target 'lint-py' failed
make: *** [lint-py] Error 1
+ cat test-eslint.tap
+ grep -v '^ok\|^TAP version 13\|^1\.\.'
+ sed '/^\s*$/d'
--------------------------------------------------------------------------------
------------------------------- JS_TEST_FAILURE --------------------------------
Reason     known_issues/test-vm-timeout-escape-queuemicrotask
Type       JS_TEST_FAILURE
Failed PR  1 (https://api.github.com/repos/nodejs/node/git/refs/heads/master)
Appeared   test-requireio_joeyvandijk-debian10-armv7l_pi2-1
Last CI    https://ci.nodejs.org/job/node-test-commit/35963/

Example: https://ci.nodejs.org/job/node-test-binary-arm-12+/RUN_SUBSET=0,label=pi2-docker/4539/console

not ok 650 known_issues/test-vm-timeout-escape-queuemicrotask
  ---
  duration_ms: 1.243
  severity: fail
  stack: |-
  ...

--------------------------------------------------------------------------------
Reason     parallel/test-worker-stdio
Type       JS_TEST_FAILURE
Failed PR  1 (https://api.github.com/repos/nodejs/node/git/refs/heads/master)
Appeared   test-requireio_securogroup-debian10-arm64_pi3-2
Last CI    https://ci.nodejs.org/job/node-test-commit/35963/

Example: https://ci.nodejs.org/job/node-test-binary-arm-12+/RUN_SUBSET=0,label=pi3-docker/4539/console

not ok 660 parallel/test-worker-stdio
  ---
  duration_ms: 251.448
  severity: fail
  exitcode: -15
  stack: |-
    timeout
  ...

--------------------------------------------------------------------------------
Reason     sequential/test-worker-prof
Type       JS_TEST_FAILURE
Failed PR  1 (https://api.github.com/repos/nodejs/node/git/refs/heads/master)
Appeared   test-packetnet-centos7-arm64-2
Last CI    https://ci.nodejs.org/job/node-test-commit/35767/

Example: https://ci.nodejs.org/job/node-test-commit-arm/nodes=centos7-arm64-gcc6/29460/console

not ok 2899 sequential/test-worker-prof
  ---
  duration_ms: 0.814
  severity: fail
  exitcode: 1
  stack: |-
    assert.js:102
      throw new AssertionError(obj);
      ^

    AssertionError [ERR_ASSERTION]: child exited with signal: {
      status: null,
      signal: 'SIGSEGV',
      output: [ null, '', '' ],
      pid: 98106,
      stdout: '',
      stderr: ''
    }
        at Object.<anonymous> (/home/iojs/build/workspace/node-test-commit-arm/nodes/centos7-arm64-gcc6/test/sequential/test-worker-pr...
--------------------------------------------------------------------------------
--------------------------------- GIT_FAILURE ----------------------------------
Reason     warning: failed to remove out/Release/.nfs00000000002caa3800000faf: Device or resource busy
Type       GIT_FAILURE
Failed PR  1 (https://api.github.com/repos/nodejs/node/git/refs/heads/master)
Appeared   test-requireio_williamkapke-debian10-arm64_pi3-1
Last CI    https://ci.nodejs.org/job/node-test-commit/35734/

Example: https://ci.nodejs.org/job/node-test-binary-arm-12+/RUN_SUBSET=0,label=pi3-docker/4370/console

warning: failed to remove out/Release/.nfs00000000002caa3800000faf: Device or resource busy
--------------------------------------------------------------------------------
Reason     warning: failed to remove out/Release/.nfs00000000005a46070000098f: Device or resource busy
Type       GIT_FAILURE
Failed PR  1 (https://api.github.com/repos/nodejs/node/git/refs/heads/master)
Appeared   test-requireio_notthetup_sayanee-debian10-arm64_pi3-1
Last CI    https://ci.nodejs.org/job/node-test-commit/35767/

Example: https://ci.nodejs.org/job/node-test-binary-arm-12+/RUN_SUBSET=1,label=pi3-docker/4399/console

warning: failed to remove out/Release/.nfs00000000005a46070000098f: Device or resource busy
--------------------------------------------------------------------------------
Reason     warning: failed to remove out/Release/.nfs00000000005e146e000009f6: Device or resource busy
Type       GIT_FAILURE
Failed PR  1 (https://api.github.com/repos/nodejs/node/git/refs/heads/master)
Appeared   test-requireio_pivotalagency-debian10-arm64_pi3-1
Last CI    https://ci.nodejs.org/job/node-test-commit/35734/

Example: https://ci.nodejs.org/job/node-test-binary-arm-12+/RUN_SUBSET=2,label=pi3-docker/4370/console

warning: failed to remove out/Release/.nfs00000000005e146e000009f6: Device or resource busy
--------------------------------------------------------------------------------
Reason     warning: failed to remove out/Release/.nfs0000000000763f2b000012bc: Device or resource busy
Type       GIT_FAILURE
Failed PR  1 (https://api.github.com/repos/nodejs/node/git/refs/heads/master)
Appeared   test-requireio_rvagg-debian10-armv7l_pi2-1
Last CI    https://ci.nodejs.org/job/node-test-commit/35734/

Example: https://ci.nodejs.org/job/node-test-binary-arm-12+/RUN_SUBSET=0,label=pi2-docker/4370/console

warning: failed to remove out/Release/.nfs0000000000763f2b000012bc: Device or resource busy
--------------------------------------------------------------------------------
Reason     warning: failed to remove out/Release/.nfs00000000007c439200001175: Device or resource busy
Type       GIT_FAILURE
Failed PR  1 (https://api.github.com/repos/nodejs/node/git/refs/heads/master)
Appeared   test-requireio_joeyvandijk-debian10-armv7l_pi2-2
Last CI    https://ci.nodejs.org/job/node-test-commit/35734/

Example: https://ci.nodejs.org/job/node-test-binary-arm-12+/RUN_SUBSET=1,label=pi2-docker/4370/console

warning: failed to remove out/Release/.nfs00000000007c439200001175: Device or resource busy
--------------------------------------------------------------------------------
Reason     warning: failed to remove out/Release/.nfs00000000007c5ad000001385: Device or resource busy
Type       GIT_FAILURE
Failed PR  1 (https://api.github.com/repos/nodejs/node/git/refs/heads/master)
Appeared   test-requireio_joeyvandijk-debian10-armv7l_pi2-1
Last CI    https://ci.nodejs.org/job/node-test-commit/35734/

Example: https://ci.nodejs.org/job/node-test-binary-arm-12+/RUN_SUBSET=2,label=pi2-docker/4370/console

warning: failed to remove out/Release/.nfs00000000007c5ad000001385: Device or resource busy
--------------------------------------------------------------------------------
core/node-core-utils (add-daily-master $% u=) %

gireeshpunathil · 2020-02-27T03:56:29Z

apologies all, half of my triages went erroneous. Everyday something new to learn, so!

tniessen · 2020-03-13T00:15:21Z

We can land PRs due to CI being yellow-ish (which means "acceptable failure"), but it is still not looking great:

Trott · 2020-03-13T04:14:44Z

We can land PRs due to CI being yellow-ish (which means "acceptable failure"), but it is still not looking great:

Perma-yellow at this time is to due to node-test-commit-osx, specifically the new osx1015. Given that we have lots of macOS users as Collaborators, I'm hopeful that someone can dig deep on those failures and get that working reliably. @nodejs/platform-macos

AshCripps · 2020-03-13T11:03:56Z

@Trott we are waiting on Apple to fix their regression to lift us out of perma yellow - #30030 (comment)

sam-github · 2020-03-13T13:28:48Z

I still think that case should be skipped, perhaps actual js code (since it doesn't need to be skipped for all OS X versions). Our CI shouldn't be yellow if Apple, at their discretion, no longer ship a mostly unused network stack feature (kq notification of zero-length udp packets). It should be yellow if we have a bug.

Trott · 2020-03-13T14:59:15Z

I still think that case should be skipped, perhaps actual js code (since it doesn't need to be skipped for all OS X versions). Our CI shouldn't be yellow if Apple, at their discretion, no longer ship a mostly unused network stack feature (kq notification of zero-length udp packets). It should be yellow if we have a bug.

There's precedence for this too. We definitely used to have code in a test that skipped if it detected it was on macOS at a certain version. We took it out when Apple fixed the bug.

I agree with Sam that an OS bug that causes a test to fail all the time should be skipped. (Even better if a second nearly-identical test is added to known_issues so that we can detect when the issue is fixed. But that's a bonus.) Tests should be marked flaky if they are unreliable, not if they are completely broken.

mmarchini · 2020-03-13T15:08:29Z

Which test(s) should be skipped on osx1015?

AshCripps · 2020-03-13T15:13:13Z

The ones marked flaky here - https://github.com/nodejs/node/pull/31936/files

would need backporting too to at least 12 (some are missing on 10 so its not as affected IIRC)

mmarchini · 2020-03-17T20:59:11Z

FWIW the last few PRs I landed I had to run CI a few times before getting a green/yellow, so the problem is not limited to the OS X 10.15 issue.

Trott · 2020-03-29T14:28:15Z

As of right now, the highest failure rate is on node-test-commit-linux-containered.

jasnell · 2020-06-25T22:56:54Z

Given that we have the flaky test project board and there really hasn't been updates on this in a while, closing. Can reopen if necessary.

tniessen added flaky-test Issues and PRs related to the tests with unstable failures on the CI. build Issues and PRs related to build files or the CI. labels Feb 25, 2020

mmarchini mentioned this issue Mar 13, 2020

Node.js future directions - any interest in online or in person summit? nodejs/TSC#797

Closed

jasnell closed this as completed Jun 25, 2020

This comment has been minimized.

Sign in to view

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

CI is unusable #31955

CI is unusable #31955

tniessen commented Feb 25, 2020

tniessen commented Feb 25, 2020

rvagg commented Feb 26, 2020

Trott commented Feb 26, 2020

gireeshpunathil commented Feb 26, 2020

sam-github commented Feb 26, 2020 •

edited

Loading

gireeshpunathil commented Feb 27, 2020

tniessen commented Mar 13, 2020

Trott commented Mar 13, 2020

AshCripps commented Mar 13, 2020

sam-github commented Mar 13, 2020

Trott commented Mar 13, 2020

mmarchini commented Mar 13, 2020

AshCripps commented Mar 13, 2020

mmarchini commented Mar 17, 2020

Trott commented Mar 29, 2020

jasnell commented Jun 25, 2020

This comment has been minimized.

CI is unusable #31955

CI is unusable #31955

Comments

tniessen commented Feb 25, 2020

tniessen commented Feb 25, 2020

rvagg commented Feb 26, 2020

Trott commented Feb 26, 2020

gireeshpunathil commented Feb 26, 2020

sam-github commented Feb 26, 2020 • edited Loading

gireeshpunathil commented Feb 27, 2020

tniessen commented Mar 13, 2020

Trott commented Mar 13, 2020

AshCripps commented Mar 13, 2020

sam-github commented Mar 13, 2020

Trott commented Mar 13, 2020

mmarchini commented Mar 13, 2020

AshCripps commented Mar 13, 2020

mmarchini commented Mar 17, 2020

Trott commented Mar 29, 2020

jasnell commented Jun 25, 2020

This comment has been minimized.

sam-github commented Feb 26, 2020 •

edited

Loading