Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Flaky linux tests #18539

Closed
maclover7 opened this issue Feb 2, 2018 · 10 comments
Closed

Flaky linux tests #18539

maclover7 opened this issue Feb 2, 2018 · 10 comments
Labels
doc Issues and PRs related to the documentations. flaky-test Issues and PRs related to the tests with unstable failures on the CI. test Issues and PRs related to the tests.

Comments

@maclover7
Copy link
Contributor

maclover7 commented Feb 2, 2018 •
edited by gibfahn
Loading

  • Version: master
  • Platform: Fedora24, Ubuntu1404, possibly other Linux distros as well
  • Subsystem: test

There seems to be a more recurring issue with running native addons tests on Linux. I have seen many fail lately with the message "Failed to add alternative version links to index". I pasted two examples below, but I remember there being more in the past.

Example from https://ci.nodejs.org/job/node-test-commit-linux/16068/nodes=ubuntu1404-64/console:

Building addon /home/iojs/build/workspace/node-test-commit-linux/nodes/ubuntu1404-64/test/addons/zlib-binding/
gyp info it worked if it ends with ok
gyp info using node-gyp@3.6.2
gyp info using node@10.0.0-pre | linux | x64
gyp info chdir /home/iojs/build/workspace/node-test-commit-linux/nodes/ubuntu1404-64/test/addons/zlib-binding/
gyp info spawn /usr/bin/python
gyp info spawn args [ '/home/iojs/build/workspace/node-test-commit-linux/nodes/ubuntu1404-64/deps/npm/node_modules/node-gyp/gyp/gyp_main.py',
gyp info spawn args   'binding.gyp',
gyp info spawn args   '-f',
gyp info spawn args   'make',
gyp info spawn args   '-I',
gyp info spawn args   '/home/iojs/build/workspace/node-test-commit-linux/nodes/ubuntu1404-64/test/addons/zlib-binding/build/config.gypi',
gyp info spawn args   '-I',
gyp info spawn args   '/home/iojs/build/workspace/node-test-commit-linux/nodes/ubuntu1404-64/deps/npm/node_modules/node-gyp/addon.gypi',
gyp info spawn args   '-I',
gyp info spawn args   '/home/iojs/build/workspace/node-test-commit-linux/nodes/ubuntu1404-64/common.gypi',
gyp info spawn args   '-Dlibrary=shared_library',
gyp info spawn args   '-Dvisibility=default',
gyp info spawn args   '-Dnode_root_dir=/home/iojs/build/workspace/node-test-commit-linux/nodes/ubuntu1404-64',
gyp info spawn args   '-Dnode_gyp_dir=/home/iojs/build/workspace/node-test-commit-linux/nodes/ubuntu1404-64/deps/npm/node_modules/node-gyp',
gyp info spawn args   '-Dnode_lib_file=/home/iojs/build/workspace/node-test-commit-linux/nodes/ubuntu1404-64/$(Configuration)/node.lib',
gyp info spawn args   '-Dmodule_root_dir=/home/iojs/build/workspace/node-test-commit-linux/nodes/ubuntu1404-64/test/addons/zlib-binding',
gyp info spawn args   '-Dnode_engine=v8',
gyp info spawn args   '--depth=.',
gyp info spawn args   '--no-parallel',
gyp info spawn args   '--generator-output',
gyp info spawn args   'build',
gyp info spawn args   '-Goutput_dir=.' ]
gyp info spawn make
gyp info spawn args [ 'BUILDTYPE=Release', '-C', 'build', '--jobs', 2 ]
make[2]: Entering directory `/home/iojs/build/workspace/node-test-commit-linux/nodes/ubuntu1404-64/test/addons/zlib-binding/build'
  CXX(target) Release/obj.target/binding/binding.o
  SOLINK_MODULE(target) Release/obj.target/binding.node
  COPY Release/binding.node
make[2]: Leaving directory `/home/iojs/build/workspace/node-test-commit-linux/nodes/ubuntu1404-64/test/addons/zlib-binding/build'
gyp info ok 
touch test/addons/.buildstamp
Failed to add alternative version links to index
Aborted (core dumped)
make[2]: *** [out/doc/api/string_decoder.json] Error 134
make[2]: *** Waiting for unfinished jobs....
make[2]: Leaving directory `/home/iojs/build/workspace/node-test-commit-linux/nodes/ubuntu1404-64'
make[1]: *** [doc-only] Error 2
make[1]: Leaving directory `/home/iojs/build/workspace/node-test-commit-linux/nodes/ubuntu1404-64'
make: *** [run-ci] Error 2
Build step 'Execute shell' marked build as failure
Run condition [Always] enabling perform for step [[]]
Run condition [Always] enabling perform for step [[]]
TAP Reports Processing: START
Looking for TAP results report in workspace using pattern: *.tap
Did not find any matching files. Setting build result to FAILURE.
Checking ^not ok
Jenkins Text Finder: File set '*.tap' is empty
Notifying upstream projects of job completion
Finished: FAILURE

Example from https://ci.nodejs.org/job/node-test-commit-linux/16068/nodes=ubuntu1404-64/console:

Building addon /home/iojs/build/workspace/node-test-commit-linux/nodes/ubuntu1404-64/test/addons/zlib-binding/
gyp info it worked if it ends with ok
gyp info using node-gyp@3.6.2
gyp info using node@10.0.0-pre | linux | x64
gyp info chdir /home/iojs/build/workspace/node-test-commit-linux/nodes/ubuntu1404-64/test/addons/zlib-binding/
gyp info spawn /usr/bin/python
gyp info spawn args [ '/home/iojs/build/workspace/node-test-commit-linux/nodes/ubuntu1404-64/deps/npm/node_modules/node-gyp/gyp/gyp_main.py',
gyp info spawn args   'binding.gyp',
gyp info spawn args   '-f',
gyp info spawn args   'make',
gyp info spawn args   '-I',
gyp info spawn args   '/home/iojs/build/workspace/node-test-commit-linux/nodes/ubuntu1404-64/test/addons/zlib-binding/build/config.gypi',
gyp info spawn args   '-I',
gyp info spawn args   '/home/iojs/build/workspace/node-test-commit-linux/nodes/ubuntu1404-64/deps/npm/node_modules/node-gyp/addon.gypi',
gyp info spawn args   '-I',
gyp info spawn args   '/home/iojs/build/workspace/node-test-commit-linux/nodes/ubuntu1404-64/common.gypi',
gyp info spawn args   '-Dlibrary=shared_library',
gyp info spawn args   '-Dvisibility=default',
gyp info spawn args   '-Dnode_root_dir=/home/iojs/build/workspace/node-test-commit-linux/nodes/ubuntu1404-64',
gyp info spawn args   '-Dnode_gyp_dir=/home/iojs/build/workspace/node-test-commit-linux/nodes/ubuntu1404-64/deps/npm/node_modules/node-gyp',
gyp info spawn args   '-Dnode_lib_file=/home/iojs/build/workspace/node-test-commit-linux/nodes/ubuntu1404-64/$(Configuration)/node.lib',
gyp info spawn args   '-Dmodule_root_dir=/home/iojs/build/workspace/node-test-commit-linux/nodes/ubuntu1404-64/test/addons/zlib-binding',
gyp info spawn args   '-Dnode_engine=v8',
gyp info spawn args   '--depth=.',
gyp info spawn args   '--no-parallel',
gyp info spawn args   '--generator-output',
gyp info spawn args   'build',
gyp info spawn args   '-Goutput_dir=.' ]
gyp info spawn make
gyp info spawn args [ 'BUILDTYPE=Release', '-C', 'build', '--jobs', 2 ]
make[2]: Entering directory `/home/iojs/build/workspace/node-test-commit-linux/nodes/ubuntu1404-64/test/addons/zlib-binding/build'
  CXX(target) Release/obj.target/binding/binding.o
  SOLINK_MODULE(target) Release/obj.target/binding.node
  COPY Release/binding.node
make[2]: Leaving directory `/home/iojs/build/workspace/node-test-commit-linux/nodes/ubuntu1404-64/test/addons/zlib-binding/build'
gyp info ok 
touch test/addons/.buildstamp
Failed to add alternative version links to index
Aborted (core dumped)
make[2]: *** [out/doc/api/string_decoder.json] Error 134
make[2]: *** Waiting for unfinished jobs....
make[2]: Leaving directory `/home/iojs/build/workspace/node-test-commit-linux/nodes/ubuntu1404-64'
make[1]: *** [doc-only] Error 2
make[1]: Leaving directory `/home/iojs/build/workspace/node-test-commit-linux/nodes/ubuntu1404-64'
make: *** [run-ci] Error 2
Build step 'Execute shell' marked build as failure
Run condition [Always] enabling perform for step [[]]
Run condition [Always] enabling perform for step [[]]
TAP Reports Processing: START
Looking for TAP results report in workspace using pattern: *.tap
Did not find any matching files. Setting build result to FAILURE.
Checking ^not ok
Jenkins Text Finder: File set '*.tap' is empty
Notifying upstream projects of job completion
Finished: FAILURE
@maclover7 maclover7 added test Issues and PRs related to the tests. addons Issues and PRs related to native addons. flaky-test Issues and PRs related to the tests with unstable failures on the CI. labels Feb 2, 2018
@maclover7
Copy link
Contributor Author

maclover7 commented Feb 3, 2018 •
edited by gibfahn
Loading

Happened on https://ci.nodejs.org/job/node-test-commit-plinux/15054/nodes=ppcle-ubuntu1404/console

Building addon /home/iojs/build/workspace/node-test-commit-plinux/nodes/ppcle-ubuntu1404/test/addons/zlib-binding/
gyp info it worked if it ends with ok
gyp info using node-gyp@3.6.2
gyp info using node@10.0.0-pre | linux | ppc64
gyp info chdir /home/iojs/build/workspace/node-test-commit-plinux/nodes/ppcle-ubuntu1404/test/addons/zlib-binding/
gyp info spawn /usr/bin/python
gyp info spawn args [ '/home/iojs/build/workspace/node-test-commit-plinux/nodes/ppcle-ubuntu1404/deps/npm/node_modules/node-gyp/gyp/gyp_main.py',
gyp info spawn args   'binding.gyp',
gyp info spawn args   '-f',
gyp info spawn args   'make',
gyp info spawn args   '-I',
gyp info spawn args   '/home/iojs/build/workspace/node-test-commit-plinux/nodes/ppcle-ubuntu1404/test/addons/zlib-binding/build/config.gypi',
gyp info spawn args   '-I',
gyp info spawn args   '/home/iojs/build/workspace/node-test-commit-plinux/nodes/ppcle-ubuntu1404/deps/npm/node_modules/node-gyp/addon.gypi',
gyp info spawn args   '-I',
gyp info spawn args   '/home/iojs/build/workspace/node-test-commit-plinux/nodes/ppcle-ubuntu1404/common.gypi',
gyp info spawn args   '-Dlibrary=shared_library',
gyp info spawn args   '-Dvisibility=default',
gyp info spawn args   '-Dnode_root_dir=/home/iojs/build/workspace/node-test-commit-plinux/nodes/ppcle-ubuntu1404',
gyp info spawn args   '-Dnode_gyp_dir=/home/iojs/build/workspace/node-test-commit-plinux/nodes/ppcle-ubuntu1404/deps/npm/node_modules/node-gyp',
gyp info spawn args   '-Dnode_lib_file=/home/iojs/build/workspace/node-test-commit-plinux/nodes/ppcle-ubuntu1404/$(Configuration)/node.lib',
gyp info spawn args   '-Dmodule_root_dir=/home/iojs/build/workspace/node-test-commit-plinux/nodes/ppcle-ubuntu1404/test/addons/zlib-binding',
gyp info spawn args   '-Dnode_engine=v8',
gyp info spawn args   '--depth=.',
gyp info spawn args   '--no-parallel',
gyp info spawn args   '--generator-output',
gyp info spawn args   'build',
gyp info spawn args   '-Goutput_dir=.' ]
make[2]: Entering directory `/home/iojs/build/workspace/node-test-commit-plinux/nodes/ppcle-ubuntu1404/test/addons/zlib-binding/build'
  CXX(target) Release/obj.target/binding/binding.o
gyp info spawn make
gyp info spawn args [ 'BUILDTYPE=Release', '-C', 'build', '--jobs', 4 ]
  SOLINK_MODULE(target) Release/obj.target/binding.node
  COPY Release/binding.node
make[2]: Leaving directory `/home/iojs/build/workspace/node-test-commit-plinux/nodes/ppcle-ubuntu1404/test/addons/zlib-binding/build'
gyp info ok 
touch test/addons/.buildstamp
make[1]: Leaving directory `/home/iojs/build/workspace/node-test-commit-plinux/nodes/ppcle-ubuntu1404'
make: *** [run-ci] Error 2
Process leaked file descriptors. See https://jenkins.io/redirect/troubleshooting/process-leaked-file-descriptors for more information
Build step 'Execute shell' marked build as failure
Run condition [Always] enabling perform for step [[]]
TAP Reports Processing: START
Looking for TAP results report in workspace using pattern: *.tap
Did not find any matching files. Setting build result to FAILURE.
Checking ^not ok
Jenkins Text Finder: File set '*.tap' is empty
Sending e-mails to: michael_dawson@ca.ibm.com gib@uk.ibm.com
Notifying upstream projects of job completion
Finished: FAILURE

@richardlau
Copy link
Member

Failed to add alternative version links to index is a docs build error rather than addons.

@joyeecheung
Copy link
Member

AFAIK Failed to add alternative version links to ... does not cause the build to fail, it just means that the docs built don't have the correct version links but they would build just fine.

@joyeecheung
Copy link
Member

joyeecheung commented Feb 4, 2018 •
edited
Loading

The real issue in these logs is this

Aborted (core dumped)

simply building the docs should not trigger something like this...it would be helpful if we can get one of these cores to debug.

@gibfahn
Copy link
Member

gibfahn commented Feb 5, 2018

The real issue in these logs is this

The third failure doesn't include that error, so maybe a different cause.

Not sure if the core file actually gets generated (if ulimit -c is set), but if it does it probably gets cleaned up when git clean runs on the next run. So to get a core you'd probably want to take the machine offline, and try to reproduce manually to get a core.

@maclover7 maclover7 added doc Issues and PRs related to the documentations. and removed flaky-test Issues and PRs related to the tests with unstable failures on the CI. labels Feb 6, 2018
@maclover7 maclover7 changed the title Flaky native addons tests Flaky linux tests Feb 6, 2018
@maclover7 maclover7 added flaky-test Issues and PRs related to the tests with unstable failures on the CI. and removed addons Issues and PRs related to native addons. labels Feb 6, 2018
@maclover7
Copy link
Contributor Author

Slightly tweaked title for now, until can gather more information.

@maclover7
Copy link
Contributor Author

There is also an issue about this in nodejs/build: nodejs/build#1049

cc @Trott

@maclover7 maclover7 mentioned this issue Feb 7, 2018
Closed
@richardlau
Copy link
Member

And maybe nodejs/build#1114.

@maclover7
Copy link
Contributor Author

https://ci.nodejs.org/job/node-test-commit-linuxone/nodes=rhel72-s390x/13533/console

@maclover7
Copy link
Contributor Author

maclover7 commented May 24, 2018 •
edited
Loading

I forget what issue/PR, but I believe this is resolved now. At the very least, hasn't happened recently enough that people have noticed and commented. Please reopen if I'm mistaken.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
doc Issues and PRs related to the documentations. flaky-test Issues and PRs related to the tests with unstable failures on the CI. test Issues and PRs related to the tests.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@maclover7 @joyeecheung @richardlau @gibfahn