-
Notifications
You must be signed in to change notification settings - Fork 29.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Flaky parallel/test-trace-events-fs-sync #21038
Comments
test-trace-events-fs-sync has been failing ocasinally on FreeBSD. While we don't have a fix, it should be marked as flaky. Ref: nodejs#21038
Seems like this assertion should print the entire |
Not just FreeBSD. Here it is failing on LinuxONE: https://ci.nodejs.org/job/node-test-commit-linuxone/1863/nodes=rhel72-s390x/console
|
I updated the title of this to remove |
Stress test to confirm this is reproducible on FreeBSD: https://ci.nodejs.org/job/node-stress-single-test/1906/nodes=freebsd11-x64/ |
PR to improve output: #21120 |
test-trace-events-fs-sync has been failing ocasinally on FreeBSD. While we don't have a fix, it should be marked as flaky. Ref: #21038 PR-URL: #21039 Refs: #21038 Reviewed-By: James M Snell <jasnell@gmail.com> Reviewed-By: Refael Ackermann <refack@gmail.com> Reviewed-By: Trivikram Kamat <trivikr.dev@gmail.com> Reviewed-By: Rich Trott <rtrott@gmail.com> Reviewed-By: Anatoli Papirovski <apapirovski@mac.com> Reviewed-By: Lance Ball <lball@redhat.com>
test-trace-events-fs-sync has been failing ocasinally on FreeBSD. While we don't have a fix, it should be marked as flaky. Ref: #21038 PR-URL: #21039 Refs: #21038 Reviewed-By: James M Snell <jasnell@gmail.com> Reviewed-By: Refael Ackermann <refack@gmail.com> Reviewed-By: Trivikram Kamat <trivikr.dev@gmail.com> Reviewed-By: Rich Trott <rtrott@gmail.com> Reviewed-By: Anatoli Papirovski <apapirovski@mac.com> Reviewed-By: Lance Ball <lball@redhat.com>
|
Stress test now that we have more useful output: https://ci.nodejs.org/job/node-stress-single-test/1918/ |
So... we've got a SIGSEGV... :|
This could be related to the trace events failures we've previously seen on Windows. ping @jasnell @ofrobots since one of you previously mentioned some segfaults that you saw in the trace events code and were planning on fixing? |
I'm investigating some SIGSEGVs in tracing today. |
There are a few flaws in the choreography of the tracing event loop on shutdown that cause the segfault: basically the libuv threadpool mutex gets destroyed as part of the static destructors, but the tracing agent shutdown happens after that. It tries to do some IO to write the terminating JSON characters, which end up crashing. I'm still looking into what might be the best way to fix this. This explains the segfault. It is not clear to me whether the other original failures are related to this same underlying issue, but first things first. |
/cc @eugeneo |
Failed on LinuxONE again today. https://ci.nodejs.org/job/node-test-commit-linuxone/2386/nodes=rhel72-s390x/console 10:12:08 not ok 1848 parallel/test-trace-events-fs-sync
10:12:08 ---
10:12:08 duration_ms: 1.462
10:12:08 severity: fail
10:12:08 exitcode: 1
10:12:08 stack: |-
10:12:08 assert.js:80
10:12:08 throw new AssertionError(obj);
10:12:08 ^
10:12:08
10:12:08 AssertionError [ERR_ASSERTION]: fs.sync.fstat:
10:12:08 { status: null,
10:12:08 signal: 'SIGSEGV',
10:12:08 output: [ null, '', '' ],
10:12:08 pid: 51890,
10:12:08 stdout: '',
10:12:08 stderr: '' }
10:12:08 at Object.<anonymous> (/data/iojs/build/workspace/node-test-commit-linuxone/nodes/rhel72-s390x/test/parallel/test-trace-events-fs-sync.js:136:10)
10:12:08 at Module._compile (internal/modules/cjs/loader.js:702:30)
10:12:08 at Object.Module._extensions..js (internal/modules/cjs/loader.js:713:10)
10:12:08 at Module.load (internal/modules/cjs/loader.js:612:32)
10:12:08 at tryModuleLoad (internal/modules/cjs/loader.js:551:12)
10:12:08 at Function.Module._load (internal/modules/cjs/loader.js:543:3)
10:12:08 at Function.Module.runMain (internal/modules/cjs/loader.js:744:10)
10:12:08 at startup (internal/bootstrap/node.js:237:19)
10:12:08 at bootstrapNodeJSCore (internal/bootstrap/node.js:561:3)
10:12:08 ... |
Failed again on LinuxONE today. https://ci.nodejs.org/job/node-test-commit-linuxone/nodes=rhel72-s390x/3018/console 22:34:36 not ok 1910 parallel/test-trace-events-fs-sync
22:34:36 ---
22:34:36 duration_ms: 3.131
22:34:36 severity: fail
22:34:36 exitcode: 1
22:34:36 stack: |-
22:34:36 assert.js:80
22:34:36 throw new AssertionError(obj);
22:34:36 ^
22:34:36
22:34:36 AssertionError [ERR_ASSERTION]: fs.sync.stat:
22:34:36 { status: null,
22:34:36 signal: 'SIGSEGV',
22:34:36 output: [ null, '', '' ],
22:34:36 pid: 60920,
22:34:36 stdout: '',
22:34:36 stderr: '' }
22:34:36 at Object.<anonymous> (/data/iojs/build/workspace/node-test-commit-linuxone/nodes/rhel72-s390x/test/parallel/test-trace-events-fs-sync.js:139:10)
22:34:36 at Module._compile (internal/modules/cjs/loader.js:689:30)
22:34:36 at Object.Module._extensions..js (internal/modules/cjs/loader.js:700:10)
22:34:36 at Module.load (internal/modules/cjs/loader.js:599:32)
22:34:36 at tryModuleLoad (internal/modules/cjs/loader.js:538:12)
22:34:36 at Function.Module._load (internal/modules/cjs/loader.js:530:3)
22:34:36 at Function.Module.runMain (internal/modules/cjs/loader.js:742:12)
22:34:36 at startup (internal/bootstrap/node.js:266:19)
22:34:36 at bootstrapNodeJSCore (internal/bootstrap/node.js:596:3)
22:34:36 ... |
LinuxONE again: https://ci.nodejs.org/job/node-test-commit-linuxone/nodes=rhel72-s390x/3551/console 21:48:50 not ok 1954 parallel/test-trace-events-fs-sync
21:48:50 ---
21:48:50 duration_ms: 2.975
21:48:50 severity: fail
21:48:50 exitcode: 1
21:48:50 stack: |-
21:48:50 assert.js:84
21:48:50 throw new AssertionError(obj);
21:48:50 ^
21:48:50
21:48:50 AssertionError [ERR_ASSERTION]: fs.sync.open:
21:48:50 { status: null,
21:48:50 signal: 'SIGSEGV',
21:48:50 output: [ null, '', '' ],
21:48:50 pid: 24620,
21:48:50 stdout: '',
21:48:50 stderr: '' }
21:48:50 at Object.<anonymous> (/data/iojs/build/workspace/node-test-commit-linuxone/nodes/rhel72-s390x/test/parallel/test-trace-events-fs-sync.js:139:10)
21:48:50 at Module._compile (internal/modules/cjs/loader.js:689:30)
21:48:50 at Object.Module._extensions..js (internal/modules/cjs/loader.js:700:10)
21:48:50 at Module.load (internal/modules/cjs/loader.js:599:32)
21:48:50 at tryModuleLoad (internal/modules/cjs/loader.js:538:12)
21:48:50 at Function.Module._load (internal/modules/cjs/loader.js:530:3)
21:48:50 at Function.Module.runMain (internal/modules/cjs/loader.js:742:12)
21:48:50 at startup (internal/bootstrap/node.js:266:19)
21:48:50 at bootstrapNodeJSCore (internal/bootstrap/node.js:596:3)
21:48:50 ... |
Original commit message: [tracing] do not add traces when disabled #21038 Change-Id: Ic4c9f403b5e54a97d3170b2311dd5aab8c8357c8 Reviewed-on: https://chromium-review.googlesource.com/1217726 Commit-Queue: Ali Ijaz Sheikh <ofrobots@google.com> Reviewed-by: Yang Guo <yangguo@chromium.org> Cr-Commit-Position: refs/heads/master@{#55809} Refs: v8/v8@2363cdf PR-URL: #22812 Reviewed-By: Refael Ackermann <refack@gmail.com> Reviewed-By: James M Snell <jasnell@gmail.com> Reviewed-By: Anna Henningsen <anna@addaleax.net> Reviewed-By: Eugene Ostroukhov <eostroukhov@google.com>
#22812 has landed. Hopefully it fixes it. There might still be some other thread races, but I haven't been able to observe any so far in my testing. |
@ofrobots 🎉 Thank you so much! I'll open a PR to unmark the test as flaky and close this issue. |
Original commit message: [tracing] do not add traces when disabled #21038 Change-Id: Ic4c9f403b5e54a97d3170b2311dd5aab8c8357c8 Reviewed-on: https://chromium-review.googlesource.com/1217726 Commit-Queue: Ali Ijaz Sheikh <ofrobots@google.com> Reviewed-by: Yang Guo <yangguo@chromium.org> Cr-Commit-Position: refs/heads/master@{#55809} Refs: v8/v8@2363cdf PR-URL: #22812 Reviewed-By: Refael Ackermann <refack@gmail.com> Reviewed-By: James M Snell <jasnell@gmail.com> Reviewed-By: Anna Henningsen <anna@addaleax.net> Reviewed-By: Eugene Ostroukhov <eostroukhov@google.com>
It's flaky still, but in a different way (SIGABRT rather than SIGSEGV). Will open a new issue, but ref this one... |
Original commit message: [tracing] do not add traces when disabled nodejs#21038 Change-Id: Ic4c9f403b5e54a97d3170b2311dd5aab8c8357c8 Reviewed-on: https://chromium-review.googlesource.com/1217726 Commit-Queue: Ali Ijaz Sheikh <ofrobots@google.com> Reviewed-by: Yang Guo <yangguo@chromium.org> Cr-Commit-Position: refs/heads/master@{nodejs#55809} Refs: v8/v8@2363cdf
Original commit message: [tracing] do not add traces when disabled #21038 Change-Id: Ic4c9f403b5e54a97d3170b2311dd5aab8c8357c8 Reviewed-on: https://chromium-review.googlesource.com/1217726 Commit-Queue: Ali Ijaz Sheikh <ofrobots@google.com> Reviewed-by: Yang Guo <yangguo@chromium.org> Cr-Commit-Position: refs/heads/master@{#55809} Refs: v8/v8@2363cdf PR-URL: #22812 Reviewed-By: Refael Ackermann <refack@gmail.com> Reviewed-By: James M Snell <jasnell@gmail.com> Reviewed-By: Anna Henningsen <anna@addaleax.net> Reviewed-By: Eugene Ostroukhov <eostroukhov@google.com>
Original commit message: [tracing] do not add traces when disabled #21038 Change-Id: Ic4c9f403b5e54a97d3170b2311dd5aab8c8357c8 Reviewed-on: https://chromium-review.googlesource.com/1217726 Commit-Queue: Ali Ijaz Sheikh <ofrobots@google.com> Reviewed-by: Yang Guo <yangguo@chromium.org> Cr-Commit-Position: refs/heads/master@{#55809} Refs: v8/v8@2363cdf PR-URL: #22812 Reviewed-By: Refael Ackermann <refack@gmail.com> Reviewed-By: James M Snell <jasnell@gmail.com> Reviewed-By: Anna Henningsen <anna@addaleax.net> Reviewed-By: Eugene Ostroukhov <eostroukhov@google.com>
Original commit message: [tracing] do not add traces when disabled nodejs#21038 Change-Id: Ic4c9f403b5e54a97d3170b2311dd5aab8c8357c8 Reviewed-on: https://chromium-review.googlesource.com/1217726 Commit-Queue: Ali Ijaz Sheikh <ofrobots@google.com> Reviewed-by: Yang Guo <yangguo@chromium.org> Cr-Commit-Position: refs/heads/master@{nodejs#55809} Refs: v8/v8@2363cdf
Original commit message: [tracing] do not add traces when disabled #21038 Change-Id: Ic4c9f403b5e54a97d3170b2311dd5aab8c8357c8 Reviewed-on: https://chromium-review.googlesource.com/1217726 Commit-Queue: Ali Ijaz Sheikh <ofrobots@google.com> Reviewed-by: Yang Guo <yangguo@chromium.org> Cr-Commit-Position: refs/heads/master@{#55809} Refs: v8/v8@2363cdf PR-URL: #22754 Reviewed-By: Matteo Collina <matteo.collina@gmail.com> Reviewed-By: James M Snell <jasnell@gmail.com> Reviewed-By: Refael Ackermann <refack@gmail.com> Reviewed-By: Ali Ijaz Sheikh <ofrobots@google.com> Reviewed-By: Colin Ihrig <cjihrig@gmail.com>
nodejs/node#21038 Change-Id: Ic4c9f403b5e54a97d3170b2311dd5aab8c8357c8 Reviewed-on: https://chromium-review.googlesource.com/1217726 Commit-Queue: Ali Ijaz Sheikh <ofrobots@google.com> Reviewed-by: Yang Guo <yangguo@chromium.org> Cr-Commit-Position: refs/heads/master@{#55809}
Unreliability for test-trace-events-fs-sync is believed to have been fixed. Remove flaky designation. Ref: nodejs#22812 Ref: nodejs#21038 (comment)
Unreliability for test-trace-events-fs-sync is believed to have been fixed. Remove flaky designation. Ref: nodejs#22812 Ref: nodejs#21038 (comment) PR-URL: nodejs#22856 Refs: nodejs#22812 Reviewed-By: Richard Lau <riclau@uk.ibm.com> Reviewed-By: Gireesh Punathil <gpunathi@in.ibm.com> Reviewed-By: Weijia Wang <starkwang@126.com> Reviewed-By: Colin Ihrig <cjihrig@gmail.com> Reviewed-By: James M Snell <jasnell@gmail.com> Reviewed-By: Sakthipriyan Vairamani <thechargingvolcano@gmail.com> Reviewed-By: Yuta Hiroto <hello@hiroppy.me> Reviewed-By: Refael Ackermann <refack@gmail.com>
Unreliability for test-trace-events-fs-sync is believed to have been fixed. Remove flaky designation. Ref: #22812 Ref: #21038 (comment) PR-URL: #22856 Refs: #22812 Reviewed-By: Richard Lau <riclau@uk.ibm.com> Reviewed-By: Gireesh Punathil <gpunathi@in.ibm.com> Reviewed-By: Weijia Wang <starkwang@126.com> Reviewed-By: Colin Ihrig <cjihrig@gmail.com> Reviewed-By: James M Snell <jasnell@gmail.com> Reviewed-By: Sakthipriyan Vairamani <thechargingvolcano@gmail.com> Reviewed-By: Yuta Hiroto <hello@hiroppy.me> Reviewed-By: Refael Ackermann <refack@gmail.com>
Unreliability for test-trace-events-fs-sync is believed to have been fixed. Remove flaky designation. Ref: #22812 Ref: #21038 (comment) PR-URL: #22856 Refs: #22812 Reviewed-By: Richard Lau <riclau@uk.ibm.com> Reviewed-By: Gireesh Punathil <gpunathi@in.ibm.com> Reviewed-By: Weijia Wang <starkwang@126.com> Reviewed-By: Colin Ihrig <cjihrig@gmail.com> Reviewed-By: James M Snell <jasnell@gmail.com> Reviewed-By: Sakthipriyan Vairamani <thechargingvolcano@gmail.com> Reviewed-By: Yuta Hiroto <hello@hiroppy.me> Reviewed-By: Refael Ackermann <refack@gmail.com>
Unreliability for test-trace-events-fs-sync is believed to have been fixed. Remove flaky designation. Ref: #22812 Ref: #21038 (comment) PR-URL: #22856 Refs: #22812 Reviewed-By: Richard Lau <riclau@uk.ibm.com> Reviewed-By: Gireesh Punathil <gpunathi@in.ibm.com> Reviewed-By: Weijia Wang <starkwang@126.com> Reviewed-By: Colin Ihrig <cjihrig@gmail.com> Reviewed-By: James M Snell <jasnell@gmail.com> Reviewed-By: Sakthipriyan Vairamani <thechargingvolcano@gmail.com> Reviewed-By: Yuta Hiroto <hello@hiroppy.me> Reviewed-By: Refael Ackermann <refack@gmail.com>
Test failing occasionally on FreeBSD with different assertion failures:
https://ci.nodejs.org/job/node-test-commit-freebsd/18147/ (#20962)
https://ci.nodejs.org/job/node-test-commit-freebsd/18103/ (#21003)
https://ci.nodejs.org/job/node-test-commit-freebsd/18143/ (#21032)
The text was updated successfully, but these errors were encountered: