Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

flaky test-inspector-connect-main-thread #31226

Closed
sam-github opened this issue Jan 6, 2020 · 4 comments
Closed

flaky test-inspector-connect-main-thread #31226

sam-github opened this issue Jan 6, 2020 · 4 comments
Assignees
@addaleax
Labels
flaky-test Issues and PRs related to the tests with unstable failures on the CI. inspector Issues and PRs related to the V8 inspector protocol worker Issues and PRs related to Worker support.

Comments

@sam-github
Copy link
Contributor

  • Version: master
  • Platform: Linux/Alpine
  • Subsystem: inspector

Node daily failed, looks like a SIGSEGV:

00:07:15 not ok 1374 parallel/test-inspector-connect-main-thread
00:07:15   ---
00:07:15   duration_ms: 0.612
00:07:15   severity: crashed
00:07:15   exitcode: -11
00:07:15   stack: |-
00:07:15     Skip child is done
00:07:15     Message for a test
00:07:15     Worker is done
00:07:15     Message NodeWorker.enable was sent
00:07:15     Message Debugger.enable was sent
00:07:15     Message Runtime.enable was sent
00:07:15     Message Debugger.setBreakpointByUrl was sent
00:07:15     Message Debugger.evaluateOnCallFrame was sent
00:07:15     Message Debugger.resume was sent
@sam-github sam-github added flaky-test Issues and PRs related to the tests with unstable failures on the CI. linux Issues and PRs related to the Linux platform. inspector Issues and PRs related to the V8 inspector protocol labels Jan 6, 2020
@richardlau
Copy link
Member

Crash on freebsd:
https://ci.nodejs.org/job/node-test-commit-freebsd/30802/nodes=freebsd11-x64/consoleFull

01:28:58 not ok 1248 parallel/test-inspector-connect-main-thread
01:28:58   ---
01:28:58   duration_ms: 1.289
01:28:58   severity: crashed
01:28:58   exitcode: -6
01:28:58   stack: |-
01:28:58     Skip child is done
01:28:58     Message for a test
01:28:58     Worker is done
01:28:58     Message NodeWorker.enable was sent
01:28:58     Message Debugger.enable was sent
01:28:58     Message Runtime.enable was sent
01:28:58     Message Debugger.setBreakpointByUrl was sent
01:28:58     Message Debugger.evaluateOnCallFrame was sent
01:28:58     Message Debugger.resume was sent
01:28:58     out/Release/node[37608]: ../src/api/callback.cc:147:MaybeLocal<v8::Value> node::InternalMakeCallback(node::Environment *, Local<v8::Object>, const Local<v8::Function>, int, Local<v8::Value> *, node::async_context): Assertion `!recv.IsEmpty()' failed.
01:28:58      1: 0x9d8f60 node::Abort(void) [/usr/home/iojs/build/workspace/node-test-commit-freebsd/nodes/freebsd11-x64/out/Release/node]
01:28:58      2: 0x9d8d19 node::Assert(node::AssertionInfo const&) [/usr/home/iojs/build/workspace/node-test-commit-freebsd/nodes/freebsd11-x64/out/Release/node]
01:28:58      3: 0x937e27 v8<node::Environment*::Function> node::InternalMakeCallback(node::Environment*, v8::Local<node::Environment*::Object>(int, v8<node::Environment*::Value>*, node::async_context) [/usr/home/iojs/build/workspace/node-test-commit-freebsd/nodes/freebsd11-x64/out/Release/node]
01:28:58      4: 0x9491d1 int node::AsyncWrap::MakeCallback(v8::Local<node::AsyncWrap::MakeCallback::Function>(v8<node::AsyncWrap::MakeCallback::Value>*) [/usr/home/iojs/build/workspace/node-test-commit-freebsd/nodes/freebsd11-x64/out/Release/node]
01:28:58      5: 0xaa1fe4 _register_inspector(void) [/usr/home/iojs/build/workspace/node-test-commit-freebsd/nodes/freebsd11-x64/out/Release/node]
01:28:58      6: 0xaa966b node::inspector::MainThreadInterface::DispatchMessages(void) [/usr/home/iojs/build/workspace/node-test-commit-freebsd/nodes/freebsd11-x64/out/Release/node]
01:28:58      7: 0xaaabae node::inspector::MainThreadHandle::Expired(void) [/usr/home/iojs/build/workspace/node-test-commit-freebsd/nodes/freebsd11-x64/out/Release/node]
01:28:58      8: 0xa420e2 node::PerIsolatePlatformData::RunForegroundTask(std::__1::unique_ptr<v8::Task, node::PerIsolatePlatformData::RunForegroundTask::default_delete<v8> >) [/usr/home/iojs/build/workspace/node-test-commit-freebsd/nodes/freebsd11-x64/out/Release/node]
01:28:58      9: 0xa40b03 node::PerIsolatePlatformData::FlushForegroundTasksInternal(void) [/usr/home/iojs/build/workspace/node-test-commit-freebsd/nodes/freebsd11-x64/out/Release/node]
01:28:58     10: 0xa423d1 node::NodePlatform::DrainTasks(v8::Isolate*) [/usr/home/iojs/build/workspace/node-test-commit-freebsd/nodes/freebsd11-x64/out/Release/node]
01:28:58     11: 0xa671a8 node::worker::Worker::Run(void) [/usr/home/iojs/build/workspace/node-test-commit-freebsd/nodes/freebsd11-x64/out/Release/node]
01:28:58     12: 0xa68e25 _register_worker(void) [/usr/home/iojs/build/workspace/node-test-commit-freebsd/nodes/freebsd11-x64/out/Release/node]
01:28:58     13: 0x805103c06 operator-> [/lib/libthr.so.3]
01:28:58   ...

@Trott
Copy link
Member

Trott commented Jan 11, 2020

With a relatively-recent debug build and running tools/test.py --repeat=100 --mode=debug test/parallel/test-inspector-connect-main-thread.js, I got a crash on macOS Mojave:

=== debug test-inspector-connect-main-thread ===                   
Path: parallel/test-inspector-connect-main-thread
Skip child is done
Message for a test
Worker is done
Message NodeWorker.enable was sent
Message Debugger.enable was sent
Message Runtime.enable was sent
Message Debugger.setBreakpointByUrl was sent
Message Debugger.evaluateOnCallFrame was sent
Message Debugger.resume was sent
out/Debug/node[5252]: ../src/api/callback.cc:147:MaybeLocal<v8::Value> node::InternalMakeCallback(node::Environment *, Local<v8::Object>, const Local<v8::Function>, int, Local<v8::Value> *, node::async_context): Assertion `!recv.IsEmpty()' failed.
 1: 0x10005fdfc node::DumpBacktrace(__sFILE*) [/Users/trott/io.js/out/Debug/node]
 2: 0x10016ef7b node::Abort() [/Users/trott/io.js/out/Debug/node]
 3: 0x10016ea13 node::Assert(node::AssertionInfo const&) [/Users/trott/io.js/out/Debug/node]
 4: 0x100002fc7 node::InternalMakeCallback(node::Environment*, v8::Local<v8::Object>, v8::Local<v8::Function>, int, v8::Local<v8::Value>*, node::async_context) [/Users/trott/io.js/out/Debug/node]
 5: 0x100033a39 node::AsyncWrap::MakeCallback(v8::Local<v8::Function>, int, v8::Local<v8::Value>*) [/Users/trott/io.js/out/Debug/node]
 6: 0x10036ecdc node::inspector::(anonymous namespace)::JSBindingsConnection<node::inspector::(anonymous namespace)::MainThreadConnection>::OnMessage(v8::Local<v8::Value>) [/Users/trott/io.js/out/Debug/node]
 7: 0x10036ec3c node::inspector::(anonymous namespace)::JSBindingsConnection<node::inspector::(anonymous namespace)::MainThreadConnection>::JSBindingsSessionDelegate::SendMessageToFrontend(v8_inspector::StringView const&) [/Users/trott/io.js/out/Debug/node]
 8: 0x100390452 node::inspector::(anonymous namespace)::ThreadSafeDelegate::SendMessageToFrontend(v8_inspector::StringView const&)::'lambda'(node::inspector::InspectorSessionDelegate*)::operator()(node::inspector::InspectorSessionDelegate*) const [/Users/trott/io.js/out/Debug/node]
 9: 0x1003903ba node::inspector::(anonymous namespace)::CallRequest<node::inspector::InspectorSessionDelegate, node::inspector::(anonymous namespace)::ThreadSafeDelegate::SendMessageToFrontend(v8_inspector::StringView const&)::'lambda'(node::inspector::InspectorSessionDelegate*)>::Call(node::inspector::MainThreadInterface*) [/Users/trott/io.js/out/Debug/node]
10: 0x100388d71 node::inspector::MainThreadInterface::DispatchMessages() [/Users/trott/io.js/out/Debug/node]
11: 0x100393d2f node::inspector::(anonymous namespace)::DispatchMessagesTask::Run() [/Users/trott/io.js/out/Debug/node]
12: 0x10026ca2b node::PerIsolatePlatformData::RunForegroundTask(std::__1::unique_ptr<v8::Task, std::__1::default_delete<v8::Task> >) [/Users/trott/io.js/out/Debug/node]
13: 0x10026b232 node::PerIsolatePlatformData::FlushForegroundTasksInternal() [/Users/trott/io.js/out/Debug/node]
14: 0x10026ce2e node::NodePlatform::DrainTasks(v8::Isolate*) [/Users/trott/io.js/out/Debug/node]
15: 0x1002d7690 node::worker::Worker::Run()::$_0::operator()() const [/Users/trott/io.js/out/Debug/node]
16: 0x1002d74e7 node::OnScopeLeaveImpl<node::worker::Worker::Run()::$_0>::~OnScopeLeaveImpl() [/Users/trott/io.js/out/Debug/node]
17: 0x1002c9ac5 node::OnScopeLeaveImpl<node::worker::Worker::Run()::$_0>::~OnScopeLeaveImpl() [/Users/trott/io.js/out/Debug/node]
18: 0x1002c9364 node::worker::Worker::Run() [/Users/trott/io.js/out/Debug/node]
19: 0x1002d03fe node::worker::Worker::StartThread(v8::FunctionCallbackInfo<v8::Value> const&)::$_2::operator()(void*) const [/Users/trott/io.js/out/Debug/node]
20: 0x1002d03b5 node::worker::Worker::StartThread(v8::FunctionCallbackInfo<v8::Value> const&)::$_2::__invoke(void*) [/Users/trott/io.js/out/Debug/node]
21: 0x7fff6f4552eb _pthread_body [/usr/lib/system/libsystem_pthread.dylib]
22: 0x7fff6f458249 _pthread_start [/usr/lib/system/libsystem_pthread.dylib]
23: 0x7fff6f45440d thread_start [/usr/lib/system/libsystem_pthread.dylib]
Command: out/Debug/node /Users/trott/io.js/test/parallel/test-inspector-connect-main-thread.js
--- CRASHED (Signal: 6) ---

@Trott Trott removed the linux Issues and PRs related to the Linux platform. label Jan 11, 2020
@Trott
Copy link
Member

Trott commented Jan 11, 2020

/ping @addaleax @joyeecheung

The relevant CHECK() (as seen by @richardlau and me above) was moved/refactored by @joyeecheung in e54d11e which landed after this issue was first reported (which may explain why @sam-github has a leaner error message in his original report above--in my time zone, this issue was opened on January 6 and e54d11e landed on January 8).

Prior to that, the CHECK() had been in src/node.cc and was last modified 2 years ago by @addaleax in f27b5e4. In fact, that's the commit where the CHECK() was introduced.

@mscdex
Copy link
Contributor

mscdex commented Jan 24, 2020

Failed on rhel7-s390x:

not ok 2750 parallel/test-inspector-connect-main-thread
  ---
  duration_ms: 120.50
  severity: fail
  exitcode: -15
  stack: |-
    timeout
    Skip child is done
    Message NodeWorker.enable was sent

@addaleax addaleax self-assigned this Feb 4, 2020
@addaleax addaleax added the worker Issues and PRs related to Worker support. label Feb 4, 2020
addaleax added a commit to addaleax/node that referenced this issue Feb 4, 2020
@addaleax
test: fix flaky test-inspector-connect-main-thread
f74813e 
Previously, the test waited for a (any) message from the workers,
and then attached another event listener to a specific kind of
message. However, it was possible that the second listener was
attached after the Worker had already exited, thus never receiving
the message it was supposed to receive. (This is the race condition
here – usually, the Worker thread would exit *after* the second
listener was attached.)

Solve this by keeping a single `'message'` event listener attached
to the worker instance during its entire lifetime.

Fixes: nodejs#31226
@addaleax addaleax mentioned this issue Feb 4, 2020
Closed
3 tasks
@Trott Trott closed this as completed in 0279c2f Feb 8, 2020
codebytere pushed a commit that referenced this issue Feb 17, 2020
@addaleax @codebytere
test: fix flaky test-inspector-connect-main-thread
6139d4e 
Previously, the test waited for a (any) message from the workers,
and then attached another event listener to a specific kind of
message. However, it was possible that the second listener was
attached after the Worker had already exited, thus never receiving
the message it was supposed to receive. (This is the race condition
here – usually, the Worker thread would exit *after* the second
listener was attached.)

Solve this by keeping a single `'message'` event listener attached
to the worker instance during its entire lifetime.

Fixes: #31226

PR-URL: #31637
Reviewed-By: Eugene Ostroukhov <eostroukhov@google.com>
Reviewed-By: Rich Trott <rtrott@gmail.com>
codebytere pushed a commit that referenced this issue Mar 15, 2020
@addaleax @codebytere
test: fix flaky test-inspector-connect-main-thread
e77597c 
Previously, the test waited for a (any) message from the workers,
and then attached another event listener to a specific kind of
message. However, it was possible that the second listener was
attached after the Worker had already exited, thus never receiving
the message it was supposed to receive. (This is the race condition
here – usually, the Worker thread would exit *after* the second
listener was attached.)

Solve this by keeping a single `'message'` event listener attached
to the worker instance during its entire lifetime.

Fixes: #31226

PR-URL: #31637
Reviewed-By: Eugene Ostroukhov <eostroukhov@google.com>
Reviewed-By: Rich Trott <rtrott@gmail.com>
codebytere pushed a commit that referenced this issue Mar 17, 2020
@addaleax @codebytere
test: fix flaky test-inspector-connect-main-thread
896690d 
Previously, the test waited for a (any) message from the workers,
and then attached another event listener to a specific kind of
message. However, it was possible that the second listener was
attached after the Worker had already exited, thus never receiving
the message it was supposed to receive. (This is the race condition
here – usually, the Worker thread would exit *after* the second
listener was attached.)

Solve this by keeping a single `'message'` event listener attached
to the worker instance during its entire lifetime.

Fixes: #31226

PR-URL: #31637
Reviewed-By: Eugene Ostroukhov <eostroukhov@google.com>
Reviewed-By: Rich Trott <rtrott@gmail.com>
codebytere pushed a commit that referenced this issue Mar 30, 2020
@addaleax @codebytere
test: fix flaky test-inspector-connect-main-thread
795a21d 
Previously, the test waited for a (any) message from the workers,
and then attached another event listener to a specific kind of
message. However, it was possible that the second listener was
attached after the Worker had already exited, thus never receiving
the message it was supposed to receive. (This is the race condition
here – usually, the Worker thread would exit *after* the second
listener was attached.)

Solve this by keeping a single `'message'` event listener attached
to the worker instance during its entire lifetime.

Fixes: #31226

PR-URL: #31637
Reviewed-By: Eugene Ostroukhov <eostroukhov@google.com>
Reviewed-By: Rich Trott <rtrott@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@addaleax addaleax

Labels
flaky-test Issues and PRs related to the tests with unstable failures on the CI. inspector Issues and PRs related to the V8 inspector protocol worker Issues and PRs related to Worker support.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

test: fix flaky test-inspector-connect-main-thread addaleax/node
5 participants
@sam-github @mscdex @Trott @addaleax @richardlau