[K9VULN-2477] Fix (additional) PKU-related v8 segfaults

[jasonforal]

What problem are you trying to solve?

#560 fixed an issue with v8 initialization that caused segfaults on Linux with PKU support. However, this was only tested on Intel Xeon 3rd gen CPUs.

We're still seeing segfaults on Xeon 1st/2nd gen CPUs, which are used by some cloud providers (for example, the t3 instance family on AWS EC2). A screenshot below shows a reproduction:

Processor Information
	Socket Designation: CPU 0
	Type: Central Processor
	Family: Xeon
	Manufacturer: Intel(R) Corporation
	Signature: Type 0, Family 6, Model 85, Stepping 7
	Version: Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz

What is your solution?

Seemingly, when we initialize rayon before v8, v8 segfaults when trying to initialize a default platform. I haven't dug enough to determine if this is something inherent to what rayon does or if this is simply a timing thing masking a concurrency bug somewhere (perhaps initializing a rayon pool takes enough time such that something in v8 is sequenced correctly).

Before

(Condensed) output from GDB:

  Id   GId  Target Id                                            Frame 
* 1    1    Thread 0x7ffff7e7ebc0 (LWP 118172) "datadog-static-" rayon_core::registry::Registry::new<rayon_core::registry::DefaultSpawn> (builder=...) at src/registry.rs:237
#0  rayon_core::registry::Registry::new<rayon_core::registry::DefaultSpawn> (builder=...) at src/registry.rs:237
#1  0x0000555558e293d8 in rayon_core::thread_pool::ThreadPool::build<rayon_core::registry::DefaultSpawn> (builder=...) at src/thread_pool/mod.rs:69
#2  0x0000555558e2e6fe in rayon_core::ThreadPoolBuilder<rayon_core::registry::DefaultSpawn>::build<rayon_core::registry::DefaultSpawn> (self=...) at src/lib.rs:267
#3  0x000055555597a9dc in datadog_static_analyzer::main () at crates/bins/src/bin/datadog-static-analyzer.rs:480

  Id   GId  Target Id                                            Frame 
* 1    1    Thread 0x7ffff7e7ebc0 (LWP 118172) "datadog-static-" v8::platform::Platform::new (thread_pool_size=2, idle_task_support=false) at /home/ubuntu/.cargo/registry/src/index.crates.io-6f17d22bba15001f/v8-130.0.2/src/platform.rs:134
  104  104  Thread 0x7ffff7a006c0 (LWP 118277) "datadog-static-" syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
  105  105  Thread 0x7ffff76006c0 (LWP 118278) "datadog-static-" syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#0  v8::platform::Platform::new (thread_pool_size=2, idle_task_support=false) at /home/ubuntu/.cargo/registry/src/index.crates.io-6f17d22bba15001f/v8-130.0.2/src/platform.rs:134
#1  v8::platform::new_default_platform (thread_pool_size=2, idle_task_support=false) at /home/ubuntu/.cargo/registry/src/index.crates.io-6f17d22bba15001f/v8-130.0.2/src/platform.rs:80
#2  static_analysis_kernel::analysis::ddsa_lib::v8_platform::V8Platform<static_analysis_kernel::analysis::ddsa_lib::v8_platform::Uninitialized>::initialize (thread_pool_size=2) at crates/static-analysis-kernel/src/analysis/ddsa_lib/v8_platform.rs:31
#3  0x0000555556c3e48a in static_analysis_kernel::analysis::ddsa_lib::v8_platform::initialize_v8::{closure#0} () at crates/static-analysis-kernel/src/analysis/ddsa_lib/v8_platform.rs:92
#4  0x00005555569c0d98 in std::sync::once::{impl#2}::call_once::{closure#0}<static_analysis_kernel::analysis::ddsa_lib::v8_platform::initialize_v8::{closure_env#0}> () at /rustc/f6e511eec7342f59a25f7c0534f1dbea00d01b14/library/std/src/sync/once.rs:158
#5  0x0000555555931733 in std::sys::sync::once::futex::Once::call () at std/src/sys/sync/once/futex.rs:168
#6  0x00005555569c0c86 in std::sync::once::Once::call_once<static_analysis_kernel::analysis::ddsa_lib::v8_platform::initialize_v8::{closure_env#0}> (self=0x55555bcd78b8 <static_analysis_kernel::analysis::ddsa_lib::v8_platform::initialize_v8::V8>, f=...) at /rustc/f6e511eec7342f59a25f7c0534f1dbea00d01b14/library/std/src/sync/once.rs:158
#7  0x00005555569810a1 in static_analysis_kernel::analysis::ddsa_lib::v8_platform::initialize_v8 (thread_pool_size=2) at crates/static-analysis-kernel/src/analysis/ddsa_lib/v8_platform.rs:90
#8  0x000055555597aa3a in datadog_static_analyzer::main () at crates/bins/src/bin/datadog-static-analyzer.rs:483

[Switching to Thread 0x7ffff76006c0 (LWP 118278)]
  Id   GId  Target Id                                            Frame 
  1    1    Thread 0x7ffff7e7ebc0 (LWP 118172) "datadog-static-" syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
  104  104  Thread 0x7ffff7a006c0 (LWP 118277) "datadog-static-" v8::internal::wasm::WasmCodePointerTable::AllocateUninitializedEntry() () at ../../../../v8/src/wasm/wasm-code-pointer-table-inl.h:73
* 105  105  Thread 0x7ffff76006c0 (LWP 118278) "datadog-static-" v8::internal::wasm::WasmCodePointerTable::AllocateUninitializedEntry() () at ../../../../v8/src/wasm/wasm-code-pointer-table-inl.h:73
  106  106  Thread 0x7ffff72006c0 (LWP 118279) "V8 DefaultWorke" 0x00007ffff7c98d61 in __futex_abstimed_wait_common64 (private=32767, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x55555c3aa340) at ./nptl/futex-internal.c:57
  107  107  Thread 0x7ffff68006c0 (LWP 118280) "V8 DefaultWorke" 0x00007ffff7c98d61 in __futex_abstimed_wait_common64 (private=32767, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x55555c3bcc30) at ./nptl/futex-internal.c:57
#0  v8::internal::wasm::WasmCodePointerTable::AllocateUninitializedEntry() () at ../../../../v8/src/wasm/wasm-code-pointer-table-inl.h:73
#1  0x0000555557534ffd in v8::internal::Isolate::Isolate(v8::internal::IsolateGroup*) () at ../../../../v8/src/execution/isolate.cc:4110
#2  0x0000555557533c9b in v8::internal::Isolate::New() () at ../../../../v8/src/execution/isolate.cc:3974
#3  0x0000555557926b8b in v8::internal::SnapshotCreatorImpl::SnapshotCreatorImpl(v8::Isolate::CreateParams const&) () at ../../../../v8/src/snapshot/snapshot.cc:903
#4  0x000055555747be48 in v8::SnapshotCreator::SnapshotCreator(v8::Isolate::CreateParams const&) () at ../../../../v8/src/api/api.cc:552
#5  0x00005555574468f3 in v8__SnapshotCreator__CONSTRUCT () at ../../../../src/binding.cc:2774
#6  0x000055555743ee50 in v8::snapshot::SnapshotCreator::new_impl<&[u8]> (external_references=..., existing_snapshot_blob=..., params=<error reading variable: Cannot access memory at address 0x88>) at src/snapshot.rs:143
#7  v8::snapshot::SnapshotCreator::new (external_references=..., params=...) at src/snapshot.rs:107
#8  v8::isolate::Isolate::snapshot_creator (external_references=..., params=...) at src/isolate.rs:662

Thread 104 "datadog-static-" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7ffff7a006c0 (LWP 118277)]
0x000055555753f6df in v8::internal::wasm::WasmCodePointerTable::AllocateUninitializedEntry() () at ../../../../v8/src/wasm/wasm-code-pointer-table-inl.h:73
warning: 73	../../../../v8/src/wasm/wasm-code-pointer-table-inl.h: No such file or directory
  Id   GId  Target Id                                            Frame 
  1    1    Thread 0x7ffff7e7ebc0 (LWP 118172) "datadog-static-" syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
* 104  104  Thread 0x7ffff7a006c0 (LWP 118277) "datadog-static-" 0x000055555753f6df in v8::internal::wasm::WasmCodePointerTable::AllocateUninitializedEntry() () at ../../../../v8/src/wasm/wasm-code-pointer-table-inl.h:73
  105  105  Thread 0x7ffff76006c0 (LWP 118278) "datadog-static-" 0x000055555753f6ab in v8::internal::wasm::WasmCodePointerTable::AllocateUninitializedEntry() () at ../../../../v8/src/wasm/wasm-code-pointer-table-inl.h:73
  106  106  Thread 0x7ffff72006c0 (LWP 118279) "V8 DefaultWorke" 0x00007ffff7c98d61 in __futex_abstimed_wait_common64 (private=32767, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x55555c3aa340) at ./nptl/futex-internal.c:57
  107  107  Thread 0x7ffff68006c0 (LWP 118280) "V8 DefaultWorke" 0x00007ffff7c98d61 in __futex_abstimed_wait_common64 (private=32767, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x55555c3bcc30) at ./nptl/futex-internal.c:57

After

(Condensed) output from GDB:

  Id   GId  Target Id                                            Frame 
* 1    1    Thread 0x7ffff7e7ebc0 (LWP 118489) "datadog-static-" v8::platform::Platform::new (thread_pool_size=2, idle_task_support=false) at /home/ubuntu/.cargo/registry/src/index.crates.io-6f17d22bba15001f/v8-130.0.2/src/platform.rs:134
#0  v8::platform::Platform::new (thread_pool_size=2, idle_task_support=false) at /home/ubuntu/.cargo/registry/src/index.crates.io-6f17d22bba15001f/v8-130.0.2/src/platform.rs:134
#1  v8::platform::new_default_platform (thread_pool_size=2, idle_task_support=false) at /home/ubuntu/.cargo/registry/src/index.crates.io-6f17d22bba15001f/v8-130.0.2/src/platform.rs:80
#2  static_analysis_kernel::analysis::ddsa_lib::v8_platform::V8Platform<static_analysis_kernel::analysis::ddsa_lib::v8_platform::Uninitialized>::initialize (thread_pool_size=2) at crates/static-analysis-kernel/src/analysis/ddsa_lib/v8_platform.rs:31
#3  0x0000555556c3e48a in static_analysis_kernel::analysis::ddsa_lib::v8_platform::initialize_v8::{closure#0} () at crates/static-analysis-kernel/src/analysis/ddsa_lib/v8_platform.rs:92
#4  0x00005555569c0d98 in std::sync::once::{impl#2}::call_once::{closure#0}<static_analysis_kernel::analysis::ddsa_lib::v8_platform::initialize_v8::{closure_env#0}> () at /rustc/f6e511eec7342f59a25f7c0534f1dbea00d01b14/library/std/src/sync/once.rs:158
#5  0x0000555555931733 in std::sys::sync::once::futex::Once::call () at std/src/sys/sync/once/futex.rs:168
#6  0x00005555569c0c86 in std::sync::once::Once::call_once<static_analysis_kernel::analysis::ddsa_lib::v8_platform::initialize_v8::{closure_env#0}> (self=0x55555bcd78b8 <static_analysis_kernel::analysis::ddsa_lib::v8_platform::initialize_v8::V8>, f=...) at /rustc/f6e511eec7342f59a25f7c0534f1dbea00d01b14/library/std/src/sync/once.rs:158
#7  0x00005555569810a1 in static_analysis_kernel::analysis::ddsa_lib::v8_platform::initialize_v8 (thread_pool_size=2) at crates/static-analysis-kernel/src/analysis/ddsa_lib/v8_platform.rs:90
#8  0x000055555597a998 in datadog_static_analyzer::main () at crates/bins/src/bin/datadog-static-analyzer.rs:480

  Id   GId  Target Id                                            Frame 
* 1    1    Thread 0x7ffff7e7ebc0 (LWP 118489) "datadog-static-" rayon_core::registry::Registry::new<rayon_core::registry::DefaultSpawn> (builder=...) at src/registry.rs:237
  104  104  Thread 0x7ffff72006c0 (LWP 118594) "V8 DefaultWorke" 0x00007ffff7c98d61 in __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x55555c3acb50) at ./nptl/futex-internal.c:57
  105  105  Thread 0x7ffff68006c0 (LWP 118595) "V8 DefaultWorke" 0x00007ffff7c98d61 in __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x55555c3a9110) at ./nptl/futex-internal.c:57
#0  rayon_core::registry::Registry::new<rayon_core::registry::DefaultSpawn> (builder=...) at src/registry.rs:237
#1  0x0000555558e293d8 in rayon_core::thread_pool::ThreadPool::build<rayon_core::registry::DefaultSpawn> (builder=...) at src/thread_pool/mod.rs:69
#2  0x0000555558e2e6fe in rayon_core::ThreadPoolBuilder<rayon_core::registry::DefaultSpawn>::build<rayon_core::registry::DefaultSpawn> (self=...) at src/lib.rs:267
#3  0x000055555597a9e9 in datadog_static_analyzer::main () at crates/bins/src/bin/datadog-static-analyzer.rs:481
[New Thread 0x7ffff7a006c0 (LWP 118596)]
[New Thread 0x7ffff76006c0 (LWP 118597)]
[Switching to Thread 0x7ffff76006c0 (LWP 118597)]
  Id   GId  Target Id                                            Frame 
  1    1    Thread 0x7ffff7e7ebc0 (LWP 118489) "datadog-static-" syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
  104  104  Thread 0x7ffff72006c0 (LWP 118594) "V8 DefaultWorke" 0x00007ffff7c98d61 in __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x55555c3acb50) at ./nptl/futex-internal.c:57
  105  105  Thread 0x7ffff68006c0 (LWP 118595) "V8 DefaultWorke" 0x00007ffff7c98d61 in __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x55555c3a9110) at ./nptl/futex-internal.c:57
  106  106  Thread 0x7ffff7a006c0 (LWP 118596) "datadog-static-" v8::internal::wasm::WasmCodePointerTable::AllocateUninitializedEntry() () at ../../../../v8/src/wasm/wasm-code-pointer-table-inl.h:73
* 107  107  Thread 0x7ffff76006c0 (LWP 118597) "datadog-static-" v8::internal::wasm::WasmCodePointerTable::AllocateUninitializedEntry() () at ../../../../v8/src/wasm/wasm-code-pointer-table-inl.h:73
#0  v8::internal::wasm::WasmCodePointerTable::AllocateUninitializedEntry() () at ../../../../v8/src/wasm/wasm-code-pointer-table-inl.h:73
#1  0x0000555557534ffd in v8::internal::Isolate::Isolate(v8::internal::IsolateGroup*) () at ../../../../v8/src/execution/isolate.cc:4110
#2  0x0000555557533c9b in v8::internal::Isolate::New() () at ../../../../v8/src/execution/isolate.cc:3974
#3  0x0000555557926b8b in v8::internal::SnapshotCreatorImpl::SnapshotCreatorImpl(v8::Isolate::CreateParams const&) () at ../../../../v8/src/snapshot/snapshot.cc:903
#4  0x000055555747be48 in v8::SnapshotCreator::SnapshotCreator(v8::Isolate::CreateParams const&) () at ../../../../v8/src/api/api.cc:552
#5  0x00005555574468f3 in v8__SnapshotCreator__CONSTRUCT () at ../../../../src/binding.cc:2774
#6  0x000055555743ee50 in v8::snapshot::SnapshotCreator::new_impl<&[u8]> (external_references=..., existing_snapshot_blob=..., params=<error reading variable: Cannot access memory at address 0x88>) at src/snapshot.rs:143
#7  v8::snapshot::SnapshotCreator::new (external_references=..., params=...) at src/snapshot.rs:107

[Switching to Thread 0x7ffff7a006c0 (LWP 118596)]
  Id   GId  Target Id                                            Frame 
  1    1    Thread 0x7ffff7e7ebc0 (LWP 118489) "datadog-static-" syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
  104  104  Thread 0x7ffff72006c0 (LWP 118594) "V8 DefaultWorke" 0x00007ffff7c98d61 in __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x55555c3acb50) at ./nptl/futex-internal.c:57
  105  105  Thread 0x7ffff68006c0 (LWP 118595) "V8 DefaultWorke" 0x00007ffff7c98d61 in __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x55555c3a9110) at ./nptl/futex-internal.c:57
* 106  106  Thread 0x7ffff7a006c0 (LWP 118596) "datadog-static-" v8::internal::wasm::WasmCodePointerTable::AllocateUninitializedEntry() () at ../../../../v8/src/wasm/wasm-code-pointer-table-inl.h:73
  107  107  Thread 0x7ffff76006c0 (LWP 118597) "datadog-static-" 0x000055555753f6a3 in v8::internal::wasm::WasmCodePointerTable::AllocateUninitializedEntry() () at ../../../../v8/src/wasm/wasm-code-pointer-table-inl.h:73
#0  v8::internal::wasm::WasmCodePointerTable::AllocateUninitializedEntry() () at ../../../../v8/src/wasm/wasm-code-pointer-table-inl.h:73
#1  0x0000555557534f95 in v8::internal::Isolate::Isolate(v8::internal::IsolateGroup*) () at ../../../../v8/src/execution/isolate.cc:4110
#2  0x0000555557533c9b in v8::internal::Isolate::New() () at ../../../../v8/src/execution/isolate.cc:3974
#3  0x0000555557926b8b in v8::internal::SnapshotCreatorImpl::SnapshotCreatorImpl(v8::Isolate::CreateParams const&) () at ../../../../v8/src/snapshot/snapshot.cc:903
#4  0x000055555747be48 in v8::SnapshotCreator::SnapshotCreator(v8::Isolate::CreateParams const&) () at ../../../../v8/src/api/api.cc:552
#5  0x00005555574468f3 in v8__SnapshotCreator__CONSTRUCT () at ../../../../src/binding.cc:2774
#6  0x000055555743ee50 in v8::snapshot::SnapshotCreator::new_impl<&[u8]> (external_references=..., existing_snapshot_blob=..., params=<error reading variable: Cannot access memory at address 0x88>) at src/snapshot.rs:143
#7  v8::snapshot::SnapshotCreator::new (external_references=..., params=...) at src/snapshot.rs:107
#8  v8::isolate::Isolate::snapshot_creator (external_references=..., params=...) at src/isolate.rs:662

GDB interpretation

Breakpoints were set on

• v8::platform::Platform::new (initialization of v8 platform)
• rayon_core::registry::Registry::new (creation of a new rayon pool)
• v8::internal::wasm::WasmCodePointerTable::AllocateUninitializedEntry (a proxy for creation of v8 isolate)

In the before, we see that our attempts to create v8 isolates (threads 104, 105) occur before the "V8 DefaultWorker" threads are created. It's my interpretation that the presence of all v8 worker threads is an indicator that v8 has "properly" initialized.

...
  104  104  Thread 0x7ffff7a006c0 (LWP 118277) "datadog-static-" v8::internal::wasm::WasmCodePointerTable::AllocateUninitializedEntry() () at ../../../../v8/src/wasm/wasm-code-pointer-table-inl.h:73
* 105  105  Thread 0x7ffff76006c0 (LWP 118278) "datadog-static-" v8::internal::wasm::WasmCodePointerTable::AllocateUninitializedEntry() () at ../../../../v8/src/wasm/wasm-code-pointer-table-inl.h:73
  106  106  Thread 0x7ffff72006c0 (LWP 118279) "V8 DefaultWorke" 0x00007ffff7c98d61 in __futex_abstimed_wait_common64 (private=32767, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x55555c3aa340) at ./nptl/futex-internal.c:57
  107  107  Thread 0x7ffff68006c0 (LWP 118280) "V8 DefaultWorke" 0x00007ffff7c98d61 in __futex_abstimed_wait_common64 (private=32767, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x55555c3bcc30) at ./nptl/futex-internal.c:57
  ...

In the after, the threads that create v8 isolates (threads 106, 107) are created after the "V8 DefaultWorker" threads (104, 105) are created:

...
  104  104  Thread 0x7ffff72006c0 (LWP 118594) "V8 DefaultWorke" 0x00007ffff7c98d61 in __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x55555c3acb50) at ./nptl/futex-internal.c:57
  105  105  Thread 0x7ffff68006c0 (LWP 118595) "V8 DefaultWorke" 0x00007ffff7c98d61 in __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x55555c3a9110) at ./nptl/futex-internal.c:57
* 106  106  Thread 0x7ffff7a006c0 (LWP 118596) "datadog-static-" v8::internal::wasm::WasmCodePointerTable::AllocateUninitializedEntry() () at ../../../../v8/src/wasm/wasm-code-pointer-table-inl.h:73
  107  107  Thread 0x7ffff76006c0 (LWP 118597) "datadog-static-" 0x000055555753f6a3 in v8::internal::wasm::WasmCodePointerTable::AllocateUninitializedEntry() () at ../../../../v8/src/wasm/wasm-code-pointer-table-inl.h:73
...

Testing

We currently don't have the infrastructure to test this automatically, however you can manually confirm this by spinning up an Amazon EC2 instance with the t3 family (either Amazon Linux or Ubuntu) and attempting to run datadog-static-analyzer before and after this PR.

Alternatives considered

What the reviewer should know

• A true root cause deserves further digging, as it could be a v8 bug (or even perhaps a bug in the bindings).
• We can implement the automated testing to catch this at a later date -- I think this deserves a quick merge given the severity.