Thread sanitizer tests failing in GitHub actions

[colesbury]

EDIT: The cause was due to a change in LLVM's Ubuntu package.

I think this is due to a change in GitHub actions, not any particular commit. Commits that previously succeeded are now failing. I cannot reproduce the issue locally.

The only difference I see is in the reported clang build:

CC.version: Ubuntu clang version 18.1.8 (++20240717045746+3b5b5c1ec4a3-1~exp1~20240717165903.140) (bad)
CC.version: Ubuntu clang version 18.1.8 (++20240714071213+3b5b5c1ec4a3-1~exp1~20240714191314.138) (good)

However, the runner IMAGE_VERSION doesn't look like it's changed:

IMAGE_VERSION: 20240714.1.0

Nearly every test is failing:

https://github.com/python/cpython/actions/runs/9982293102/job/27587742397

0:00:00 load avg: 2.83 Run 22 tests in parallel using 4 worker processes
0:00:01 load avg: 2.93 [ 1/22/1] test_capi.test_pyatomic worker non-zero exit code (Exit code 66)
0:00:03 load avg: 2.93 [ 2/22/2] test_code worker non-zero exit code (Exit code 66)
0:00:06 load avg: 3.01 [ 3/22/3] test_functools worker non-zero exit code (Exit code 66)
0:00:11 load avg: 3.01 [ 4/22/4] test_capi.test_mem worker non-zero exit code (Exit code 66)
0:00:12 load avg: 3.01 [ 5/22/5] test_httpservers worker non-zero exit code (Exit code 66)
0:00:16 load avg: 3.01 [ 6/22/6] test_enum worker non-zero exit code (Exit code 66)
0:00:21 load avg: 3.01 [ 7/22/7] test_imaplib worker non-zero exit code (Exit code 66)
0:00:39 load avg: 3.83 [ 8/22/8] test_importlib worker non-zero exit code (Exit code 66)
0:00:41 load avg: 4.01 [ 9/22/9] test_logging worker non-zero exit code (Exit code 66)
0:01:03 load avg: 3.79 [10/22/10] test_io worker non-zero exit code (Exit code 66) -- running (1): test_queue (42.2 sec)
0:01:08 load avg: 4.21 [11/22/11] test_signal worker non-zero exit code (Exit code 66) -- running (1): test_queue (47.1 sec)
0:01:09 load avg: 4.21 [12/22/12] test_sqlite3 worker non-zero exit code (Exit code 66) -- running (1): test_queue (48.0 sec)
0:01:11 load avg: 4.11 [13/22/13] test_syslog worker non-zero exit code (Exit code 66) -- running (2): test_queue (50.3 sec), test_socket (30.1 sec)
0:01:15 load avg: 4.82 [14/22/14] test_thread worker non-zero exit code (Exit code 66) -- running (2): test_queue (54.4 sec), test_socket (34.3 sec)
0:01:17 load avg: 4.82 [15/22/15] test_threadedtempfile worker non-zero exit code (Exit code 66) -- running (2): test_queue (56.2 sec), test_socket (36.1 sec)
0:01:20 load avg: 4.82 [16/22/16] test_queue worker non-zero exit code (Exit code 66) -- running (1): test_socket (39.0 sec)
0:01:23 load avg: 4.68 [17/22/17] test_threading_local worker non-zero exit code (Exit code 66) -- running (1): test_socket (41.8 sec)
0:01:24 load avg: 4.68 [18/22/18] test_ssl worker non-zero exit code (Exit code 66) -- running (1): test_socket (43.3 sec)
0:01:29 load avg: 4.54 [19/22/19] test_threadsignals worker non-zero exit code (Exit code 66) -- running (1): test_socket (48.1 sec)
0:01:50 load avg: 3.91 [20/22/20] test_weakref worker non-zero exit code (Exit code 66) -- running (2): test_threading (32.6 sec), test_socket (1 min 8 sec)
0:01:53 load avg: 3.68 [21/22/21] test_socket worker non-zero exit code (Exit code 66) -- running (1): test_threading (35.4 sec)
0:01:56 load avg: 3.46 [22/22/22] test_threading worker non-zero exit code (Exit code 66)

TSan logs are useless:

WARNING: ThreadSanitizer: data race (pid=8535)
  Write of size 8 at 0x56136380acd0 by main thread:
    #0 <null> <null> (python+0x31a5ee) (BuildId: 243a59[16](https://github.com/python/cpython/actions/runs/9982293102/job/27587742397#step:13:17)b2751fda530fabbdb48342727b53c0b8)
    #1 <null> <null> (python+0x31a4a6) (BuildId: 243a5916b2751fda530fabbdb48342727b53c0b8)
    #2 <null> <null> (python+0x4f1242) (BuildId: 243a5916b2751fda530fabbdb48342727b53c0b8)
    #3 <null> <null> (python+0x522ef7) (BuildId: 243a5916b2751fda530fabbdb48342727b53c0b8)
...

Linked PRs

• gh-121946: Temporarily switch to llvm-17 in TSan CI #121975
• [3.13] gh-121946: Temporarily switch to llvm-17 in TSan CI (GH-121975) #121981
• Revert "gh-121946: Temporarily switch to llvm-17 in TSan CI (#121975)" #122032
• [3.13] Revert "gh-121946: Temporarily switch to llvm-17 in TSan CI (GH-121975)" (GH-122032) #122131
• gh-121946: Temporarily switch to llvm-17 in TSan CI again #122466
• [3.13] gh-121946: Temporarily switch to llvm-17 in TSan CI again (GH-122466) #122467

[vstinner]

Can we make the two TSan jobs as non-mandatory until the problem is solved? Right now, it's no longer possible to merge any change.

cc @ambv

[colesbury]

I'm looking at this now

[colesbury]

Ok, it looks like the llvm.sh script from https://apt.llvm.org/ no longer installs the llvm-18 (or llvm-18-dev) package. That means that llvm-symbolizer is missing, which apparently means that:

1. The stack traces are garbage
2. The suppressions don't work

[colesbury]

Looks like this is related: llvm/llvm-project#99502

[brandtbucher]

I believe this is affecting JIT builds as well (llvm.sh isn't installing llvm-readobj). Downgrading is more painful there, so hopefully it's resolved soon.

[colesbury]

This looks like it's resolved now.

[encukou]

I'm seeing the same failures again:

• https://github.com/python/cpython/actions/runs/10162667281/job/28103905238?pr=122457
• https://github.com/python/cpython/actions/runs/10161251843/job/28103762841?pr=122446
• https://github.com/python/cpython/actions/runs/10163762600/job/28109078173?pr=122233

[colesbury]

Ugh... I guess we try the llvm-17 package again

[mhsmith]

I'm still seeing this even after the merge of #122466:

• https://github.com/python/cpython/actions/runs/10170893077/job/28130900982?pr=122487