AddressSanitizer: Skip tests directly in Python, not with external config

[vstinner]

BPO	46633
Nosy	@vstinner, @pablogsal
PRs	bpo-46633: Skip tests on ASAN and/or MSAN builds #31632 [3.10] bpo-46633: Skip tests on ASAN and/or MSAN builds (GH-31632) #31634 [3.9] bpo-46633: Skip tests on ASAN and/or MSAN builds (GH-31632) (GH-31634) #31644

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2022-03-02.17:13:40.631>
created_at = <Date 2022-02-04.10:09:11.092>
labels = ['tests', '3.11']
title = 'AddressSanitizer: Skip tests directly in Python, not with external config'
updated_at = <Date 2022-03-02.17:13:40.630>
user = 'https://github.com/vstinner'

bugs.python.org fields:

activity = <Date 2022-03-02.17:13:40.630>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2022-03-02.17:13:40.631>
closer = 'vstinner'
components = ['Tests']
creation = <Date 2022-02-04.10:09:11.092>
creator = 'vstinner'
dependencies = []
files = []
hgrepos = []
issue_num = 46633
keywords = ['patch']
message_count = 11.0
messages = ['412499', '412501', '412506', '414247', '414248', '414250', '414262', '414266', '414370', '414379', '414380']
nosy_count = 2.0
nosy_names = ['vstinner', 'pablogsal']
pr_nums = ['31632', '31634', '31644']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue46633'
versions = ['Python 3.11']

Linked PRs

• gh-90791: Enable test___all__ on ASAN build #108286
• gh-90791: test.pythoninfo logs ASAN_OPTIONS env var #108289

[vstinner]

It seems like bpo-45200: "Address Sanitizer: libasan dead lock in pthread_create() (test_multiprocessing_fork.test_get() hangs)" is not fixed yet.

In the GitHub Action job, test_multiprocessing_fork is skipped because it's too slow, so the job doesn't hang.

Yesterday, I modified the ASAN buildbot to double its timeout from 20 to 40 minutes:
python/buildmaster-config@5a37411

But it's useful, when it hangs, it hangs forever. Exmaple on the AMD64 Arch Linux Asan Debug 3.9 buildbot (with the new config):

---
(test.test_multiprocessing_fork.WithProcessesTestPicklingConnections) ... ok
Timeout (0:35:00)!
---
https://buildbot.python.org/all/#/builders/588/builds/332

Tests are tuned for ASAN, but the configuration is copied and inconsistent between the GitHub Actions job and the buildbot configuration.

I propose to move this configuration directly into Python.

test_decimal.py checks for "-fsanitize=address" in CFLAGS and skip some tests if it's present.

[vstinner]

Notes:

• test.support has check_sanitizer() function. Tests using it:

  • test_crypt
  • test_idle
  • test_tix
  • test_tk
  • test_ttk_guionly

• test_decimal also checks for memory sanitizer and skip some tests if it's present:

MEMORY_SANITIZER = (
    '-fsanitize=memory' in _cflags or
    '--with-memory-sanitizer' in _config_args
)   

• test_faulthandler suppress crash report, similir to support.SuppressCrashReport, directly in the C code used by tests: faulthandler_suppress_crash_report() function.

• Objects/obmalloc.c checks "#if __has_feature(address_sanitizer)": is ASAN enabled?

• GitHub Action config: .github/workflows/build.yml. Skipped tests:

  • test___all__
  • test_concurrent_futures
  • test_multiprocessing_fork
  • test_multiprocessing_forkserver
  • test_multiprocessing_spawn
  • test_peg_generator
  • test_tools

Comment:
---
# Skip test_tools test_peg_generator test_concurrent_futures because
# there are too slow: between 5 and 20 minutes on this CI.

# Skip multiprocessing and concurrent.futures tests which are affected by
# bpo-45200 bug: libasan dead lock in pthread_create().

# test___all__ is skipped because importing some modules directly can trigger
# known problems with ASAN (like tk or crypt).
---

• Buildbot configuration: UnixAsanBuild class of master/custom/factories.py. Options:

  ASAN_OPTIONS='detect_leaks=0:allocator_may_return_null=1:handle_segv=0'

With the comment:
---
# See https://bugs.python.org/issue42985 for more context on why
# SIGSEGV is ignored on purpose.
---

Skipped tests:

• test_ctypes
• test_capi
• test_crypt
• test_decimal
• test_faulthandler
• test_interpreters

With the comment:
---
# These tests are currently raising false positives or are interfering with the ASAN mechanism,
# so we need to skip them unfortunately.
---

[pablogsal]

I have recently added some decorators in test.
support to deactivate tests if running under the sanitizers.

[vstinner]

I created a PR to no longer skip tests in buildbots:
python/buildmaster-config#314

[vstinner]

Status with the 2 pending PRs:

• Asyncio: SSL transport does not support set_protocol() #75813
• ASAN and UB workers no longer exclude tests buildmaster-config#314

Tests only skipped on ASAN:

• _test_multiprocessing

Skip on ASAN and MSAN:

• test___all__
• test_concurrent_futures
• test_crypt
• test_decimal.test_maxcontext_exact_arith()
• test_idle
• test_peg_generator
• test_tix
• test_tk
• test_tools
• test_ttk_guionly

No longer skipped:

• test_capi
• test_ctypes
• test_faulthandler
• test_interpreters

[vstinner]

I created bpo-46887: ./Programs/_freeze_module fails with MSAN: Uninitialized value was created by an allocation of 'stat.i'.

[vstinner]

"""
No longer skipped:

• test_capi
• test_ctypes
• test_faulthandler
• test_interpreters
  """

I built Python manually with:

./configure --with-pydebug CC=clang LD=clang --with-address-sanitizer

These tests no longer with ASAN_OPTIONS used on our CI:

ASAN_OPTIONS='detect_leaks=0:allocator_may_return_null=1:handle_segv=0' ./python -m test (...)

[vstinner]

New changeset 9204bb7 by Victor Stinner in branch 'main':
bpo-46633: Skip tests on ASAN and/or MSAN builds (GH-31632)
9204bb7

[vstinner]

New changeset 9326445 by Victor Stinner in branch '3.10':
[3.10] bpo-46633: Skip tests on ASAN and/or MSAN builds (GH-31632) (GH-31634)
9326445

[vstinner]

New changeset 359bc39 by Victor Stinner in branch '3.9':
[3.10] bpo-46633: Skip tests on ASAN and/or MSAN builds (GH-31632) (GH-31634) (GH-31644)
359bc39

[vstinner]

python/buildmaster-config#314

I merged this PR as well. The new buildbot configuration will be deployed soon.

[vstinner]

Lib/test/test_asyncio/test_subprocess.py and Lib/test/test_distutils.py are now skipped on ASAN tests by the commit: f6dd14c (cc @gpshead)

[gpshead]

FWIW I disabled those two because they triggered ASAN on Ubuntu 20.04's gcc 9.4 and in github's CI (which might use that?). They do not trigger ASAN on more recent Debian bullseye or Ubuntu 22.04 based systems with a more recent gcc.

In the _posixsubprocess.c case it was at a callsite that has ~25 parameters (ugh) and just gained one more. But the asan assertion would only trigger during one consistent but seemingly random test_asyncio.test_subprocess test rather than all the time; so if there is a bug in the ASAN implementation itself it isn't clear what it was.

[vstinner]

I enabled test_faulthandler on ASAN: commit 58f9c63.

[vstinner]

I enabled more tests in test___all__: commit a541e01