Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

gh-117657: TSAN Fix races in PyMember_Get and PyMember_Set, for C extensions #123211

Open
wants to merge 31 commits into
base: main
Choose a base branch
from

Conversation

dpdani
Copy link
Contributor

@dpdani dpdani commented Aug 21, 2024

Fix data races that would only be visible when using C extensions.

This is a follow-up on #119368.

I'm intentionally not testing:

  • _Py_T_NONE (deprecated)
  • _Py_T_OBJECT (deprecated)
  • Py_T_STRING (immutable)
  • Py_T_STRING_INPLACE (immutable)

Py_T_CHAR

For some reason Py_T_CHAR is untested also in test_capi.test_structmembers.
In fact, it's not even in the supporting C types, which I'm using for the TSAN tests as well: old api, new api.
I'm wondering if there's a specific reason for this, or if it should be in the test suite instead.

I'm guessing that the TSAN suite should cover the thread-safety of Py_T_CHAR regardless?
Or should we dismiss it for TSAN tests as well?

I've added a new member to the T_CHAR member to the _testcapi module, and the rest of the tests don't seem to break.
I can revert this change if needed.

Copy link
Contributor

@picnixz picnixz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Quite hard to review since everything looks the same and the functions are not ordered by pairs apparently. I didn't go through everything but I had a question and an observation.

Include/cpython/pyatomic.h Show resolved Hide resolved
Include/internal/pycore_pyatomic_ft_wrappers.h Outdated Show resolved Hide resolved
@dpdani
Copy link
Contributor Author

dpdani commented Aug 22, 2024

I'm not exactly sure why test_importlib fails, but I don't want to push a dummy commit just to retry it.

Anyways, the TSAN checks for test_free_threading.test_slots are 🟢

@picnixz
Copy link
Contributor

picnixz commented Aug 22, 2024

I'm not exactly sure why test_importlib fails, but I don't want to push a dummy commit just to retry it.

I've relaunched the test manually (I can't relaunch the SSL test though, I don't know why)

Copy link
Contributor

@colesbury colesbury left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks. I left a few comments below.

How long do the added tests takes to run?

Python/structmember.c Outdated Show resolved Hide resolved
Python/structmember.c Outdated Show resolved Hide resolved
Python/structmember.c Outdated Show resolved Hide resolved
@dpdani
Copy link
Contributor Author

dpdani commented Aug 29, 2024

How long do the added tests takes to run?

./python -m test test_free_threading.test_slots

...
Total duration: 2.3 sec
...

@dpdani
Copy link
Contributor Author

dpdani commented Sep 16, 2024

@colesbury can you take another look?

Copy link
Contributor

@colesbury colesbury left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This mostly looks good. Please add a char test to test_structmembers.py. I think the implementation in the PR is broken (see inline comment) and it's not caught by any unit test.

It's fine to ping me on PRs like you did here, but please also use GitHub's "re-request review" button (top-right corner, in the reviewers section). It means that the PR shows up in my list of "awaiting reviews".

Python/structmember.c Outdated Show resolved Hide resolved
Python/structmember.c Outdated Show resolved Hide resolved
Python/structmember.c Outdated Show resolved Hide resolved
Python/structmember.c Outdated Show resolved Hide resolved
Include/cpython/pyatomic_msc.h Outdated Show resolved Hide resolved
Include/cpython/pyatomic_msc.h Outdated Show resolved Hide resolved
Include/cpython/pyatomic_msc.h Outdated Show resolved Hide resolved
Include/cpython/pyatomic_msc.h Outdated Show resolved Hide resolved
Include/cpython/pyatomic_msc.h Outdated Show resolved Hide resolved
@dpdani
Copy link
Contributor Author

dpdani commented Sep 16, 2024

Sorry, will do.

dpdani and others added 3 commits September 22, 2024 12:02
Co-authored-by: Sam Gross <colesbury@gmail.com>
@dpdani
Copy link
Contributor Author

dpdani commented Nov 2, 2024

I finally had some time to come back to this.

I guess that moving the stores after error checking does slightly change the externally-visible behavior.
Should we write a news entry?

@dpdani dpdani requested a review from colesbury November 2, 2024 17:54
Copy link
Contributor

@colesbury colesbury left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants