Skip to content

[NetBSD] test_socket creates too many locks #76106

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
serhiy-storchaka opened this issue Nov 2, 2017 · 8 comments
Closed

[NetBSD] test_socket creates too many locks #76106

serhiy-storchaka opened this issue Nov 2, 2017 · 8 comments
Labels
3.7 (EOL) end of life tests Tests in the Lib/test dir type-bug An unexpected behavior, bug, or error

Comments

@serhiy-storchaka
Copy link
Member

serhiy-storchaka commented Nov 2, 2017 •
edited by bedevere-app bot
Loading

BPO 31925
Nosy @rbtcollins, @ezio-melotti, @voidspace, @serhiy-storchaka

Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2017-11-02.14:16:30.938>
labels = ['3.7', 'type-bug', 'tests']
title = '[NetBSD] test_socket creates too many locks'
updated_at = <Date 2017-11-06.17:05:57.177>
user = 'https://github.com/serhiy-storchaka'

bugs.python.org fields:

activity = <Date 2017-11-06.17:05:57.177>
actor = 'serhiy.storchaka'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Tests']
creation = <Date 2017-11-02.14:16:30.938>
creator = 'serhiy.storchaka'
dependencies = []
files = []
hgrepos = []
issue_num = 31925
keywords = []
message_count = 2.0
messages = ['305425', '305659']
nosy_count = 4.0
nosy_names = ['rbcollins', 'ezio.melotti', 'michael.foord', 'serhiy.storchaka']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue31925'
versions = ['Python 2.7', 'Python 3.6', 'Python 3.7']

Linked PRs
@serhiy-storchaka
Copy link
Member Author

test_socket creates a lock in the constructor of ThreadSafeCleanupTestCase. But since unittest creates an instance of test class for every test method, it creates around 400 locks. This exceeds the limit of 200 locks in NetBSD 8 and causes a failure.

0:00:00 load avg: 0.00 [1/1] test_socket
sem_init: Unknown error: 4294967295
sem_init: Unknown error: 4294967295
Traceback (most recent call last):
  File "/home/serhiy/py/cpython3.7/Lib/test/libregrtest/runtest.py", line 176, in runtest_inner
sem_init: Unknown error: 4294967295
  File "/home/serhiy/py/cpython3.7/Lib/test/test_socket.py", line 5801, in test_main
sem_init: Unknown error: 4294967295
  File "/home/serhiy/py/cpython3.7/Lib/test/support/__init__.py", line 1934, in run_unittest
sem_init: Unknown error: 4294967295
  File "/home/serhiy/py/cpython3.7/Lib/unittest/loader.py", line 503, in makeSuite
sem_init: Unknown error: 4294967295
  File "/home/serhiy/py/cpython3.7/Lib/unittest/loader.py", line 92, in loadTestsFromTestCase
sem_init: Unknown error: 4294967295
  File "/home/serhiy/py/cpython3.7/Lib/unittest/suite.py", line 24, in __init__
sem_init: Unknown error: 4294967295
  File "/home/serhiy/py/cpython3.7/Lib/unittest/suite.py", line 57, in addTests
sem_init: Unknown error: 4294967295
  File "/home/serhiy/py/cpython3.7/Lib/test/test_socket.py", line 544, in __init__
sem_init: Unknown error: 4294967295
  File "/home/serhiy/py/cpython3.7/Lib/test/test_socket.py", line 144, in __init__
sem_init: Unknown error: 4294967295
  File "/home/serhiy/py/cpython3.7/Lib/threading.py", line 86, in RLock
sem_init: Unknown error: 4294967295
sem_init: Unknown error: 4294967295
RuntimeError: can't allocate lock

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/serhiy/py/cpython3.7/Lib/runpy.py", line 193, in _run_module_as_main
sem_init: Unknown error: 4294967295
sem_init: Unknown error: 4294967295
  File "/home/serhiy/py/cpython3.7/Lib/runpy.py", line 85, in _run_code
sem_init: Unknown error: 4294967295
sem_init: Unknown error: 4294967295
  File "/home/serhiy/py/cpython3.7/Lib/test/__main__.py", line 2, in <module>
sem_init: Unknown error: 4294967295
  File "/home/serhiy/py/cpython3.7/Lib/test/libregrtest/main.py", line 585, in main
sem_init: Unknown error: 4294967295
  File "/home/serhiy/py/cpython3.7/Lib/test/libregrtest/main.py", line 510, in main
sem_init: Unknown error: 4294967295
  File "/home/serhiy/py/cpython3.7/Lib/test/libregrtest/main.py", line 536, in _main
sem_init: Unknown error: 4294967295
  File "/home/serhiy/py/cpython3.7/Lib/test/libregrtest/main.py", line 458, in run_tests
sem_init: Unknown error: 4294967295
  File "/home/serhiy/py/cpython3.7/Lib/test/libregrtest/main.py", line 379, in run_tests_sequential
sem_init: Unknown error: 4294967295
  File "/home/serhiy/py/cpython3.7/Lib/test/libregrtest/runtest.py", line 140, in runtest
sem_init: Unknown error: 4294967295
  File "/home/serhiy/py/cpython3.7/Lib/test/libregrtest/runtest.py", line 200, in runtest_inner
sem_init: Unknown error: 4294967295
  File "/home/serhiy/py/cpython3.7/Lib/traceback.py", line 163, in format_exc
sem_init: Unknown error: 4294967295
sem_init: Unknown error: 4294967295
  File "/home/serhiy/py/cpython3.7/Lib/traceback.py", line 117, in format_exception
sem_init: Unknown error: 4294967295
sem_init: Unknown error: 4294967295
  File "/home/serhiy/py/cpython3.7/Lib/traceback.py", line 496, in __init__
sem_init: Unknown error: 4294967295
sem_init: Unknown error: 4294967295
  File "/home/serhiy/py/cpython3.7/Lib/traceback.py", line 357, in extract
sem_init: Unknown error: 4294967295
sem_init: Unknown error: 4294967295
  File "/home/serhiy/py/cpython3.7/Lib/traceback.py", line 281, in line
sem_init: Unknown error: 4294967295
sem_init: Unknown error: 4294967295
  File "/home/serhiy/py/cpython3.7/Lib/linecache.py", line 16, in getline
sem_init: Unknown error: 4294967295
sem_init: Unknown error: 4294967295
  File "/home/serhiy/py/cpython3.7/Lib/linecache.py", line 47, in getlines
sem_init: Unknown error: 4294967295
sem_init: Unknown error: 4294967295
  File "/home/serhiy/py/cpython3.7/Lib/linecache.py", line 136, in updatecache
sem_init: Unknown error: 4294967295
sem_init: Unknown error: 4294967295
  File "/home/serhiy/py/cpython3.7/Lib/tokenize.py", line 447, in open
sem_init: Unknown error: 4294967295
sem_init: Unknown error: 4294967295
RuntimeError: can't allocate read lock

@serhiy-storchaka serhiy-storchaka added 3.7 (EOL) end of life tests Tests in the Lib/test dir type-bug An unexpected behavior, bug, or error labels Nov 2, 2017
@pppery pppery mannequin changed the title test_socket creates too much locks test_socket creates too many locks Nov 2, 2017
@vstinner vstinner changed the title test_socket creates too many locks [NetBSD] test_socket creates too many locks Nov 6, 2017
@serhiy-storchaka
Copy link
Member Author

test_socket creates many locks on all platforms.

@ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
@serhiy-storchaka
Copy link
Member Author

serhiy-storchaka commented Oct 9, 2023 •
edited
Loading

It is a weird test. Initially the lock was introduced in bpo-31370 (#75551) bpo-6560 (#50809) to wrap addCleanup() (which appends to the list of callbacks) and doCleanups() (which pops from that list until it is empty). But in #110167 the lock was removed from doCleanups(), it now only wraps addCleanup(), that does not make sense, because list.append() is atomic.

Should the lock be removed at all? Or it is still needed in doCleanups() and should be restored?

cc @pitrou, @vstinner

@serhiy-storchaka
Copy link
Member Author

Sorry, actually it was introduced in bpo-6560 (#50809). cc @ncoghlan.

serhiy-storchaka added a commit to serhiy-storchaka/cpython that referenced this issue Oct 9, 2023
@serhiy-storchaka
pythongh-76106: Remove the cleanup lock in test_socket
1d9e321
@bedevere-app bedevere-app bot mentioned this issue Oct 9, 2023
Merged
@serhiy-storchaka
Copy link
Member Author

ThreadableTest now uses threading_helper.wait_threads_exit(). It means that if the client thread is still running at test cleanup stage, doCleanups() waits until it exit, and then calls all callbacks added in the client thread while it waited for thread exit. No race condition, no leaked resources, no need to guard addCleanup() and doCleanups() with a lock.

@vstinner
Copy link
Member

vstinner commented Oct 9, 2023

Yeah, it's really bad when a test leaves a thread running a thread in background. It's good that test_socket got fixed. I suppose that this issue can now be closed?

@vstinner vstinner closed this as completed Oct 9, 2023
@vstinner vstinner reopened this Oct 9, 2023
@vstinner
Copy link
Member

vstinner commented Oct 9, 2023

Sorry, I clicked on close by mistake.

@serhiy-storchaka
Copy link
Member Author

Removing no longer needed lock will fix the original issue.

serhiy-storchaka added a commit that referenced this issue Oct 11, 2023
@serhiy-storchaka
gh-76106: Remove the cleanup lock in test_socket (GH-110539)
f27b830 
It does not already work (because it locks only addCleanup(), not doCleanups()),
and it is no longer needed since the clean up procedure waits for all test threads to join.
miss-islington pushed a commit to miss-islington/cpython that referenced this issue Oct 11, 2023
@serhiy-storchaka @miss-islington
pythongh-76106: Remove the cleanup lock in test_socket (pythonGH-110539)
09b894e 
It does not already work (because it locks only addCleanup(), not doCleanups()),
and it is no longer needed since the clean up procedure waits for all test threads to join.
(cherry picked from commit f27b830)

Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>
This was referenced Oct 11, 2023
Merged
Merged
miss-islington pushed a commit to miss-islington/cpython that referenced this issue Oct 11, 2023
@serhiy-storchaka @miss-islington
pythongh-76106: Remove the cleanup lock in test_socket (pythonGH-110539)
4b7fb72 
It does not already work (because it locks only addCleanup(), not doCleanups()),
and it is no longer needed since the clean up procedure waits for all test threads to join.
(cherry picked from commit f27b830)

Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>
serhiy-storchaka added a commit that referenced this issue Oct 11, 2023
@miss-islington @serhiy-storchaka
[3.11] gh-76106: Remove the cleanup lock in test_socket (GH-110539) (G…
79b81d1 
…H-110700)

It does not already work (because it locks only addCleanup(), not doCleanups()),
and it is no longer needed since the clean up procedure waits for all test threads to join.
(cherry picked from commit f27b830)

Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>
serhiy-storchaka added a commit that referenced this issue Oct 11, 2023
@miss-islington @serhiy-storchaka
[3.12] gh-76106: Remove the cleanup lock in test_socket (GH-110539) (G…
bfb1f00 
…H-110699)

It does not already work (because it locks only addCleanup(), not doCleanups()),
and it is no longer needed since the clean up procedure waits for all test threads to join.
(cherry picked from commit f27b830)

Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>
Glyphack pushed a commit to Glyphack/cpython that referenced this issue Sep 2, 2024
@serhiy-storchaka @Glyphack
pythongh-76106: Remove the cleanup lock in test_socket (pythonGH-110539
8b37f98 
)

It does not already work (because it locks only addCleanup(), not doCleanups()),
and it is no longer needed since the clean up procedure waits for all test threads to join.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
3.7 (EOL) end of life tests Tests in the Lib/test dir type-bug An unexpected behavior, bug, or error
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@vstinner @serhiy-storchaka