Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Close open resources #3250

Open
wants to merge 4 commits into
base: master
Choose a base branch
from
Open

Close open resources #3250

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
49618be
Close a temporary file acting as a fake socket
kurtmckee Jul 25, 2024
62530ef
Close open sockets when an exception is encountered
kurtmckee Jul 25, 2024
dc197ff
Close open files to prevent `ResourceWarning` in the test suite
kurtmckee Jul 25, 2024
7756175
Escalate warnings to errors during testing
kurtmckee Jul 25, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 4 additions & 1 deletion gunicorn/instrument/statsd.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,11 +32,14 @@ def __init__(self, cfg):
else:
address_family = socket.AF_INET

self.sock = None
try:
self.sock = socket.socket(address_family, socket.SOCK_DGRAM)
self.sock.connect(cfg.statsd_host)
except Exception:
self.sock = None
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

shouldn't the garbage collector takes care about it?

Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ddon't try to close the cocket there. None should be ok.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

shouldn't the garbage collector takes care about it?

Even GC reliably working in our favor is no excuse. We should never make the WSGI application developer figure out which warning is our fault and harmless, and which one is triggered by their code and should be looked into.

Copy link
Owner

@benoitc benoitc Aug 11, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

closing sockets like this is unwanted also. The socket is not supposed to be created at this step. So just making it going to None will be enough.

However this is missing a more elaborated error handler to display what's going wrong.

Copy link
Contributor Author

@kurtmckee kurtmckee Aug 12, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've answered more fully below; the summary is that relying on the garbage collector without calling .close() will cause Python to throw a ResourceWarning. Simply creating the socket instance is sufficient to cause this problem, as demonstrated in the reproducer code I wrote below.

if self.sock is not None:
self.sock.close()
self.sock = None

self.dogstatsd_tags = cfg.dogstatsd_tags

Expand Down
3 changes: 3 additions & 0 deletions pyproject.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -76,6 +76,9 @@ main = "gunicorn.app.pasterapp:serve"
norecursedirs = ["examples", "lib", "local", "src"]
testpaths = ["tests/"]
addopts = "--assert=plain --cov=gunicorn --cov-report=xml"
filterwarnings = [
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've answered more fully below; the summary is that this ensures the test suite catches ResourceWarnings that might be introduced in the future.

"error",
]

[tool.setuptools]
zip-safe = false
Expand Down
3 changes: 3 additions & 0 deletions tests/t.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -65,3 +65,6 @@ def send(self, data):

def seek(self, offset, whence=0):
self.tmp.seek(offset, whence)

def close(self):
self.tmp.close()
13 changes: 8 additions & 5 deletions tests/test_http.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -184,11 +184,14 @@ def test_socket_unreader_chunk():
fake_sock = t.FakeSocket(io.BytesIO(b'Lorem ipsum dolor'))
sock_unreader = SocketUnreader(fake_sock, max_chunk=5)

assert sock_unreader.chunk() == b'Lorem'
assert sock_unreader.chunk() == b' ipsu'
assert sock_unreader.chunk() == b'm dol'
assert sock_unreader.chunk() == b'or'
assert sock_unreader.chunk() == b''
try:
assert sock_unreader.chunk() == b'Lorem'
assert sock_unreader.chunk() == b' ipsu'
assert sock_unreader.chunk() == b'm dol'
assert sock_unreader.chunk() == b'or'
assert sock_unreader.chunk() == b''
finally:
fake_sock.close()


def test_length_reader_read():
Expand Down
32 changes: 19 additions & 13 deletions tests/test_reload.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -39,9 +39,12 @@ def test_reload_on_syntax_error():
log = mock.Mock()
worker = MyWorker(age=0, ppid=0, sockets=[], app=app, timeout=0, cfg=cfg, log=log)

worker.init_process()
reloader.start.assert_called_with()
reloader.add_extra_file.assert_called_with('syntax_error_filename')
try:
worker.init_process()
reloader.start.assert_called_with()
reloader.add_extra_file.assert_called_with('syntax_error_filename')
finally:
worker.tmp.close()


def test_start_reloader_after_load_wsgi():
Expand All @@ -56,13 +59,16 @@ def test_start_reloader_after_load_wsgi():
log = mock.Mock()
worker = MyWorker(age=0, ppid=0, sockets=[], app=app, timeout=0, cfg=cfg, log=log)

worker.load_wsgi = mock.Mock()
mock_parent = mock.Mock()
mock_parent.attach_mock(worker.load_wsgi, 'load_wsgi')
mock_parent.attach_mock(reloader.start, 'reloader_start')

worker.init_process()
mock_parent.assert_has_calls([
mock.call.load_wsgi(),
mock.call.reloader_start(),
])
try:
worker.load_wsgi = mock.Mock()
mock_parent = mock.Mock()
mock_parent.attach_mock(worker.load_wsgi, 'load_wsgi')
mock_parent.attach_mock(reloader.start, 'reloader_start')

worker.init_process()
mock_parent.assert_has_calls([
mock.call.load_wsgi(),
mock.call.reloader_start(),
])
finally:
worker.tmp.close()
Loading