Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Leaked Connections should close socket #323

Closed
percontation opened this issue Jul 5, 2018 · 4 comments
Closed

Leaked Connections should close socket #323

percontation opened this issue Jul 5, 2018 · 4 comments
Labels

Comments

@percontation
Copy link
Contributor

  • asyncpg version: 0.17.0.dev0+78ea13a
  • PostgreSQL version: 10.4 (Debian 10.4-2.pgdg90+1)
  • Do you use a PostgreSQL SaaS? If so, which? Can you reproduce
    the issue with a local PostgreSQL install?
    : It's a docker postgres:latest
  • Python version: 3.6.5
  • Platform: OS X 10.11.6
  • Do you use pgbouncer?: no
  • Did you install asyncpg with pip?: yes (but github, not pypi)
  • If you built asyncpg locally, which version of Cython did you use?: 0.28.2
  • Can the issue be reproduced under both asyncio and
    uvloop?
    : yes

In the face of timeouts/other exceptions, it may not always be possible to have a reference to a Connection (that was successfully initiated) and call .close() on it. So, leaked Connections should clean up after themselves. For example, running asyncio.get_event_loop().run_until_complete(asyncpg.connect("postgres://postgres:postgres@localhost/postgres")) seems to leave the socket open forever (at least on my platform).

When attempting to address this, I noticed a couple of things that may or may not be related bugs:

  1. Connection.terminate() seems to not close the socket, and also leave it open forever.
  2. Connection and its associated Protocol never seem to get garbage collected after calling .terminate() (or getting leaked); my guess is that it has something to do with the underlying _SelectorSocketTransport not getting GCed, but I'm not confident about what's happening here.
1st1 added a commit that referenced this issue Jul 5, 2018
Connection objects currently stay alive and open if they are not
explicitly closed or terminated.  GC won't happen to them because event
loops have a strong reference to their underlying Transport object.

By replacing a strong Connection<->Protocol reference with a weak one,
we are able to implement Connection.__del__() method that:

* issues a warning if a Connection object is being GCed prior
  to be explicitly closed;

* terminates the underlying Protocol and Transport, effectively closing
  the open network connection to the Postgres server.

When in asyncio debug mode (enabled by PYTHONASYNCIODEBUG env variable
or explicitly with `loop.set_debug(True)`) Connection objects save the
traceback of their origin and later use it to make the GC warning
clarer.

Addresses #323.
@1st1
Copy link
Member

1st1 commented Jul 5, 2018

In the face of timeouts/other exceptions, it may not always be possible to have a reference to a Connection (that was successfully initiated) and call .close() on it. So, leaked Connections should clean up after themselves.

PR #324 partially addresses this by:

  • enabling Connection.__del__ by breaking a strong reference cycle between Connection, Protocol, Transport, and Loop objects.

  • issuing a warning about unclosed connections being GCed in Connection.__del__.

  • forcibly closing the underlying Protocol / Transport.

Connection.terminate() seems to not close the socket, and also leave it open forever.

You need to allow the event loop to run for a little bit for the termination callbacks to fire. If you close the event loop immediately after calling connection.terminate() it won't have an opportunity to actually shut down the transport. In such an event, it will take much longer for the event loop to get GCed with all other objects attached to it.

1st1 added a commit that referenced this issue Jul 5, 2018
Connection objects currently stay alive and open if they are not
explicitly closed or terminated.  GC won't happen to them because event
loops have a strong reference to their underlying Transport object.

By replacing a strong Connection<->Protocol reference with a weak one,
we are able to implement Connection.__del__() method that:

* issues a warning if a Connection object is being GCed prior
  to be explicitly closed;

* terminates the underlying Protocol and Transport, effectively closing
  the open network connection to the Postgres server.

When in asyncio debug mode (enabled by PYTHONASYNCIODEBUG env variable
or explicitly with `loop.set_debug(True)`) Connection objects save the
traceback of their origin and later use it to make the GC warning
clarer.

Addresses #323.
@1st1 1st1 added the bug label Jul 5, 2018
@percontation
Copy link
Contributor Author

Awesome, thanks for the fix (and for the terminate explanation).

This fixes the the leak in the context that I noticed it (but I'll let a maintainer close this issue, since the "PR #324 partially addresses this" makes it sound like there may have been more work left to do here?)

@1st1
Copy link
Member

1st1 commented Jul 6, 2018

makes it sound like there may have been more work left to do here?

I used "partially" because it's impossible to fully solve the problem in all cases. The loop holds strong references to its transports, so there might be some hypothetical situation where the only solution is to manually close the Connection object. I.e. if you forget to close an event loop in your long-living application, that zombie event loop will keep its connections forever open. It's highly unlikely that this happens in a properly engineered application though. :)

@elprans
Copy link
Member

elprans commented Jul 10, 2018

This is fixed in #324

@elprans elprans closed this as completed Jul 10, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

3 participants