media repo worker leaks open FDs

[richvdh]

There is a sawtooth on the "Open FDs" graph for the media repository: they accumulate until a Gen 2 GC happens:

The accumulated FDs appear to be CLOSE_WAIT sockets; they mostly appear to be outbound connections to port 80 or 443 so they are probably related to url preview requests.

[ShadowJonathan]

This looks to me like twisted not properly explicitly closing these, is this an upstream bug?

[clokep]

I'd be curious if we're seeing this on a server that is not using the S3 provider, just to narrow down the potential cause.

There was also a useful comment from the previous issue: #8653 (comment)

We seem to be leaking sockets in CLOSE_WAIT:

$ lsof -n -p 54367 | grep -Eo "\(.*\)$" | sort | uniq -c
   1229 (CLOSE_WAIT)
    190 (ESTABLISHED)
      3 (LISTEN)

[richvdh]

given the destinations in question for those CLOSE_WAIT sockets, I'd be surprised if S3 was relevant here.

[clokep]

Unfortunately it seems like #9497 did not fix this. 😢 I'll need to keep investigating.

[clokep]

I was unable to reproduce this, so this needs a bit more investigation.