Is there a way to disable the persistant plugin in the faraday adapter?

[m-arav]

Context:
Using faraday with httpx adapter in background jobs(Sidekiq). We have 2 processes with 10 threads each and end up making a lot of tcp connections. Because the connections are not closed but kept in a CLOSE_WAIT state, we end up running out of file descriptors.

Is there a way to disable the persistent plugin while using the faraday adapter? or should I monkey patch the faraday adapter for this use case?

[HoneyryderChuck]

Hi @m-arav ,

You can disable persistence of connections by setting the persistent option to false:

# while setting yourr faraday conn
conn.adapter :httpx, persistent: false

That sucks, your use-case is actually something I mention as a caveat when using the persistent plugin in the wiki.

I've enabled persistent connections in the faraday adapter because I thought that it would serve the majority of use-cases, as most faraday users complain about it not reusing tcp connections for same-origin requests by default (I believe the faraday maintainers maintain a page listing adapters with socket persistence support).

My assessment may be wrong here, and there may be more faraday users which prefer the approach you mentioned. Sadly, there's not a faraday "handle" to signal to the adapter which configuration is preferred in this case. I'd recommend discussing this with the faraday maintainers to see what do they think about this, what should be the default behaviour, and whether it's worth to add an option to signal to the adapters whether sockets should be kept open for re-usage (for the adapters which support this at least).

[m-arav]

Thank you.

I must have completely missed the persistent: false option when going through the wiki. I feel the same way, enabling persistent connections is the sane default to have.

One of the ideas that came up while discussing this with my team was the ability to set a max shared pool size. This would benefit sharing connections for the most used connections once we hit the limit we could close the least used connection and create a new one. Would this feature be beneficial for httpx?

[HoneyryderChuck]

must have completely missed the persistent: false option when going through the wiki.

That's because it was not documented anywhere :) I've since updated the faraday adapter section with this workaround.

One of the ideas that came up while discussing this with my team was the ability to set a max shared pool size.

Not sure what the whole proposal would be, but I would welcome it. The current connection pool isn't shared across threads; it has the benefit of making httpx interactions thread-safe without the synchronization overhead, with the caveat that you may needlessly initialize multiple connections to the same origin in multithread scenarios. I think that, given enough scale, you'd tend to do it anyway (otherwise you'd starve thread workers waiting on an available connection), but it's not an argument for doing an alternative implementation.