Unable to do TLSInterception when using proxy_pool plugin

[musicderp]

Describe the bug
I am using the proxy_pool plugin as a base for my own proxy selector. I was expecting to be able to use TLS interception in order to read the uri in order to determine which upstream proxy to send the request to. Unfortunately, as the plugin is written, I'm unable to retrieve the request.path unless it's an http request. I'm unsure why the TLS interception doesn't take place because in before_upstream_connection we should have the entire request before we even attempt to connect to the upstream proxy, so I don't see a reason we can't take a peak at the contents and base the upstream proxy on the content of the uri

To Reproduce
Steps to reproduce the behavior:

1. Run proxy.py with TLS interception parameters
2. Add debug to the proxy_pool plugin to attempt to print the uri/request path before choosing an upstream proxy
3. Make an https request and see that you get None back

Expected behavior
I expect to be able to snoop on the contents of the https request because I have TLS Interception enabled

Version information

• OS: Docker
• proxy.py Version: https://github.com/abhinavsingh/proxy.py.git@develop

[musicderp]

Additionally, it looks like it prevents other plugins that would usually work from doing tls interception. I spun up the modify request plugin, and if the proxy_pool is active, no interception happens, but just the modify requests plugin works on its own.

[jryantz]

Would be interested to know if there is a resolution for this. The proxy pool / tls interception incompatibility is a blocker for my solution as well.

[musicderp]

Would be interested to know if there is a resolution for this. The proxy pool / tls interception incompatibility is a blocker for my solution as well.

I ended up moving my project to mitmproxy where I got the tls interception working with the upstream proxy

[abhinavsingh]

@musicderp @jryantz Thank you folks for reporting this. Proxy pool was originally added as an example, since community demanded such a feature. But I am unsure if proxy pool uses/honours the TLS interception pipeline. Its been long and I'll need to verify the same.

[abhinavsingh]

@musicderp @jryantz I checked the code back and looks like there is a confusion regarding how proxy pool plugin actually works. TL;DR -- It does not support TLS interception, it expects upstream proxies to perform TLS interception.

However, I can understand that at-times upstream proxies might not be under our control and hence you might want to TLS intercept locally on the host system itself. To support this we'll need refactoring of the code, to assume, upstream endpoint is not HTTPS server but a HTTP PROXY server.

I started a draft PR to start experimenting with it #1387 but it needs more time and attention. I'll hope someone sees this through via contribution. I'll try to come back to it later.