Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Feature request] Add PAC file support #30

Closed
Scrapz opened this issue Feb 27, 2018 · 13 comments
Closed

[Feature request] Add PAC file support #30

Scrapz opened this issue Feb 27, 2018 · 13 comments
Labels
enhancement fixed

Comments

@Scrapz
Copy link

Scrapz commented Feb 27, 2018

Maybe also the possibility to add proxy autodiscovery (WPAD), which returns a PAC file.
@genotrance
Copy link
Owner

This is exactly what I'm looking at for the last few hours. The pypac package will make this super easy along with urllib to get manually defined proxies in Internet Options.

#2 already discussed this at length but that's more about noproxy so will keep this issue open to track PAC and WPAD support separately.

@genotrance
Copy link
Owner

This feature has been implemented in the latest release. Please try it out and let me know how it works for you.

@genotrance genotrance mentioned this issue Feb 28, 2018
Closed
@w1sht0l1v3
Copy link

Doesn't work for me...
-config:

[proxy]
server =
port = 3128
listen = 127.0.0.1
allow = *.*.*.*
gateway = 0
hostonly = 0
noproxy = 
useragent = 

[settings]
workers = 2
threads = 5
idle = 30
socktimeout = 5.0
proxyreload = 60
foreground = 0
log = 1

-output log

C:\px>MainProcess: MainThread: 1519830664: verify_request: Client address: 127.0.0.1
MainProcess: Thread_0: 1519830664: do_CONNECT: Entering
MainProcess: Thread_0: 1519830664: do_transaction: Entering
MainProcess: Thread_0: 1519830664: get_destination: www.google.ro:443
MainProcess: Thread_0: 1519830665: load_proxy: Proxy mode = 2
MainProcess: Thread_0: 1519830665: find_proxy_for_url: pypac = [('prox1.intra', '8080'), ('prox2.intra', '8080'), ('prox3.intra', '8080'), ('prox4.intra', '8080'), ('prox5.intra', '8080'), ('prox7.intra', '8080'), ('prox8.intra', '8080'), ('prox9.intra', '8080')] -1
MainProcess: Thread_0: 1519830665: get_destination: Proxy from PAC = [('prox1.intra', '8080'), ('prox2.intra', '8080'), ('prox3.intra', '8080'), ('prox4.intra', '8080'), ('prox5.intra', '8080'), ('prox7.intra', '8080'), ('prox8.intra', '8080'), ('prox9.intra', '8080')]
MainProcess: Thread_0: 1519830665: get_response: winkerberos SSPI
MainProcess: Thread_0: 1519830665: do_socket: Entering
MainProcess: Thread_0: 1519830665: do_socket_connect: New connection: ('prox1.intra', '8080')
MainProcess: Thread_0: 1519830665: do_socket_connect: Connect failed
MainProcess: Thread_0: 1519830665: do_socket_connect: New connection: ('prox2.intra', '8080')
MainProcess: Thread_0: 1519830665: do_socket_connect: Connect failed
MainProcess: Thread_0: 1519830665: do_socket_connect: New connection: ('prox3.intra', '8080')
MainProcess: Thread_0: 1519830665: do_socket_connect: Connect failed
MainProcess: Thread_0: 1519830665: do_socket_connect: New connection: ('prox4.intra', '8080')
MainProcess: Thread_0: 1519830665: do_socket_connect: Connect failed
MainProcess: Thread_0: 1519830665: do_socket_connect: New connection: ('prox5.intra', '8080')
MainProcess: Thread_0: 1519830665: do_socket_connect: Connect failed
MainProcess: Thread_0: 1519830665: do_socket_connect: New connection: ('prox7.intra', '8080')
MainProcess: Thread_0: 1519830665: do_socket_connect: Connect failed
MainProcess: Thread_0: 1519830665: do_socket_connect: New connection: ('prox8.intra', '8080')
MainProcess: Thread_0: 1519830665: do_socket_connect: Connect failed
MainProcess: Thread_0: 1519830665: do_socket_connect: New connection: ('prox9.intra', '8080')
MainProcess: Thread_0: 1519830665: do_socket_connect: Connect failed
MainProcess: Thread_0: 1519830665: do_CONNECT: Error 408
MainProcess: Thread_0: 1519830665: log_message: code 408, message Request Timeout
MainProcess: Thread_0: 1519830665: log_message: "CONNECT www.google.ro:443 HTTP/1.1" 408 -
MainProcess: Thread_0: 1519830665: do_CONNECT: Transferred 0 bytes
MainProcess: Thread_0: 1519830665: do_CONNECT: Done

If i use in config "server = prox1.intra:8080,etc" it works fine.

@jpjoux jpjoux mentioned this issue Feb 28, 2018
Merged
@genotrance
Copy link
Owner

Thanks @jpjoux, just pulled his PR32 to fix this PAC port issue which was treating the proxy port as a string instead of an int and failing to connect. Please try the latest vHEAD release to see if it now works for you.

@Scrapz
Copy link
Author

Scrapz commented Feb 28, 2018 •
edited
Loading

Still having issues, I get an "ERR_PROXY_AUTH_UNSUPPORTED", in Chrome, IE displays a 407 code.

This is the log output (with some private stuff removed):

MainProcess: MainThread: 1519860952: load_proxy: Proxy mode = 2 Serving at 127.0.0.1:3128 proc MainProcess
MainProcess: MainThread: 1519860952: parse_config: proxy:server =
MainProcess: MainThread: 1519860952: parse_config: proxy:port = 3128
MainProcess: MainThread: 1519860952: parse_config: proxy:listen = 127.0.0.1
MainProcess: MainThread: 1519860952: parse_config: proxy:allow = ...
MainProcess: MainThread: 1519860952: parse_config: proxy:gateway = 0
MainProcess: MainThread: 1519860952: parse_config: proxy:noproxy =
MainProcess: MainThread: 1519860952: parse_config: proxy:useragent =
MainProcess: MainThread: 1519860952: parse_config: proxy:hostonly = 0
MainProcess: MainThread: 1519860952: parse_config: settings:workers = 2
MainProcess: MainThread: 1519860952: parse_config: settings:threads = 5
MainProcess: MainThread: 1519860952: parse_config: settings:idle = 30
MainProcess: MainThread: 1519860952: parse_config: settings:socktimeout = 5.0
MainProcess: MainThread: 1519860952: parse_config: settings:log = 1
MainProcess: MainThread: 1519860952: parse_config: settings:proxyreload = 60
MainProcess: MainThread: 1519860952: parse_config: settings:foreground = 0
MainProcess: MainThread: 1519860952: detach_console: Freed console successfully
MainProcess: MainThread: 1519860959: verify_request: Client address: 127.0.0.1
MainProcess: Thread_0: 1519860959: do_CONNECT: Entering
MainProcess: Thread_0: 1519860959: do_transaction: Entering
MainProcess: Thread_0: 1519860959: get_destination: www.google.com.au:443
MainProcess: Thread_0: 1519860959: load_proxy: Skip proxy refresh
MainProcess: Thread_0: 1519860959: find_proxy_for_url: pypac = [('proxyserver', 8080)] -1
MainProcess: Thread_0: 1519860959: get_destination: Proxy from PAC = [('proxyserver', 8080)]
MainProcess: Thread_0: 1519860959: get_response: winkerberos SSPI
MainProcess: Thread_0: 1519860959: do_socket: Entering
MainProcess: Thread_0: 1519860959: do_socket_connect: New connection: ('proxyserver', 8080)
MainProcess: Thread_0: 1519860959: do_socket: b'CONNECT www.google.com.au:443 HTTP/1.1\r\n'
MainProcess: Thread_0: 1519860959: do_socket: Sending b'Host: www.google.com.au:443\r\n'
MainProcess: Thread_0: 1519860959: do_socket: Sending b'Proxy-Connection: keep-alive\r\n'
MainProcess: Thread_0: 1519860959: do_socket: Sending b'User-Agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/64.0.3282.167 Safari/537.36\r\n'
MainProcess: Thread_0: 1519860959: do_socket: Sending extra b'Proxy-Authorization: Negotiate {redacted}\r\n'
MainProcess: Thread_0: 1519860959: do_socket: Reading response code
MainProcess: Thread_0: 1519860959: do_socket: Response code: 407 False
MainProcess: Thread_0: 1519860959: do_socket: Reading response headers
MainProcess: Thread_0: 1519860959: do_socket: Received header Mime-Version = 1.0
MainProcess: Thread_0: 1519860959: do_socket: Received header Date = Thu, 01 Mar 2018 10:36:00 AEDT
MainProcess: Thread_0: 1519860959: do_socket: Received header Via = 1.1 proxy01:80 (Cisco-WSA/9.1.2-022)
MainProcess: Thread_0: 1519860959: do_socket: Received header Content-Type = text/html
MainProcess: Thread_0: 1519860959: do_socket: Received header Proxy-Authenticate = NTLM
MainProcess: Thread_0: 1519860959: do_socket: Received header Proxy-Authenticate = Basic realm="Cisco IronPort Web Security Appliance"
MainProcess: Thread_0: 1519860959: do_socket: Received header Connection = close
MainProcess: Thread_0: 1519860959: do_socket: Received header Proxy-Connection = close
MainProcess: Thread_0: 1519860959: do_socket: Received header Content-Length = 2244
MainProcess: Thread_0: 1519860959: do_socket: Reading response data
MainProcess: Thread_0: 1519860959: do_socket: Content length 2244
MainProcess: Thread_0: 1519860960: do_transaction: Auth required
MainProcess: Thread_0: 1519860960: do_transaction: Didn't get challenge, not NTLM proxy
MainProcess: Thread_0: 1519860960: do_CONNECT: Error 407
MainProcess: Thread_0: 1519860960: log_message: code 407, message Proxy Authentication Required
MainProcess: Thread_0: 1519860960: log_message: "CONNECT www.google.com.au:443 HTTP/1.1" 407 -
MainProcess: Thread_0: 1519860960: do_CONNECT: Transferred 0 bytes
MainProcess: Thread_0: 1519860960: do_CONNECT: Done

@genotrance
Copy link
Owner

Looks like you are also running into issue #29. It's getting the PAC proxy info correctly.

@genotrance
Copy link
Owner

PAC support is now functional but Px has gone from a 6MB binary to 14MB which isn't too bad, but where it used 10MB of RAM per process, it now uses 140MB of RAM per process.

This is because pypac uses js2py to convert the PAC file, which is Javascript, into Python code. js2py takes up 120MB of RAM by itself.

I'm not especially thrilled by this and am looking into alternatives. I've also requested the pypac module owner for his opinion on the matter.

I'm curious what the Px community's opinion is on the matter as well and whether time should be spent optimizing this or prioritizing some other feature.

@Scrapz
Copy link
Author

Scrapz commented Mar 1, 2018

It's not an absolute requirement, it just saves me from having to statically specify a bunch of noproxy options. I'd be happy to revert version in order to keep the footprint low, and have you focus on other features.

@genotrance
Copy link
Owner

I definitely benefit myself from PAC support - when I'm not on my VPN, I can go direct. Without PAC support, I had to have two ini files and stop/start Px. Now, with PAC support, it is zero effort.

I just found a very simple PAC implementation - just leverage Window's built-in infrastructure.
http://markmail.org/download.xqy?id=4lc5svoe5wecuyg4&number=1

I'll work on moving over over the next few weeks. This is a no-brainer since the code is ready made.

genotrance added a commit that referenced this issue Mar 5, 2018
@genotrance
Improved PAC memory usage #30, added setup.py for pip #24
21ff7dd
@genotrance
Copy link
Owner

I just replaced the pypac dependency with WinHttp and we are back to a 6MB binary and 10MB RAM size.

Hope it still works for everyone as expected.

@jpjoux
Copy link

jpjoux commented Mar 6, 2018 •
edited
Loading

@genotrance I have tested the head version. it works fine for me. It slow but i do not know if it python or not

@genotrance genotrance removed the waiting label Mar 7, 2018
@gophynna
Copy link

How do we use this on the windows binary?

@hellboy81
Copy link

hellboy81 commented Feb 5, 2019 •
edited
Loading

The same question:

px
Serving at 127.0.0.1:3128 proc MainProcess
Serving at 127.0.0.1:3128 proc Process-1

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement fixed
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
6 participants
@genotrance @hellboy81 @Scrapz @w1sht0l1v3 @gophynna @jpjoux