Is there "no proxy" configuration support?

[nathan-alden-sr]

One cool feature of Cntlm (and most other proxies) is the ability to ignore certain IP addresses/ranges and hostnames/ranges. e.g., in Cntlm, I could do this:

NoProxy localhost,127.0.0.*,*.corporatenetwork.com

This is a critical feature because without it, I can't force certain IPs and hostnames to resolve locally.

[genotrance]

Hello Nathan,
I can look into implementing the NoProxy option as Cntlm defines it.

Standalone proxy. Cntlm has been reworked in a way that allows you either to use it like before, with a parent proxy, or as an independent proxy altogether (think Squid, Tinyproxy, etc). The most common configuration, however, will probably be a combination of the two. Use the new NoProxy option to specify which URL's should go through the parent proxy and which URL's should Cntlm process directly, via direct ("intranet") connections. This allows you to set Cntlm permanently in your applications and use it for all links within your company and on the Internet.

My use case for Px has primarily been to use it with CLI tools since most modern browsers can handle the Windows proxy directly, along with any NoProxy declarations. It will help if you could describe your use case a bit further since it will help me understand the utility of such a capability.

Thanks for your feedback.

[nathan-alden-sr]

That's how we're using it, as well. We've got a whole host of originated-on-Linux CLI tools like git.exe and none of them know how to perform NTLM negotiation. The way we use px is to define the forward proxy to be the corporate proxy. The thing is, it is nearly impossible (politically) to get new ignore rules added to the corporate proxy. A good example on our network is 192.168.100.4, which is an IP address used by a VirtualBox VM on several of our machines. Without a NoProxy-like setting, traffic is forwarded to the corporate proxy, which doesn't consider it (for some reason) to be a local address. We get connection failures as a result. With Cntlm, we simply add it to the cntlm.ini file and we're good to go.

If you did add ignore capability, it would be useful to allow us to specify them in both wildcard and CIDR format. Windows' Internet Options control panel only allows wildcards, which makes ignoring non-octet-bounded networks irritating.

localhost,::1,127.0.0.*,*.corporatenetwork.com,192.168.100.0/28

Thanks for writing such a great tool! The only other useful thing I think px is missing is an installer that runs it as a Windows service. It's a bit cumbersome to set that up manually.

[genotrance]

It's been a while but I've finally finished v0.2.0 which has the noproxy feature you requested. I discovered several issues with Px while I built out the feature so it took longer than expected. Please download and try it out, your feedback will be appreciated. I'll leave this issue open until I hear back.

Note that the noproxy declaration only supports IP addresses at this time (specific IPs, ranges, with wildcards and CIDR format), I haven't gotten around to domain names and related variations.

Another note, I've not been able to test Px on an IPv6 network so while the implementation should support IPv6 addresses, there could be issues.

I considered your suggestion to provide an installer that runs Px as a Windows service. However, given Px has to run as a domain user (and not a local system account) in order to successfully perform NTLM authentication, running it would require manual configuration anyway (user login and password). Using a service manager such as NSSM might make things easier without requiring functionality within Px. A lighter alternative would be to run Px via startup, though that would require the user to login first.

However, I'm open to your comments on the matter. Please feel free to open a new feature request on the subject so that it is tracked independently from this noproxy thread.

[nathan-alden-sr]

Thanks, @genotrance! The noproxy stuff is exciting! I'll let you know what I find.

[nathan-alden-sr]

Hi, @genotrance. I'm getting 0.2 installed on my machine now. I have a question: Does noproxy support IPv6? Right now, I'm defining my NO_PROXY system environment variable like this: 127.0.0.*,::1,localhost,.corporatedomain.com. My goal is to eliminate the need to define NO_PROXY.

[genotrance]

I mentioned this above - the noproxy declaration only supports IP addresses at this time (specific IPs, ranges, with wildcards and CIDR format), I haven't gotten around to domain names and related variations so corporatedomain.com won't work.

I've also not been able to test Px on an IPv6 network so while the implementation should support IPv6 addresses, there could be issues since I've not been able to try it out. I'm interested to know how it works for you.

[nathan-alden-sr]

Yep, I understood that domain names are not currently supported; I was just pasting what my existing before-px environment variable was set to.

So far, I haven't seen any issues. I am able to use commands like git pull just fine.

[genotrance]

That's good to hear, hopefully, performance is also better since I fixed several bugs that were causing long delays.

Are you seeing the IPv6 portion also working as expected?

[Kronos11]

Great job on this, I was going to actually start implementing this for my company.
I also looked into pulling the proxy settings from the IE or Windows default and then incorporating that directly into the noproxy list. Do you think that would be possible?

[genotrance]

Yes, that can be done, I can add that in a few days. Adding support for PAC proxy configurations will be a lot harder though.

[Kronos11]

I also looked into doing some pac parsing, I think I got it to work just running it as javascript, however I am very concerned about the performance of checking that every call. I do not believe that would be feasible. I think there would probably have to be some sort of parsing done at startup which is very much more complicated.

[genotrance]

Kronos11, can you please clarify what you mean by this:

I also looked into pulling the proxy settings from the IE or Windows default and then incorporating that directly into the noproxy list.

When you say incorporating into the noproxy list, what do you mean? I initially thought you meant that you'd set the NTLM proxy to the value set in IE but I don't see what noproxy has to do with that.

[nathan-alden-sr]

Windows allows you to configure an exceptions list. Maybe that's what @Kronos11 is referring to?

[Kronos11]

Nathan, yes that is what I am referring to.
@genotrance
My company also uses a proxy.pac file that parses the outgoing url and determines if it should go through the proxy or be bypassed. I would also be willing to help out with this type of feature if you are willing to incorporate it. My Python is a little rusty but I think I have a simple prototype of executing javascript inside python. I am still unsure what the performance impact of this is though.

[genotrance]

Okay, that makes sense - so here's the design I'm moving towards:-

• If proxy server is defined in px.ini, detect and use that server on startup
  • Ignore anything defined in Internet Properties altogether including exceptions
  • Existing behavior for everything else
• Else if AutoConfigUrl defined in Internet Properties
  • Load PAC file using pypac
  • Check PAC configuration in registry every 30 seconds to detect changes (VPN)
  • If noproxy defined in px.ini, check every host first with noproxy for direct connection
  • If not in noproxy, check with pypac to obtain proxy host or direct and act accordingly
• Else if proxy server is configured in Internet Properties, detect and use that as the server on startup
  • If Exceptions defined in Internet Properties, load those into noproxy
  • If noproxy defined in px.ini, extend noproxy with those values

The pypac Python module makes the PAC load and check pretty easy. Python has a js2py module as well which can run the PAC Javascript so most of the heavy lifting is done.

Performance will be impacted with PAC though since each host will have to be checked. I could cache the PAC responses within Python but will need to see how to synchronize that data across all threads and processes, might be easier just to use pypac and deal with the overhead for simplicity.

Let me know if you have any feedback on this approach.

[Kronos11]

that sounds like a great plan

[genotrance]

Took a while but I finally just added support for PAC files and auto discovery of the proxy. Issue #30 is being used to track that in specific.

I'll continue using this issue to track the noproxy idea. Px already allows noproxy for IP addresses and supports bypassing the proxy if told so by the PAC file. The things that remain are as follows:-

• Support for hostnames in noproxy definitions
• Loading noproxy info from Internet Options

[PandarinDev]

Any updates on supporting domain names for noproxy? Both issues related to this have been closed as of 2018. I can look into adding support for this feature if you are not planning on working on it in the near future.

[genotrance]

Now that Px relies on libcurl, this should be much easier to implement. We simply need to forward the info to mcurl.py.

Right now, wproxy.py is doing the noproxy processing (but IP only, not domains). We simply need to rewire Px to just delegate everything to mcurl.py.

[aadiene]

Hello @genotrance,

First of all, thanks for writing such a great tool.

I'm thinking about using PX instead of CNTLM but still facing an issue regarding the noproxy rule.
In MODE_CONFIG_PAC, no matter the rules (IP/Hostname/Pattern), its seems that noproxy is ignored. Am I missing something ?

I also noticed this Proxy and noproxy details need to come from the same source - they are not merged from wproxy.py.
Can you elaborate about different sources please.

[EDIT] : I am on Windows platform and using the v0.8.3 release

Thanks for your help.

[aadiene]

After forking repo, I found out the issue and created a PR #177

[genotrance]

I also noticed this Proxy and noproxy details need to come from the same source - they are not merged from wproxy.py. Can you elaborate about different sources please.

The idea here is that MODE_CONFIG_PAC will download a PAC file which will handle all noproxy situations within the PAC file. The noproxy configuration within px.ini is only relevant when you configure the proxy server within px.ini. That's what it means when it says - same sources - the PAC file should include everything within it, including noproxy handling.

What's your scenario? Is your PAC file not adding the right exceptions?

[aadiene]

Exactly ! PAC file is managed by company and does not include all of the exceptions.
Thats why I think local exceptions should be processed in MODE_CONFIG_PAC (the same for MODE_PAC as well)

[genotrance]

This is fixed in v0.9.0 still in development - see branch.

Fixes include noproxy host support as well as overriding NOPROXY for all modes from the Px configuration. Appreciate any tests.

[penenkel]

Could you create a prerelease with the binaries? Then I could test it.

By the way, I appreciate your hard work on this, it has made my work-live inside a corporate network much easier.

[genotrance]

v0.9.0 has been released finally!