Add support for NTLM proxies

[hickford]

Pip doesn't work—it times out—in my office because our computers are behind an evil proxy. It still doesn't work even after specifying the http_proxy environment variable per http://docs.python.org/2/library/urllib.html

The problem is the proxy uses a proprietary technology called NTLM that urllib (or whatever http client Pip uses) can't cope with https://en.wikipedia.org/wiki/NTLM

This problem isn't peculiar to Pip--other Python and Ruby and Nodejs break too--but the problem is fatal to Pip users. Python's strength is its great libraries (Numpy, Beautiful Soup, Pandas and so on). Behind a NTLM proxy, these are inaccessible and Python development essentially impossible.

On my computer, I've installed all these packages from source (T-E-D-I-O-U-S), but that doesn't help my colleagues when I send them a 'handy ten line Python script'. It fails with an import error. We spend a quarter of an hour installing the dependency chain from source. Oh, they say, I heard Python was easy to use.

---

Maybe these libraries could help https://code.google.com/p/python-ntlm/ https://github.com/requests/requests-ntlm

[pfmoore]

+1 to adding NTLM support. I have been in this situation and the commonly suggested workaround (cntlm) is not very usable in practice. The OP may be able to get his corporate IT support to sanction use of "Firewall Client for ISA Server" (which is a download from MS) - this provides transparent access but needs central configuration AIUI so it's not a complete solution.

But including NTLM support out of the box in pip would be a definite usability benefit for users behind such proxies.

[dstufft]

If my requests branch works correctly and lands it's possible we can use something like https://github.com/requests/requests-ntlm to make this happen.

[pfmoore]

That sounds reasonable. I'd be willing to look at making a patch (although I'd need the OP to test, as I no longer have access to an NTLM environment that needs this functionality) once the requests branch lands.

[thedrow]

@pfmoore If you can't set up one on Travis it will be hard to ensure that the implementation doesn't break.
Can you think about a way to set up an NTLM environment or maybe even fake one?

[pfmoore]

@dstufft Nope, I know of no way of setting up or faking an NTLM environment. I've never been anything more than a victim of it. Having said that, personally I'd rather see support added and given a one-off test by @matt-hickford, than not add it just because we can't set up automated tests. That way when IT policy at work changes and I lose my current workaround, I won't also lose access to PyPI... :-)

[hickford]

For interest, corresponding Ruby Gems issue http://stackoverflow.com/q/4418/284795

[espdev]

As for me, I have also had some problems with pip because of the corporate ISA Proxy server.

I have tried to set proxy parameters through option --proxy and environment variable HTTP_PROXY, however, it does not work. An error has occurred:

HTTP Error 407: Proxy Authentication Required (The ISA Server requires authorization to fulfill the request. Access to the Web Proxy filter is denied.)

First and foremost, I have installed "Firewall Client for ISA Server". With "Firewall Client for ISA Server" one does not need to set proxy parameters for each program, as in this case the authentication is automatic.

It works very well with other programs, for example, Dropbox can be easily connected (its client is written in Python), however, there are some minor problems with urllib. I have decided to carry out some experiments with urllib, and I have found the following:

Firstly, I have viewed the list of current proxy:

print urllib2.getproxies()
{'ftp': 'ftp://prxoxy:3128', 'http': 'http://prxoxy:3128', 'https': 'https://prxoxy:3128'}

Then I have entered a very simple example:

u = urllib2.urlopen(url)
data = u.read()

In this example I have not set any proxy parameters. The program does not work. And again the same error has occurred: HTTP Error 407. .... urllib does not work with the default settings of proxy and "Firewall Client for ISA Server" does not help at all in this case.

Then I have set proxy parameters with authentication:

proxy = urllib2.ProxyHandler({'http': 'http://user:password@proxy:3128'})
opener = urllib2.build_opener(proxy)
urllib2.install_opener(opener)

u = urllib2.urlopen(url)
data = u.read()

And again the same error has occurred: `HTTP Error 407``. It seems to me there is a problem with NTLM support.

And here is the most interesting moment! I have decided to change an example and replace ProxyHandler by UnknownHandler:

proxy = urllib2.ProxyHandler({})
opener = urllib2.build_opener(proxy)
urllib2.install_opener(opener)

u = urllib2.urlopen(url)
data = u.read()

The program works! In this case urllib does not use any information about proxy; instead "Firewall Client for ISA Server" automatically authenticates the user.

As far as I understand pip uses urllib settings by default, if proxy parameters have not been set. Therefore, all these problems with ISA Proxy server and "Firewall Client for ISA Server" appear.

[dstufft]

pip doesn't currently support NTLM authentication (and in the development version we've switched to requests).

[jonsedar]

Currently experiencing the same issues:

• Is the simplest solution to use the dev version of pip?
• Should I also need to use the Firewall Client for ISA Server?

Cheers, Jon

[hickford]

@jonsedar Alas, there isn't any fix. This is what you can do:

1. Petition your office to stop using an NTLM proxy because it excludes free software.
2. Install the proxy proxy cntlm on your computer to "help you break free from the chains of Microsoft proprietary world"

---

To explain, this is what would need to happen before the issue can be fixed in Pip. Each step depends on those after it.

1. NTLM proxy support added to requests (understand that the requests-ntlm plugin does web authentication, not proxy authentication)
2. NTLM proxy support added to urllib3
3. NTLM proxy support added to python-ntlm
4. The stagnant python-ntlm project moved from Google Code to Github
   so it can receive pull requests and attract new active maintainer(s)

This is the crux. python-ntlm is a great library (one of few open-source ntlm implementations) but hasn't received much attention in recent years. Presumably the original developers are busy with other projects / hobbies / family -- completely understandable. However, as the world around it changes, the project has unfortunately suffered software rot. (For example, the last package published on pypi was in 2011 and isn't Python 3 compatible.)

GitHub is great for reviving projects--if an owner becomes busy to write new features, they can still easily merge pull requests from the public. And a young keen contributor can win the trust of the owner to be made an official maintainer.

We should approach the developers courteously asking them to consider moving the project to GitHub.

[pfmoore]

@jonsedar the ISA firewall client fixes this problem. I use it at work with the release version of pip (and a lot of other software) and have had no problem. So if you have the option to use that, go for it.

[jonsedar]

Thanks guys,

@pfmoore I'll try the ISA Firewall Client and hope that it does the trick. I couldn't get cntlm to work for me despite a few different configs - I think the ISA server wanted additional info in the authentication, it's a very tightly controlled network.

@matt-hickford Thats a shame about the library not being maintained, I don't really have the skills to do it myself but would happily join the courteous petition. Would Continuum Analytics be interested in writing it into conda I wonder?

[espdev]

@jonsedar Did you read my comment about ISA Firewall client and urllib/urllib2? A problem exists in the urllib and urllib2, I think.

For example, this code work correctly:

proxy = urllib2.ProxyHandler({})
opener = urllib2.build_opener(proxy)
urllib2.install_opener(opener)

u = urllib2.urlopen(url)
data = u.read()

[jonsedar]

Thanks @espdev, a combination of MS Firewall Client for ISA Server 2004 (FCIS) (this on a Windows 7 machine) and your suggestion seems to work:

1. FCIS is disabled, 'normal' default system setup. Attempt to reach Anaconda Continuum repo 'http://repo.continuum.io/pkgs/free/win-64/' via, Chrome browser, works correctly.

2. Enable FCIS. No additional changes. Attempt to reach Anaconda Continuum repo 'http://repo.continuum.io/pkgs/free/win-64/' via, Chrome browser, works correctly.

3. Boot up an ipython notebook and run:

   import urllib2
   url = 'http://repo.continuum.io/pkgs/free/win-64/'
   ## Named proxy
   proxy = urllib2.ProxyHandler({'http': 'http://user:password@proxy:port'})
   opener = urllib2.build_opener(proxy)
   urllib2.install_opener(opener)
   u = urllib2.urlopen(url)
   print(u.read())
   
   HTTPError: HTTP Error 407: Proxy Authentication Required

4. Now run the same code without proxy details:

   ## Unknown proxy
   proxy = urllib2.ProxyHandler({})
   opener = urllib2.build_opener(proxy)
   urllib2.install_opener(opener)
   u = urllib2.urlopen(url)
   print(u.read())
   
   ##Success! The HTML payload is returned.

Note: That if FCIS is not enabled in step 2, then step 4 does not work

URLError: <urlopen error [Errno 10060] A connection attempt failed

EDIT:

Since I actually want to use conda for package maintenance, I went on to mod the file C:\Anaconda\Lib\site-packages\conda\connection.py: line 59:

    def installopener():
        opener = urllib2.build_opener(
            urllib2.ProxyHandler({}), # <- HACK: replaced proxies_dict with empty dict
            ...

Hope someone else in this situation finds this thread helpful!

[pfmoore]

Apologies for not being clearer - yes, ISA firewall client works when you don't specify any proxy details (i.e., it's transparent to the user). Glad you worked it out and got it working.

[asmeurer]

No need to hack conda. That dictionary comes from the proxy_servers setting of the .condarc (or urllib2.getproxies() if that is not set). You can have the same effect by adding

- proxy_servers: {}

to your ~/.condarc.

[espdev]

@pfmoore, how is it possible to set unknown proxy for pip? I think it would solve problem for the majority of people who use ISA firewall client.

[pfmoore]

@espdev I don't understand. In my experience, you set up ISA firewall client, and don't set up any proxy for pip. You should find that pip works as if there was no firewall and no proxy. If that doesn't work, then all I can say is that your setup differs from mine.

If you set proxy details up when the ISA firewall client is active (via HTTP_PROXY or proxy command line arguments) then it won't work (again, to be precise it doesn't work in my configuration).

[espdev]

@pfmoore, I use activated ISA firewall client. I do not set any HTTP_PROXY settings for pip. But in the my corporate network pip does not work.

For example, I run pip:

pip install <package_name>

And I am getting error:

Cannot fetch index base URL https://pypi.python.org/simple/

And in pip.log:

Could not fetch URL https://pypi.python.org/simple/<package_name>/: <urlopen error Tunnel connection failed: 407 Proxy Authentication Required ( Forefront TMG requires authorization to fulfill the request. Access to the Web Proxy filter is denied.  )>

But, this code works (I am getting the data from the Internet url):

proxy = urllib2.ProxyHandler({})
opener = urllib2.build_opener(proxy)
urllib2.install_opener(opener)

url = 'https://pypi.python.org/simple/'
u = urllib2.urlopen(url)
data = u.read()

And I do not understand why pip does not work.

[pfmoore]

Hmm you may need to get approval/access to the firewall client. All I can say is that it works for me. Do tools like wget or git work with no proxy config? If so, I don't see why Python (pip) wouldn't.

Sorry I can't offer any more help.

[espdev]

A problem exists in pip vendor "requests" in the module sessions (\site-packages\pip\_vendor\requests\sessions.py in the line range 356-358 (https://github.com/pypa/pip/blob/develop/pip/_vendor/requests/sessions.py#L356). In my corporate network function get_environ_proxies returns a dict of proxies, and it does not work. I need an empty proxy dict {} for the correct work with ISA. I think it's a bug.

PS
I have commented these lines (356-358) and it has solved the problem for me. But this way is not the best one.

[dbstraight]

@espdev I had the same problem, could make a request using urllib but pip didn't work. I modified the getproxies function in urllib to always return an empty dictionary and that fixed it. Maybe it'll work for you. (I was on Python 3, that's why I'm not posting exactly what I changed, because it won't be the same.)

[karsibali]

@dbstraight Can you please post how you modified it. I'm kind of newbie and cannot solve the issue. Thanks...

[espdev]

@karsibali You can set the NO_PROXY environment variable and use MS ISA firewall client. See also this comment:
https://github.com/kennethreitz/requests/issues/2036#issuecomment-42526147

It works for me.

[karsibali]

@espdev I cannot use ISA firewall client since I'm using Linux. I'm using cntlm here. cntlm works fine for many software except some, like pip, easy_install, composer... I could not find solution to this problem.

[DavidJFelix]

+1. This has caused major headaches. Our solution was to take a box out of network, use basket and scp the basket directory onto the server.

[karsibali]

@DavidJFelix Does basket itself work behind corporate proxy?

[pradyunsg]

The relevant upstream issue is urllib3/urllib3#242?

[jaxspades]

I keep running into this too, is there any traction on this issue?

[myidealab]

Finally!! I've been searching all over for a solution. I actually read this thread multiple times.

I decided to try the solution from @voltagex after a colleague also brought up Fiddler. Thank you so much! Sometimes, you just don't know you are passing over the correct solution until you try it.

[KindDragon]

Solution with cntlm works too

[iyanmv]

As many people here I face this problem every day at work. I use cntlm without problems with apt, wget, curl, etc. but it does not work when I try to connect with pip/conda. My solution for the moment is creating a local repo in one of the nodes with specific packages using rsync and force the rest of the nodes to use that repo.

Anyway, I'd love to see pip/conda running directly with CNTLM. I even spoke with some IT colleagues in the company and the assure me than certain packages the proxy blocks (coming from pip) do not correctly authenticate using NTLMv2 (although all traffic is redirected thorugh CNTLM).

Can you tell paste here your cntlm.conf and pip.conf? @KindDragon

[iyanmv]

By the way, I just noticed that both pip and conda work for me in Windows using CNTLM but I still have the same problem in GNU/Linux. Do you know if there are any differences between how pip/conda get packages in GNU/Linux and Windows?

[pfmoore]

For what it's worth, my cntlm.ini is as follows (all security-sensitive data blanked out):

Username        XXXXXX
Domain          XXXXX
Auth            NTLMv2
PassLM          XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
PassNT          XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
PassNTLMv2      XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
Proxy           xxx.xxx.xxx.xxx:nn
NoProxy         localhost, 127.0.0.*, 10.*, 192.168.*
Listen          3128
Gateway yes

As you can see, basically everything is default (the "Gateway yes" setting is something I added recently when experimenting with docker - pip works fine whether it's there or not).

I have environment variables:

http_proxy                     http://localhost:3128
https_proxy                    http://localhost:3128

I also have

[global]
proxy = localhost:3128

in my pip.ini, but I believe the environment variables take precedence. If I remove the environment variables, though, pip still works (presumably because of the pip.ini setting).

I use this daily, and have for a number of years. Whatever the issues people are having on this thread, I can confirm they are not affecting me. I'm no longer using the ISA firewall client I was using back in 2014 when I last commented on this thread - I've been using cntlm for some time now.

I'm not trying to claim that the people commenting here are not experiencing this issue, but if there's any hope of confirming that there's a problem in pip and getting it fixed, we're going to need someone to provide a reproducible test case that will allow us to see the problem first hand. And yes, that's hard as it needs an NTLM proxy set up somewhere accessible on the internet - maybe someone can build an environment somewhere like Appveyor? I don't know. Or if someone experiencing the issue can dive into the pip code and find a solution they can submit as a PR, we can test it doesn't break other currently-working setups, and assuming it's good, merge it.

Otherwise we're going to have to continue to assume it's an issue somewhere in the way cntlm or the NTLM proxy it's interfacing to, are configured.

@iyanmv Just saw your last comment - that's very interesting. I can't help in terms of what differences there might be between your two environments except to say "pip has basically no differences in how it works on Windows and Linux", but it does clearly indicate that there's something environmental involved in this issue... (Could it be something as simple as differing CNTLM versions? I'm using version 0.92.2).

[iyanmv]

Just saw your last comment - that's very interesting. I can't help in terms of what differences there might be between your two environments except to say "pip has basically no differences in how it works on Windows and Linux", but it does clearly indicate that there's something environmental involved in this issue... (Could it be something as simple as differing CNTLM versions? I'm using version 0.92.2).

Yeah, I also have the Gateway option set to yes in the GNU/Linux servers (for docker and other stuff) and I have additional Listen entries, but I tried to change it to have it exactly as in the Windows machine and nothing, still not working. I am using CNTLM 0.92.3 in both systems by the way.

Even if I have a "solution" to avoid this issue I'd love to understand why is not working. I'll write if I figure something out...

[JaimieMurdock]

Just wanted to note that I've noticed this issue (or a similar one) twice with the eduroam university network on Macs.

[iyanmv]

@JaimieMurdock I don't think it's the same issue. Eduroam is just a network of hotspots and it uses WPA2 Enterprise to authenticate users with your university credentials. If you have a similar problem could be because your university uses internally (but it has nothing to do with eduroam) a proxy with NTLMv2. Are you using cntlm?

[TCT9]

Thank you very much "voltagex" (commented on 22 May 2015) your solution with "Fiddler" (free) was perfect !!! Now I can use PIP on a corporate proxy !!!

[flamelizard]

Superb advice. It really works using Fiddler to authenticate against proxy requiring NTLM auth.

[jaxspades]

You can also use Squid if you want, that worked for me. Here's an example working for emacs, but it also works for pip:

https://stackoverflow.com/questions/10787087/use-elpa-emacs-behind-a-proxy-requiring-authentication

[iyanmv]

For those struggling with the same issue in Linux servers in Windows networks (Internet through a NTLMv2 authentication proxy) I recommend you to try Nexus Repository Manager. It is possible to configure an http or https proxy with NTLM authentication and manage proxy repositories that point to PyPI. For me it was the only solution after several tries...

In Windows, CNTLM local proxy works just fine with both pip and conda.

[chinsoon12]

In case if CNTLM and Fiddler are blocked as well, the workaround proposed by YuMan-Tam in Experimentation connection works well (i.e. amending the _tunnel function in Anaconda\Lib\http\client.py)

In short, the first http request returns a 40* error code and urllib3 thinks that its unsuccessful and throws an exception. NTLM requires you to send the request again after auth. You can learn about this via curl.

This works only if files.pythonhosted.org is not blocked in the first place (i.e. you can open https://files.pythonhosted.org in your browser)

After client.py is amended, you can then pip install -v --proxy http://proxyserver:proxyport --trusted-host pypi.org --trusted-host files.pythonhosted.org packagename

[pfmoore]

To bring this discussion to a close - contrary to my previous comment (from 7 years ago!) I have changed my mind, and I no longer think pip should add specific support NTLM proxies. The ideal solution for handling such proxies would be if core Python's network API wrapped the platform native APIs directly. This would mean that all Python programs would work exactly the same as your browser and the rest of the OS. This has been discussed in the past, but it is a non-trivial piece of work and there isn't currently any plan for implementing it.

In the absence of that, a realistic alternative would be if core networking libraries like requests handled NTLM proxies transparently - again, this is so that all Python programs benefit. Currently, requests has support via extra plugins, but (as I understand it) that support requires client code changes, meaning that it isn't transparent. I don't know why the support isn't currently transparent (it may be that the NTLM protocol isn't amenable to that - that's something that would need to be looked at) but IMO making it so is definitely the best approach.

If either of the two solutions above were implemented, pip would gain NTLM support "for free".

In the absence of such a solution, workarounds exist such as cntlm and Fiddler. These seem to work for the majority of users, and to be honest, if they don't work, then you should probably be talking to your corporate security staff anyway, to confirm that what you are trying to do is acceptable to them. Having pip give you a way of avoiding that conversation doesn't seem reasonable to me.

If people need help in setting up cntlm or Fiddler, this tracker isn't really the right place to get that help - probably a forum like Stack Overflow is more reasonable. So in order to direct such questions correctly, I am locking this conversation.

Finally, as with anything, if corporate users really need custom pip support for NTLM proxies, they could consider funding such work. I'd still suggest that funding work to add transparent support to requests would be a better investment, as it would have far wider benefits than a pip-only change.

[pradyunsg]

Finally, as with anything, if corporate users really need custom pip support for NTLM proxies, they could consider funding such work. I'd still suggest that funding work to add transparent support to requests would be a better investment, as it would have far wider benefits than a pip-only change.

And, if you're wondering who to reach out to, I suggest reaching out to the PSF's Funding Working Group. From the announcement of the working group's creation:

initial resource we are able to provide is advice, so if you have any questions about funding, you can email us at project-funding-wg@python.org, and we will do our best to help.