functionapp: Proxy returns 407 when default credentials supplied

[AjkayAlan]

Describe the bug
While behind a corporate proxy as defined in the HTTP_PROXY and HTTPS_PROXY environment variables as defined as "http://proxyhost:port", I am able to login using az login but as soon as I try to deploy a functionapp using something like az functionapp deployment source config-zip -g resourceGroup -n \functionAppName --src zipfile.zip I receive a 407 proxy authentication required stacktrace. The full stacktrace is as follows:

HTTPSConnectionPool(host='hostobfiscatedhereonpurpose', port=443): Max retries exceeded with url: /api/zipdeploy?isAsync=true (Caused by ProxyError('Cannot connect to proxy.', OSError('Tunnel connection failed: 407 authenticationrequired',)))
Traceback (most recent call last):
  File "C:\Users\VSSADM~1\AppData\Local\Temp\pip-install-e5ce5s7r\urllib3\urllib3\connectionpool.py", line 594, in urlopen
  File "C:\Users\VSSADM~1\AppData\Local\Temp\pip-install-e5ce5s7r\urllib3\urllib3\connectionpool.py", line 805, in _prepare_proxy
  File "C:\Users\VSSADM~1\AppData\Local\Temp\pip-install-e5ce5s7r\urllib3\urllib3\connection.py", line 308, in connect
  File "C:\Program Files (x86)\Microsoft SDKs\Azure\CLI2\lib\http\client.py", line 919, in _tunnel
    message.strip()))
OSError: Tunnel connection failed: 407 authenticationrequired

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\VSSADM~1\AppData\Local\Temp\pip-install-e5ce5s7r\requests\requests\adapters.py", line 449, in send
  File "C:\Users\VSSADM~1\AppData\Local\Temp\pip-install-e5ce5s7r\urllib3\urllib3\connectionpool.py", line 638, in urlopen
  File "C:\Users\VSSADM~1\AppData\Local\Temp\pip-install-e5ce5s7r\urllib3\urllib3\util\retry.py", line 398, in increment
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='hostobfiscatedhereonpurpose', port=443): Max retries exceeded with url: /api/zipdeploy?isAsync=true (Caused by ProxyError('Cannot connect to proxy.', OSError('Tunnel connection failed: 407 authenticationrequired',)))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\VSSADM~1\AppData\Local\Temp\pip-install-e5ce5s7r\knack\knack\cli.py", line 206, in invoke
  File "C:\Users\VSSADM~1\AppData\Local\Temp\pip-install-e5ce5s7r\azure-cli-core\azure\cli\core\commands\__init__.py", line 346, in execute
  File "C:\Users\VSSADM~1\AppData\Local\Temp\pip-install-e5ce5s7r\six\six.py", line 693, in reraise
  File "C:\Users\VSSADM~1\AppData\Local\Temp\pip-install-e5ce5s7r\azure-cli-core\azure\cli\core\commands\__init__.py", line 320, in execute
  File "C:\Users\VSSADM~1\AppData\Local\Temp\pip-install-e5ce5s7r\azure-cli-core\azure\cli\core\commands\__init__.py", line 169, in __call__
  File "C:\Users\VSSADM~1\AppData\Local\Temp\pip-install-e5ce5s7r\azure-cli-core\azure\cli\core\__init__.py", line 440, in default_command_handler
  File "C:\Users\VSSADM~1\AppData\Local\Temp\pip-install-e5ce5s7r\azure-cli-appservice\azure\cli\command_modules\appservice\custom.py", line 262, in enable_zip_deploy
  File "C:\Users\VSSADM~1\AppData\Local\Temp\pip-install-e5ce5s7r\requests\requests\api.py", line 116, in post
  File "C:\Users\VSSADM~1\AppData\Local\Temp\pip-install-e5ce5s7r\requests\requests\api.py", line 60, in request
  File "C:\Users\VSSADM~1\AppData\Local\Temp\pip-install-e5ce5s7r\requests\requests\sessions.py", line 533, in request
  File "C:\Users\VSSADM~1\AppData\Local\Temp\pip-install-e5ce5s7r\requests\requests\sessions.py", line 646, in send
  File "C:\Users\VSSADM~1\AppData\Local\Temp\pip-install-e5ce5s7r\requests\requests\adapters.py", line 510, in send
requests.exceptions.ProxyError: HTTPSConnectionPool(host='hostobfiscatedhereonpurpose', port=443): Max retries exceeded with url: /api/zipdeploy?isAsync=true (Caused by ProxyError('Cannot connect to proxy.', OSError('Tunnel connection failed: 407 authenticationrequired',)))

However, if I set environment variables to "http://username:password@proxyhost:port" I am able to deploy without issue. While I agree that I have a workaround, I would prefer to not specify my username and password in my environment variables due to my organizations rotating password requirements.

To Reproduce

1. Install latest Azure CLI on Windows 10 (as of posting, 2.0.52)
2. Set HTTP_PROXY and HTTPS_PROXY environment variables to a proxy server which doesn't require username and password to be defined (an NTLM proxy?)
3. Login to the cli using az login
4. Attempt to deploy using az functionapp deployment source config-zip -g resourceGroup -n \functionAppName --src zipfile.zip

Expected behavior
az functionapp should honor authentication on the proxy the same way that the az login command works. A 407 proxy authentication required error should not be thrown

Environment summary
Install Method: MSI
CLI Version: 2.0.52
OS: Windows 10
Console: Powershell

[AjkayAlan]

Recommended labels: Functions-cli, Workaround
Recommended priority: Minor (since there is a workaround).

[jonlorusso]

This workaround does not work for me. With proxy environment variables defined, attempting to login az login I get:

Please ensure you have network connection. Error detail: HTTPSConnectionPool(host='login.microsoftonline.com', port=443): Max retries exceeded with url: /common/oauth2/token (Caused by ProxyError('Cannot connect to proxy.', OSError('Tunnel connection failed: 407 authenticationrequired',)))

[ghost]

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @ ahmedelnably, @fabiocav

[jiasli]

@jonlorusso, using HTTP_PROXY and HTTPS_PROXY requires the proxy to work without authentication or with basic authentication. They don't work with NTLM authentication.

[jiasli]

Due to the poor support of NTLM in Python community, this 407 issue has been mentioned in all 3 libraries which Azure CLI relies on: pypa/pip#1182, psf/requests#2036, urllib3/urllib3#242

Adding requests-ntlm in msrest is not practical at the moment. cc @lmazuel

There seems to be these solutions (I don’t have an NTLM environment so I can’t do the test):

1. cntlm, mentioned by Add support for NTLM proxies pypa/pip#1182 (comment), but this project has been out of maintenance for years
2. ISA firewall client, mentioned by Add support for NTLM proxies pypa/pip#1182 (comment), but the download link is down and it doesn't seem to support Windows 10
3. Fiddler, mentioned by Add support for NTLM proxies pypa/pip#1182 (comment), but Python doesn’t trust Fiddler’s root certificate which requires extra steps to configure root CA on Azure CLI for Fiddler's root certificate
4. Use Cloud Shell on Azure Portal (recommended)
5. Use an Azure VM as a jump server and install Azure CLI on that machine (recommended)

Please let us know if you can get unblocked.