SSL: CERTIFICATE_VERIFY_FAILED error on "az bicep install" Resurfaced #20842

Zerthick · 2021-12-27T20:43:25Z

Describe the bug
This is a reopening of #19420 the az bicep install command is once again throwing an SSL: CERTIFICATE_VERIFY_FAILED error

To Reproduce

$ az bicep install
Error while attempting to retrieve the latest Bicep version: HTTPSConnectionPool(host='aka.ms', port=443): Max retries exceeded with url: /BicepLatestRelease (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:852)'),)).

With debug:

$ az bicep install --debug
cli.knack.cli: Command arguments: ['bicep', 'install', '--debug']
cli.knack.cli: __init__ debug log:
Enable color in terminal.
cli.knack.cli: Event: Cli.PreExecute []
cli.knack.cli: Event: CommandParser.OnGlobalArgumentsCreate [<function CLILogging.on_global_arguments at 0x7f603acf9b70>, <function OutputProducer.on_global_arguments at 0x7f603ac2b730>, <function CLIQuery.on_global_arguments at 0x7f603ac4b840>]
cli.knack.cli: Event: CommandInvoker.OnPreCommandTableCreate []
cli.azure.cli.core: Modules found from index for 'bicep': ['azure.cli.command_modules.resource']
cli.azure.cli.core: Loading command modules:
cli.azure.cli.core: Name                  Load Time    Groups  Commands
cli.azure.cli.core: resource                  0.010        40       187
cli.azure.cli.core: Total (1)                 0.010        40       187
cli.azure.cli.core: These extensions are not installed and will be skipped: ['azext_ai_examples', 'azext_next']
cli.azure.cli.core: Loading extensions:
cli.azure.cli.core: Name                  Load Time    Groups  Commands  Directory
cli.azure.cli.core: Total (0)                 0.000         0         0
cli.azure.cli.core: Loaded 40 groups, 187 commands.
cli.azure.cli.core: Found a match in the command table.
cli.azure.cli.core: Raw command  : bicep install
cli.azure.cli.core: Command table: bicep install
cli.knack.cli: Event: CommandInvoker.OnPreCommandTableTruncate [<function AzCliLogging.init_command_file_logging at 0x7f603a76c598>]
cli.knack.cli: Event: CommandInvoker.OnPreArgumentLoad [<function register_global_subscription_argument.<locals>.add_subscription_parameter at 0x7f603a725620>, <function register_global_query_examples_argument.<locals>.register_query_examples at 0x7f603a6af840>]
cli.knack.cli: Event: CommandInvoker.OnPostArgumentLoad []
cli.knack.cli: Event: CommandInvoker.OnPostCommandTableCreate [<function register_ids_argument.<locals>.add_ids_arguments at 0x7f603a6af8c8>, <function register_cache_arguments.<locals>.add_cache_arguments at 0x7f603a6af9d8>]
cli.knack.cli: Event: CommandInvoker.OnCommandTableLoaded []
cli.knack.cli: Event: CommandInvoker.OnPreParseArgs []
cli.knack.cli: Event: CommandInvoker.OnPostParseArgs [<function OutputProducer.handle_output_argument at 0x7f603ac2b7b8>, <function CLIQuery.handle_query_parameter at 0x7f603ac4b8c8>, <function register_global_query_examples_argument.<locals>.handle_example_parameter at 0x7f603a6af7b8>, <function register_ids_argument.<locals>.parse_ids_arguments at 0x7f603a6af950>]
urllib3.connectionpool: Starting new HTTPS connection (1): aka.ms:443
cli.azure.cli.core.util: azure.cli.core.util.handle_exception is called with an exception:
cli.azure.cli.core.util: Traceback (most recent call last):
  File "/opt/az/lib/python3.6/site-packages/urllib3/connectionpool.py", line 706, in urlopen
    chunked=chunked,
  File "/opt/az/lib/python3.6/site-packages/urllib3/connectionpool.py", line 382, in _make_request
    self._validate_conn(conn)
  File "/opt/az/lib/python3.6/site-packages/urllib3/connectionpool.py", line 1010, in _validate_conn
    conn.connect()
  File "/opt/az/lib/python3.6/site-packages/urllib3/connection.py", line 426, in connect
    tls_in_tls=tls_in_tls,
  File "/opt/az/lib/python3.6/site-packages/urllib3/util/ssl_.py", line 450, in ssl_wrap_socket
    sock, context, tls_in_tls, server_hostname=server_hostname
  File "/opt/az/lib/python3.6/site-packages/urllib3/util/ssl_.py", line 493, in _ssl_wrap_socket_impl
    return ssl_context.wrap_socket(sock, server_hostname=server_hostname)
  File "/opt/az/lib/python3.6/ssl.py", line 407, in wrap_socket
    _context=self, _session=session)
  File "/opt/az/lib/python3.6/ssl.py", line 817, in __init__
    self.do_handshake()
  File "/opt/az/lib/python3.6/ssl.py", line 1077, in do_handshake
    self._sslobj.do_handshake()
  File "/opt/az/lib/python3.6/ssl.py", line 689, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:852)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/az/lib/python3.6/site-packages/requests/adapters.py", line 449, in send
    timeout=timeout
  File "/opt/az/lib/python3.6/site-packages/urllib3/connectionpool.py", line 756, in urlopen
    method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
  File "/opt/az/lib/python3.6/site-packages/urllib3/util/retry.py", line 574, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='aka.ms', port=443): Max retries exceeded with url: /BicepLatestRelease (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:852)'),))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/az/lib/python3.6/site-packages/azure/cli/command_modules/resource/_bicep.py", line 156, in get_bicep_latest_release_tag
    response = requests.get("https://aka.ms/BicepLatestRelease", verify=ca_file)
  File "/opt/az/lib/python3.6/site-packages/requests/api.py", line 75, in get
    return request('get', url, params=params, **kwargs)
  File "/opt/az/lib/python3.6/site-packages/requests/api.py", line 61, in request
    return session.request(method=method, url=url, **kwargs)
  File "/opt/az/lib/python3.6/site-packages/requests/sessions.py", line 542, in request
    resp = self.send(prep, **send_kwargs)
  File "/opt/az/lib/python3.6/site-packages/requests/sessions.py", line 655, in send
    r = adapter.send(request, **kwargs)
  File "/opt/az/lib/python3.6/site-packages/requests/adapters.py", line 514, in send
    raise SSLError(e, request=request)
requests.exceptions.SSLError: HTTPSConnectionPool(host='aka.ms', port=443): Max retries exceeded with url: /BicepLatestRelease (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:852)'),))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/az/lib/python3.6/site-packages/knack/cli.py", line 231, in invoke
    cmd_result = self.invocation.execute(args)
  File "/opt/az/lib/python3.6/site-packages/azure/cli/core/commands/__init__.py", line 658, in execute
    raise ex
  File "/opt/az/lib/python3.6/site-packages/azure/cli/core/commands/__init__.py", line 721, in _run_jobs_serially
    results.append(self._run_job(expanded_arg, cmd_copy))
  File "/opt/az/lib/python3.6/site-packages/azure/cli/core/commands/__init__.py", line 692, in _run_job
    result = cmd_copy(params)
  File "/opt/az/lib/python3.6/site-packages/azure/cli/core/commands/__init__.py", line 328, in __call__
    return self.handler(*args, **kwargs)
  File "/opt/az/lib/python3.6/site-packages/azure/cli/core/commands/command_operation.py", line 121, in handler
    return op(**command_args)
  File "/opt/az/lib/python3.6/site-packages/azure/cli/command_modules/resource/custom.py", line 3588, in install_bicep_cli
    ensure_bicep_installation(release_tag=version, target_platform=target_platform)
  File "/opt/az/lib/python3.6/site-packages/azure/cli/command_modules/resource/_bicep.py", line 106, in ensure_bicep_installation
    release_tag = release_tag if release_tag else get_bicep_latest_release_tag()
  File "/opt/az/lib/python3.6/site-packages/azure/cli/command_modules/resource/_bicep.py", line 160, in get_bicep_latest_release_tag
    raise ClientRequestError(f"Error while attempting to retrieve the latest Bicep version: {err}.")
azure.cli.core.azclierror.ClientRequestError: Error while attempting to retrieve the latest Bicep version: HTTPSConnectionPool(host='aka.ms', port=443): Max retries exceeded with url: /BicepLatestRelease (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:852)'),)).

cli.azure.cli.core.azclierror: Error while attempting to retrieve the latest Bicep version: HTTPSConnectionPool(host='aka.ms', port=443): Max retries exceeded with url: /BicepLatestRelease (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:852)'),)).
az_command_data_logger: Error while attempting to retrieve the latest Bicep version: HTTPSConnectionPool(host='aka.ms', port=443): Max retries exceeded with url: /BicepLatestRelease (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:852)'),)).
cli.knack.cli: Event: Cli.PostExecute [<function AzCliLogging.deinit_cmd_metadata_logging at 0x7f603a76c7b8>]
cli.__main__: Command ran in 0.447 seconds (init: 0.089, invoke: 0.358)

Expected behavior
Bicep successfully installs.

Environment summary

Linux-5.10.16.3-microsoft-standard-WSL2-x86_64-with-debian-bullseye-sid, Ubuntu 20.04.3 LTS
Python 3.6.10
Installer: DEB

azure-cli 2.31.0

Extensions:
ml 2.0.3

Additional context
Like the pervious ticket I am also behind a corporate proxy, other commands that require the proxy certificate like az login are working as expected. The REQUESTS_CA_BUNDLE environment variable has been set to point to a file including the proxy certificate. In addition, it doesn't not appear that bicep is obeying the AZURE_CLI_DISABLE_CONNECTION_VERIFICATION environment variable as running the following command export AZURE_CLI_DISABLE_CONNECTION_VERIFICATION=1 before attempting to do the install is having no effect.

The text was updated successfully, but these errors were encountered:

ghost · 2021-12-28T00:40:27Z

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @armleads-azure.

Issue Details

Describe the bug
This is a reopening of #19420 the az bicep install command is once again throwing an SSL: CERTIFICATE_VERIFY_FAILED error

To Reproduce

$ az bicep install
Error while attempting to retrieve the latest Bicep version: HTTPSConnectionPool(host='aka.ms', port=443): Max retries exceeded with url: /BicepLatestRelease (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:852)'),)).

With debug:

$ az bicep install --debug
cli.knack.cli: Command arguments: ['bicep', 'install', '--debug']
cli.knack.cli: __init__ debug log:
Enable color in terminal.
cli.knack.cli: Event: Cli.PreExecute []
cli.knack.cli: Event: CommandParser.OnGlobalArgumentsCreate [<function CLILogging.on_global_arguments at 0x7f603acf9b70>, <function OutputProducer.on_global_arguments at 0x7f603ac2b730>, <function CLIQuery.on_global_arguments at 0x7f603ac4b840>]
cli.knack.cli: Event: CommandInvoker.OnPreCommandTableCreate []
cli.azure.cli.core: Modules found from index for 'bicep': ['azure.cli.command_modules.resource']
cli.azure.cli.core: Loading command modules:
cli.azure.cli.core: Name                  Load Time    Groups  Commands
cli.azure.cli.core: resource                  0.010        40       187
cli.azure.cli.core: Total (1)                 0.010        40       187
cli.azure.cli.core: These extensions are not installed and will be skipped: ['azext_ai_examples', 'azext_next']
cli.azure.cli.core: Loading extensions:
cli.azure.cli.core: Name                  Load Time    Groups  Commands  Directory
cli.azure.cli.core: Total (0)                 0.000         0         0
cli.azure.cli.core: Loaded 40 groups, 187 commands.
cli.azure.cli.core: Found a match in the command table.
cli.azure.cli.core: Raw command  : bicep install
cli.azure.cli.core: Command table: bicep install
cli.knack.cli: Event: CommandInvoker.OnPreCommandTableTruncate [<function AzCliLogging.init_command_file_logging at 0x7f603a76c598>]
cli.knack.cli: Event: CommandInvoker.OnPreArgumentLoad [<function register_global_subscription_argument.<locals>.add_subscription_parameter at 0x7f603a725620>, <function register_global_query_examples_argument.<locals>.register_query_examples at 0x7f603a6af840>]
cli.knack.cli: Event: CommandInvoker.OnPostArgumentLoad []
cli.knack.cli: Event: CommandInvoker.OnPostCommandTableCreate [<function register_ids_argument.<locals>.add_ids_arguments at 0x7f603a6af8c8>, <function register_cache_arguments.<locals>.add_cache_arguments at 0x7f603a6af9d8>]
cli.knack.cli: Event: CommandInvoker.OnCommandTableLoaded []
cli.knack.cli: Event: CommandInvoker.OnPreParseArgs []
cli.knack.cli: Event: CommandInvoker.OnPostParseArgs [<function OutputProducer.handle_output_argument at 0x7f603ac2b7b8>, <function CLIQuery.handle_query_parameter at 0x7f603ac4b8c8>, <function register_global_query_examples_argument.<locals>.handle_example_parameter at 0x7f603a6af7b8>, <function register_ids_argument.<locals>.parse_ids_arguments at 0x7f603a6af950>]
urllib3.connectionpool: Starting new HTTPS connection (1): aka.ms:443
cli.azure.cli.core.util: azure.cli.core.util.handle_exception is called with an exception:
cli.azure.cli.core.util: Traceback (most recent call last):
  File "/opt/az/lib/python3.6/site-packages/urllib3/connectionpool.py", line 706, in urlopen
    chunked=chunked,
  File "/opt/az/lib/python3.6/site-packages/urllib3/connectionpool.py", line 382, in _make_request
    self._validate_conn(conn)
  File "/opt/az/lib/python3.6/site-packages/urllib3/connectionpool.py", line 1010, in _validate_conn
    conn.connect()
  File "/opt/az/lib/python3.6/site-packages/urllib3/connection.py", line 426, in connect
    tls_in_tls=tls_in_tls,
  File "/opt/az/lib/python3.6/site-packages/urllib3/util/ssl_.py", line 450, in ssl_wrap_socket
    sock, context, tls_in_tls, server_hostname=server_hostname
  File "/opt/az/lib/python3.6/site-packages/urllib3/util/ssl_.py", line 493, in _ssl_wrap_socket_impl
    return ssl_context.wrap_socket(sock, server_hostname=server_hostname)
  File "/opt/az/lib/python3.6/ssl.py", line 407, in wrap_socket
    _context=self, _session=session)
  File "/opt/az/lib/python3.6/ssl.py", line 817, in __init__
    self.do_handshake()
  File "/opt/az/lib/python3.6/ssl.py", line 1077, in do_handshake
    self._sslobj.do_handshake()
  File "/opt/az/lib/python3.6/ssl.py", line 689, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:852)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/az/lib/python3.6/site-packages/requests/adapters.py", line 449, in send
    timeout=timeout
  File "/opt/az/lib/python3.6/site-packages/urllib3/connectionpool.py", line 756, in urlopen
    method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
  File "/opt/az/lib/python3.6/site-packages/urllib3/util/retry.py", line 574, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='aka.ms', port=443): Max retries exceeded with url: /BicepLatestRelease (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:852)'),))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/az/lib/python3.6/site-packages/azure/cli/command_modules/resource/_bicep.py", line 156, in get_bicep_latest_release_tag
    response = requests.get("https://aka.ms/BicepLatestRelease", verify=ca_file)
  File "/opt/az/lib/python3.6/site-packages/requests/api.py", line 75, in get
    return request('get', url, params=params, **kwargs)
  File "/opt/az/lib/python3.6/site-packages/requests/api.py", line 61, in request
    return session.request(method=method, url=url, **kwargs)
  File "/opt/az/lib/python3.6/site-packages/requests/sessions.py", line 542, in request
    resp = self.send(prep, **send_kwargs)
  File "/opt/az/lib/python3.6/site-packages/requests/sessions.py", line 655, in send
    r = adapter.send(request, **kwargs)
  File "/opt/az/lib/python3.6/site-packages/requests/adapters.py", line 514, in send
    raise SSLError(e, request=request)
requests.exceptions.SSLError: HTTPSConnectionPool(host='aka.ms', port=443): Max retries exceeded with url: /BicepLatestRelease (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:852)'),))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/az/lib/python3.6/site-packages/knack/cli.py", line 231, in invoke
    cmd_result = self.invocation.execute(args)
  File "/opt/az/lib/python3.6/site-packages/azure/cli/core/commands/__init__.py", line 658, in execute
    raise ex
  File "/opt/az/lib/python3.6/site-packages/azure/cli/core/commands/__init__.py", line 721, in _run_jobs_serially
    results.append(self._run_job(expanded_arg, cmd_copy))
  File "/opt/az/lib/python3.6/site-packages/azure/cli/core/commands/__init__.py", line 692, in _run_job
    result = cmd_copy(params)
  File "/opt/az/lib/python3.6/site-packages/azure/cli/core/commands/__init__.py", line 328, in __call__
    return self.handler(*args, **kwargs)
  File "/opt/az/lib/python3.6/site-packages/azure/cli/core/commands/command_operation.py", line 121, in handler
    return op(**command_args)
  File "/opt/az/lib/python3.6/site-packages/azure/cli/command_modules/resource/custom.py", line 3588, in install_bicep_cli
    ensure_bicep_installation(release_tag=version, target_platform=target_platform)
  File "/opt/az/lib/python3.6/site-packages/azure/cli/command_modules/resource/_bicep.py", line 106, in ensure_bicep_installation
    release_tag = release_tag if release_tag else get_bicep_latest_release_tag()
  File "/opt/az/lib/python3.6/site-packages/azure/cli/command_modules/resource/_bicep.py", line 160, in get_bicep_latest_release_tag
    raise ClientRequestError(f"Error while attempting to retrieve the latest Bicep version: {err}.")
azure.cli.core.azclierror.ClientRequestError: Error while attempting to retrieve the latest Bicep version: HTTPSConnectionPool(host='aka.ms', port=443): Max retries exceeded with url: /BicepLatestRelease (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:852)'),)).

cli.azure.cli.core.azclierror: Error while attempting to retrieve the latest Bicep version: HTTPSConnectionPool(host='aka.ms', port=443): Max retries exceeded with url: /BicepLatestRelease (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:852)'),)).
az_command_data_logger: Error while attempting to retrieve the latest Bicep version: HTTPSConnectionPool(host='aka.ms', port=443): Max retries exceeded with url: /BicepLatestRelease (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:852)'),)).
cli.knack.cli: Event: Cli.PostExecute [<function AzCliLogging.deinit_cmd_metadata_logging at 0x7f603a76c7b8>]
cli.__main__: Command ran in 0.447 seconds (init: 0.089, invoke: 0.358)

Expected behavior
Bicep successfully installs.

Environment summary

Linux-5.10.16.3-microsoft-standard-WSL2-x86_64-with-debian-bullseye-sid, Ubuntu 20.04.3 LTS
Python 3.6.10
Installer: DEB

azure-cli 2.31.0

Extensions:
ml 2.0.3

Additional context
Like the pervious ticket I am also behind a corporate proxy, other commands that require the proxy certificate like az login are working as expected. The REQUESTS_CA_BUNDLE environment variable has been set to point to a file including the proxy certificate. In addition, it doesn't not appear that bicep is obeying the AZURE_CLI_DISABLE_CONNECTION_VERIFICATION environment variable as running the following command export AZURE_CLI_DISABLE_CONNECTION_VERIFICATION=1 before attempting to do the install is having no effect.

Author:	Zerthick
Assignees:	-
Labels:	`Service Attention`, `ARM`, `customer-reported`
Milestone:	-

yonzhan · 2021-12-28T00:40:33Z

route to service team

Zerthick · 2021-12-28T15:11:44Z

As a temporary workaround, you can install bicep manually outside the az cli following the instructions here: https://docs.microsoft.com/en-us/azure/azure-resource-manager/bicep/install#install-manually

This however does not allow you to deploy bicep from the az cli though

Jaffacakes82 · 2022-01-13T14:57:09Z

Also experiencing this in version 2.32.0 behind a corporate proxy.

jtrivett · 2022-01-20T19:03:42Z

I have also confirmed that this issue exists in version 2.32.0. I posted additional details in my comment on the previously-closed issue, #19420.

Unfortunately, this is a showstopper that prevents me from adopting bicep in a corporate setting at the moment, which is a shame - the DSL and tooling offered appear to be a great evolution & streamlining of the ARM template approach!

jtrivett · 2022-01-21T03:42:16Z

Upon inspecting the previous commit for #19420, the issue is that the bicep module is hardcoded to use the default cacert.pem certificate bundle (which is provided by the certifi python library dependency) when invoking HTTP requests via the requests python library. This approach ignores the REQUESTS_CA_BUNDLE environment variable that many of us are setting to utilize the az cli behind a corporate proxy. I believe these lines need to be adjusted to match how other az cli modules are setting the verify parameter for their calls to requests.get().

@Zerthick and @Jaffacakes82 - in the meantime, I found another workaround that allowed az bicep install to complete successfully for me on a Windows machine:

Append your corporate proxy certificate(s) to the end of the certifi library's cacert.pem file. You'll likely find the file at one of the following locations:

On Windows: C:\Program Files (x86)\Microsoft SDKs\Azure\CLI2\Lib\site-packages\certifi\cacert.pem
On Linux: /opt/az/lib/python3.6/site-packages/certifi/cacert.pem

steve-sellers · 2022-01-24T20:36:33Z

Upon inspecting the previous commit for #19420, the issue is that the bicep module is hardcoded to use the default cacert.pem certificate bundle (which is provided by the certifi python library dependency) when invoking HTTP requests via the requests python library. This approach ignores the REQUESTS_CA_BUNDLE environment variable that many of us are setting to utilize the az cli behind a corporate proxy. I believe these lines need to be adjusted to match how other az cli modules are setting the verify parameter for their calls to requests.get().

@Zerthick and @Jaffacakes82 - in the meantime, I found another workaround that allowed az bicep install to complete successfully for me on a Windows machine:

Append your corporate proxy certificate(s) to the end of the certify library's cacert.pem file. You'll likely find the file at one of the following locations:

On Windows: C:\Program Files (x86)\Microsoft SDKs\Azure\CLI2\Lib\site-packages\certifi\cacert.pem

On Linux: /opt/az/lib/python3.6/site-packages/certifi/cacert.pem

Thanks for this - worked for me. In case it helps others: I'm on a mac and found the correct cacert.pem location by running az --version which spit out

Python location '/Users/<username>/.brew/Cellar/azure-cli/2.32.0/libexec/bin/python'

and subsequently found the right cacert.pem file here: ~/.brew/Cellar/azure-cli/2.32.0/libexec/lib/python3.10/site-packages/certifi/cacert.pem. Appending my corporate certs onto it allowed me to successfully run az bicep install 👍

baileydoestech · 2022-02-11T09:36:35Z

These suggestions work for us and provide a temporary workaround, for anyone looking to run Bicep from a Docker container (such as VS devcontainers) we use the following solution (with the cert fix that @jtrivett mentions)

FROM ubuntu
RUN apt-get update && \
    apt-get install -y --no-install-recommends \
        gnupg && \
    apt-get clean && \
    rm -rf /var/lib/apt/lists/*

# Get Azure CLI
# https://docs.microsoft.com/en-us/cli/azure/install-azure-cli-apt?view=azure-cli-latest
RUN wget -q -O - https://packages.microsoft.com/keys/microsoft.asc | gpg --dearmor | tee /etc/apt/trusted.gpg.d/microsoft.asc.gpg > /dev/null && \
    echo "deb [arch=amd64] https://packages.microsoft.com/repos/azure-cli/ bionic main" | tee /etc/apt/sources.list.d/azure-cli.list && \
    apt-get update && \
    apt-get install -y --no-install-recommends azure-cli && \
    apt-get clean && \
    rm -rf /var/lib/apt/lists/*

# Azure Bicep
RUN curl -Lo bicep https://github.com/Azure/bicep/releases/latest/download/bicep-linux-x64 && \
    chmod +x ./bicep && \
    mv ./bicep /usr/local/bin/bicep && \
    bicep --help

# Hack to fix Bicep cert issue: https://github.com/Azure/azure-cli/issues/20842#issuecomment-1018139759
RUN cp /etc/ssl/certs/<CORPORATE CERT CHAIN>.pem /opt/az/lib/python3.6/site-packages/certifi/cacert.pem

navba-MSFT · 2022-03-03T10:52:45Z

@Zerthick Apologies for the late reply. Could you please let us know if you have been able to test using the above mentioned suggestions ? Please let us know if you are still running into the same issue. Awaiting your reply.

Zerthick · 2022-03-08T19:54:13Z

@navba-MSFT The suggestions above do indeed appear to work, though as others on this ticket have commented they are a workaround, bicep should be ideally obeying the REQUESTS_CA_BUNDLE environment variable to handle including corporate certificates and the AZURE_CLI_DISABLE_CONNECTION_VERIFICATION environment variable to disable the ssl check completely.

Jaffacakes82 · 2022-03-12T09:51:46Z

Thanks @ssvalkyrie & @jtrivett, worked a treat. @navba-MSFT, agree with @Zerthick, this needs addressing by Microsoft.

jtrivett · 2022-03-14T15:04:12Z

@navba-MSFT thanks for checking in on this issue. Is this something that can be reviewed & prioritized by Microsoft for an upcoming AZ CLI release? Us corporate users would greatly appreciate it!

MiguelElGallo · 2022-03-25T07:10:20Z

@navba-MSFT Yeah, let us know what is the plan. As mentioned before there is a lot of people behind a corporate firewall.

ghost · 2022-03-25T07:25:28Z

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @josephkwchan, @jennyhunter-msft.

Issue Details

Describe the bug
This is a reopening of #19420 the az bicep install command is once again throwing an SSL: CERTIFICATE_VERIFY_FAILED error

To Reproduce

$ az bicep install
Error while attempting to retrieve the latest Bicep version: HTTPSConnectionPool(host='aka.ms', port=443): Max retries exceeded with url: /BicepLatestRelease (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:852)'),)).

With debug:

$ az bicep install --debug
cli.knack.cli: Command arguments: ['bicep', 'install', '--debug']
cli.knack.cli: __init__ debug log:
Enable color in terminal.
cli.knack.cli: Event: Cli.PreExecute []
cli.knack.cli: Event: CommandParser.OnGlobalArgumentsCreate [<function CLILogging.on_global_arguments at 0x7f603acf9b70>, <function OutputProducer.on_global_arguments at 0x7f603ac2b730>, <function CLIQuery.on_global_arguments at 0x7f603ac4b840>]
cli.knack.cli: Event: CommandInvoker.OnPreCommandTableCreate []
cli.azure.cli.core: Modules found from index for 'bicep': ['azure.cli.command_modules.resource']
cli.azure.cli.core: Loading command modules:
cli.azure.cli.core: Name                  Load Time    Groups  Commands
cli.azure.cli.core: resource                  0.010        40       187
cli.azure.cli.core: Total (1)                 0.010        40       187
cli.azure.cli.core: These extensions are not installed and will be skipped: ['azext_ai_examples', 'azext_next']
cli.azure.cli.core: Loading extensions:
cli.azure.cli.core: Name                  Load Time    Groups  Commands  Directory
cli.azure.cli.core: Total (0)                 0.000         0         0
cli.azure.cli.core: Loaded 40 groups, 187 commands.
cli.azure.cli.core: Found a match in the command table.
cli.azure.cli.core: Raw command  : bicep install
cli.azure.cli.core: Command table: bicep install
cli.knack.cli: Event: CommandInvoker.OnPreCommandTableTruncate [<function AzCliLogging.init_command_file_logging at 0x7f603a76c598>]
cli.knack.cli: Event: CommandInvoker.OnPreArgumentLoad [<function register_global_subscription_argument.<locals>.add_subscription_parameter at 0x7f603a725620>, <function register_global_query_examples_argument.<locals>.register_query_examples at 0x7f603a6af840>]
cli.knack.cli: Event: CommandInvoker.OnPostArgumentLoad []
cli.knack.cli: Event: CommandInvoker.OnPostCommandTableCreate [<function register_ids_argument.<locals>.add_ids_arguments at 0x7f603a6af8c8>, <function register_cache_arguments.<locals>.add_cache_arguments at 0x7f603a6af9d8>]
cli.knack.cli: Event: CommandInvoker.OnCommandTableLoaded []
cli.knack.cli: Event: CommandInvoker.OnPreParseArgs []
cli.knack.cli: Event: CommandInvoker.OnPostParseArgs [<function OutputProducer.handle_output_argument at 0x7f603ac2b7b8>, <function CLIQuery.handle_query_parameter at 0x7f603ac4b8c8>, <function register_global_query_examples_argument.<locals>.handle_example_parameter at 0x7f603a6af7b8>, <function register_ids_argument.<locals>.parse_ids_arguments at 0x7f603a6af950>]
urllib3.connectionpool: Starting new HTTPS connection (1): aka.ms:443
cli.azure.cli.core.util: azure.cli.core.util.handle_exception is called with an exception:
cli.azure.cli.core.util: Traceback (most recent call last):
  File "/opt/az/lib/python3.6/site-packages/urllib3/connectionpool.py", line 706, in urlopen
    chunked=chunked,
  File "/opt/az/lib/python3.6/site-packages/urllib3/connectionpool.py", line 382, in _make_request
    self._validate_conn(conn)
  File "/opt/az/lib/python3.6/site-packages/urllib3/connectionpool.py", line 1010, in _validate_conn
    conn.connect()
  File "/opt/az/lib/python3.6/site-packages/urllib3/connection.py", line 426, in connect
    tls_in_tls=tls_in_tls,
  File "/opt/az/lib/python3.6/site-packages/urllib3/util/ssl_.py", line 450, in ssl_wrap_socket
    sock, context, tls_in_tls, server_hostname=server_hostname
  File "/opt/az/lib/python3.6/site-packages/urllib3/util/ssl_.py", line 493, in _ssl_wrap_socket_impl
    return ssl_context.wrap_socket(sock, server_hostname=server_hostname)
  File "/opt/az/lib/python3.6/ssl.py", line 407, in wrap_socket
    _context=self, _session=session)
  File "/opt/az/lib/python3.6/ssl.py", line 817, in __init__
    self.do_handshake()
  File "/opt/az/lib/python3.6/ssl.py", line 1077, in do_handshake
    self._sslobj.do_handshake()
  File "/opt/az/lib/python3.6/ssl.py", line 689, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:852)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/az/lib/python3.6/site-packages/requests/adapters.py", line 449, in send
    timeout=timeout
  File "/opt/az/lib/python3.6/site-packages/urllib3/connectionpool.py", line 756, in urlopen
    method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
  File "/opt/az/lib/python3.6/site-packages/urllib3/util/retry.py", line 574, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='aka.ms', port=443): Max retries exceeded with url: /BicepLatestRelease (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:852)'),))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/az/lib/python3.6/site-packages/azure/cli/command_modules/resource/_bicep.py", line 156, in get_bicep_latest_release_tag
    response = requests.get("https://aka.ms/BicepLatestRelease", verify=ca_file)
  File "/opt/az/lib/python3.6/site-packages/requests/api.py", line 75, in get
    return request('get', url, params=params, **kwargs)
  File "/opt/az/lib/python3.6/site-packages/requests/api.py", line 61, in request
    return session.request(method=method, url=url, **kwargs)
  File "/opt/az/lib/python3.6/site-packages/requests/sessions.py", line 542, in request
    resp = self.send(prep, **send_kwargs)
  File "/opt/az/lib/python3.6/site-packages/requests/sessions.py", line 655, in send
    r = adapter.send(request, **kwargs)
  File "/opt/az/lib/python3.6/site-packages/requests/adapters.py", line 514, in send
    raise SSLError(e, request=request)
requests.exceptions.SSLError: HTTPSConnectionPool(host='aka.ms', port=443): Max retries exceeded with url: /BicepLatestRelease (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:852)'),))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/az/lib/python3.6/site-packages/knack/cli.py", line 231, in invoke
    cmd_result = self.invocation.execute(args)
  File "/opt/az/lib/python3.6/site-packages/azure/cli/core/commands/__init__.py", line 658, in execute
    raise ex
  File "/opt/az/lib/python3.6/site-packages/azure/cli/core/commands/__init__.py", line 721, in _run_jobs_serially
    results.append(self._run_job(expanded_arg, cmd_copy))
  File "/opt/az/lib/python3.6/site-packages/azure/cli/core/commands/__init__.py", line 692, in _run_job
    result = cmd_copy(params)
  File "/opt/az/lib/python3.6/site-packages/azure/cli/core/commands/__init__.py", line 328, in __call__
    return self.handler(*args, **kwargs)
  File "/opt/az/lib/python3.6/site-packages/azure/cli/core/commands/command_operation.py", line 121, in handler
    return op(**command_args)
  File "/opt/az/lib/python3.6/site-packages/azure/cli/command_modules/resource/custom.py", line 3588, in install_bicep_cli
    ensure_bicep_installation(release_tag=version, target_platform=target_platform)
  File "/opt/az/lib/python3.6/site-packages/azure/cli/command_modules/resource/_bicep.py", line 106, in ensure_bicep_installation
    release_tag = release_tag if release_tag else get_bicep_latest_release_tag()
  File "/opt/az/lib/python3.6/site-packages/azure/cli/command_modules/resource/_bicep.py", line 160, in get_bicep_latest_release_tag
    raise ClientRequestError(f"Error while attempting to retrieve the latest Bicep version: {err}.")
azure.cli.core.azclierror.ClientRequestError: Error while attempting to retrieve the latest Bicep version: HTTPSConnectionPool(host='aka.ms', port=443): Max retries exceeded with url: /BicepLatestRelease (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:852)'),)).

cli.azure.cli.core.azclierror: Error while attempting to retrieve the latest Bicep version: HTTPSConnectionPool(host='aka.ms', port=443): Max retries exceeded with url: /BicepLatestRelease (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:852)'),)).
az_command_data_logger: Error while attempting to retrieve the latest Bicep version: HTTPSConnectionPool(host='aka.ms', port=443): Max retries exceeded with url: /BicepLatestRelease (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:852)'),)).
cli.knack.cli: Event: Cli.PostExecute [<function AzCliLogging.deinit_cmd_metadata_logging at 0x7f603a76c7b8>]
cli.__main__: Command ran in 0.447 seconds (init: 0.089, invoke: 0.358)

Expected behavior
Bicep successfully installs.

Environment summary

Linux-5.10.16.3-microsoft-standard-WSL2-x86_64-with-debian-bullseye-sid, Ubuntu 20.04.3 LTS
Python 3.6.10
Installer: DEB

azure-cli 2.31.0

Extensions:
ml 2.0.3

Additional context
Like the pervious ticket I am also behind a corporate proxy, other commands that require the proxy certificate like az login are working as expected. The REQUESTS_CA_BUNDLE environment variable has been set to point to a file including the proxy certificate. In addition, it doesn't not appear that bicep is obeying the AZURE_CLI_DISABLE_CONNECTION_VERIFICATION environment variable as running the following command export AZURE_CLI_DISABLE_CONNECTION_VERIFICATION=1 before attempting to do the install is having no effect.

Author:	Zerthick
Assignees:	-
Labels:	`Service Attention`, `ARM`, `customer-reported`, `needs-team-attention`
Milestone:	-

navba-MSFT · 2022-03-25T07:26:08Z

Adding the Service team to look into this issue and share the update on this.

@josephkwchan, @jennyhunter-msft could you please provide an update on this once you get a chance ? Awaiting your reply.

MiguelElGallo · 2022-03-25T11:39:45Z

By the way I'm facing this issue when running: az bicep build

…es for CA bundle

CiaraSHoulihan · 2022-04-06T12:27:03Z

By the way I'm facing this issue when running: az bicep build

I also had the same issue but I noticed there should be a fix (#20183) and my az cli was not up to date

Have you tried az upgrade before doing az bicep build again?

woeterman94 · 2022-09-07T12:40:16Z

By the way I'm facing this issue when running: az bicep build

I also had the same issue but I noticed there should be a fix (#20183) and my az cli was not up to date

Have you tried az upgrade before doing az bicep build again?

Having the same issue with the latest version of azure cli. So No upgrading does not solve the issue.

"azure-cli": "2.40.0",
  "azure-cli-core": "2.40.0",
  "azure-cli-telemetry": "1.0.8",

haciz · 2023-02-01T10:06:28Z

Experiencing exactly the same issue, @navba-MSFT What is the plan for solving this issue?

aslan-im · 2023-08-01T14:06:12Z

I do not use the proxy, but have the same issue when trying to run:
az bicep decompile --file main.json

OS: MacOS Ventura 13.5
azure-cli 2.51.0
core 2.51.0
telemetry 1.1.0
Extensions:
azure-devops 0.26.0
Dependencies:
msal 1.24.0b1
azure-mgmt-resource 23.1.0b2
Bicep CLI version 0.19.5 (87ca110fc0)

yonzhan added the ARM az resource/group/lock/tag/deployment/policy/managementapp/account management-group label Dec 28, 2021

ghost removed the needs-triage This is a new issue that needs to be triaged to the appropriate team. label Dec 28, 2021

yonzhan added Service Attention This issue is responsible by Azure service team. and removed question The issue doesn't require a change to the product in order to be resolved. Most issues start as that labels Dec 28, 2021

jtrivett mentioned this issue Feb 10, 2022

SSL: CERTIFICATE_VERIFY_FAILED error on "az bicep install" #19420

Closed

navba-MSFT added the needs-author-feedback More information is needed from author to address the issue. label Mar 3, 2022

ghost added needs-team-attention This issue needs attention from Azure service team or SDK team and removed needs-author-feedback More information is needed from author to address the issue. labels Mar 8, 2022

navba-MSFT removed the Service Attention This issue is responsible by Azure service team. label Mar 25, 2022

ghost added the needs-team-triage This issue needs the team to triage. label Mar 25, 2022

navba-MSFT added Service Attention This issue is responsible by Azure service team. and removed needs-team-triage This issue needs the team to triage. labels Mar 25, 2022

wwmoraes mentioned this issue Mar 25, 2022

[ARM] Fix #20842: az bicep: Fix to use requests environment variables for CA bundle #21807

Merged

3 tasks

zhoxing-ms closed this as completed in #21807 Mar 30, 2022

zhoxing-ms pushed a commit that referenced this issue Mar 30, 2022

[ARM] Fix #20842: az bicep: Fix to use requests environment variabl…

e0bb745

…es for CA bundle

navba-MSFT self-assigned this Mar 30, 2022

navba-MSFT mentioned this issue Apr 28, 2022

Error while attempting to retrieve the latest Bicep version: #22194

Closed

navba-MSFT added the bug This issue requires a change to an existing behavior in the product in order to be resolved. label Sep 29, 2022

github-actions bot mentioned this issue Jul 25, 2024

Azure CLI not able to install bicep extension #29474

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

SSL: CERTIFICATE_VERIFY_FAILED error on "az bicep install" Resurfaced #20842

SSL: CERTIFICATE_VERIFY_FAILED error on "az bicep install" Resurfaced #20842

Zerthick commented Dec 27, 2021 •

edited

Loading

ghost commented Dec 28, 2021

yonzhan commented Dec 28, 2021

Zerthick commented Dec 28, 2021 •

edited

Loading

Jaffacakes82 commented Jan 13, 2022

jtrivett commented Jan 20, 2022

jtrivett commented Jan 21, 2022 •

edited

Loading

steve-sellers commented Jan 24, 2022

baileydoestech commented Feb 11, 2022

navba-MSFT commented Mar 3, 2022

Zerthick commented Mar 8, 2022

Jaffacakes82 commented Mar 12, 2022 •

edited

Loading

jtrivett commented Mar 14, 2022

MiguelElGallo commented Mar 25, 2022

ghost commented Mar 25, 2022

navba-MSFT commented Mar 25, 2022

MiguelElGallo commented Mar 25, 2022

CiaraSHoulihan commented Apr 6, 2022

woeterman94 commented Sep 7, 2022

haciz commented Feb 1, 2023

aslan-im commented Aug 1, 2023

SSL: CERTIFICATE_VERIFY_FAILED error on "az bicep install" Resurfaced #20842

SSL: CERTIFICATE_VERIFY_FAILED error on "az bicep install" Resurfaced #20842

Comments

Zerthick commented Dec 27, 2021 • edited Loading

ghost commented Dec 28, 2021

yonzhan commented Dec 28, 2021

Zerthick commented Dec 28, 2021 • edited Loading

Jaffacakes82 commented Jan 13, 2022

jtrivett commented Jan 20, 2022

jtrivett commented Jan 21, 2022 • edited Loading

steve-sellers commented Jan 24, 2022

baileydoestech commented Feb 11, 2022

navba-MSFT commented Mar 3, 2022

Zerthick commented Mar 8, 2022

Jaffacakes82 commented Mar 12, 2022 • edited Loading

jtrivett commented Mar 14, 2022

MiguelElGallo commented Mar 25, 2022

ghost commented Mar 25, 2022

navba-MSFT commented Mar 25, 2022

MiguelElGallo commented Mar 25, 2022

CiaraSHoulihan commented Apr 6, 2022

woeterman94 commented Sep 7, 2022

haciz commented Feb 1, 2023

aslan-im commented Aug 1, 2023

Zerthick commented Dec 27, 2021 •

edited

Loading

Zerthick commented Dec 28, 2021 •

edited

Loading

jtrivett commented Jan 21, 2022 •

edited

Loading

Jaffacakes82 commented Mar 12, 2022 •

edited

Loading