UnicodeEncodeError: 'charmap' codec can't encode characters in position 192-231: character maps to <undefined>

[mattmazzola]

Describe the bug

When attempting to build a Docker image using az acr build the build fails in the ADO pipeline.
However, the same az acr build works locally.

Perhaps this issue could be transferred to ADO if appropriate.

Command Name
az acr build <removed>

Errors:

Step 4/14 : RUN pip install -U pip   && apt-get update   && apt install -y curl netcat   && pip install --user poetry
ERROR: The command failed with an unexpected error. Here is the traceback:
ERROR: 'charmap' codec can't encode characters in position 192-231: character maps to <undefined>
Traceback (most recent call last):
  File "D:\a\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\knack/cli.py", line 231, in invoke
  File "D:\a\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/commands/__init__.py", line 663, in execute
  File "D:\a\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/commands/__init__.py", line 726, in _run_jobs_serially
  File "D:\a\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/commands/__init__.py", line 697, in _run_job
  File "D:\a\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/commands/__init__.py", line 333, in __call__
  File "D:\a\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/commands/command_operation.py", line 121, in handler
  File "D:\a\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/command_modules/acr/build.py", line 143, in acr_build
  File "D:\a\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/command_modules/acr/_stream_utils.py", line 58, in stream_logs
  File "D:\a\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/command_modules/acr/_stream_utils.py", line 143, in _stream_logs
  File "D:\a\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\colorama/ansitowin32.py", line 41, in write
  File "D:\a\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\colorama/ansitowin32.py", line 162, in write
  File "D:\a\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\colorama/ansitowin32.py", line 190, in write_and_convert
  File "D:\a\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\colorama/ansitowin32.py", line 195, in write_plain_text
  File "encodings\cp1252.py", line 19, in encode
UnicodeEncodeError: 'charmap' codec can't encode characters in position 192-231: character maps to <undefined>
To open an issue, please run: 'az feedback'

To Reproduce:

Steps to reproduce the behavior. Note that argument values have been redacted, as they may contain sensitive information.

It's not clear how to reproduce since we can't run ADO pipeline environment locally. The error makes it seems like file encoding issue.

I am attempting to build and push 3 images in the pipeline.
For images 1 and 2 az acr build executes without issues.
This makes it seems like there is something wrong with the 3rd Dockefile.
However, I cans run az acr build for that file from my local machine and it succeeds which suggests the Dockerfile is ok, and it is in fact something specific to running az cli in ADO

As I understand the first thing az acr build does is create a .tar.gz of the build context and send it to an ACR agent.
It would suggest that the files it is packaging and sending locally are encoded differently than those it packages and sends during the pipeline run. This would explain why the agent succeeds with the former but fails with the latter.

Although both the local machine and pipeline are using Windows

Expected Behavior

az acr build should work the same in ADO build as it does locally.

Environment Summary

Windows-10-10.0.22622-SP0
Python 3.10.5
Installer: MSI

azure-cli 2.38.0 *

Extensions:
application-insights 0.1.16
containerapp 0.3.8
log-analytics 0.2.2
storage-blob-preview 0.6.1
storage-preview 0.8.3
stream-analytics 0.1.2

Dependencies:
msal 1.18.0b1
azure-mgmt-resource 21.1.0b1

Additional Context

[yonzhan]

route to CXP team

[navba-MSFT]

@mattmazzola Thanks for reaching out to us and reporting this issue. We are looking into this issue and we will provide an update.

[navba-MSFT]

@mattmazzola I think it's a platform issue related to the unique combination Windows + Python.

Could you please try to update the Windows system language setting as shown below ?

Control panel -> Clock and Region -> Region -> Administrative -> Change system locale -> Check Beta: Use Unicode UTF-8 for worldwide language support.

The change will require reboot. Please let me know if you still face the same issue. Awaiting your reply.

[mattmazzola]

Could you please try to update the Windows system language setting as shown below?

You have listed instructions that show using the Windows GUI.
In my case, this is running in an Azure Dev Ops pipeline using the windows-2022 image

https://docs.microsoft.com/en-us/azure/devops/pipelines/agents/hosted?view=azure-devops&tabs=yaml#software

https://github.com/actions/runner-images/blob/main/images/win/Windows2022-Readme.md

I assume the setting could be controlled via the Registry or enabled by via PowerShell, although I don't think I would have opportunity to "restart" the container

1. Can you reproduce the issue?
   (Perhaps minimal pipeline with python related az acr build)

2. Are there alternate instructions to achieve the same goal of system wide utf8 support in containers?

[navba-MSFT]

@mattmazzola Could you please try with the below registry key which does the same in the background.

New-ItemProperty -LiteralPath 'HKLM:\SYSTEM\CurrentControlSet\Control\Nls\CodePage' -Name 'ACP' -Value '65001' -PropertyType String -Force;
New-ItemProperty -LiteralPath 'HKLM:\SYSTEM\CurrentControlSet\Control\Nls\CodePage' -Name 'OEMCP' -Value '65001' -PropertyType String -Force;
New-ItemProperty -LiteralPath 'HKLM:\SYSTEM\CurrentControlSet\Control\Nls\CodePage' -Name 'MACCP' -Value '65001' -PropertyType String -Force;

[mattmazzola]

I added those commands to modify Registry and It seems to have worked!
The pipeline that attempted to build the image was successful!

Before the pipeline was failing on step 4/14 (see fulll snippet above)
But now it continues:

Step 4/14 : RUN pip install -U pip   && apt-get update   && apt install -y curl netcat   && pip install --user poetry
 ---> Running in 74180ea2968d
Requirement already satisfied: pip in /usr/local/lib/python3.10/site-packages (22.2.1)

Collecting pip
  Downloading pip-22.2.2-py3-none-any.whl (2.0 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.0/2.0 MB 38.1 MB/s eta 0:00:00
Installing collected packages: pip
  Attempting uninstall: pip
    Found existing installation: pip 22.2.1
    Uninstalling pip-22.2.1:
      Successfully uninstalled pip-22.2.1
Successfully installed pip-22.2.2
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
Get:1 http://deb.debian.org/debian bullseye InRelease [116 kB]
Get:2 http://deb.debian.org/debian-security bullseye-security InRelease [48.4 kB]
Get:3 http://deb.debian.org/debian bullseye-updates InRelease [44.1 kB]
Get:4 http://deb.debian.org/debian bullseye/main amd64 Packages [8182 kB]
Get:5 http://deb.debian.org/debian-security bullseye-security/main amd64 Packages [182 kB]
Get:6 http://deb.debian.org/debian bullseye-updates/main amd64 Packages [2596 B]
Fetched 8574 kB in 1s (6341 kB/s)

Reading package lists...

<truncated log>

It says this is a beta feature, but I imagine this is causing issues for many others trying to create images for python based applications.

I will close with these two future items you may consider:

• Adding notes about these Registry keys to the Azure Pipeline documentation specifically calling out this issue
• Consider talking with the compliance and ADO teams to make these settings enabled by default in the windows-2022 base image so that it works by default for future people.

[sbertolawork]

New-ItemProperty -LiteralPath 'HKLM:\SYSTEM\CurrentControlSet\Control\Nls\CodePage' -Name 'ACP' -Value '65001' -PropertyType String -Force;
New-ItemProperty -LiteralPath 'HKLM:\SYSTEM\CurrentControlSet\Control\Nls\CodePage' -Name 'OEMCP' -Value '65001' -PropertyType String -Force;
New-ItemProperty -LiteralPath 'HKLM:\SYSTEM\CurrentControlSet\Control\Nls\CodePage' -Name 'MACCP' -Value '65001' -PropertyType String -Force;

How would you go about doing this for an acr task generated through "az acr task create". My task is failing due to this same UnicodeEncodeError