Character set issue on Windows

[mikepowell]

I'm seeing the following error when running yamllint on a UTF-8 encoded file that includes characters that don't appear in the Windows 1252 character set:

Traceback (most recent call last):
  File "c:\users\mpowell\appdata\local\programs\python\python38-32\lib\runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "c:\users\mpowell\appdata\local\programs\python\python38-32\lib\runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "C:\Users\mpowell\AppData\Roaming\Python\Python38\Scripts\yamllint.exe\__main__.py", line 7, in <module>
  File "C:\Users\mpowell\AppData\Roaming\Python\Python38\site-packages\yamllint\cli.py", line 206, in run
    problems = linter.run(f, conf, filepath)
  File "C:\Users\mpowell\AppData\Roaming\Python\Python38\site-packages\yamllint\linter.py", line 240, in run
    content = input.read()
  File "c:\users\mpowell\appdata\local\programs\python\python38-32\lib\encodings\cp1252.py", line 23, in decode
    return codecs.charmap_decode(input,self.errors,decoding_table)[0]
UnicodeDecodeError: 'charmap' codec can't decode byte 0x9d in position 96: character maps to <undefined>

To reproduce, run yamllint on a file containing the following:

# “these quotes” will cause the error

If I edit line 205 in yamllint's cli.py file like this:

--- with io.open(file, newline='') as f:
+++ with io.open(file, newline='', encoding='utf-8') as f:

the error goes away. It seems io.open is just using the default Windows character set, rather than the actual encoding of the file.

I don't know much python so I doubt this is a robust enough fix to submit as a PR, but if I'm wrong let me know and I'll gladly submit the PR!

Maybe adding a character set parameter to the CLI would be a more flexible way to address?

[mikepowell]

Forgot to mention I also tried setting locale: en_US.UTF-8 in my config, but it didn't change this behavior. Maybe using the locale setting's character set in the io.open call would be a nice way to fix this?

[adrienverge]

Hello Mike, have you looked for exisiting issues and pull requests before opening a new one? Looks like #238 and #240 are very similar to your problem.

[mikepowell]

Sorry about that, I did look but somehow did not find that issue/PR.

[CAM-Gerlach]

@adrienverge Thanks for your continued attention to this. This still appears to be an issue in my testing on Python 3 on any platform that doesn't happen to have UTF-8 as the default locale (e.g. all Windows, some Linux). Given that implementing an ideal fix seems to have stalled for a year now, it would seem prudent until one could be implemented to simply assume UTF-8 rather than the current OS's locale, as that would presumably be straightforward to implement, ensure consistent cross-platform behavior, be a much less likely to trigger an error on well-formed YAML files than the status quo, and be closer to the spec.

It may not be perfect, but this would seem to be a situation where the perfect is the enemy of the good, given this appears a widespread problem and this approach, unless I'm misunderstanding something, should solve the overwhelming majority of cases without adding a dependency or non-trivial additional complexity, and without handling any valid cases worse than the status quo (since no OS I'm aware of has a locale that defaults to a non-UTF-8 Unicode encoding, and YAML files that are not Unicode encoded, as I understand, do not follow the spec and are thus not well-formed to begin with). But maybe I'm misunderstanding everything :)

In light of that, would you be willing to accept a PR with such a solution until a more robust fix can be implemented? Or how can this be moved forward? For the time being, this essentially limits us to either using only ASCII characters in YAML files, or not using yamllint (via pre-commit and CI) to lint them. Thanks!

[davidmoremad]

Also happened to me because I'm using emojis on some YAML files.

Python==3.10.4
yamllint==1.26.3

[MarcHagen]

Jetbrains IDEs have "yamllint build in" but is complaining about some file, but i don't know which one...

Python 3.11.3
yamllint 1.31.0

[Serge3leo]

Workaround for Python 3.7+ on Windows: set PYTHONUTF8=1

The probability of a false negative result of yamllint is small, since it is required that UTF-8 encoded bytes are not included in the system encoding. For example, for CP1251 there is one such byte - 0x98, for KOI8-R there are no such bytes at all, but for CP1252 there are 5/256.

On the other hand, there is a possibility of a false positive result of yamllint, for example, in the case of a correct YAML document in the system encoding, which is not decoded as UTF-8.