Use UTF-8 on Windows 10 Version 1903, fix #1195

[jhasse]

Allows Ninja to use descriptions, filenames and environment variables with characters outside of the ANSI codepage on Windows. Build manifests are now UTF-8 by default (this change needs to be emphasized in the release notes).

WriteConsoleOutput doesn't support UTF-8, but it's deprecated on newer Windows 10 versions anyway (or as Microsoft likes to put it: "no longer a part of our ecosystem roadmap"). We'll use the VT100 sequence just as we do on Linux and macOS.

https://docs.microsoft.com/en-us/windows/uwp/design/globalizing/use-utf8-code-page
https://docs.microsoft.com/en-us/windows/console/writeconsoleoutput
https://docs.microsoft.com/de-de/windows/console/console-virtual-terminal-sequences

Fixes #1195 (I hope).

[tristanlabelle]

@jhasse I'm happy to see Ninja supporting Unicode, but I want to point out consequences to this change that should be taken into account:

This is a breaking change for ninja build files. Since Ninja feeds the raw bytes from those files into -A Win32 functions, this means that Ninja build files will now have to be encoded as utf-8, not ANSI (which is good, but a breaking change). In particular, cmake will have to be updated to not generate Ninja build files as ANSI.

There might also be repercussions with the "include prefix" feature. Ninja does binary comparison of the string coming from its build file and that coming from the cl.exe process output. Previously the first of these strings was ANSI and now it will be UTF-8, so this might cause mismatches depending on the encoding that cl.exe uses.

[jhasse]

Indeed, thanks for pointing that out.

Note that this will only result in issues when there are non-ASCII characters in the build manifest. Something that would have resulted in problems on Windows anyway so I doubt that many people relied on that. It will still be the first point in the release notes ;)

[bradking]

I've opened CMake Issue 21866 for this, thanks.

[bradking]

#1918 proposes an additional tool to help generators determine the correct encoding for build.ninja files.

[penagos]

It appears this fix isn't providing full UTF-8 support on Windows 10 (build 19043). With a simple build.ninja file below:

cflags = -Wall
CC = gcc

rule cc
  command = $CC $cflags -c $in -o $out
build foo°.o: cc foo.c

And a source file, foo.c below:

int main(int argc, char** argv) { return 0; }

ninja incorrectly generates an intermediate .o file with the name foo°.o. build.ninja was saved using the UTF-8 encoding. Similarly, ninja does not seem to correctly handle input source files with UTF-8 filenames. If instead we use the following build.ninja file:

cflags = -Wall
CC = gcc

rule cc
  command = $CC $cflags -c $in -o $out
build foo.o: cc foo°.c

and rename the aforementioned C source file to foo°.c, ninja produces the build error:

ninja: error: 'foo┬░.c', needed by 'foo.o', missing and no known rule to make it

Note that the lexer used by ninja did not encounter issues when the build.ninja file had comments, environment variables or target names including UTF-8 characters. The two failure patterns above are reproducible on both cmd and Powershell. Is this change intended to support UTF-8 characters in filenames / filepaths or lay the groundwork for such future support (it wasn't immediately clear from the PR description)?

[jhasse]

Is this change intended to support UTF-8 characters in filenames / filepaths [...]?

Yes. Looks like you've found a bug.