Detect .NET SDK & VSLANG Custom Language Settings & Apply To MSBuild

[nagilson]

Fixes #1596

Changes Made

SetConsoleUI now calls into a helper which sets the encoding to support non-en languages and checks if an environment variable exists to change the language to.

Testing

Setting DOTNET_CLI_UI_LANGUAGE=ja now changes msbuild correctly:

Doing a complicated build (aka building MSBuild) to use multiple threads shows other threads seem to use the same UI culture:

See that chcp remains the same after execution:

(Was set to 65001 temporarily but back to the original page before execution.)

Notes

Much of this code is a port of this code: dotnet/sdk#29755
There are some details about the code here.

[!] In addition, it will introduce a breaking change for msbuild just like the SDK.
The break is documented here for the sdk: dotnet/docs#34250

[rainersigwald]

Two things:

1. Can you expand on the breaking change some more, please? Is it only that we now respect VSLANG and will emit UTF-8 when it is specified?
2. Can you test this in the SDK when opted into MSBuild Server? I'm especially curious about the "do a build/start the server with no DOTNET_CLI_UI_LANGUAGE, then set it, then do another build" case.

[nagilson]

Can you expand on the breaking change some more, please? Is it only that we now respect VSLANG and will emit UTF-8 when it is specified?

When you use VSLANG or DOTNET_CLI_UI_LANGUAGE it will change the encoding to UTF-8. But that was only officially supported in the Windows 10 (November 2019 update). What happens if you set the environment variable in a version of windows 10 below that is undefined to me. I should change this to only run if Environment.OSVersion's last string is less than the 2019 Nov build number: 18363. I think we would tell people then to set the code page themselves if they are out of date.

If we didn't do that (which is what the SDK is currently doing): probably it will work for some versions, but maybe with some encoding bugs from Windows. Maybe it will fail/reject the encoding in very early versions. Outside of VS at the very least, if you set the environment variable without the encoding it would give you a bunch of ??? or garbled missing-no's.

For VS it is more ... hrmmm. For non-Windows it should not matter, encoding is not an issue. For Win 8 and below setting the environment variables will probably result in garbage outputs until the user manually edits the code page. Thinking about it more, maybe we should try to set the language specific encoding for these languages. But each language has its own quirks with characters output by MSBuild that probably won't work well, and emojis (like the live logger) will be completely broken.

In addition, I need to get to your point in 2) later. But from the testing I did yesterday the chcp change is preserved upon MsBuild exit. If MSBuild is killed and doesn't get to restore the code page, for programs on the same console I believe they will be impacted by the encoding change if they don't manage their own encoding. This could cause various breakages, exemplified by one posted by the customer linked above. In addition, even for modern windows 10 versions, I would expect there to be legacy / custom consoles that for whatever reason don't work well with the UTF-8 encoding.

[nagilson]

This is waiting for dotnet/sdk#30963 to be merged. Once that's merged, I will ping again for review.

[nagilson]

For 1) Now, we only set utf-8 if its officially supported or opted in. And we also restore the encoding. So the breaking change is that if VS doesn't set the encoding for win 8 or below then the characters could become garbled for non-en scenarios. Let's discuss this.

For 2) It doesn't work and that may relate to the VS issue in 1. This is because xmake is skipped in these cases per @Forgind. I've pinged you offline @rainersigwald.

[rainersigwald]

@rokonec what do you think the best way to change the stdout codepage in the msbuild client is, based primarily on an environment variable? Should it be done in the client init code, or maybe we should add a new ServerChangeCodepage packet type?

[nagilson]

@rainersigwald This is good to consider, though we also must consider the VS Case not calling into XMake. What about BuildManager's BeginBuild methods?

cc @Forgind

[rainersigwald]

we also must consider the VS Case not calling into XMake. What about BuildManager's BeginBuild methods?

If you're using the API, the log events should always be standard 16-bit UCS-2 .NET strings and shouldn't care about codepage one way or another.

Does the "output window" in VS have problems with encodings in these cases?

[nagilson]

Does the "output window" in VS have problems with encodings in these cases?

Nope! Tested it myself at least and it didn't.... there were issues but they all came from calling into dotnet first, so seems like it was an SDK thing.

So @rokonec sorry for the delay, please let us know your opinion on the above as it still doesn't work in msbuild server.

@rokonec what do you think the best way to change the stdout codepage in the msbuild client is, based primarily on an environment variable? Should it be done in the client init code, or maybe we should add a new ServerChangeCodepage packet type?

[rokonec]

I would recommend to move whole logic around SetConsoleUI() from Execute to Main before the if (server) , about here

msbuild/src/MSBuild/XMake.cs

Line 236 in 5a06292

so it will become on both, classic and server path.

That shall cause to properly set CurrentThread.UICulture which is then passed to server in build command, and all should work "fine" since.

However, if it does not work "fine" for server scenarios, please focus on another aspects of this PR and I volunteer to fixing it for server in another consequent PR.

[nagilson]

The encoding needs to be added to logging as well before this is merged.

[JanKrivanek]

Looks good to go

[nagilson]

I was able to get it working with logs!

A new trade-off I made: I put the code to occur BEFORE debugging so it is literally the first thing to run. If I didn't do that and you got build errors in a different language, they would be junk. Maybe that's acceptable if we want to be obstinate on the principle that NOTHING should run before the debug attachment code.

What it looks like if we move it back and you get a build error building msbuild:

What it looks like now:

[nagilson]

When working on this I happened to discover a deadlock (I think) that is unrelated to my changes.

You have to set set MSBUILDDEBUGONSTART=2 and then do artifacts\bin\bootstrap\net472\MSBuild\Current\Bin\MSBuild.exe MSBuild.Dev.slnf /m  /flp:v=diag

This happens even on main without my changes, then MSBuild will hang forever in a deadlock waiting for the _syncLock. If you press ctrl-c it will say Attempting to cancel the build but no matter how many times you press it or wait youll need to task kill it. The thread window shows all the threads are waiting.

[nagilson]

🥳