Exec ConsoleToMSBuild=true doesn't preserve colors

[jzabroski]

@rainersigwald I've spent some time pondering some deep, Confucious stuff lately regarding my build toolchain, and started asking:

"How can I run a .NET Core CLI Tool in an MSBuild Exec Task AND preserve the color output?"

Oh, boy.

Along the way, I found that MSBuild output itself can emit ANSI Escape Codes to processes like Travis CI... so it got me wondering, why can't MSBuild parse ANSI Escape Codes rather than generate them? Searching StackOverflow, I don't see any good answers. People seem to think fundamentally there is no way to capture color, it's drastically impractical, because color is a property of the console, not the standard output and error streams. I say, let there be color. Pastel, maybe.

Here is a sample repro of the issue:

build.targets

<?xml version="1.0" encoding="utf-8" ?>
  <Project ToolsVersion="15.0" xmlns="http://schemas.microsoft.com/developer/msbuild/2003" DefaultTargets="PowershellColoredOutput">
  <Target Name="PowershellColoredOutput">
    <!-- Clear your mind with this zen action: -->
    <Exec Command="pwsh.exe -Command &quot;Write-Host 'hi' -ForegroundColor red&quot;" ConsoleToMSBuild="true">
	  <Output TaskParameter="ConsoleOutput" ItemName="OutputOfExec" />
    </Exec>
  </Target>
</Project>

Expected Output

"hi" in red.

Actual Output

"hi" in console default foreground color and default background color

Deep Thoughts

“I for one believe that if you give people a thorough understanding of what confronts them and the basic causes that produce it, they’ll create their own program, and when the people create a program, you get action.” — Malcolm X

[jzabroski]

Per my new friend Mark Junker, it should be possible to do this thanks to Windows 10's Windows Subsystem for Linux (WSL). In looking this up, it appears he is right, and it is possible thanks to microsoft/WSL#406

[rainersigwald]

Thanks for filing this; I've been thinking about it for a long time, too. Unfortunately, I haven't found a good solution either. There are a few problems:

1. It's extremely common for tools to detect that they're being redirected and stop emitting color codes. Powershell does this:
   
2. On Windows, there are two ways to change the console color: ANSI escape codes, and Win32 Console APIs. We could only ever hope to capture the former--I definitely don't think it's worth building a ConDrv-aware Exec into core MSBuild just for colorization.
3. We have to decide what to do with the output. Right now, messages exist only as text. That means they can be streamed to the console (with our minimal colorization applied) and to text logs and to the binary log and to custom loggers. It's very likely that a subset of those things would be very confused if VT escape codes suddenly appeared in the text.

This came up in the context of dotnet test, which runs inside MSBuild but wants to control its output completely, which currently requires various workarounds.

[jzabroski]

A couple of thoughts:

1. We need a movement to emancipate color so that programs can share color with one another. Can you please share PowerShell naughty behavior in the PowerShell "vZeroTechnicalDebt" issue here: Is it time for "PowerShell vZeroTechnicalDebt" ("vNext") and/or for an opt-in mechanism into new/fixed features that break backward compatibility? PowerShell/PowerShell#6745
2. Right. It really doesn't belong in MSBuild in the first place. There are enough useful APIs buried in obscure places as-is. System.Console.Palette (or similar) would be a fine common namespace.
3. No, we don't have to decide. With a Builder or Interpreter design pattern, it is up to the client caller to decide when to materialize. Examples include StringBuilder.ToString() & DbSet<T>().ToList(). In both situations, there is a degree of "call-by-intention" function evaluation semantics. Another example of call-by-intention is a -WhatIf flag in PowerShell commands that traces an effectful execution path without side effects. The generalization therefore is providing a similar way to trace effects called an Output Renderer.

My last point may be too clever for some people to understand, so I'll break it down another way:

When you write logging code, you could write: Log.Info(ex.ToString()), and given the evaluation rules of C#, you will always evaluate ex.ToString() as a side-effect. Alternatively, you could "lift" the entire logging expression into a delegate Log.Info(() => ex.ToString()) and it can be optimized away by a JIT compiler if your Logger isn't configured for Info-level messages. All you are doing is quoting your log message and level as data, and dynamically deciding whether to evaluate it. Once it's data and not code, it becomes call-by-intention and there is no eager evaluation of values. Because it's structured as data, you can even optimize how you store that data, which is what logging frameworks like Serilog and Its.Log effectively do. With both of those frameworks, the templates for how log data actually materializes output are downstream and part of the evaluation process.

It's this sort of abstraction people want, and it just needs to exist and be documented for it to work really well. So, let's Process.Start() a revolution, baby!

[jzabroski]

@rainersigwald Let me know if the above replies make sense. We'll fight the good fight. In the mean time, since it seems like my Trojan Horse about Exec needing Colors was unnecessary, since you already seem to understand the general problem, let me know if I can rename this to something a bit more serendipitous with your thoughts.

But, seriously though: I would be very happy just to get MSBuild to preserve dotnet cli colors.

[jzabroski]

@rainersigwald Thanks for the reference to ConDrv. I was unaware that a Console Team even exists. It seems their solution is a bit too clever by half, and seems to prevent the very scenario I want to achieve.

It's hard to fully understand their architecture drawing, but it seems to me that they boxed developers in by only having a "Command Line API" on the left hand side of ConDrv, and no "Command Line API" on the right hand side of ConDrv. I guess I'll try to track down Rich to understand his drawing better.

Do you happen to know if ConPTY is exposed on .NET Core, and, if so, how? TYVM! Could it be this? https://github.com/Microsoft/console/tree/07d06f62aa5a883f70cbe8572bf8ea1f8577f53f/samples/ConPTY/GUIConsole/GUIConsole.ConPTY

[zadjii-msft]

Hey so lemme chime in real quick before we get off the rails on ConPty.

Conpty exists to enable applications to act as terminal emulators on Windows. It acts as the console host, servicing console API calls in the same way the console normally does, but then "renders" the side effects of the API calls to a stream of VT sequences. This means that client applications (like cmd.exe, msbuild.exe, etc) can use the same old Console API's they've been using since Windows 3.1, but now instead being forced to use conhost as the terminal window, another application could step in as the terminal instead, and the new terminal application could be written just the same as a terminal emulator on linux.

We (the console team) certainly haven't done any work to expose ConPTY in any sort of managed sense. The sample you linked is community code, but there's no official support currently.

Conpty is not a new magic commandline client API. It will not magically make your client app emit output as VT.

For commandline-client application developers targeting Win10+, our general recommendation is to use VT sequences always, and enable VT processing with SetConsoleMode. We're expanding our support for VT sequences, but we're leaving the console API as it is, for compatibility reasons. As noted here, the Console API isn't really portable to other platforms.

[jzabroski]

@zadjii-msft But does my "call-by-intention" remarks make sense to you, and what I think a "good API" for this aesthetically has? For example, awhile ago author Joe Albahari created a library called Linqkit for extending Linq SQL libaries with common sub-expression substitutions. However, this library is fundamentally the wrong user experience for "call-by-intention" programming: The API producer has to "know" the end-user plans to require such substitutions, and the API end-use has to "know" the API producer provides such substitute-ability: The API Producer calls linqQuery.AsExpandable(). A much better approach is in the erecruit.Expr library, where the client caller can call Expand() at the very end of a an IQueryable builder pattern, without needing to know whether the IQueryable "supports" any special functionality.

Sorry if this is overly verbose, but it's a Interpreter design pattern I have discovered some programmers have experience with, and others do not, and somewhere in the middle where there is a huge portion of engineers who don't even see the difference. THIS is the architecture we need.

[zadjii-msft]

Yea I'm gonna say that's out of my area of expertise. I'm mostly only really concerned about the characters that end up getting written to the console. How apps want to pipe information to each other and preserve metadata such as color is not something I spend a lot of time thinking about :/

[jzabroski]

Thanks. One last question: When you say:

We're expanding our support for VT sequences, but we're leaving the console API as it is, for compatibility reasons. As noted here, the Console API isn't really portable to other platforms.

Are you referring to the Win32 Console API or the .NET surface area exposed via System.Console? As a .NET engineer discussing a .NET project MSBuild, I hear "the Console API isn't really portable to other platforms" as "the [.NET] Console API isn't really portable to other platforms."

[zadjii-msft]

I'm talking about the Win32 Console API, as that's the layer we maintain. I'm not entirely sure there's anyone maintaining the .NET Console API.

[jzabroski]

Got it. After comparing to what UNIX systems do, they are not too far off from how Windows behaves. The key thing is that GNU Core Utilities tend to have a --color switch that accepts always, never and auto/tty/if-tty. Programs like ls and grep can auto-detect if the output is redirected to a tty (console).

In this sense, I suppose you might be right: the Win32 Console API shouldn't necessarily care what programs do, but the general Windows ecosystem sucks at having any support for this.

[danmoseley]

I'm not entirely sure there's anyone maintaining the .NET Console API.

Yes there are. For .NET Core, we live in https://github.com/dotnet/corefx

[jzabroski]

I pinged @mscraigloewen and @bitcrazed on twitter about this issue. My "call by intention" idea has outstanding merit.

[mcartoixa]

This came up in the context of dotnet test, which runs inside MSBuild but wants to control its output completely, which currently requires various workarounds.

@rainersigwald I am realizing that custom colorization also prevents tests output from being processed by MSBuild loggers (microsoft/vstest#680)! By the way dotnet test calls MSBuild which itself (via VSTestTask) calls dotnet exec vstest.console.dll, so maybe the way forward would be to cut the middle man? Which is easier written than done, I am sure.

If I may add my 2 cents here, having used MSBuild for many years, I love the austerity of the MSBuild output which gives me 2 simple and essential things: errors are red and warnings are yellow. And further information I can dig in the logs, should I need them. Other environments (think Rake, gulp.js, Phing...) are lacking in this regard IMHO, most often in their (in)ability to produce detailed logs.

[bitcrazed]

@jzabroski

The problem of Windows' command-line tools not handling VT sequences is a long and historical one. But to summarize:

Windows was architected upon object-based principles. UNIX was designed upon the principle that everything is a stream/file. Thus there are MANY differences to the design and implementation of both OS' and the tools that run upon them.

This is easily seen in the way that UNIX and Windows command-lines work:

In UNIX, apps emit streams of characters. Those streams of characters may contain embedded formatting codes (ANSI/VT sequences) that instruct the recipient to render the subsequent characters in bold, italic, or specific foreground / background colors, etc.

In Windows, command-line apps emit text, but call APIs on the Console (the traditional Windows command-line GUI app that you see on-screen) to set colors, move the cursor around, etc. However, this breaks-down when you try to execute a command-line app on a remote machine - how does the remote command-line app call a method on your local Console from the remote machine upon which the command-line app is running? This is why Windows command-line apps suddenly start rendering in monochrome when accessed via ssh etc.

We've been overhauling Windows' command-line infrastructure over the last several years to allow us to build a modern Windows Terminal app, and to eventually allow Windows command-line apps themselves to start emitting text/VT directly, rather than calling Windows' Console APIs (via System.Console in .NET).

We built the "magic" ConPTY to render text and API calls to an off-screen buffer and then stream the updated chars as text/VT so that Terminal doesn't have to continue to implement legacy Console APIs.

This ConPTY will also allow comms gateways like OpenSSH to locally "render and stream" the text UI of Windows command-line apps as text/VT meaning you'll see color text coming back from remote command-line apps at some point when the work gets done 😜

And now that Console and Terminal have pretty decent support for VT, command-line apps like MSBuild could emit text/VT directly rather than calling System.Console (and thus Win32 Console APIs on Windows). This would take a little work, clearly, but may well be worth considering. Happy to discuss with the MSBuild team if they'd like to reach out.

We're also working with @jonsequitur & @KathleenDollard (System.CommandLine) & others to figure out a modern command-line infrastructure which command-line apps can use to simply emit text/VT, avoiding Console APIs and the need to "render and stream" completely, also making it easier to write x-plat .NET and native code too.

HTH.

[jzabroski]

@bitcrazed Thanks for the thoughtful reply.

My point was something much simpler, though.

I was pointing out that if you look at how various UNIX utilities like troff behave and what formatting they support, there is a concept of:

--color switch that accepts always, never and auto/tty/if-tty

Think of it this way. Color switch is like an HTTP media type. The terminal can query the program and the program can query the terminal and negotiate content types. In some cases, the terminal is just a browser/intermediary for two programs to communicate in a client-server pattern. The terminal can't and shouldn't know about the "magic conPTY" you describe. There has to be basic infrastructure for IPC.

I understand for portability reasons to UNIX and general platform goals of supporting Linux, you chose what you chose. It's a defensible choice. I am simply trying to argue what I think the next 30 years of my career would ideally look like: content-based negotiation.

I can also come up with more interesting examples to further my argument. But first I'm just seeking understanding of my viewpoint, as it is likely non-obvious to both Linux and Windows users.

[bitcrazed]

The color arg passed to troff etc. is just that - a switch that the code has to recognize and prevent emitting VT color sequences.

This has nothing to do with the Terminal. Why? Terminals don't know what they're connected to - it could be a bash shell on the local machine, it could be ping.exe, it could be SSH which connects via WIFi to a weather station in Llanfairpwllgwyngyllgogerychwyrngogogoch

To launch and connect to an tool, a terminal:

1. Creates a pipe for stdout through which it sends all key chars
2. Creates a pipe through which it receives any text/VT which it renders to the screen
3. Launches the necessary shell/tool connected to the two pipes

Everything else is transparent to the Terminal. That's why the ConPTY is magic.

Note: There is no content negotiation available in today's command-line on any platform. There's a database of terminal capabilities but it's incomplete and not dynamic, which is why tools like troff accept a command-line arg to enable/disable emission of color VT sequences.

[jzabroski]

I get that you're focused on pseudoterminals. Would it be more helpful to use a different word than pseudoterminal / pty, as ConPTY magic is basically just giving Windows what Linux has had for a long time? Would BrowserTerminal or bty better communicate the idea?

troff indeed is not the future, but I'm simply describing how you can think of the various *NIX utilities as truly non-modular, by first recasting a simple feature (color) as a "terminal capability". Rather than having to explicitly specify the capability on the command line, I'd rather it just be auto. But, ideally, in the future, I have a terminal.config.json or something that controls what capabilities I prefer. And I just download that config from github wherever I go, and there I have my preferences. That's where the world is going with things like Firefox and Google Chrome preferences, but also tools for composing OS tech stacks like Nix with textual rules checked into GitHub.

In the pseudoterminal world, you're forced to tell troff these switches, BECAUSE the terminal has to intercept all these things BECAUSE its in charge of the pipes. So when you pipe something into troff, that input has to be massaged a certain way. There is an implicit contract between the left and right hand side of the pipe. If it's an implicit contract, why can't I have middleware (call it bty for BrowserTerminal) that negotiates the details of that contract for me? Because then I don't have tight coupling on the representation of characters at all.

Rather than all this talk of pipes, what I'd rather see is simpler abstractions of Processor and ProcessorContext. The Programs communicating over what you call pipes are just sharing a context. Then my command line expression could be a continuation instead of just a co-routine, and I could have deep Control+C SIGINT.

As Rainier said, some programs already sniff for things like whether their output is being redirected. It would be a lot nicer if that sniffing layer was uniform and we had canonical names for all these wonderful smells we can sniff.

To be honest, this way of thinking about the problem is new to me, too. I looked around for others with similar ideas and probably the closest person is Thomas Lord (of GNU Arch DVCS and GNU Guile Lisp fame). I only thought about it recently because I had seen a large number of problems in Continuous Integration servers and their toolchains around color and formatting expectations.

[bitcrazed]

ConPTY is much, MUCH more than simply bringing a Linux-style Pty to Windows. If that's all we'd done, no existing Windows command-line app would work on any terminal connecting to the simpler pty.

The point is that pty should be an unintelligent pipe and should not be responsible for nor involved in content/capability negotiation.

There's an important architectural quirk here: There is no meta-communication channel between the Terminal and the command-line app. There's no way for the Terminal to send commands or requests for information to a command-line app.

SHOULD there be one? Perhaps. But know that for this to work, there'd need to be an industry-wide, standardized, multi-decade initiative to make this happen.

Re. sniffing:

What @rainersigwald is seeing is not redirection sniffing

Calling echo specifying -ForegroundColor results in pwsh calling Console APIs but the output of the command is pure text (no VT) and so the API calls do nothing. Thus, pure text is piped into WSL's cat and it just displays plain text.

If you emit VT, however, you'll see the color is preserved:

$esc = "$([char]27)"
echo "$esc[91mHello$esc[0mWorld" | wsl cat

So, if you want to color/stylize the output of your command-line apps, you should start planning to adopt VT, and avoid Console APIs if you want your styling to survive being piped through environments, between machines, and displayed even on non-Windows terminals.

[jzabroski]

SHOULD there be one? Perhaps. But know that for this to work, there'd need to be an industry-wide, standardized, multi-decade initiative to make this happen.

This makes me happy. I mean, it's 2020 and I'm finally enjoying web apps.

[jzabroski]

What @rainersigwald is seeing is not redirection sniffing

I actually re-read this three times before I understood your point. Welp. I think I over complicated my immediate needs. Good discussion, though.

you should start planning to adopt VT, and avoid Console APIs

This is the big takeaway. I'll give it a try and see what problems I run into.

[bitcrazed]

LOLZ. Sorry if I dove deep, but I wanted to be sure that people realize that the command-line today is LITERALLY a mirror of the command-line as it was in 1960: It's chars out and chars in. That's it; everything else is just details! 😜

It's for this very reason that I wrote this multi-part series on the command-line and how it evolved, and how Windows differs from *NIX, etc.

[jonsequitur]

Color switch is like an HTTP media type. The terminal can query the program and the program can query the terminal and negotiate content types. In some cases, the terminal is just a browser/intermediary for two programs to communicate in a client-server pattern. The terminal can't and shouldn't know about the "magic conPTY" you describe. There has to be basic infrastructure for IPC.
-- @jzabroski

It's chars out and chars in. That's it; everything else is just details! 😜
-- @bitcrazed

Supporting some form of content negotiation in a way that can work everywhere is why we have this "directive" concept in System.CommandLine:

> myapp  [output:ansi] --some-args --and stuff 

The syntax is weird because we wanted it to be amenable to standardization regardless of your CLI's own grammar.

[bitcrazed]

@jonsequitur So System.CommandLine will strip VT if [output:ansi] is set?

[jonsequitur]

System.CommandLine.Rendering will include VT if [output:ansi] is set, it will attempt to render using System.Console APIs if [output:nonansi] is set, and it will render plain text (no VT, table layouts via whitespace) if [output:plaintext] is set.

We typically try to detect the terminal capabilities and set this for you, so these are explicit overrides to that behavior.

[jzabroski]

@bitcrazed One discussion @jonsequitur and I had months ago (years ago??) in private was whether there was any standard for representing command line grammars, so that tools could interop with command line grammars and provide services like auto-complete and rich GUIs similar to the PowerGUI concept that was acquired by Dell and subsequently buried after budget cuts.

If you happen to know of any standard, that would probably be helpful towards bty type goals. When you mentioned Kathleen and Jon, I didn't mention that Jon and I have had discussions about system engineering dating back to Its.Log prototype.

[bitcrazed]

@jonsequitur - Got it, thanks. Though one suggestion: You may want to rename ANSI to VT since ANSI stopped standardizing escape codes and VT sequences in the '80s, deferring instead to industry standards. ISO/IEC and ECMA both maintain standards, but haven't really kept up with industry's advancements to support, for example, 24-bit color, etc.

And perhaps nonansi could be replaced with legacy or similar?

[bitcrazed]

@jzabroski Alas, the only shell <-> command-line protocol for things like arg expansion, etc. that I am aware of are ad-hoc approaches like those supported by bash et. al.

However, John, Kathleen, and I (and others) have discussed how we might be able to systematize this somewhat - which is why @jonsequitur introduced the ability to dynamically reflect-over/query command-line apps built atop System.CommandLine for its list of supported args, etc. Support for such a capability, however, as one can imagine, will require work by a lot of teams to introduce, adapt, and support.

But, from small acorns ... 😜

[jonsequitur]

the ability to dynamically reflect-over/query command-line apps built atop System.CommandLine for its list of supported args

And a footnote to this is that the mechanism by which this query is performed is also the directive syntax I mentioned earlier, e.g.:

> dotnet-interactive [suggest] stdio h
--help
--log-path
-h
/h
http