ANSI escape handling and PowerShell

[SteveL-MSFT]

User scenario

Since we've added more ANSI escape sequences for coloring in PS7, if you pipe or redirect that text output, the embedded ANSI escape sequences are sent down the pipeline. This can make log files harder to read. Also, for those wanting to leverage ANSI escape sequences, they are difficult to create/read for scripters and also harder to use formatting presets rather than hardcoding specific colors.

Fundamental philosophy

On Linux, if you use ls --color then different file types use ANSI escape sequences as color indicators. If you pipe this output to less, then you get paging while retaining the color information. If you redirect this output to a file, that file contains the ANSI escape sequences. If you then use the cat command on the file, you see the coloring as the ANSI escape sequences are rendered by the terminal.

On macOS, the similar command is ls -G, however, piping this output to less or redirecting to a file you lose the ANSI escape sequences so it is just plain text. I believe ls on macOS is detecting if output is redirected and turning off coloring automatically.

So for PowerShell, we should have a consistent experience regardless if Windows, Linux, or macOS and interop with native commands that may output ANSI escape sequences in addition to cmdlets/formatting.

Since ANSI escape sequences are de facto standardized, we should not introduce a new intermediate format and should just embed ANSI escape sequences within strings. This will ensure compatibility with other tools that emit or handle ANSI escape sequences.

Proposed technical implementation details

A new automatic variable $PSStyle will be added:

   TypeName: System.Management.Automation.PSStyle

Name            MemberType Definition
----            ---------- ----------
Reset           Property   string AttributesOff {get;set;}
Background      Property   System.Management.Automation.PSStyle+BackgroundColor Background {get;set;}
Blink           Property   string Blink {get;set;}
BlinkOff        Property   string BlinkOff {get;set;}
Bold            Property   string Bold {get;set;}
BoldOff         Property   string BoldOff {get;set;}
Foreground      Property   System.Management.Automation.PSStyle+ForegroundColor Foreground {get;set;}
Formatting      Property   System.Management.Automation.PSStyle+FormattingData Formatting {get;set;}
Hidden          Property   string Hidden {get;set;}
HiddenOff       Property   string HiddenOff {get;set;}
OutputRendering Property   System.Management.Automation.OutputRendering OutputRendering {get;set;}
Reverse         Property   string Reverse {get;set;}
ReverseOff      Property   string ReverseOff {get;set;}
Standout        Property   string Standout {get;set;}
StandoutOff     Property   string StandoutOff {get;set;}
Underline       Property   string Underlined {get;set;}
UnderlineOff    Property   string UnderlinedOff {get;set;}

$PSStyle

The base members return ANSI escape sequences mapped to their names. These are also settable so the user can change bold to underlined, for example. This makes it easier for scripters to author decorated strings with tab completion:

"$($PSStyle.Background.LightCyan)Power$($PSStyle.Underlined)$($PSStyle.Bold)Shell$($PSStyle.Reset)"

$PSStyle.OutputRendering

This is of type System.Management.Automation.OutputRendering which is an enum with the values:

• Automatic This is the default. If the host supports VirtualTerminal, then ANSI is always passed as-is, otherwise plaintext
• ANSI ANSI is always passed through as-is
• PlainText ANSI escape sequences are always stripped so that it is only plain text
• HostOnly This would be the macOS behavior where redirected or piped output the ANSI escape sequences are removed

$PSStyle.Formatting

This effectively replaces $Host.PrivateData as the way to read or configure colors for formatting rendering. $Host.PrivateData will continue to exist for backwards compatibility, but is not connected to $PSStyle.Formatting.

   TypeName: System.Management.Automation.PSStyle+FormattingData

Name         MemberType Definition
----         ---------- ----------
Debug        Property   string Debug {get;set;}
Error        Property   string Error {get;set;}
ErrorAccent  Property   string ErrorAccent {get;set;}
FormatAccent Property   string FormatAccent {get;set;}
Verbose      Property   string Verbose {get;set;}
Warning      Property   string Warning {get;set;}

One difference here is that instead of being of type ConsoleColor, these are all strings and doesn't separate foreground and background colors. This means that a single member can have foreground and background colors defined as well as other attributes like bold, underlined, etc...

Scripters can easily leverage this:

"$($PSStyle.Formatting.ErrorAccent)Power$($PSStyle.Formatting.Verbose)Shell$($PSStyle.AttributesOff)"

$PSStyle.Foreground and $PSStyle.Background

These members contain the standard 16 console colors as well as a RGB() methods to specify 24-bit color. For the colors, the values are settable and because they are strings can be any string content and any number of ANSI escape sequences.

   TypeName: System.Management.Automation.PSStyle+ForegroundColor

Name         MemberType Definition
----         ---------- ----------
Rgb          Method     string Rgb(byte red, byte green, byte blue), string Rgb(int rgb)
Black        Property   string Black {get;set;}
Blue         Property   string Blue {get;set;}
Cyan         Property   string Cyan {get;set;}
DarkGray     Property   string DarkGray {get;set;}
Green        Property   string Green {get;set;}
LightBlue    Property   string LightBlue {get;set;}
LightCyan    Property   string LightCyan {get;set;}
LightGray    Property   string LightGray {get;set;}
LightGreen   Property   string LightGreen {get;set;}
LightMagenta Property   string LightMagenta {get;set;}
LightRed     Property   string LightRed {get;set;}
LightYellow  Property   string LightYellow {get;set;}
Magenta      Property   string Magenta {get;set;}
Red          Property   string Red {get;set;}
White        Property   string White {get;set;}
Yellow       Property   string Yellow {get;set;}

PowerShell engine changes

The formatting system, pipelining, and redirection will be updated to respect the value of $PSStyle.OutputRendering.

StringDecorated type

This is an internal type to handle ANSI escaped strings.

Constructor

Single constructor taking a string as input.

bool IsDecorated property

Returns if the string contains ANSI escape sequences based on if the string contains ESC or C1 CSI.

int Length property

Returns the length of just the text content of the string sans ANSI escape sequences.

StringDecorated Substring(int contentLength) method

Returns a substring starting at index 0 up to the contentLength for content that is not part of an ANSI escape sequence. This is needed for table formatting to truncate strings preserving ANSI escape sequences that don't take up printable character space.

string ToString() method

Returns the plaintext version of the string.

string ToString(bool Ansi) method

Returns the raw ANSI embedded string if Ansi parameter is true, otherwise returns plain text with ANSI escape sequences removed.

Out-String update

To enable scripts to easily allow plain text redirection, Out-String will have a -RemoveAnsi switch.

Select-String update

By default, it would make sense for Select-String to ignore ANSI escape sequences, perhaps a -IncludeAnsi switch should be added.

$Host.PrivateData

This will be available for legacy scripts/modules that read from it. However, the engine changes (and console host changes) would mean these settings will no longer be observed and instead use the settings from $PSStyle.

Alternate considerations

Original prototype was built on top of System.CommandLine.Rendering, however, that is still a work in progress and is not targeted to be 1.0 in time for 7.1. It is also geared towards C# developers and the user experience is not great for scripts.

Related Issues

#10811
#7744
#3611

[adamskt]

Possibly stupid question here: Will I be able to use Export-Clixml on the $PSSyle automatic variable to "round-trip" my thoroughly tricked-out and gorgeous settings to a file that I could use to move them to another environment? Will the ANSI escape codes in the string members get serialized in such a way as to not trip up version control systems? In any case, this seems like a fabulous idea!

[SteveL-MSFT]

@adamskt good question. I don't see any reason why you couldn't serialize $PSStyle to Clixml and deserialize it to overwrite $PSStyle as a way to have the same settings across systems.

[jhoneill]

So in summary.
(1) A standard, fairly readable way for scripts to do formatting
(2) A simple way to turn off ANSI formatting whether the author used (1) or did it their own way
That solves what I see as the two main problems with ansi codes (wanting to turn it off - sometimes - and ugly scripts) .

I liked the early draft of this idea, and I like this even more. A simple thumbs up wasn't enough :-)

Since common parameters set the preference variables for the local scope, what would your view be on adding a common parameter which sets output-rendering for a single command ? Among other things that would mean if one really hates the authors choice of colors a simple entry in $PSDefaultParamterValues gets rid of it.

I also wonder if more choices should be available so that a user can set their own warning, error, emphasis styles and the author then says "format this as a warning" not "use orange text"

[SeeminglyScience]

If PowerShell is going to tackle theming I'd really really like to see some extra settings that a TUI might be able to tap into. Taking bootstrap as an example, I'd like to see the styles be under names like:

• Primary
• Secondary
• Success
• Danger
• Warning
• Info
• Light
• Dark
• Muted

Every time I've started making a TUI of any sort, trying to make something that looks nice and also keeps to the spirit of the user's color choices is very challenging.

[mklement0]

Re terminology: I wonder if we should use "Virtual Terminal (VT) [Escape] Sequences" rather than "ANSI escape codes" in order to avoid confusion with the "ANSI" character encoding - though, arguably, that confusion is more likely for Windows users and, conversely, Unix users may be less familiar with the term "Virtual Terminal".

FWIW, looking at a few man pages for the various Unix utilities capable of producing colored output suggests that they rarely mention the coloring mechanism and often just talk about "color", and less frequently "escape sequences".

Avoiding "ANSI" would also mean: string ToString(bool Ansi) -> string ToString(bool decorated)

---

Re output modes (controlling when to use the sequences, $PSStyle.OutputRendering):

Note that GNU ls (and implicitly also macOS/BSD ls) has 3 modes (as do both GNU and macOS/BSD grep):

• auto (the default): if stdout is connected to a (VT-enabled) terminal, use color; otherwise (redirected output (pipe, file)) don't.
• never: never use color
• always: always use color, irrespective of where the output goes.

GNU ls without a color option defaults to auto; using --color is the same as --color=always (the other options obviously being --color=never and --color=auto); note that this default is an unfortunate choice; GNU and macOS/BSD grep more sensibly interpret --color as --color=auto.

macOS/BSD ls, via arguments, offers auto behavior (only) via -G (default is never); environment variable CLICOLOR_FORCE can be set to achieve always behavior.

In short: I think these modes are sufficient and I think they're worth adopting - including the terminology.

---

Use of environment variables, new CLI parameter, new common parameter to control coloring:

We should consider exposing setting $PSStyle.OutputRendering and the potential theming proposed by @SeeminglyScience via environment variables as well, so that there's an easy way for automation environments that call the PowerShell CLI to preset unconditional (always) coloring (e.g., from a POSIX-like shell; name negotiable, but it should be reasonably short: PSSTYLE_MODE=always pwsh -NoProfile -File ...)

Update: #10811 (comment) mentions @lzybkr proposing a new CLI parameter:

@lzybkr had a good suggestion to have a param on pwsh like -color with always, auto, never values.

@jhoneill above suggested a new common parameter for cmdlet-individual control (caveat is that unless this parameter is opt-in, conflicts with existing user-defined parameters are possible).

[mklement0]

@SteveL-MSFT, this issue mistakenly links to itself under "Related Issues"; I think you meant to link to #10811.

[vexx32]

This already seems kind of similar to what @Jaykul has done with PANSIES, only less flexible and harder to work with. In that module, a PSProvider is added that is capable of generating escape codes for common (named) colors as well as arbitrary hex color values. That can then be used mid-string to insert values in a flexible way.

If users are still going to have to look up the escape sequences for these settings / download a third party module to handle that anyway, I'm not sure this really adds much. Having a base theming framework is a good start, but without some more user-friendly features (VT escapes are not user friendly, period) I don't see it getting a ton of use.

[SteveL-MSFT]

@mklement0 you're right, had the wrong URL in my clipboard at the time. Fixed.

[SteveL-MSFT]

@vexx32 the $PSStyle variable is explicitly to handle not knowing ESC sequences and allowing for tab-completion in strings. It also has the RGB method for arbitrary 24-bit color.

[SteveL-MSFT]

@SeeminglyScience I think those are interesting style names and can see how they may be more future proof. Were you thinking those would replace the ones I have under the Formatting member or in addition? How would you map yours to the existing ones (some are obvious, others not as much like ErrorAccent).

[mklement0]

@vexx32, as implied by @SteveL-MSFT's comment, unless you want to redefine the default colors (which probably isn't a good idea) or redefine the $PSStyle.Formatting properties, you won't have to deal with escape sequences.

However, in order to better support these - advanced - use cases, perhaps we could extend Write-Host to support all the styles supported by $PSStyle and add an -AsDecorated switch that outputs the result as a string with escape sequences.