Override the color translation heuristic by detecting terminal colors with OSC codes

[rprichard]

winpty currently uses a color translation heuristic to translate console colors into terminal colors. It's described in detail here. Basically, the console always use LtGray-on-Black (overriding the default), even if the terminal is Black-on-White, Green-on-Black, etc.

So far, I haven't had any complaints about the heuristic, but even still, if it's possible to get the colors exactly right, it seems worth trying.

There are a few issues that make color translation difficult:

1. (In theory) Some Windows programs might not behave properly with a non-default color scheme. I'm not sure whether this is an issue in practice, but if it were, a winpty user might appreciate the current translation strategy.
2. winpty doesn't know what colors the terminal is using.
3. Forcing the terminal text to have the same color as the console looks very ugly, even when the color schemes mostly match.

It turns out that there are OSC terminal codes that would let winpty query the color table, solving # 2:

• ESC ] 10 ; ? ; ESC \ produces the foreground color (as an RGB color).
• ESC ] 11 ; ? ; ESC \ produces the background color.
• ESC ] 4 ; nn ; ESC \ produces palette color nn.

Here's how I think it would work. At startup, winpty would print 18 OSC codes--two for foreground and background, 16 for the palette. Once it received the replies, it would search for the two terminal palette colors closest to the terminal fore/back colors. It would configure the console to use those two colors as its fore/back colors, then it would map those two console colors to the terminal's default coloring (i.e. "no coloring"). The other 14 console colors would map to their corresponding terminal colors.

Some difficulties:

• Many terminals don't reply to these OSC commands. I think I can work around this using the same DSR trick that I'm using to recognize ESC keypresses. (i.e. print ESC [ 6 n and wait for the row+col reply.)
• Some terminal don't even reply to DSR. So far, I only know of the Cygwin Console (i.e. TERM=cygwin). I think I can work around that terminal by whitelisting TERM values.
• Nonetheless, some terminal might exist that sets TERM=xterm and doesn't reply to anything. In that case, it's not clear how initialization works. I think I need to clear the screen, with the appropriate fore/back color, before starting the user's child process. I suppose winpty could wait a second or two, but that'd be awful.

Alternatively, there could be an interface to winpty for specifying the foreground and background colors. Perhaps a command-line argument to console.exe or an environment variable. (e.g. --color-scheme LtGray-on-Black)

[mkq]

Well, here's the missing "complaint". If this enhancement made winpty a better workaround for this gradle issue, I would appreciate it.