Win10 New Console: Enable ENABLE_VIRTUAL_TERMINAL_PROCESSING by default (or with a flag)

[rprichard]

Windows 10's console recently began interpreting escapes, but only with ENABLE_VIRTUAL_TERMINAL_PROCESSING, which is off by default. Maybe winpty should turn it on for its underlying console. Or, at the least, there could be a command-line flag to turn it on.

My use case was running Clang under Ninja under winpty. I configured Clang to output ANSI escapes, and it seems that Ninja wrote the escapes to the console (e.g. with WriteFile) rather than interpret them. If ENABLE_VIRTUAL_TERMINAL_PROCESSING were enabled, I think the escapes would still work.

[fpqc]

MS's built in processing is kind of suuuuuuper broken atm. I trust your escape processing over theirs any time. The other problem is that you would need to somehow push the escapes to the terminfo databases, and MS's escape sequence interpretation is not yet stable.

[rprichard]

The issue is what happens when a console program uses WriteConsole (or WriteFile) to write an escape sequence:

1. If ENABLE_VIRTUAL_TERMINAL_PROCESSING is on, then Windows (i.e. conhost.exe) interprets the escape sequence. If a program writes \e[32m, then Windows changes the current foreground color to Green.
2. If ENABLE_VIRTUAL_TERMINAL_PROCESSING is off (the default and pre-Win10 behavior), then Windows writes each character of the sequence to a successive cell of the screen buffer. If it's written near the end of a line, then it will even wrap around to the next line. winpty doesn't see the WriteConsole calls directly--instead it scrapes the contents of the screen buffer--so it's not really in a position to fix this behavior. Therefore, when it sees an invisible ESC character in a console cell, it converts it to a question mark, and the user see a garbled escape sequence.

In other words, the ENABLE_VIRTUAL_TERMINAL_PROCESSING flag isn't choosing between a winpty or Microsoft implementation. Only conhost.exe (or some other program that intercepts WriteConsole calls) can successfully interpret the escape sequences.

[fpqc]

Alright, just as long as you are aware that MS's escape coverage of xterm esc sequences is buggy and incomplete. I know this from extensive use of bash.exe in the default console. It won't do anything unless you export a different TERM (xterm instead of cygwin).

[CMCDragonkai]

I'm guessing this is not an environment variable but something that needs to be defined in the C source code. I had ghci giving back weird characters because I ported it from my linux one which used escape codes. Would be interested to see what it looks like with the option switched on.

[fpqc]

@CMCDragonkai Yep, it's something you define in the sourcecode that needs to be set on the I/O handles to the console.

[rprichard]

Most of the programs I've seen that try to turn ENABLE_VIRTUAL_TERMINAL_PROCESSING on don't turn it back off. I tested these programs:

• Docker version 17.03.1-ce, build c6d412e
• node.js v7.8.0
• PowerShell (Windows 10.0.15063)
• CMD (Windows 10.0.15063)

All four set the VT mode flag when they started.

CMD seemed to do something extra -- when it spawned a child process, it reset the console mode to the value it had on CMD start. Once the child exited, CMD set the VT mode flag again. CMD didn't restore the original mode when it exited.

I also ran PowerShell, then a Cygwin 2.7.0 bash subshell under that, and finally another PowerShell subshell under the two. When Cygwin bash started, and when I exited the nested PowerShell, Cygwin turned the VT mode flag off. The topmost PowerShell instance did not turn the VT flag back on when I exited Cygwin bash.

FWIW, I'm more inclined now to have winpty turn the flag on.

[rprichard]

This link is interesting: https://wpdev.uservoice.com/forums/266908-command-prompt-console-bash-on-ubuntu-on-windo/suggestions/15617610--re-enable-enable-virtual-terminal-processing-by

[rprichard]

Actually, since Cygwin turns the VT flag off, I'm more inclined to have winpty do nothing by default. Maybe there could be an optional flag, though.

[dragonwolf83]

This sounds like PowerShell/PowerShell#1177 which was fixed in PowerShell 6.0 Alpha 15 FWIW.

[rprichard]

@dragonwolf83 Thanks for the link. It looks like PowerShell is going to behave more like the CMD behavior I described above, where the shell cleans up after programs that change the VT flag.

The commit message also restates the idea that applications need to opt-in to VT100 support:

An application can't rely on the shell enabling VT100 because enabling
by default can cause problems for some applications, therefore
Windows applications must enable VT100 themselves.