Output errors after setting text red, then normal while in raw mode

[clinton-r]

Environment

Microsoft Windows [Version 10.0.19041.685]
Windows Terminal Version: 1.4.3243.0
Microsoft Visual Studio Community 2019 Version 16.8.3

Steps to reproduce

1. Compile the program below
2. Open Windows Terminal, with tabs for both cmd and Powershell
3. Run the program in the cmd tab
4. Run the program in the Powershell tab
5. Type 'exit' in the Powershell tab
6. Open conhost running cmd
7. Run the program in conhost

#include <stdio.h>
#include <stdlib.h>
#include <windows.h>

static HANDLE hStdin = NULL;
static HANDLE hStdout = NULL;
static int savedConsoleOutputModeIsValid = 0;
static DWORD savedConsoleOutputMode = 0;
static int savedConsoleInputModeIsValid = 0;
static DWORD savedConsoleInputMode = 0;

static void restoreConsoleState(void)
{
    if (savedConsoleOutputModeIsValid)
    {
        SetConsoleMode(hStdout, savedConsoleOutputMode);
    }
    if (savedConsoleInputModeIsValid)
    {
        SetConsoleMode(hStdin, savedConsoleInputMode);
    }
    //printf("\r\nRestored console state\r\n");
}

#define USE_WRITECONSOLE 0

void writeString(char *str)
{
#if USE_WRITECONSOLE
    WriteConsoleA(hStdout, str, strlen(str), NULL, NULL);
#else
    printf(str);
#endif
}

int main(void)
{
    DWORD newInputMode;
    DWORD newOutputMode;

    // Make sure the console state will be returned to its original state
    // when this program ends
    atexit(restoreConsoleState);

    // Get handles for stdin and stdout
    hStdin = GetStdHandle(STD_INPUT_HANDLE);
    if (hStdin == INVALID_HANDLE_VALUE || hStdin == NULL)
    {
        puts("Failed to get handle for stdin.  Exiting.");
        return 1;
    }
    hStdout = GetStdHandle(STD_OUTPUT_HANDLE);
    if (hStdout == INVALID_HANDLE_VALUE || hStdout == NULL)
    {
        puts("Failed to get handle for stdout.  Exiting.");
        return 1;
    }

    // Get original modes to be restored later

    if (!GetConsoleMode(hStdout, &savedConsoleOutputMode))
    {
        puts("Failed to get console mode for stdout.  Exiting.");
        return 1;
    }
    savedConsoleOutputModeIsValid = 1;

    if (!GetConsoleMode(hStdin, &savedConsoleInputMode))
    {
        puts("Failed to get console mode for stdin.  Exiting.");
        return 1;
    }
    savedConsoleInputModeIsValid = 1;

    // Print message normal, then red, then normal
    char buf[200];
    sprintf_s(buf, sizeof(buf), "\nInput mode 0x%08x, Output mode 0x%08x\n", (int)savedConsoleInputMode, (int)savedConsoleOutputMode);
    writeString(buf);
    writeString("This text should be normal\n");
    writeString("\x1b[31m");
    writeString("I think (?) this text should be red\n");
    writeString("\x1b[0m");
    writeString("This text should be normal\n");

    // Enable escape sequences

    newOutputMode = savedConsoleOutputMode;
    newOutputMode |= ENABLE_VIRTUAL_TERMINAL_PROCESSING;
    if (!SetConsoleMode(hStdout, newOutputMode))
    {
        puts("Failed to set console mode for stdout.  Exiting.");
        return 1;
    }

    newInputMode = savedConsoleInputMode;
    newInputMode |= ENABLE_VIRTUAL_TERMINAL_INPUT;
    if (!SetConsoleMode(hStdin, newInputMode))
    {
        puts("Failed to set console mode for stdin.  Exiting.");
        return 1;
    }

    // Print message normal, then red, then normal
    sprintf_s(buf, sizeof(buf), "\nInput mode 0x%08x, Output mode 0x%08x\n", (int)newInputMode, (int)newOutputMode);
    writeString(buf);
    writeString("This text should be normal\n");
    writeString("\x1b[31m");
    writeString("This text should be red\n");
    writeString("\x1b[0m");
    writeString("This text should be normal\n");

    // Set console to "raw" mode

    newOutputMode = savedConsoleOutputMode;
    newOutputMode &= ~ENABLE_PROCESSED_OUTPUT;
    newOutputMode &= ~ENABLE_WRAP_AT_EOL_OUTPUT;
    newOutputMode |= ENABLE_VIRTUAL_TERMINAL_PROCESSING;
    if (!SetConsoleMode(hStdout, newOutputMode))
    {
        puts("Failed to set console mode for stdout.  Exiting.");
        return 1;
    }

    newInputMode = savedConsoleInputMode;
    newInputMode &= ~ENABLE_ECHO_INPUT;
    newInputMode &= ~ENABLE_LINE_INPUT;
    newInputMode &= ~ENABLE_PROCESSED_INPUT;
    newInputMode |= ENABLE_VIRTUAL_TERMINAL_INPUT;
    if (!SetConsoleMode(hStdin, newInputMode))
    {
        puts("Failed to set console mode for stdin.  Exiting.");
        return 1;
    }

    // Print message normal, then red, then normal
    sprintf_s(buf, sizeof(buf), "\r\nInput mode 0x%08x, Output mode 0x%08x\r\n", (int)newInputMode, (int)newOutputMode);
    writeString(buf);
    writeString("This text should be normal\r\n");
    writeString("\x1b[31m");
    writeString("This text should be red\r\n");
    writeString("\x1b[0m");
    writeString("This text should be normal\r\n");

    return 0;
}

Expected behavior

I expect to see as output in all 3 cases:

Input mode 0x000001f7, Output mode 0x00000007
This text should be normal
I think (?) this text should be red
This text should be normal

Input mode 0x000003f7, Output mode 0x00000007
This text should be normal
This text should be red
This text should be normal

Input mode 0x000003f0, Output mode 0x00000004
This text should be normal
This text should be red
This text should be normal

In the case of the Powershell tab I also expect that when 'exit' is typed it will be echoed to the right of the prompt after the above output.

Actual behavior

In all 3 cases, the final "This text should be normal" line is truncated.
In the Powershell tab when 'exit' is typed, the first 2 characters 'ex' are echoed where they should be, then the cursor jumps up 5 lines and the complete word 'exit' is written there.

Extra notes and questions

Happens whether using printf() or WriteConsoleA()

Am I using the modes properly? I'm just learning about console programming...

Also, is VT output supposed to be disabled when conhost is first started? That surprised me.

Thanks!

[DHowett]

Hey @clinton-r! Thanks for reaching out.

So, here's the skinny.

You almost never want to turn off ENABLE_PROCESSED_OUTPUT on your output handle. This is doubly true if you're trying to use control sequences (like the ones you have above for changing the color) or using ENABLE_VIRTUAL_TERMINAL_PROCESSING.

When you turn off ENABLE_PROCESSED_OUTPUT, the control sequences are written directly to the console buffer, as you can see in your second screenshot in conhost. Once they've been inserted into the backing buffer as-is, the escape characters and other control sequences take up physical space and do not get interpreted.

Thanks to a choice we made in the console hosting API that Terminal (and Visual Studio, VS Code, Alacritty, ...) uses, those control sequences are passed through directly from the console session into the terminal. Now, this is where things go off the rails. The terminal takes a second chance to parse them. This is clearly a mistake (on our end).

The terminal, when it parses them, is effectively taking what the console host told it ("put this literal escape character HERE, and this one HERE") and corrupting the buffer contents by trying to interpret those control sequences. That is tracked by #4363.

In short, always use PROCESSED_OUTPUT and TERMINAL_PROCESSING at the same time when you're using VT.

---

As for your other question about why conhost and Terminal act differently, that's an artifact of our compatibility history. The console host must start up in the mode most likely to keep old applications working. The terminal doesn't really have the same constraint, so it starts out by default with VIRTUAL_TERMINAL_PROCESSING enabled.

CMD and PowerShell do some additional work on top of that, because they do want VIRTUAL_TERMINAL_PROCESSING. If they turn it on (and they were the first ones to turn it on), they turn it back off before running another application. Again, for maximal compatibility.

If you have an application that is sensitive to these things, you should just always enforce the specific modes that work best for your app. 😄

We're working on making the output modes per-process so that they don't need to be saved/restored by each/every application (which is clearly crazy). That work is tracked in #4954.

Hope that helps! I'm gonna close this one out as an answered question, but do feel free to ask more.