.NET Console.WriteLine throws 'The handle is invalid.' on large output when using ENABLE_VIRTUAL_TERMINAL_PROCESSING

[gerardog]

Environment

Windows build number: 10.0.18362.356
Windows Terminal version (if applicable): Using windows conHost or Terminal 0.7.3291.0
Visual Studio 2019 Professional 16.3.7
Net Framework 4.6, 4.6.1 or 4.7.2

Steps to reproduce

Create a simple C# Process that set console mode ENABLE_VIRTUAL_TERMINAL_PROCESSING, and then writes unlimited output to StdOut.
Eventually StdOut gets broken and throws System.IO.IOException: The handle is invalid.

C# Repro code:

using Microsoft.Win32.SafeHandles;
using System;
using System.Runtime.InteropServices;

class Program
{
    static void Main(string[] args)
    {
        if (SetConsoleModeVT())
        {
            for (int i = 0; ; i++)
            {
                try
                {
                    Console.WriteLine(i);
                }
                catch (Exception ex)
                {
                    Console.Error.WriteLine(ex.ToString());
                    Console.ReadLine();
                }
            }
        }
    }

    private static bool SetConsoleModeVT()
    {
        var hStdOut = GetStdHandle(STD_OUTPUT_HANDLE);
        if (!GetConsoleMode(hStdOut, out uint outConsoleMode))
        {
            return false;
        }

        outConsoleMode |= ENABLE_VIRTUAL_TERMINAL_PROCESSING;
        if (!SetConsoleMode(hStdOut, outConsoleMode))
        {
            return false;
        }

        return true;
    }

    internal const int STD_OUTPUT_HANDLE = -11;
    internal const uint ENABLE_VIRTUAL_TERMINAL_PROCESSING = 0x0004;

    [DllImport("kernel32.dll", SetLastError = true)]
    internal static extern SafeFileHandle GetStdHandle(int nStdHandle);

    [DllImport("kernel32.dll", SetLastError = true)]
    internal static extern bool SetConsoleMode(SafeFileHandle hConsoleHandle, uint mode);

    [DllImport("kernel32.dll", SetLastError = true)]
    internal static extern bool GetConsoleMode(SafeFileHandle handle, out uint mode);
}

Expected behavior

The app should not throw IOException.

Actual behavior

This is the output:

1
2
(...)
163543
163544
163545
System.IO.IOException: The handle is invalid.

   at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
   at System.IO.__ConsoleStream.Write(Byte[] buffer, Int32 offset, Int32 count)
   at System.IO.StreamWriter.Flush(Boolean flushStream, Boolean flushEncoder)
   at System.IO.StreamWriter.Write(String value)
   at System.IO.TextWriter.Write(UInt64 value)
   at System.IO.TextWriter.WriteLine(UInt64 value)
   at System.IO.TextWriter.SyncTextWriter.WriteLine(UInt64 value)
   at System.Console.WriteLine(UInt64 value)
   at ConsoleApp5.Program.Main(String[] args) in C:\Users\Me\source\repos\ConsoleApp5\Program.cs:line 17

The exception is only thrown with ENABLE_VIRTUAL_TERMINAL_PROCESSING console mode.
Either launching the app from a Windows CMD/ConHost, or the new Windows Terminal/PowerShell.

Thanks!

[gerardog]

If I add a Thread.Sleep(60 seconds) inside the for loop, it also throws after just 5 minutes. So this does not seems to be related to the amount of output.
I was also able to reproduce it using Windows Preview version 2004 (build 19033.1) from the fast ring, and/or .Net Core 3.0.

[amd4dnow]

i've the same issue, but have no idea how to solve it.

[gerardog]

@amd4dnow I have no idea either, but for what it's worth:

(A non-recommended way to fix this issue that will cause long-term application stability issues follows.)

What I found out is that when using Windows Terminal, Cmder or ConEmu, I can send VT sequences and they get processes even without setting ENABLE_VIRTUAL_TERMINAL_PROCESSING flag. So my current code detects those terminals (so far, reading WT_SESSION or ConEmuANSI environment variables) and only then send VT sequences.

[DHowett-MSFT]

@gerardog please, whatever you do, don't do that. You'll run right into #1965 which will cause terribly unusual behavior when you reach the edge of the screen.

[DHowett-MSFT]

ConPTY can't pass raw escape sequences in a way that the terminal can differentiate from escape sequences that are in the text buffer, and if they're in the text buffer they're subject to wrapping and splitting.

[gerardog]

Thanks for the information @DHowett-MSFT . Do you have any insight about how to workaround the The handle is invalid exception in the meantime? Is this bug rooted on ConHost? or Kernel32.dll? Does plain C apps work fine with this flag?

[DHowett-MSFT]

We'll have to have a look-- sorry, things have been slow on account of the impending holidays 😄

[nszeitli]

I am getting the same error when writing to console in an ASP.NET core 3.0 app after a long running process. It appears to be related to high memory usage but not sure.

[ogamespec]

Happens to me when I try to interact with console in native DLL:

DLL code init:

	AllocConsole();
	if (freopen("CONOUT$", "w", stdout) == 0)
		return;

DLL code shutdown:

	fclose(stdout);
	FreeConsole();

[ogamespec]

For your C # application to work properly, you need to make sure that no one else touches the console.

There "somewhere else" you can use GetConsoleWindow function.

[gerardog]

@DHowett Can a .Net app render its content sending VT sequences? maybe with ENABLE_VIRTUAL_TERMINAL_PROCESSING? or with a passthrough mode #1173...
Is ENABLE_VIRTUAL_TERMINAL_PROCESSING completely broken or am I doing something wrong?

[DHowett-MSFT]

@gerardog this should definitely work.

Setting the console output handle (retrieved from GetStdHandle(STD_OUTPUT_HANDLE) or CreateFile("CONOUT$") (please only use this if you absolutely require a console handle to function. It isn't good to do this as a default for writing a console application that might be used in non-console scenarios) into ENABLE_VIRTUAL_TERMINAL_PROCESSING mode should be sufficient for sending all types of VT.

You may need to fiddle with the ..NEWLINE_AUTO_RETURN mode and related modes, but at its core this should work fine.

I just got your reproducer under a debugger, and the strangest thing is happening.

1. Your Main() is still running, and still writing to the console handle.
2. The standard output handle is being closed by a ".NET Finalizer" thread.

It looks like the SafeHandle that you used to set the output mode is being finalized when its scope ends on line 40. It's closing the only output handle when it does so.

If you replace the use of SafeHandle in GetStdHandle with IntPtr, it works perfectly fine forever as the handle isn't prematurely slain in its sleep.

Chalking this one up to user error :)

[gerardog]

Many thanks @DHowett-MSFT. You really nailed this one!