console redirection is broken

[jkotas]

From @Spongman on June 15, 2017 21:5

using System;
class Test {
  public static void Main () {
    while(true)
      Console.WriteLine(Console.ReadLine());
  }
}

stdout redirection:

test command: dotnet run > test.txt
expected behavior is to show typed text in the console/tty, and to write the typed lines out to test.txt

on windows: works fine.
on linux: writes stdin to the file ok, but the typed text is not echoed to tty.

stdin redirection:

test command: dotnet run < test.txt (test.txt exists and contains some lines of text)
expected behavior is to write the contents of test.txt to the console/tty.

on both windows & linux: console clears and the CPU is pegged!!

version: 1.0.4

Microsoft .NET Core Shared Framework Host

  Version  : 1.1.0
  Build    : 928f77c4bc3f49d892459992fb6e1d5542cb5e86

Copied from original issue: dotnet/coreclr#12308

[Spongman]

ugh, please forget the 'stdin redirection' part.

[joshfree]

related https://github.com/dotnet/corefx/issues/25916

[carlossanlop]

Triage:

on linux: writes stdin to the file ok, but the typed text is not echoed to tty.
@eiriktsarpalis and I were able to reproduce this issue. @Spongman would you be interested in investigating this issue further?

[cfbao]

To add some more details:

For stdout redirection, not only is the input not displayed, the output isn't fine either - it contains the user input + lines written by Console.WriteLine().
I.e. for such a program

Console.WriteLine(Console.ReadLine());

If the user enters "abc" when prompted, it'd behave like this

> ./test > output
> cat output
abc
abc

---

More context on this issue's impact:

We're rewriting a CLI app from Go to C#, and it's a requirement that user input isn't written to stdout, because one main use case for this app is that user will run it inside $(...), answer a few interactive prompts, and the app will output commands that are directly executed. Having stdin redirected to stdout breaks the output commands, and causes errors.

[cfbao]

In case this issue won't be fixed soon, any comment on this potential workaround on Linux & Mac?

var stdinReader= new StreamReader(new FileStream(
    new Microsoft.Win32.SafeHandles.SafeFileHandle(0, ownsHandle: false),
    FileAccess.Read
));
stdinReader.ReadLine();

Should ownsHandle be true or false in this case? Or maybe it doesn't matter if stdinReader lives for the entire lifetime of the app?

[adamsitnik]

Should ownsHandle be true or false in this case?

ownsHandle tells us whether the handle should be released when it's disposed (in explicit way) and/or finalized (when it's no longer used by anyone and GC collects memory). In case of handle to standard input, it should rather be false (it will be released by the OS, when the app exits), so other parts of the code that can potentially use it won't report an exception when they try to.

[cfbao]

I might have an idea of what's going on with user terminal input when stdout is redirected.

When Console.ReadLine() is called and stdin isn't redirected, the runtime calls StdInReader.ReadLineCore, which first sets the terminal to a sort of raw mode (so every key stroke is captured & user input isn't directly displayed on screen):

runtime/src/native/libs/System.Native/pal_console.c

Line 188 in 6cb44ff

termios.c_lflag &= (uint32_t)(~(ECHO | ICANON | IEXTEN));

then read input byte-by-byte, and prints the characters back out using Console.Write:

runtime/src/libraries/System.Console/src/System/IO/StdInReader.cs

Line 254 in 3eb4407

Console.Write(keyInfo.KeyChar);

This of course means that user input will end up in stdout (redirected or not) and not on the terminal (when stdout is redirected).

It's a similar process for other read methods like Console.ReadKey.

Might be a naive suggestion, but can the runtime prints user input back out to /dev/tty instead of stdout to resolve the first part of this issue?

[cfbao]

Just noticed that the "user input ending up in stdout" problem can occur on Windows too, though to a much smaller extent - only when Console.ReadKey is called.
The problematic code in the runtime:

runtime/src/libraries/System.Console/src/System/ConsolePal.Windows.cs

Lines 397 to 398 in 9f8f653

	if (!intercept)
	Console.Write(ir.keyEvent.uChar);

[tmds]

This of course means that user input will end up in stdout (redirected or not) and not on the terminal (when stdout is redirected).

That's right. These writes are are meant to emulate a terminal ECHO. When the output is redirected, we shouldn't write these characters.

@cfbao do you want to make a PR that implements that change?

[cfbao]

That's right. These writes are are meant to emulate a terminal ECHO. When the output is redirected, we shouldn't write these characters.

To confirm - When stdout is redirected, I believe users would still expect whatever they type on the terminal to appear on screen (just not end up in stdout), so do we want to write their input to /dev/tty instead? (or maybe a better approach that I'm not aware of)

@cfbao do you want to make a PR that implements that change?

I'm interested, but will probably need some handholding 😅

[tmds]

I believe users would still expect whatever they type on the terminal to appear on screen (just not end up in stdout), so do we want to write their input to /dev/tty instead?

Yes, that is desired.

I'm interested, but will probably need some handholding

If you want to have a go at it, I'll assist you.

The documentation here explains how you can build and test on your machine.

[krwq]

@cfbao would be much appreciated to get your help 😄

[tmds]

@cfbao feel free to pick this up, and I'll provide guidance.
Otherwise, I can look into this in October or so.

[tmds]

I've made a PR that addresses the stdout redirection issue: #94414.

The stdin redirection issue is not a real issue. The problem is that the while loop doesn't quit when ReadLine returns null.