System.Console.Read() needlessly changes terminal settings on linux

[webczat]

I have noticed that when using Console.Read on linux, the method disables canonical mode (ICANON) flag when reading, may apply to mac too.
Generally while ReadKey reads key by key and to do this it's needed to disable canonical mode and possibly echo when requested, read reads single characters, but actually blocks until newline before starting to return characters, so works similarly to scanf/cin/whatever in other languages.
However, linux itself would handle that just fine in canonical mode, so that the Read call does not have to turn it off. Turning it off makes it impossible to, for example, erase typed characters before pressing return key, or use standard terminal editing features like ctrl+u to erase line.
In canonical mode linux would start returning characters one by one, but only when the user presses enter on a terminal. Pretty sure it's standard unix behavior, and likely same for OSX.
Of course it is also an expected behavior when the program behaves like standard unix tools and just reads from standard input, if user runs it interactively, input is accepted line by line even if the program is not really doing line based input parsing. That way user still has limited edit capability but the program itself still has freedom when it goes to the way it reads input.

[ghost]

Tagging subscribers to this area:
See info in area-owners.md if you want to be subscribed.

Issue Details

---

I have noticed that when using Console.Read on linux, the method disables canonical mode (ICANON) flag when reading, may apply to mac too.
Generally while ReadKey reads key by key and to do this it's needed to disable canonical mode and possibly echo when requested, read reads single characters, but actually blocks until newline before starting to return characters, so works similarly to scanf/cin/whatever in other languages.
However, linux itself would handle that just fine in canonical mode, so that the Read call does not have to turn it off. Turning it off makes it impossible to, for example, erase typed characters before pressing return key, or use standard terminal editing features like ctrl+u to erase line.
In canonical mode linux would start returning characters one by one, but only when the user presses enter on a terminal. Pretty sure it's standard unix behavior, and likely same for OSX.
Of course it is also an expected behavior when the program behaves like standard unix tools and just reads from standard input, if user runs it interactively, input is accepted line by line even if the program is not really doing line based input parsing. That way user still has limited edit capability but the program itself still has freedom when it goes to the way it reads input.

Author:	webczat
Assignees:	-
Labels:	area-System.Console, untriaged
Milestone:	-

[webczat]

after some investigation I cannot quite find the code responsible for the difference of behavior, but it seems like even ReadLine disables canonical mode and has it's own handling for things like backspace/etc, but ctrl+u does not work.
That makes the thing behave a little bit unnaturally on unix.
Is there any particular reason why it's not directly handled by the os? Not sure how console reading works on windows, but in unix in default canonical mode ReadLine could be implemented as Read() till eol, and Read() could just Read from stdin. Unless that makes some functionality stop working.
For example would ReadKey work properly assuming it would constantly switch canonical mode off/on per call? Not sure if it switches echo that way or that's also directly handled by the ReadKey logic.

[stephentoub]

cc: @tmds

[tmds]

@webczat yes, .NET disables canonical mode when Console is used. And ReadLine is implemented by calling ReadKey and doing some limited processing. It has been like this since .NET Core 1.0.

We can see what happens if we enable canonical mode while doing ReadLine (and disable it when ReadKey). There may be some side-effects and the best way to find out is by trying.

Is this something you'd want to experiment with? If not, I or someone else can take a look.

The relevant code lives in https://github.com/dotnet/runtime/blob/main/src/libraries/System.Console/src/System/IO/StdInReader.cs. Look for InitializeConsoleBeforeRead.

cc @eiriktsarpalis

[webczat]

mm, I found the code yesterday, just not specifically the difference between Read() and ReadLine().
I am not sure if I feel competent enough to make changes here.
What I am afraid of is possible races between successive ReadKey calls because they would switch terminal to and from canon back and forth, back and forth... Especially if you disable echo on ReadKey, wouldn't the result be that some keys would be displayed when pressed fast enough?
On native linux console apps you would normally just disable canon once and set other terminal settings like disable echo, and then you can read keys freely, but it probably does not fit the model here...

[tmds]

What I am afraid of is possible races between successive ReadKey calls because they would switch terminal to and from canon back and forth, back and forth... Especially if you disable echo on ReadKey, wouldn't the result be that some keys would be displayed when pressed fast enough?

Ah yes. ReadLine would need to echo, and ReadKey(true) would need to not echo. Flipping the echoing on an off between these calls may cause characters to disappear or appear unintentionally.

I guess Windows doesn't effectively echo until there is a read call, while Linux echos as the user is typing (even when there is no read call yet).

[webczat]

I didn't read enough to see whether you disable echo and manually echo yourself, or really toggle echo on ReadKey currently? it's not the ICANON flag, but similar problem.

[tmds]

Independent of how you Read from Console, the configuration is: disable echo, and disable canon:

runtime/src/libraries/Native/Unix/System.Native/pal_console.c

Line 185 in 9849163

termios.c_lflag &= (uint32_t)(~(ECHO | ICANON | IEXTEN));

[webczat]

not sure how this can be solved, if at all, without breaking something.
Was wondering about things like leave canon/echo enabled and disable it before first readkey, then leave it disabled until read/readline,
or, disable it as usual but reenable both on read/readline then disable again, both of them actually are newline delimited.
Both cases look a bit unnatural on linux, although there is probably no value in being able to see pressed arrow characters when nothing is trying to read at the moment... and I doupt apps using ReadKey mix it with ReadLine, and especially not with Read.

[webczat]

oh, they likely do for things like press any key to continue. but programs affected by terminal state when doing successive readkey may not

[webczat]

how did mono handle that? or ... how does it handle that now...
As in, it was more linux oriented back in the days and it's console code could handle this correctly. Of course I don't know whether it did.

[tmds]

I took a look at the mono sources, I see it also disables echo and disables canonical mode. So the behavior is similar to that of .NET.

[webczat]

it's actually frustrating, because that way it doesn't behave like a native linux app. Not quite sure how likely are you to create an app that depends on it, although I definitely hit one case where it made a difference that backspace didn't work, and backspace didn't work when using the Read() method. And things like ctrl+u not working, kinda useful shortcut.

[tmds]

@webczat it was an interesting discussion. Things behave the way they do to be closer to Windows behavior, which makes it less native Linux like. If you care about it for an app you're building, you can try calling tcsetattr using https://github.com/tmds/Tmds.LibC before you use System.ReadLine to enable ECHO and ICANON.

I don't think we found a way to improve what .NET is doing, so I think we can close this issue?

[webczat]

well the frustrating part is that it's something that would never require any action if it was not dotnet doing such things. I understand the thing about windows compat, although of course it would be nice to be able to make both sides happy here when not running on windows, if there's any way to do it that doesn't break things.
The actual trigger for this issue was the fact that when calling ReadLine() I could use backspace key to erase chars, but when calling Read(), even though it actually also has to first buffer a full line, backspace didn't work, and it broke my use case.

[tmds]

well the frustrating part is that it's something that would never require any action if it was not dotnet doing such things. I understand the thing about windows compat, although of course it would be nice to be able to make both sides happy here when not running on windows, if there's any way to do it that doesn't break things.

From our discussion, the fundamental difference is that Windows delays echoing until there is a Read call, and there is no way to make Linux delay the echoing until the user calls read. The echoing already happens as the user is typing, and we can't undo it if the user calls an API that is not meant to echo.

The actual trigger for this issue was the fact that when calling ReadLine() I could use backspace key to erase chars, but when calling Read(), even though it actually also has to first buffer a full line, backspace didn't work, and it broke my use case.

I think this is a bug due to not proper handling Backspace when consumeKeys is false here:

runtime/src/libraries/System.Console/src/System/IO/StdInReader.cs

Lines 176 to 180 in 877a8df

	else if (keyInfo.Key == ConsoleKey.Backspace)
	{
	int len = _readLineSB.Length;
	if (len > 0)
	{

I will take a look at fixing this.

[webczat]

wondering if it would be feasible to actually enable canon/echo when doing Read/ReadLine and disable them before ReadKey. wondering about possible racy behaviors in that case. as in the reverse of having it enabled by default and disabling on ReadKey.

[stephentoub]

disable them before ReadKey

It was originally that way. But the user can be typing long before ReadKey is called.

[webczat]

ehh...

[webczat]

well I meant more like, disable them before readkey if changed, but don't enable them by default. although even that could pose a problem.

[tmds]

wondering if it would be feasible to actually enable canon/echo when doing Read/ReadLine and disable them before ReadKey. wondering about possible racy behaviors in that case. as in the reverse of having it enabled by default and disabling on ReadKey.

When ReadKey is followed by ReadLine, characters from the line may end up not echoed.
When ReadLine is followed by ReadKey, the read key char may end up echoed.