Console.ReadKey behaves differently in .NET 6 and .NET 7-preview.7 comparing to .NET 5 when TERM is set to rxvt

[daxian-dbw]

Description

Description

We got a report in PowerShell repo about "arrow keys not functioning properly in PowerShell 7.2 with rxvt terminals", see PowerShell/PowerShell#16606.

It turns out Console.ReadKey behaves differently in .NET 6 and .NET 7-preview.7 comparing to .NET 5 when TERM is set to rxvt.
In .NET 5, the ConsoleKeyInfo returned for LeftArrow doesn't has any modifiers, as shown below:

However, in .NET 6 and .NET 7-preview.7, the ConsoleKeyInfo returned for LeftArrow has Control and Shift modifiers specified, as shown below:

Note that, this issue was originally reported as #63387, which was closed as completed. However, it's not fixed in .NET 7-preview.7.

Reproduction Steps

1. Run export TERM=rxvt
2. Start PowerShell 7.2.0, or 7.2.1, or 7.3.0-preview.7
3. Try using arrow keys to navigate the cursor position

Expected behavior

LeftArrow and RightArrow should move the cursor left and right;
LeftArrow and RightArrow should go through history commands.

Actual behavior

Arrow keys are not working as expected.
Home and End prints OH and OF respectively.

Regression?

Yes.

Known Workarounds

Setting TERM=xterm would make Console.ReadKey work as expected with rxvt terminals.

Configuration

Which version of .NET is the code running on?
.NET 7-preview.7

What OS and version, and what distro if applicable?
Ubuntu 18.04

What is the architecture (x64, x86, ARM, ARM64)?
x64

Other information

No response

[ghost]

Tagging subscribers to this area: @dotnet/area-system-console
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Description

Description

We got a report in PowerShell repo about "arrow keys not functioning properly in PowerShell 7.2 with rxvt terminals", see PowerShell/PowerShell#16606.

It turns out Console.ReadKey behaves differently in .NET 6 and .NET 7-preview.7 comparing to .NET 5 when TERM is set to rxvt.
In .NET 5, the ConsoleKeyInfo returned for LeftArrow doesn't has any modifiers, as shown below:

However, in .NET 6 and .NET 7-preview.7, the ConsoleKeyInfo returned for LeftArrow has Control and Shift modifiers specified, as shown below:

Note that, this issue was originally reported as #63387, which was closed as completed. However, it's not fixed in .NET 7-preview.7.

Reproduction Steps

1. Run export TERM=rxvt
2. Start PowerShell 7.2.0, or 7.2.1, or 7.3.0-preview.7
3. Try using arrow keys to navigate the cursor position

Expected behavior

LeftArrow and RightArrow should move the cursor left and right;
LeftArrow and RightArrow should go through history commands.

Actual behavior

Arrow keys are not working as expected.
Home and End prints OH and OF respectively.

Regression?

Yes.

Known Workarounds

Setting TERM=xterm would make Console.ReadKey work as expected with rxvt terminals.

Configuration

Which version of .NET is the code running on?
.NET 7-preview.7

What OS and version, and what distro if applicable?
Ubuntu 18.04

What is the architecture (x64, x86, ARM, ARM64)?
x64

Other information

No response

Author:	daxian-dbw
Assignees:	-
Labels:	area-System.Console
Milestone:	-

[daxian-dbw]

/cc @adamsitnik

[vcsjones]

Based on the https://github.com/dotnet/runtime/commits/v7.0.0-preview.7.22375.6 tag, it's last commit was July 25th.

The fix wasn't merged until August 1st. f1de614

So it would seem the fix is not present in preview7, however I do see it in the RC1 branch.

[daxian-dbw]

Please keep this issue open until it's verified fixed with RC1 release.

[adamsitnik]

@vcsjones is right, #72193 was not included in Preview 7, but it in RC 1

Now we have tests for these particular keys for rxvt and I am 100% sure they work as expected:

Arrow keys and all modifiers:

runtime/src/libraries/System.Console/tests/KeyParserTests.cs

Lines 1088 to 1107 in 1f6ebd0

	yield return (new byte[] { 27, 91, 68 }, new ConsoleKeyInfo(default, ConsoleKey.LeftArrow, false, false, false)); // LeftArrow
	yield return (new byte[] { 27, 79, 100 }, new ConsoleKeyInfo(default, ConsoleKey.LeftArrow, false, false, true)); // Ctrl+LeftArrow
	yield return (new byte[] { 27, 27, 91, 68 }, new ConsoleKeyInfo(default, ConsoleKey.LeftArrow, false, true, false)); // Alt+LeftArrow
	yield return (new byte[] { 27, 91, 100 }, new ConsoleKeyInfo(default, ConsoleKey.LeftArrow, true, false, false)); // Shift+LeftArrow
	yield return (new byte[] { 27, 27, 91, 100 }, new ConsoleKeyInfo(default, ConsoleKey.LeftArrow, true, true, false)); // Shift+Alt+LeftArrow
	yield return (new byte[] { 27, 91, 65 }, new ConsoleKeyInfo(default, ConsoleKey.UpArrow, false, false, false)); // UpArrow
	yield return (new byte[] { 27, 79, 97 }, new ConsoleKeyInfo(default, ConsoleKey.UpArrow, false, false, true)); // Ctrl+UpArrow
	yield return (new byte[] { 27, 27, 91, 65 }, new ConsoleKeyInfo(default, ConsoleKey.UpArrow, false, true, false)); // Alt+UpArrow
	yield return (new byte[] { 27, 91, 97 }, new ConsoleKeyInfo(default, ConsoleKey.UpArrow, true, false, false)); // Shift+UpArrow
	yield return (new byte[] { 27, 27, 91, 97 }, new ConsoleKeyInfo(default, ConsoleKey.UpArrow, true, true, false)); // Shift+Alt+UpArrow
	yield return (new byte[] { 27, 91, 66 }, new ConsoleKeyInfo(default, ConsoleKey.DownArrow, false, false, false)); // DownArrow
	yield return (new byte[] { 27, 79, 98 }, new ConsoleKeyInfo(default, ConsoleKey.DownArrow, false, false, true)); // Ctrl+DownArrow
	yield return (new byte[] { 27, 27, 91, 66 }, new ConsoleKeyInfo(default, ConsoleKey.DownArrow, false, true, false)); // Alt+DownArrow
	yield return (new byte[] { 27, 91, 98 }, new ConsoleKeyInfo(default, ConsoleKey.DownArrow, true, false, false)); // Shift+DownArrow
	yield return (new byte[] { 27, 27, 91, 98 }, new ConsoleKeyInfo(default, ConsoleKey.DownArrow, true, true, false)); // Shift+Alt+DownArrow
	yield return (new byte[] { 27, 91, 67 }, new ConsoleKeyInfo(default, ConsoleKey.RightArrow, false, false, false)); // RightArrow
	yield return (new byte[] { 27, 79, 99 }, new ConsoleKeyInfo(default, ConsoleKey.RightArrow, false, false, true)); // Ctrl+RightArrow
	yield return (new byte[] { 27, 27, 91, 67 }, new ConsoleKeyInfo(default, ConsoleKey.RightArrow, false, true, false)); // Alt+RightArrow
	yield return (new byte[] { 27, 91, 99 }, new ConsoleKeyInfo(default, ConsoleKey.RightArrow, true, false, false)); // Shift+RightArrow
	yield return (new byte[] { 27, 27, 91, 99 }, new ConsoleKeyInfo(default, ConsoleKey.RightArrow, true, true, false)); // Shift+Alt+RightArrow

Home:

runtime/src/libraries/System.Console/tests/KeyParserTests.cs

Line 1072 in 1f6ebd0

yield return (new byte[] { 27, 91, 55, 126 }, new ConsoleKeyInfo(default, ConsoleKey.Home, false, false, false)); // Home

End:

runtime/src/libraries/System.Console/tests/KeyParserTests.cs

Line 1076 in 1f6ebd0

yield return (new byte[] { 27, 91, 56, 126 }, new ConsoleKeyInfo(default, ConsoleKey.End, false, false, false)); // End

Please keep this issue open until it's verified fixed with RC1 release.

@daxian-dbw RC1 is going to be released in four days (13th of September) Could you please test PowerShell with RC1 and re-open it if needed? We close all the issues once PR with a fix gets merged (it lowers the bookkeeping cost on our end).

[daxian-dbw]

@adamsitnik Thanks for confirming! I opened this issue only because the original one was locked and hence I cannot ask for confirmation. I will test with RC1 once it's out.

[daxian-dbw]

@adamsitnik This issue still reproduces with PowerShell built against .NET 7-RC1, on Ubuntu 18.04 WSL2. The env var TERM is set to rxvt as in the repro step

Here are the observed behaviors:

1. When directly running [Console]::ReadKey($true) in PowerShell, it returns the expected ConsoleKeyInfo results when pressing arrow keys (same as with PowerShell 7.2.6 that is built against .NET 6)
2. When running a console application that simply calls Console.ReadKey(true) (with .NET 7-RC1), it returns the expected ConsoleKeyInfo results when pressing arrow keys (didn't try building and running the application with .NET 6)
3. When calling Console.ReadKey(true) from PSReadLine which is running in the same PowerShell session, it returns the arrow ConsoleKeyInfo with Shift | Control modifiers. (same as with PowerShell 7.2.6 that is built against .NET 6)

This is really puzzling ☹️ I would really appreciate your help to understand the root cause of this.

[adamsitnik]

When calling Console.ReadKey(true) from PSReadLine which is running in the same PowerShell session, it returns the arrow ConsoleKeyInfo with Shift | Control modifiers. (same as with PowerShell 7.2.6 that is built against .NET 6)

@daxian-dbw I need to attach a debugger to find out why. Is there any chance you could describe how I can repro that? (What to build and how to run PSReadLine)?

[adamsitnik]

For now I've set the milestone to 8.0.0 as I am expecting that the fix will be rather in PowerShell than in .NET. If I am wrong, we are going to fix it in System.Console and backport to 7.0

[adamsitnik]

@daxian-dbw Is there any chance that PSReadLine uses tcsetattr to set some custom Terminal settings? Currently it's the only scenario I see

[daxian-dbw]

1. Download the package powershell-7.3.0-preview.8-linux-x64.tar.gz that is built with .NET SDK 7.0.100-rc.1.22431.12 to your Linux environment (e.g. I'm using WSLv2 with Ubuntu 18.04) from this Microsoft OneDrive link: https://microsoft-my.sharepoint.com/:u:/p/dongbow/EY9x6tF4wFpGktSbijYKblUBk2P-EqT1iwSFLFwDmINFjw?e=Ebgsw2
2. Extract the package to a folder, say ~/ps/
3. Open bash and run export TERM=rxvt. Then run ~/ps/pwsh -noprofile to start PowerShell without loading the profile file in case there is one in your environment.
4. PSReadLine will be automatically loaded when PowerShell starts up, and now, press the arrow keys, and you will find they don't work as expected.

If you don't need the PSReadLine code for your debugging, then these 4 steps are all you need to reproduce the issue. If you want to set breakpoints in PSReadLine and see the ConsoleKeyInfo returned from Console.ReadKey(), then the following steps are needed:

1. Clone the PSReadLine repository in your Linux environment: git clone https://github.com/PowerShell/PSReadLine.git
2. Run code <PSReadLine-repo-root>/PSReadLine to open the project in VSCode (use "Remote - WSL" extension in case of using WSL)
3. Run ~/ps/pwsh, then cd into the root directory of your local PSReadLine repo
4. Run ./build.ps1 -Bootstrap
5. Run ./build.ps1 -Framework net6.0 to build, and the module will be bin-placed at ./bin/Debug/PSReadLine/
   • The official PSReadLine is built against .NET Framework 4.6.2, and the produced assemblies can work on both Windows PowerShell 5.1 (.NET Framework) and PowerShell 7+ (.NET). For development on Linux/macOS, we make it build against .NET 6, but it doesn't matter because we will load the assembly in PowerShell, and thus it will be using .NET 7-RC1 runtime at execution.
6. Start another bash and run export TERM=rxvt. Then run ~/ps/pwsh -noprofile -NonInteractive. This will start the 7.3-preview.8 powershell without automatically loading its built-in PSReadLine.
7. Within that powershell session, run Import-Module '<PSReadLine-repo-root>/bin/Debug/PSReadLine' to load the PSReadLine we just built.
8. In VSCode, set a breakpoint at line 58 in the file CharMap.cs
9. From VSCode, attach to the powershell process (you can run $pid in powershell to get the process id).
10. Within that powershell session, press any of the arrow keys and the breakpoint will be hit.

[daxian-dbw]

Is there any chance that PSReadLine uses tcsetattr to set some custom Terminal settings?

No, PSReadLine doesn't set custom terminal settings.

[adamsitnik]

@daxian-dbw I've downloaded the package and run it from Rxvt Color Unicode Terminal and everything works fine:

I can use left & right arrows to move the cursor and up & down to browse the command history

it works with xterm as well:

Open bash and run export TERM=rxvt.

I think that the source of the problem is that you start one terminal that identifies as TERM=ABC and then force it to identify as different terminal by setting TERM to XYZ. Your terminal is still using ABC key mappings, while .NET thinks that it's working with XYZ terminal and fetches XYZ terminfo db from the disk. The mappings for arrow keys are different, hence .NET fails to map the keys.

I am able to reproduce it with xterm:

Most likely you were doing that to mimic PowerShell/PowerShell#16606 but what was needed was running an actual rxvt terminal.

I am going to close the issue as I am quite confident that rxvt terminals are working as expected now. If you can run any rxvt terminal on Unix and repro the bug without setting TERM to a different terminal please re-open the issue and we are going to investigate it again. Thank you!

FWIW we have released a blog post dedicated to the rewrite, you can read it here: https://devblogs.microsoft.com/dotnet/console-readkey-improvements-in-net-7/

[daxian-dbw]

@adamsitnik Thank you for the investigation and also confirming that it works on the true Rxvt terminal.

I think that the source of the problem is that you start one terminal that identifies as TERM=ABC and then force it to identify as different terminal by setting TERM to XYZ.

This makes a lot of sense.
Now I know it's a bad idea to mimic another terminal by simply setting TERM env var to a different value :)