Arrows ignored by bash when it started in ConEmu

[Maximus5]

I'm the author of ConEmu - alternative terminal for Windows. My users have created the issue Maximus5/ConEmu#629. When bash is started in ConEmu console, it totally ignores all extended keys (Arrows, Fn, etc.), so users can't access command line history with Up/Down arrows.

What is the problem with bash? Up/Down arrows properly cycle cmd.exe history, when it's started in ConEmu, but bash ignores them.

ConEmu is GUI application, so it get keyboard events via WM_KEYUP/WM_KEYDOWN and forward them to the created (usually hidden) real console using WriteConsoleInput.

[Maximus5]

I've created workaround in ConEmu build 160411. However, it would be nice to know, is it a bug or intended behavior.

[Maximus5]

Unfortunately, arrow keys still do not work in vim.

[sunilmut]

bash.exe has its own logic to read from console directly (this is as per the design today and subject to change in future releases). I am not a console expert, but I wonder if ConEmu interacting with the console directly is interfering with bash.exe's ability to process keyboard events?

[Maximus5]

ConEmu use official console API WriteConsoleInput to post keypressed into the conhost input buffer. That worked perfectly until "Bash on Ubuntu" appears.

[sunilmut]

Ok. I am not sure I fully understand the problem here. My understanding of this issue is that bash is not able to process arrow keys when launched from within ConEmu? Can you help clarify.

[Maximus5]

Short answer: Yes.

Many words below.
Let's take for example UpArrow key.

When user press and release UpArrow, normal console application
recieves two console key events:

1. bKeyDown = TRUE; wVirtualKeyCode = VK_UP;
2. bKeyDown = FALSE; wVirtualKeyCode = VK_UP;
   References: PeekConsoleInput, ReadConsoleInput, INPUT_RECORD, KEY_EVENT_RECORD.

But Unix console applications expect different key sequences.
For UpArrow keypress either ESC [ A or ESC O A sequences are expected
depending on requested ‘DECCKM’ mode.
References: ANSI sequence ESC [ ? 1 h and ESC [ ? 1 l.

The bash.exe is Windows console application, therefore it's expected
that this application would work properly with standard Windows console API
keypresses (KEY_EVENT_RECORD, VK_UP). But, I'm not sure who, the translation
is done when user presses UpArrow. And bash.exe recieves ESC [ A instead.

Well, that is rude workaround, but if bash.exe is started in ConEmu with
-cur_console:p1 switch, ConEmu does this translation and arrows magically
start working.

That is really rude, because ConEmu does not recieve ANSI sequences from
linux processes. And, when user starts vim from bash prompt, ‘DECCKM’ is not changed.
Therefore, ‘Arrows are not working in vim under ConEmu’.
There are absolutely no way to determine that vim was started from bash,
or vim has requested to change ‘DECCKM’ mode.
Seems like linux processes (init, bash, vim) somehow write their console output
directly to conhost, bypassing bash.exe. That's bad, because ConEmu has absolutely
no control over linux processes (started from service), only bash.exe may be hooked
to catch it's output and ‘fix’ input.

If you would like to check, download latest ConEmu build (alpha 160411 ATM),
and run from Win+R:

ConEmu.exe -basic -cmd bash.exe -cur_console:p1

[Maximus5]

Sure, it is. Monday, 02 May 2016, 07:09PM +03:00 from Daniel Gordon < notifications@github.com> :

So Just to be clear here - you guys are talking about bash.exe - that is or is not the same thing as the new Ubuntu Subsystem in the most recent Windows 10 dev build?
—
You are receiving this because you authored the thread.
Reply to this email directly or view it on GitHub

[DanielGGordon]

@Maximus5 I deleted my comment because I figured it out on my own by looking at the top of this page 10 seconds after I posted my comment... Sorry for the confusion.

[dariobig]

Hi, console2 and consoleZ, two other terminal emulators, have the same problem with arrows not working on bash.

[Maximus5]

@bigonazzi Of course they are. Only ConEmu is able to send xterm-ish keyboard sequences, which LX kernel expects.

[xilun]

Just would like to add that IMO this is quite important to have WSL working properly with ConEmu, because some serious console users (myself included) are not going to be satisfied with the basic Windows Console (which is now quite better than the legacy one, but still far away from any random XWindow Console we are used to)

For now I'm doing some dev under WSL on the Windows Console (for now coping with the international keyboard bugs it has by switching to a US layout) but I will be seriously disappointed if I can't use ConEmu for a vast majority of WSL software, and that obviously includes vim...

@Maximus5 thanks for your excellent ConEmu btw :)

[dariobig]

@xilun +1

@Maximus5 I like conemu, all I'm saying is that the issue is bigger than just one program and it should be addressed with high priority.

[Darktex]

@xilun +1

The Windows default terminals don't have nearly enough features for any power user

[djmott]

@Maximus5 thank you for ConEmu. It's a huge productivity improvement. MS should ditch the POS default console, give you a few million USD and ship it with every install of Windows.

[sunilmut]

@Maximus5 - Given the popularity of ConEmu, you could also convert it to a UWP using “Project Centennial." and submit it to Windows Store for even a wider audience.

[nigurr]

+1 Please fix this

[miniksa]

Sorry, this is my bad. @zadjii-msft and I overlooked this portion of the design when we implemented the virtual terminal input handling in the console host such that it doesn't present itself properly for ConEmu and friends. Unfortunately because it's a design flaw, it's a little bit more work than a quick bug fix.

I have it tracked internally as 7640020. I can't commit to a date it'll be fixed, but know that it is in our tracker, it is understood, and it is being ranked/prioritized.

[dariobig]

@miniksa good to hear that you're listening to user feedback!

[Novynn]

Ah, just came across this issue. I hope we'll see some progress soon!

[Maximus5]

@Novynn What progress do you want to see? ConEmu had implemented workaround and arrows started working months ago.

[Novynn]

@Maximus5 I was meaning more towards on Microsoft's end, I wasn't aware of a workaround!

I've just started using Cmder today, which reports it's ConEmu as being 160710 stable (I see this version is the latest release on GitHub) and I am unable to use arrow keys.

I'm using cmd.exe in Cmder to start Bash on Windows to ssh into a screen session on a Linux machine. This is on the Windows 10 Insider Preview 14393.

[Maximus5]

ConEmu.exe -basic -run {bash}

[Novynn]

Ah, that makes a lot more sense! Works perfectly thanks.

I still hope that Microsoft resolves this issue so no workarounds are required on your end.

[i336]

That's a very good point - the new console subsystem needs to have really, really good Unicode support. Modern programming languages support unicode strings and expect to be able to use it for I/O, and there are lots of prompt hacks using unicode characters as well.

I figured I might as well ride the bit of traffic that's been generated right now to chime in and add that the community remains super interested in a series of blog posts on the architectural updates. An in-depth writeup that adds every last detail that's legally possible would be widely appreciated - both because architectural upgrade trivia is always really fun, and also because it'll clearly explain why everything took so long.

I personally completely understand that this bugfix is the tail end of a lot of planets aligning being strong-armed into place, and that it'll take a little while. The console subsystem probably still has design cruft in it from Windows 3.1 - the NTVDM code doesn't ship in most (64-bit) installations, but is still builds for 32-bit, so it's still in the codebase. Knowing Microsoft's obsession with backward-compatibility, fixing this will not be easy.

For people frustrated about this, I have an open question (I don't have any Windows-capable hardware at the moment, so I unfortunately can't try this out myself): Can urxvt or xterm from Cygwin running on a Windows X server talk to WSL? If it can, that could be a very workable alternative.

For the MS devs working on this issue, when you see this update - I understand that speculating on timelines creates lots of avoidable fallout, so I won't ask for that. What I will ask for is a vague idea of the status of the internal tracking bug. How far have 7640020 and/or its dependencies/blockers progressed since the bug was initially filed (around August last year)?

I'm not trying to extricate an executive "I'm impatient, hurry up" summary. This is not ready yet, not much to do except face that. That's fine. But to know that there has been some movement would be cool.

Ah, if only you could share internal details. Then you could keep us in the loop about "the architecture, the customers, the issues, and the dependencies" :D and what the progress is with each. We wouldn't mind <3

[rprichard]

FWIW, winpty works around this issue by detecting when the console's ENABLE_VIRTUAL_TERMINAL_INPUT flag is enabled and, when it is, switching to an alternate system for generating console input:

• For most input, winpty replaces the keydown INPUT_RECORD with records representing the VT/100 encoding of the key. (See https://github.com/rprichard/winpty/blob/ce9239af5d751195ea6982a41c7d71356ac9201c/src/agent/ConsoleInput.cc#L722.)

• For navigation keys (arrows keys, Home/End), when no modifier was used (i.e. no Alt/Shift/Ctrl), winpty instead sends keydown/keyup window messages. This special case is necessary for Midnight Commander and probably vim too. (See Bash-on-Windows input handling doesn't account for alternate terminal modes rprichard/winpty#90 and https://github.com/rprichard/winpty/blob/ce9239af5d751195ea6982a41c7d71356ac9201c/src/agent/ConsoleInput.cc#L684.)

Unfortunately, sending a window message doesn't work quite right, because the resulting console input records seem to use modifier state from the keyboard. winpty would like to synthesize the modifier state, but I don't think there's a way to do that using SendMessage. This issue also affects Ctrl-C handling, which is a more noticeable problem with winpty. (See rprichard/winpty#116 and https://github.com/rprichard/winpty/blob/ce9239af5d751195ea6982a41c7d71356ac9201c/src/agent/ConsoleInput.cc#L361.)

[rprichard]

AFAIK, other console emulators on Windows (e.g. ConEmu and ConsoleZ) ought to be able to work around this issue the same way that winpty does.

@i336

Can urxvt or xterm from Cygwin running on a Windows X server talk to WSL?

AFAIK, one could run WSL urxvt or xterm connecting to any Windows X server, including Cygwin's. One could also run any Cygwin terminal emulator (including mintty, urxvt, xterm) and connect to WSL using https://github.com/rprichard/wslbridge (a program I wrote). wslbridge is also shipped as part of https://github.com/mintty/wsltty, a standalone package bundling mintty, wslbridge, and a Cygwin runtime DLL.

[Maximus5]

FYI ConEmu has implemented a workaround long long ago

[fpqc]

@Maximus5 I once got the msys2 bridge + wslbridge working together in tandem (without MinTTY), but it stopped working after like an hour. Have you done any testing with it?

[Maximus5]

Of course. It's working. Just use latest ConEmu version.

[zadjii-msft]

Hey everyone,

I finally have an update on this bug! I have a potential fix in code review right now.

To summarize the problem: We were only translating KEY_EVENTs into appropriate VT sequences when the events arrived from the window message loop. I moved the translation layer around, and now the INPUT_RECORDs written to the input buffer from the API (WriteConsoleInput) will also get translated appropriately.

It's not going to be available in the Fall Creator's Update, but it should be in the release that follows.

Thanks for your patience!

[jarun]

Good news. 👍 I just wasted an hour figuring out the issue and reached here.

[rprichard]

@zadjii-msft Good to hear. A few questions:

1. Does this also affect recognition of Ctrl-C or Ctrl-Break? That is, when ENABLE_PROCESSED_INPUT is on, will WriteConsoleInput send a signal for these key presses? It currently doesn't, and I'm guessing that won't change. (If it did, I might be able to fix Ctrl-C handling is inadequate for interactive cmd and PowerShell rprichard/winpty#116.)

2. Is mouse input affected? There's an analogous situation where mouse input should produce VT escape sequences, and the encoding is controlled by mouse mode flags (e.g. 1006 for the "SGR mouse encoding"). I haven't gotten around to testing the console yet, so maybe WriteConsoleInput calls already do this conversion.

3. Do you have suggestions on how programs can detect this behavior change (i.e. so that a workaround can be disabled)? For my situation (winpty), I think it'd be worth it to temporarily enable the VT input mode and test whether input is converted (by calling WriteConsoleInput then ReadConsoleInput). I'm guessing there isn't a better approach, though I suppose checking the OS version could be faster.

[ericblade]

That is great news (not so great that it won't make Fall.. oh well :-) ) .. just curious if this means that it ended up being significantly less work than initially expected? :-D

[zadjii-msft]

@rprichard

1. I don't believe it will. The change is pretty limited in scope to just the VT translator, so I don't believe any of the rest of the input is modified.
2. Mouse is unfortunately not fixed by this change. The keyboard events were easier to move than the mouse events unfortunately. I have a work item to address that issue before the end of the next release.
3. I don't have any good recommendations on that topic, sorry. Both the mechanisms you describe sound like they should work, though I have no idea which would be faster.

It was a bit easier than anticipated, but like I mentioned, mouse input is actually harder than anticipated unfortunately.

[bgshacklett]

Sorry, guys, I know you're being bombarded, but can you tell us how being on the Windows Insider fast track might affect the speed at which we see some of these changes?

[zadjii-msft]

@bgshacklett If you're on the fast ring, you'll see these changes pretty immediately after they hit winmain, which is usually a month or so after I commit the code on my end. Otherwise, if you're on just a non-insider's build, you'll have to wait for the entire release to be finished, which is about twice a year (based on previous Windows 10 releases)

[bitcrazed]

Thanks for the discussion.

Closing this issue since:

1. This issue is now fixed
2. This is the WSL issues repo, but this is an issue in Console which has its own Console GitHub Repo
3. GitHub doesn't allow issues to be moved between repos, preserving posters' identity :(

If you have further asks/issues, please file new issues on our Console GitHub Repo.

[maietta]

I DO NOT CARE WHAT YOU PEOPLE KEEP SAYING. THIS ISSUE IS NOT FIXED IN WINDWOS 11 AND WSL2.

Why is this still an issue in 2023?

[zadjii-msft]

@maietta This issue was closed 5 years ago. The console codebase has had a tremendous amount of churn since then. Whatever you're seeing now probably has a new root cause. I'd recommend filing a new issue over at https://github.com/microsoft/terminal/issues, so we can ID whatever's causing what you're seeing.

Thanks!

[nikhil-swamix]

for people running cli scripts in python: disable this

Terminal › Integrated: Windows Enable Conpty

key navigation will work, for packages like rich, fire, click cli tools...