Alt+Numpad input is broken (since Windows 7)

[alabuzhev]

Prior to Windows 7 it was possible to enter any "Unicode" (UCS-2 0-65535 range) character into a console app using Alt + Numpad keys.
This feature has been broken since Windows 7 (probably due to moving some console code into conhost.exe).

Environment

Any OS version since Windows 7.

Any other software?
No.

Steps to reproduce

Run the attached app:
readkey.zip

• Press and hold the Alt key
• Type any large enough decimal number on the numeric keyboard, say, 8888 or 65535.
• Release the Alt key
• Press and hold the Alt key
• Type any small enough decimal number on the numeric keyboard, say, 1 or 7.
• Release the Alt key

Expected behavior

Windows 2000 / XP / 2003 / Vista / 2008

The release of the Alt key generates a KEY_EVENT_RECORD with wVirtualKeyCode = VK_MENU and uChar.UnicodeChar = <the number you've entered>.

The expected output of the app for Alt+8888:
KEY_EVENT_RECORD: Up, Count=1, Vk=VK_MENU [18/0x12], Scan=56, uChar=[U='⊸'(0x22b8) A='¸'(0xb8)], Control=0x04000020 (casac - ecNs)

The expected output of the app for Alt+1:
KEY_EVENT_RECORD: Up, Count=1, Vk=VK_MENU [18/0x12], Scan=56, uChar=[U='☺'(0x0001) A='☺'(0x01)], Control=0x04000020 (casac - ecNs)

Actual behavior

Windows 7 / 8 / 10 classic / 10 new

The release of the Alt key generates a KEY_EVENT_RECORD with wVirtualKeyCode = VK_MENU and some rubbish in uChar.UnicodeChar.

The actual output of the app for Alt+8888:
KEY_EVENT_RECORD: Up, Count=1, Vk=VK_MENU [18/0x12], Scan=56, uChar=[U='©'(0x00a9) A='©'(0xa9)], Control=0x00000020 (casac - ecNs)

The actual output of the app for Alt+1:
KEY_EVENT_RECORD: Up, Count=1, Vk=VK_MENU [18/0x12], Scan=56, uChar=[U='☺'(0x263a) A=':'(0x3a)], Control=0x00000020 (casac - ecNs)

It looks like the host performs some dodgy internal conversions on the entered Unicode character.

And it's especially mental in the case of Alt+1 (and other "control" characters below 0x20):
instead of setting both UnicodeChar and AsciiChar to 0x1, it somehow takes a Unicode "replacement" for \1 - '☺'(0x263a), as if the wollowing code has been executed somewhere:

wchar_t W;
MultiByteToWideChar(CP_OEMCP, MB_USEGLYPHCHARS, "\1", 1, &W, 1);
assert(W == 0x263a);

Is it possible to stop doing those weird conversions and return to the pre-Windows 7 behaviour?

Thanks.

[zadjii-msft]

Let me start off - great breakdown here.

For the record, there wasn't a console team in the Windows Vista-7 timeframe. We only took over modification of the console at the start of Windows 10 development, so the fact that this is something that regressed in that timeframe is surprising to say the least.

I'd agree the new behavior doesn't seem to make any sense.

This might also have some tie-in to conversations happening in #3101/#3117, though those are more Terminal-specific and not conhost specific.

I'd love to hear the rest of the team's input on this, we'll probably discuss in triage early next week. If someone wants to try digging in deeper, I believe the following is where we handle input from the window, and convert it to INPUT_RECORDs. This is where I'd start investigating what's happening.

terminal/src/interactivity/win32/windowio.cpp

Lines 131 to 449 in 634687b

	void HandleKeyEvent(const HWND hWnd,
	const UINT Message,
	const WPARAM wParam,
	const LPARAM lParam,
	_Inout_opt_ PBOOL pfUnlockConsole)
	{
	CONSOLE_INFORMATION& gci = ServiceLocator::LocateGlobals().getConsoleInformation();
	// BOGUS for WM_CHAR/WM_DEADCHAR, in which LOWORD(lParam) is a character
	WORD VirtualKeyCode = LOWORD(wParam);
	WORD VirtualScanCode = LOBYTE(HIWORD(lParam));
	const WORD RepeatCount = LOWORD(lParam);
	const ULONG ControlKeyState = GetControlKeyState(lParam);
	const BOOL bKeyDown = WI_IsFlagClear(lParam, KEY_TRANSITION_UP);
	if (bKeyDown)
	{
	// Log a telemetry flag saying the user interacted with the Console
	// Only log when the key is a down press. Otherwise we're getting many calls with
	// Message = WM_CHAR, VirtualKeyCode = VK_TAB, with bKeyDown = false
	// when nothing is happening, or the user has merely clicked on the title bar, and
	// this can incorrectly mark the session as being interactive.
	Telemetry::Instance().SetUserInteractive();
	}
	// Make sure we retrieve the key info first, or we could chew up
	// unneeded space in the key info table if we bail out early.
	if (Message == WM_CHAR || Message == WM_SYSCHAR || Message == WM_DEADCHAR || Message == WM_SYSDEADCHAR)
	{
	// --- START LOAD BEARING CODE ---
	// NOTE: We MUST match up the original data from the WM_KEYDOWN stroke (handled at some inexact moment in the
	// past by TranslateMessageEx) with the WM_CHAR we are processing now to ensure we have the correct
	// wVirtualScanCode to associate with the message and pass down into the console input queue for further
	// processing.
	// This is required because we cannot accurately re-synthesize (using MapVirtualKey/Ex)
	// the original scan code just based on the information we have now and the scan code might be
	// required by the underlying client application, processed input handler (inside the console),
	// or other input channels to help portray certain key sequences.
	// Most notably this affects Ctrl-C, Ctrl-Break, and Pause/Break among others.
	//
	RetrieveKeyInfo(hWnd,
	&VirtualKeyCode,
	&VirtualScanCode,
	!gci.pInputBuffer->fInComposition);
	// --- END LOAD BEARING CODE ---
	}
	KeyEvent keyEvent{ !!bKeyDown, RepeatCount, VirtualKeyCode, VirtualScanCode, UNICODE_NULL, 0 };
	if (Message == WM_CHAR || Message == WM_SYSCHAR || Message == WM_DEADCHAR || Message == WM_SYSDEADCHAR)
	{
	// If this is a fake character, zero the scancode.
	if (lParam & 0x02000000)
	{
	keyEvent.SetVirtualScanCode(0);
	}
	keyEvent.SetActiveModifierKeys(GetControlKeyState(lParam));
	if (Message == WM_CHAR || Message == WM_SYSCHAR)
	{
	keyEvent.SetCharData(static_cast<wchar_t>(wParam));
	}
	else
	{
	keyEvent.SetCharData(0);
	}
	}
	else
	{
	// if alt-gr, ignore
	if (lParam & 0x02000000)
	{
	return;
	}
	keyEvent.SetActiveModifierKeys(ControlKeyState);
	keyEvent.SetCharData(0);
	}
	const INPUT_KEY_INFO inputKeyInfo(VirtualKeyCode, ControlKeyState);
	// Capture telemetry on Ctrl+Shift+ C or V commands
	if (IsInProcessedInputMode())
	{
	// Capture telemetry data when a user presses ctrl+shift+c or v in processed mode
	if (inputKeyInfo.IsShiftAndCtrlOnly())
	{
	if (VirtualKeyCode == 'V')
	{
	Telemetry::Instance().LogCtrlShiftVProcUsed();
	}
	else if (VirtualKeyCode == 'C')
	{
	Telemetry::Instance().LogCtrlShiftCProcUsed();
	}
	}
	}
	else
	{
	// Capture telemetry data when a user presses ctrl+shift+c or v in raw mode
	if (inputKeyInfo.IsShiftAndCtrlOnly())
	{
	if (VirtualKeyCode == 'V')
	{
	Telemetry::Instance().LogCtrlShiftVRawUsed();
	}
	else if (VirtualKeyCode == 'C')
	{
	Telemetry::Instance().LogCtrlShiftCRawUsed();
	}
	}
	}
	// If this is a key up message, should we ignore it? We do this so that if a process reads a line from the input
	// buffer, the key up event won't get put in the buffer after the read completes.
	if (gci.Flags & CONSOLE_IGNORE_NEXT_KEYUP)
	{
	gci.Flags &= ~CONSOLE_IGNORE_NEXT_KEYUP;
	if (!bKeyDown)
	{
	return;
	}
	}
	Selection* pSelection = &Selection::Instance();
	if (bKeyDown && gci.GetInterceptCopyPaste() && inputKeyInfo.IsShiftAndCtrlOnly())
	{
	// Intercept C-S-v to paste
	switch (VirtualKeyCode)
	{
	case 'V':
	// the user is attempting to paste from the clipboard
	Telemetry::Instance().SetKeyboardTextEditingUsed();
	Clipboard::Instance().Paste();
	return;
	}
	}
	else if (!IsInVirtualTerminalInputMode())
	{
	// First attempt to process simple key chords (Ctrl+Key)
	if (inputKeyInfo.IsCtrlOnly() && ShouldTakeOverKeyboardShortcuts() && bKeyDown)
	{
	switch (VirtualKeyCode)
	{
	case 'A':
	// Set Text Selection using keyboard to true for telemetry
	Telemetry::Instance().SetKeyboardTextSelectionUsed();
	// the user is asking to select all
	pSelection->SelectAll();
	return;
	case 'F':
	// the user is asking to go to the find window
	DoFind();
	*pfUnlockConsole = FALSE;
	return;
	case 'M':
	// the user is asking for mark mode
	Selection::Instance().InitializeMarkSelection();
	return;
	case 'V':
	// the user is attempting to paste from the clipboard
	Telemetry::Instance().SetKeyboardTextEditingUsed();
	Clipboard::Instance().Paste();
	return;
	case VK_HOME:
	case VK_END:
	case VK_UP:
	case VK_DOWN:
	// if the user is asking for keyboard scroll, give it to them
	if (Scrolling::s_HandleKeyScrollingEvent(&inputKeyInfo))
	{
	return;
	}
	break;
	case VK_PRIOR:
	case VK_NEXT:
	Telemetry::Instance().SetCtrlPgUpPgDnUsed();
	break;
	}
	}
	// Handle F11 fullscreen toggle
	if (VirtualKeyCode == VK_F11 &&
	bKeyDown &&
	inputKeyInfo.HasNoModifiers() &&
	ShouldTakeOverKeyboardShortcuts())
	{
	ServiceLocator::LocateConsoleWindow<Window>()->ToggleFullscreen();
	return;
	}
	// handle shift-ins paste
	if (inputKeyInfo.IsShiftOnly() && ShouldTakeOverKeyboardShortcuts())
	{
	if (!bKeyDown)
	{
	return;
	}
	else if (VirtualKeyCode == VK_INSERT && !(pSelection->IsInSelectingState() && pSelection->IsKeyboardMarkSelection()))
	{
	Clipboard::Instance().Paste();
	return;
	}
	}
	// handle ctrl+shift+plus/minus for transparency adjustment
	if (inputKeyInfo.IsShiftAndCtrlOnly() && ShouldTakeOverKeyboardShortcuts())
	{
	if (!bKeyDown)
	{
	return;
	}
	else
	{
	//This is the only place where the window opacity is changed NOT due to the props sheet.
	short opacityDelta = 0;
	if (VirtualKeyCode == VK_OEM_PLUS || VirtualKeyCode == VK_ADD)
	{
	opacityDelta = OPACITY_DELTA_INTERVAL;
	}
	else if (VirtualKeyCode == VK_OEM_MINUS || VirtualKeyCode == VK_SUBTRACT)
	{
	opacityDelta = -OPACITY_DELTA_INTERVAL;
	}
	if (opacityDelta != 0)
	{
	ServiceLocator::LocateConsoleWindow<Window>()->ChangeWindowOpacity(opacityDelta);
	return;
	}
	}
	}
	}
	// Then attempt to process more complicated selection/scrolling commands that require state.
	// These selection and scrolling functions must go after the simple key-chord combinations
	// as they have the potential to modify state in a way those functions do not expect.
	if (gci.Flags & CONSOLE_SELECTING)
	{
	if (!bKeyDown)
	{
	return;
	}
	Selection::KeySelectionEventResult handlingResult = pSelection->HandleKeySelectionEvent(&inputKeyInfo);
	if (handlingResult == Selection::KeySelectionEventResult::CopyToClipboard)
	{
	// If the ALT key is held, also select HTML as well as plain text.
	bool const fAlsoSelectHtml = WI_IsFlagSet(GetKeyState(VK_MENU), KEY_PRESSED);
	Clipboard::Instance().Copy(fAlsoSelectHtml);
	return;
	}
	else if (handlingResult == Selection::KeySelectionEventResult::EventHandled)
	{
	return;
	}
	}
	if (Scrolling::s_IsInScrollMode())
	{
	if (!bKeyDown || Scrolling::s_HandleKeyScrollingEvent(&inputKeyInfo))
	{
	return;
	}
	}
	// we need to check if there is an active popup because otherwise they won't be able to receive shift+key events
	if (pSelection->s_IsValidKeyboardLineSelection(&inputKeyInfo) && IsInProcessedInputMode() && gci.PopupCount.load() == 0)
	{
	if (!bKeyDown || pSelection->HandleKeyboardLineSelectionEvent(&inputKeyInfo))
	{
	return;
	}
	}
	// if the user is inputting chars at an inappropriate time, beep.
	if ((gci.Flags & (CONSOLE_SELECTING | CONSOLE_SCROLLING | CONSOLE_SCROLLBAR_TRACKING)) &&
	bKeyDown &&
	!IsSystemKey(VirtualKeyCode))
	{
	ServiceLocator::LocateConsoleWindow()->SendNotifyBeep();
	return;
	}
	if (gci.pInputBuffer->fInComposition)
	{
	return;
	}
	bool generateBreak = false;
	// ignore key strokes that will generate CHAR messages. this is only necessary while a dialog box is up.
	if (ServiceLocator::LocateGlobals().uiDialogBoxCount != 0)
	{
	if (Message != WM_CHAR && Message != WM_SYSCHAR && Message != WM_DEADCHAR && Message != WM_SYSDEADCHAR)
	{
	WCHAR awch[MAX_CHARS_FROM_1_KEYSTROKE];
	BYTE KeyState[256];
	if (GetKeyboardState(KeyState))
	{
	int cwch = ToUnicodeEx((UINT)wParam, HIWORD(lParam), KeyState, awch, ARRAYSIZE(awch), TM_POSTCHARBREAKS, nullptr);
	if (cwch != 0)
	{
	return;
	}
	}
	else
	{
	return;
	}
	}
	else
	{
	// remember to generate break
	if (Message == WM_CHAR)
	{
	generateBreak = true;
	}
	}
	}
	HandleGenericKeyEvent(keyEvent, generateBreak);
	}

[alabuzhev]

the fact that this is something that regressed in that timeframe is surprising to say the least

As I mentioned, somewhere between Vista and 7 at least one major thing happened - some parts were extracted from csrss into conhost, which was, supposedly, not a trivial copy&paste, so probably not that surprising after all.

I noticed this in 2009, but, since you didn't have this handy bugtracker back then, implemented a workaround and moved on. Today it resurfaced in a slightly different context, and while implementing another workaround I realised that now I can (finally) report it :)

[alabuzhev]

I think I found it.

terminal/src/host/stream.cpp

Lines 101 to 121 in 634687b

	// chars that are generated using alt + numpad
	if (!keyEvent->IsKeyDown() && keyEvent->GetVirtualKeyCode() == VK_MENU)
	{
	if (keyEvent->IsAltNumpadSet())
	{
	if (HIBYTE(keyEvent->GetCharData()))
	{
	char chT[2] = {
	static_cast<char>(HIBYTE(keyEvent->GetCharData())),
	static_cast<char>(LOBYTE(keyEvent->GetCharData())),
	};
	*pwchOut = CharToWchar(chT, 2);
	}
	else
	{
	// Because USER doesn't know our codepage,
	// it gives us the raw OEM char and we
	// convert it to a Unicode character.
	char chT = LOBYTE(keyEvent->GetCharData());
	*pwchOut = CharToWchar(&chT, 1);
	}

CharToWchar is implemented in terms of ConvertOutputToUnicode, which is implemented in terms of MultiByteToWideChar with the MB_USEGLYPHCHARS flag:

terminal/src/host/misc.cpp

Line 316 in 634687b

return MultiByteToWideChar(uiCodePage, MB_USEGLYPHCHARS, pchSource, cchSource, pwchTarget, cchTarget);

This explains 0x1 -> 0x263a.

And static_cast<char>(LOBYTE(L'\x22b8')) is 0xb8, MultiByteToWideChar(0xb8) is 0x00a9.

I think HIBYTE(keyEvent->GetCharData()) is 0 even for 0x22b8 here, otherwise CharToWchar would have picked the 0x22 byte and returned L'\' - probably some other code zeroes the high byte elsewhere.

P.S. It's probably not a good design choice to use ConvertOutputToUnicode for the input stream (e.g. due to unexpected effects of MB_USEGLYPHCHARS, as in this case), but that's a different question.

[alabuzhev]

some other code zeroes the high byte elsewhere

Probably here:

terminal/src/host/directio.cpp

Lines 92 to 98 in 634687b

	char inBytes[] = {
	static_cast<char>(keyEvent->GetCharData())
	};
	try
	{
	outWChar = ConvertToW(gci.CP, { inBytes, ARRAYSIZE(inBytes) });
	}

But there are also quite a few other occurrences of static_cast<char>(...).

[DHowett-MSFT]

This is definitely something we should look at, so I'm taking the Triage tag off. Thanks for the great investigation.

[Shorotshishir]

hello, sorry to make this old post alive, I faced an interesting behavior today,

Widows 10 Pro 1909
Windows Terminal 1.0.1401.0 (Installed via store)

In default PowerShell,

• if you go Alt+22, it pastes whatever you copied last.
• Alt+26 removes whatever you wrote or pasted,
• Alt+25 and Alt+29 gives a knocking sound
• Alt+18 gives you a searching option. bck-i-search. Which I figured allows you to search through your terminal input history.

In Commandline:

• Alt+22 shows ^V which we used to see in cmd when we typed Ctrl+V

Not sure why this is happening, but seemed interesting, :D

[mjvh80]

Perhaps a different issue, but unicode input also doesn't work via enablehexnumpad. So if you have enablehexnumpad enabled in the registry, some applications (as support in Windows is generally poor) allow the unicode codepoint to be entered in hex by typing e.g. alt + 3C0 (where + is the numpad +) to give π.
If I type the digits using the numpad for digits the above example gives +cπ where I expect π only. Without numpad it doesn't work at all.