Terminal hangs or prints repetitive characters when using Japanese IME

[hnakamur]

---

Important

📌 plan of record: move the console tsf implementation to the Terminal codebase

---

Windows Terminal version

1.15.2875.0

Windows build number

10.0.22621.755

Other Software

WSL2 Ubuntu 22.04.1 LTS (version 2204.1.23.0)

Steps to reproduce

Input some Japanese characters using Microsoft IME at the right side edge of the terminal window.

Expected Behavior

Characters are converted correctly.

Actual Behavior

Sometimes repetitive characters are inputted, sometimes the terminal hangs or crashes.

2022-11-06-12-46-12.mp4

2022-11-06-13-03-10.mp4

[hnakamur]

The same problem occurs with PowerShell version 5.1.22621.608.

2022-11-06-13-35-41.mp4

[lhecker]

Input some Japanese characters using Microsoft IME at the right side edge of the terminal window.

Just to be sure: You're saying this never happens if you are not at the right side edge? Because if that's true, then that would narrow down the issue quite a lot and make fixing it much easier.

[hnakamur]

I tested this again and found out this also happens when I am not at the right side edge.

Steps to reproduce:
Input some Japanese texts and delete them and repeat that.

2022-11-08-12-36-32.mp4

[hnakamur]

I uninstalled version 1.15.2875.0 and installed https://github.com/microsoft/terminal/releases/download/v1.14.2281.0/Microsoft.WindowsTerminal_Win11_1.14.2282.0_8wekyb3d8bbwe.msixbundle and it seems the problem go away.

[dogatana]

Unfortunately, 1.16 still has this issue.
As a workaround, I'm using 1.14 now, but sometimes it's updated to 1.15 automatically.
And I cannot re-install 1.14 onto 1.15 with the following message.

アプリをインストールできませんでした。エラー メッセージ: API Logging data because access was denied for file: \?\C:\Program Files\WindowsApps\Microsoft.WindowsTerminal_1.14.2281.0_x64__8wekyb3d8bbwe\CascadiaCode.ttf, user SID: S-1-5-18 からのエラー 0x0 が発生したため、AppX 展開操作に失敗しました (0x80070005)

Although this is a Japanese message, I hope you can get information from this. 😅

Every time this happens, I need to uninstall 1.15, install 1.14 and recover setting.json.

I checked the settings, but I cloud not found the setting about auto-update.
Any suggestions are most welcome.

[dogatana]

1.17 has this too.
I should use the following workaround.

https://superuser.com/questions/1676001/windows-stop-killing-my-terminals-for-an-update

[lhecker]

I genuinely can't figure out how to reproduce the issue. I've tried writing some variation of "どうぞよろしくお願いいたします。" over and over again for about 15min an hour and it didn't happen.

But clearly this regression is caused by either of these two commits, both of which first shipped in v1.15:

• 2c922e1
• ed800dc

I don't think either of the two commits themselves is "buggy", but rather that they make bugs in TSF (Text Services Framework) obvious. This is because backspacing like in hnakamur's videos causes the TermControl::_CharacterHandler event handler to be called (the character is \b). This will cause TSFInputControl().ClearBuffer() to be called which should instruct TSF to flush its entire buffer and reset its state.

I think what's happening here is that TSF is either:

• ignoring this request
• getting into a bad state
• running out of process in the "Text Input Management Service" and that may result in yet-unknown, subtle race conditions? ¹

---

2c922e1 could have triggered the regression because it drops clamping the signed range values from TSF. I believe at the time I didn't consider it possible that TSF would actually pass either negative indices or a range where StartCaretPosition is greater than EndCaretPosition. I'll be more cautious in the code I write from now on.

ed800dc could have triggered the regression because it changed this:

_activeTextStart = ::base::ClampMin(_activeTextStart, ::base::ClampedNumeric<size_t>(range.StartCaretPosition));

into this:

_activeTextStart = std::min(_activeTextStart, _inputBuffer.size());

I think the latter is technically more correct ², but if TSF is in a bad state where the internal state didn't properly reset, then this may be off.

Footnotes

1. Microservices in your OS! Got a problem? Now it's a problem and a distributed problem! 100% more value! /s ↩

2. The design of the TSF API implies that it is random access. When the buffer contains "foo" and the _activeTextStart is 3 (= at the end), then we shouldn't move the start to 0 just because TSF replaced the entire string with "foobar". It should remain at 3, because the active string is "bar". It's unclear how TSF is supposed to be used in this case because TSF lacks documentation and is not designed to be used for a terminal. ↩

[lhecker]

I've marked #15938 as closing this issue, but I'll have to unfortunately rely on you to validate it, as I was entirely unable to reproduce the issue. 😟

[hnakamur]

Can I download the binary built with #15938 included?
Then I can validate it.

As of 1.17.11461.0, I just reproduce the issue just by inputting and deleting "日本語を入力して" three times.
I'm using the US keyboard on Japanese Windows 11.

[zadjii-msft]

@hnakamur sure can!

You should be able to go to: https://dev.azure.com/ms/terminal/_build/results?buildId=494745&view=artifacts&pathAsName=false&type=publishedArtifacts

Then navigate to the build-x64-release > _unpackaged > WindowsTerminalDev_0.0.1.0_x64.zip file. (or for a shortcut: I think this link will work?)

If there are subsequent commits, then this link:

will take you to the latest CI run

[hnakamur]

Thanks, your shortcut link works.
I'm afraid the issue is not resolved.

2023-09-07.22-53-27.mp4

[hnakamur]

I review my Windows settings and I found 「以前のバージョンのMicrosoft IMEを使う」 was on.
I changed it off and the issue seems go away on both Windows Terminal 1.17.11461.0 and WindowsTerminalDev_0.0.1.0_x64.zip!

[hnakamur]

I tried again with「以前のバージョンのMicrosoft IMEを使う」 on, the issue was reproduced.
So it seems there is a problem in the combination of the old version Microsoft IME and Windows Terminal.

As for me, I am fine with the current version of Microsoft IME.

[lhecker]

Hmm, I think that's fairly good news for us (Windows Terminal) at least, because it implies that this is indeed a bug in TSF. (An application shouldn't notice a difference in behavior between the legacy and new IME after all.) Would it be fine for us to only support the new IME?

BTW I tried to reproduce the issue with the legacy IME and again couldn't get it to reproduce it. 😟
I don't speak Japanese, so maybe I'm just not using the right input? I saw you write "kokodenennkannn" and tried that and that didn't do anything for me. Do you use any specific input that reliably reproduces the issue for you after a few tries?

[hnakamur]

I suppose the support for the old IME would be nice, but I am fine without it.

I was using 「日本語を入力して」"nihongowonyuuryokusite" to reproduce the issue.

[lhecker]

I was using 「日本語を入力して」"nihongowonyuuryokusite" to reproduce the issue.

Thanks! But I still can't get it to reproduce the issue. 🥲 I just genuinely can't figure this out.
And I noticed I haven't said this before: I'm sorry that it took so long for us to try and address this bug.

I suppose the support for the old IME would be nice, but I am fine without it.

One of my main tasks has been to improve the Internationalization of this project for a while now and I don't intend to give up just yet. 🙂 I was hoping to unify conhost's IME with the one in Windows Terminal at some point. conhost's IME implementation has two benefits: It uses TSF directly (without the WinRT CoreTextEditContext wrapper; less code = less bugs), and it writes the text during composition directly into the terminal viewport, so when you finish the composition it doesn't look quite as weird as it does now:

vs.

(I prefer the latter a lot.)

I'm just not sure when I can get to that. There's a lot of remaining Internationalization bugs in this project after all... But there's a good chance that using TSF directly without the WinRT wrapper will allow us to properly support the legacy IME and fix all those weird edge cases.
(Korean and Vietnamese in particular have a very difficult IME. By fixing those 2 IMEs, I broke the Japanese legacy IME. 😟 The conhost TSF/IME implementation never had those issues to begin with, thanks to its tight TSF integration.)

---

@zadjii-msft Should we close this bug, or repurpose it to track the Japanese legacy IME being broken? I think the latter might be good. This issue would get fixed properly if we ever manage to integrate the conhost TSF implementation into WT.

[tobiBarth]

When using tab to cycle through items I noticed that when the text would hit the right side of the terminal it would start bugging out for consecutive uses of tab. It seems to print previous lines again although this is purely visual.

japanese.text.bug.mp4

Win Terminal version: 1.17.11461.0

[lhecker]

We just published a major update to our IME implementation in the nightly Canary branch. It was rewritten from the ground up and has tons of improvements! If you're interested in trying it out, you can get it here: https://aka.ms/terminal-canary-installer
If you already have the Canary build installed, you can use this link to force an update.

If you encounter any issues or have any suggestions, or if you simply like/dislike the changes, please let us know! Thank you for bearing with us. 😊