Bash on windows does not accept Japanse input method

[lateau]

Please consider support input/output with none English languages on any of locale setting. My locale setting is US English and bash does not accept Japanese input method. As a Linux user this doesn't make sense even if I understand bash on windows does not same with Linux itself.

[russalex]

We have some known issues with internationalization support. Tracking this one in #183. I am told this is fixed in the internal builds and will be out to the public soon.

[lateau]

It does not works on 14352 yet.

[benhillis]

Could you please provide us with some more information:

1. What is the locale of the NT version you are running (en-US, ja-JP, etc)
2. What is the $LANG environment variable inside bash? (echo $LANG)

I made a change recently to detect non-English locales during install and download the appropriate language packs. Would it be possible for you to try a reinstall of Bash on Ubuntu on Windows?

lxrun.exe /uninstall
lxrun.exe /install

[vyv03354]

I'm not the initial reporter, but I can also reproduce this issue. So I'm answering the questions.

1. What is the locale of the NT version you are running (en-US, ja-JP, etc)

ja-JP.

1. What is the $LANG environment variable inside bash? (echo $LANG)

$ echo $LANG
ja_JP.UTF-8
$

I made a change recently to detect non-English locales during install and download the appropriate language packs.

Yeah, now messages are displayed in Japanese by default, but Japanese IMEs do not work yet.

Would it be possible for you to try a reinstall of Bash on Ubuntu on Windows?

I install WSL and UoW from a fresh VM image every time a new Insider Preview build has arrived. So the old installation should not affect the issue.

I guess this is not a WSL problem but a Windows console problem. Windows console has supported East-Asian IMEs only if the IME locale matches the console code page. For example, Japanese IMEs work only if the console CP is 932. Similarly, Simplified Chinese IMEs will work only with CP936, Traditional Chinese IMEs with CP950, and Korean IMEs with CP949.

WSL uses CP65001 for console CP, so no East-Asian IMEs will be activated.

[miniksa]

@vyv03354, you've hit the nail on the head. It's because it's using the 65001 UTF-8 codepage.

This is Michael from the Windows Console team.

Unfortunately for all of us, the IME is still in its original incarnation from when it was authored in 1997 where it was very tightly bound to specific codepages and non-Unicode locales set in your Control Panel Region settings. As such, you can really only get a Japanese IME activated properly when in Code Page 932 on a machine that has its Non-Unicode locale set to Japanese (Japan).

There are other issues that have been revealed to us through the use of codepage 65001 by WSL in both the console input and output stack as well. I could write a novel on the issues, but I'll spare you right now. :P

We've consolidated all of these issues into a backlog item to improve support across the board for UTF-8 (codepage 65001) in a future release. It's a little close to our Anniversary Update to make any sweeping changes now. :) These sort of issue reports definitely help us appropriately rank that backlog item and push for the funding we need to make you happy, so keep them coming!

Sorry to be the bearer of bad news, though, that this isn't going to work in the near term. :(

--Michael

P.S. To you and anyone else who is experiencing issues that you suspect are in the console host or console subsystem, the best way to alert me is to cc me @miniksa.

[lateau]

@vyv03354 You're right. Thanks for your comment.
@miniksa Thanks. Please keep working the nice job. I can't wait to see a (new?) console in future release.

In my case, both of locales are en_US may not common configuration in Japan.

All Settings > Time & Language > Region & language > Country or region > United States
All Settings > Time & Language > Region & language > Languages > English

$ echo $LANG
en_US.UTF-8

I'm using cmder to avoid this issue caused by original design. It's not the best way but there're no other ways on current build & my settings.

@miniksa Can you also consider something cool fonts support for Non-English languges? This is not a WSL specific problem, but I've been encountered font problems on many other Linux distro and I think this can be considerable issue. :)

[miniksa]

@lateau Can you be more specific about what you mean by cool fonts? What sort of fonts? Do you have an example font I can look at?

In the Anniversary Update for Windows 10 coming this summer, @paulcam206 and I removed most of the limitations around font selection. If you haven't tried to load your favorite TrueType monospaced font into the console, please give it another try!

We do still attempt to match the current output code page to the charsets that are supported by the fonts installed on your system, but we're no longer restricted only to ＭＳ ゴシック for Japanese codepage 932.

[dynaxis]

Ditto type of comments won't be any use. But as a Korean trying Bash On Windows, I experience the same. IME is not activated at all and the Korean output is messed up since Korean characters normally takes up twice as much width as roman ones.

[miniksa]

@dynaxis, it's alright. Thank you for your report. We're working right now on improving the experience for IME users and for Chinese, Japanese, and Korean language processing in general.

In the coming months when Insiders Builds are rolling out again, you should be able to see improvements in this area. I don't know when that is nor when our code will reach specific build numbers for certain, but we are making fixes in this area.

[lateau]

@miniksa

Here're more details. The feature 'cool fonts' means:

• Font itself for None ascii characters: Please guess some Japanese users have to deal with ＭＳ ゴシック
  on terminal. Is Times font prefer to every users? May the answer is no.
• Configuable fallback font feature for None ascii characters: cmder configuration has this one.
• Display multiple fonts on same terminal: Consolas for ascii characters & ＭＳ ゴシック for None ascii characters.

Please check these pictures:

Do you think ascii charactes are really readable on these pictures? ＭＳ ゴシック is a good font for Japanese but it makes hard to read all ascii characters. 1? l? Oh god...




How about source code:

cmder has 'Alternative font' configurable option I've mentioned.

I don't know how Windows handles fonts on the system and may some are not easy to handle well. But please consider that WSL needs more readable and useful font features at all.

[miniksa]

@lateau That is a good idea! I didn't know that Cmder could set an alternative font for a particular Unicode range. I'd be curious how they accomplish it.

As of right now, the Windows Console can only choose a single font for the console window at a time and calculates everything in the window as a multiple of the pixel count for the "default character" in that font. It's also still using plain ol' GDI. Very simplistic. We've had a few prototypes trying to make it more advanced, but haven't finished them off yet.

Definitely a good consideration for our future work though. Thanks for the input! :)

cc: @bitcrazed

[lateau]

@miniksa cmder is on top of Conemu and the implimantation of alternative font option is inlcluded in Conemu: http://conemu.github.io/en/Source.html

Thanks for your answer and nice work!

[vyv03354]

Now Japanese IME is working on CP65001 console! Thank you!
I tested it with build 14946, but I don't know exactly when this issue was fixed.

[miniksa]

@vyv03354 Great to hear! From what I can see, all builds from 14931 and forward should have our changes to remove locale restrictions from the IME which should effectively resolve this issue and allow Japanese input method in the Bash on Windows environment. Please let us know if you have any other issues in the future! :)