Unable to enter Chinese prompt

[LainNya]

Hi!My use is compiled under Windows main.exe, when I type Chinese Prompt, I found that the model seems to be unable to understand, under debugging found that std::getline(std::cin,line) get is empty lines, then I tried Japanese, are the same result.
(Since I am a native Chinese speaker, this question was translated by DeepL)

[TingTingin]

have you tried this https://github.com/ymcui/Chinese-LLaMA-Alpaca?

[LainNya]

have you tried this https://github.com/ymcui/Chinese-LLaMA-Alpaca?

Yes, this is the model I use,I ran the command in git bash and it seems works fine?

[LainNya]

The Chinese value obtained in the debug looks like this

[boholder]

Let me provide more information about this issue. Hope this helps to solve the problem.

@LainNya found out that:
in the CMD and Powershell (also Cygwin64 Terminal as I tested) terminals of the Windows environment, the Chinese entered could not be fetched by llama.cpp. But in other terminals like git-bash and linux subsystem for windows (no it's just linux env) it works.

It seems that the std::getline doesn't support UTF-8 character set.

This is the detailed cause and solution I found, but I can't tell since I'm not familiar with C++:

https://cboard.cprogramming.com/cplusplus-programming/145590-non-english-characters-cout-post1086757.html#post1086757

I'm not sure if this should be chalked up to a Windows terminal compatibility issue with C++.

Anyway, use git-bash and linux subsystem for windows as the workaround for now.

[josStorer]

I modified the implementation of the getline part, rewrote a simple getline with _getwch, and it can work, but it's not cross-platform

Here is the repo: https://github.com/josStorer/llama.cpp-unicode-windows

[tomsnunes]

This problem for unicode characters had been fixed here for me in the following PR #420 , but I confirm that it is currently not possible to insert or display utf-8 characters at moment.

[prusnak]

#840 has been merged - try pulling the latest master and please test whether this fixed your issue

[boholder]

LGTM after replace the latest binaries in https://github.com/ggerganov/llama.cpp/releases/tag/master-aaf3b23

Modifying the prompt is also works, nice job! 👍