Skip to content

fix some unicode prob #1584

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 7 commits into from
Closed

fix some unicode prob #1584

wants to merge 7 commits into from

Conversation

CRGBS
Copy link

@CRGBS CRGBS commented May 24, 2023

make it support chinese more

@CRGBS
Copy link
Author

CRGBS commented May 24, 2023

ignore

build.yml
and
main.cpp

it just for test some item

Copy link
Contributor

@DannyDaemonic DannyDaemonic left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In summary, try setting your locale like this instead:

export LANG=zh_CN.UTF-8

@@ -522,6 +522,11 @@ void console_init(console_state & con_st) {
// Turn off ICANON (ENABLE_LINE_INPUT) and ECHO (ENABLE_ECHO_INPUT)
dwMode &= ~(ENABLE_LINE_INPUT | ENABLE_ECHO_INPUT);
SetConsoleMode(hConIn, dwMode);
}else{
// Set console input codepage to UTF8
_setmode(_fileno(stdin), _O_U8TEXT);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this necessary when your configured locale has "UTF-8" in it?

// Set console input codepage to UTF8
_setmode(_fileno(stdin), _O_U8TEXT);
// Set locale to Chinese Simplified UTF-8
setlocale(LC_ALL, "zh-CN.UTF-8");
Copy link
Contributor

@DannyDaemonic DannyDaemonic May 24, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This sets everyone to Chinese. When someone doesn't have "zh_CN.UTF-8" installed on their system, it will instead set to "C", which breaks Unicode support for them.

What we already have:

setlocale(LC_ALL, "");

When setlocale is used with an empty string, it uses your system locale settings. In Linux you can set it like this:

export LANG=zh_CN.UTF-8

See this post at Stack Overflow for more information.

@CRGBS
Copy link
Author

CRGBS commented May 25, 2023

i am use windows 10 os
my discord bot make llamacpp in cmd inside cmd
so it can't support chinese fully

maybe this pr is not good for here
next time if i have better pr i will pr again
thx for work :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants