Add support for Windows UTF-8 version of R

[SpecterShell]

An experimental build of R using UTF-8 as native encoding on Windows has been released recently, and here is the instruction: https://svn.r-project.org/R-dev-web/trunk/WindowsBuilds/winutf8/ucrt3/howto.html

I'm on Windows 10 20H2 and has set $LANG to zh_CN.UTF-8. After setting the codepage to 65001(UTF-8), the encoding in Rterm successfully switches to UTF-8, but the encoding in radian remains unchanged.

In most of the time the terminal and the graph can still handle CJK characters properly. But sometimes some messy codes show, like the red text in the screenshot.

I thought this issue came from Python itself, so I set $PYTHONUTF8 to 1 to force Python to use UTF-8, and ran python -m radian instead of radian. However it doesn't work for radian. I also tried to directly change the encoding in radian, and then it crashed.

[randy3k]

These are the relevant lines for decoding and encoding: https://github.com/randy3k/rchitect/blob/cb6e77c860ef180bd7f47416c94185fb9db04370/rchitect/utils.py#L142

Is there any prebuilt binary? The steps seem a bit involved. It seems that there is.

[SpecterShell]

https://www.r-project.org/nosvn/winutf8/ucrt3/
And R-devel-win-80146-4517-4499.exe(or newer) is the installer of the prebuilt binary.

I just found that running demo(recursion) can also reproduce the issue while R is waiting for user's response. So there is no need to prepare building environment. Just install and test.

[randy3k]

Does something like Sys.setlocale(locale = "English_United States.utf8") fix it?

[randy3k]

I have to set this

to change the default locale of python applications.

But then, things are still not working well in radian:

r$> x = "床前明月光"

r$> print(x)
[1] "�����"

r$> Sys.getlocale()
[1] "LC_COLLATE=English_United States.utf8;LC_CTYPE=English_United States.utf8;LC_MONETARY=English_United States.utf8;LC_NUMERIC=C;LC_TIME=English_United States.utf8"

r$> l10n_info()
$MBCS
[1] TRUE

$`UTF-8`
[1] TRUE

$`Latin-1`
[1] FALSE

$codepage
[1] 65001

$system.codepage
[1] 65001

[randy3k]

randy3k/rchitect@7c0d72a should fix the garbled text issue. However, it requires checking the beta UTF-8 setting in Region settings.

[randy3k]

There is some discussion on enabling UTF-8 codepage on Windows, but it won't happen soon. So for now, the unicode setting in regional settings is the only way to enable UTF-8 on radian.

[SpecterShell]

In my opinion this issue occurs when radian calls R, system tends to use system-wide encoding settings instead of one set in the console, making R think that the current codepage is not UTF-8.
Setting the unicode setting may fix this problem by making all the applications use Unicode, but will make non-Unicode applications and even system corrupted.
Therefore, before there is a better way to solve this, I would use English as default in radian or rollback R to stable version.

[randy3k]

We have to wait until python supports Unicode codepage on Windows.

See https://groups.google.com/g/python-ideas/c/iPgyhq3_zyI/m/uGXrunPWAQAJ

[SpecterShell]

UTF-8 has become the default encoding now. 🤔
https://cran.r-project.org/bin/windows/base/NEWS.R-4.2.0.html

[SpecterShell]

The bare R term uses English by default utill the LANG is set. However, radian (or rchitect) always use system locale, resulting in garbled text.
A temporary workaround is to set LANG to en_US.UTF-8 at startup.

[randy3k]

I guess we'd need to force utf8 encoding for newer version of R.

[randy3k]

Hi @SpecterShell, would it work if you set the variable LANG using Sys.setenv(LANG = "en_US.UTF-8")?

[SpecterShell]

Works

[randy3k]

Then the solution for us is simpler, we could just set it if it is unset.

[randy3k]

@SpecterShell
Could you test if f016d46 fixes it.

[SpecterShell]

Fixed now. Thank you for the fix!

[randy3k]

Great, radian 0.6.2 is on the way.

[Yunuuuu]

Only set LANG environment is not sufficient to make this work well, I must set locale manually by adding following code in my .Rprofile.

            # Set locale to utf8
            is_uft8_support <- grepl(
                "UTF-8|utf8", Sys.getenv("LANG"),
                ignore.case = TRUE, perl = TRUE
            ) &&
                R.version$major >= 4L &&
                R.version$minor >= 2.0
            if (is_uft8_support) {
                suppressWarnings(Sys.setlocale("LC_ALL", Sys.getenv("LANG")))
            }

This is my radian version:

when use raw radian without setting above code in my .Rprofile, the R sessionInfo and l10n_info indicate UTF-8 encoding isn't work. When we read file with special character using data.table::fread, some characters won't be parsed correctly

After setting locale, this works fine, and when we read file with special character using data.table::fread, special characters can be parsed successfully.

[Yunuuuu]

RStudio has setted appropriate UTF-8 locale for R 4.2

[kristy-hu]

Where is the .Rprofile in vscode? I am new to R, and have little idea as to where to configure it... Thanks in advance!

[SpecterShell]

Where is the .Rprofile in vscode? I am new to R, and have little idea as to where to configure it... Thanks in advance!

%USERPROFILE%\Documents\.Rprofile

[kristy-hu]

Hi @SpecterShell, there was no .Rprofile under my document folder.
However, when I typed ~/.Rprofile in R, it just asked me to create a new file. And after I pasted the code, the UTF-8 was turned on. And the problem was sort of solved.

Yet I have found 2 things quite smiliar to .Rprofile in R's directory:

1. Rprofile under dir R\R-4.2.1\library\base\R
2. Rprofile.site under dir R\R-4.2.1\etc

Is it possible to configure any of these to make the UTF-8 global setting? Thanks!

[SpecterShell]

You can create one if it doesn't exist.

[kristy-hu]

You can create one if it doesn't exist.

OK, thanks!

[kristy-hu]

I think perhaps the best solution for this is adding Sys.setenv(LANG = "en_US.UTF-8") directly to .Rprofile, and the garbled text will never appear again. Now everything just works fine in English. The Chinese translation is actually a bit misleading...

[kongdd]

After set the local by Sys.setlocale("LC_ALL", "Chinese (Simplified)_China.utf8"), get the following warnings when using devtools::load_all()

There were 15 warnings (use warnings() to see them)

r$> warnings()
Warning messages:
1: In Sys.setlocale("LC_CTYPE", ctype) :
  using locale code page other than 936 may cause problems

[beansrowning]

Not sure if I'm in the right place here, but R.4.2 + radian seems to be entirely non-functional in my case. No changing of env vars seems to change that.

Radian fails on start:

System setup

radian --version

radian version: 0.6.4
r executable: C:\Users\<>\AppData\Local\Programs\R\R-4.2.1\bin\R
r version: 4.2.1
python executable: C:\Users\<>\.pyenv\pyenv-win\versions\3.10.8\python.exe
python version: 3.10.8

Stack trace

radian

Traceback (most recent call last):
  File "C:\Users\<>\.pyenv\pyenv-win\versions\3.10.8\Scripts\radian-script.py", line 33, in <module>
    sys.exit(load_entry_point('radian==0.6.4', 'console_scripts', 'radian')())
  File "C:\Users\<>\.pyenv\pyenv-win\versions\3.10.8\lib\site-packages\radian\app.py", line 108, in main
    RadianApplication(r_home, ver=__version__).run(options, cleanup=cleanup)
  File "C:\Users\<>\.pyenv\pyenv-win\versions\3.10.8\lib\site-packages\radian\app.py", line 217, in run
    self.session = create_radian_prompt_session(options, settings)
  File "C:\Users\<>\.pyenv\pyenv-win\versions\3.10.8\lib\site-packages\radian\prompt_session.py", line 153, in create_radian_prompt_session
    session = RadianPromptSession(
  File "C:\Users\<>\.pyenv\pyenv-win\versions\3.10.8\lib\site-packages\radian\lineedit\prompt.py", line 72, in __init__
    super().__init__(*args, **kwargs)
  File "C:\Users\<>\.pyenv\pyenv-win\versions\3.10.8\lib\site-packages\prompt_toolkit\shortcuts\prompt.py", line 473, in __init__
    self.default_buffer = self._create_default_buffer()
  File "C:\Users\<>\.pyenv\pyenv-win\versions\3.10.8\lib\site-packages\radian\lineedit\prompt.py", line 163, in _create_default_buffer
    return ModalBuffer(
  File "C:\Users\<>\.pyenv\pyenv-win\versions\3.10.8\lib\site-packages\radian\lineedit\buffer.py", line 185, in __init__
    super().__init__(*args, **kwargs)
  File "C:\Users\<>\.pyenv\pyenv-win\versions\3.10.8\lib\site-packages\radian\lineedit\buffer.py", line 22, in __init__
    super().__init__(*args, **kwargs)
  File "C:\Users\<>\.pyenv\pyenv-win\versions\3.10.8\lib\site-packages\prompt_toolkit\buffer.py", line 313, in __init__
    self.reset(document=document)
  File "C:\Users\<>\.pyenv\pyenv-win\versions\3.10.8\lib\site-packages\radian\lineedit\buffer.py", line 264, in reset
    self._reset_history()
  File "C:\Users\<>\.pyenv\pyenv-win\versions\3.10.8\lib\site-packages\radian\lineedit\buffer.py", line 257, in _reset_history
    for m, item in self.history.load():
  File "C:\Users\<>\.pyenv\pyenv-win\versions\3.10.8\lib\site-packages\radian\lineedit\history.py", line 9, in load
    self._loaded_strings = list(self.load_history_strings())
  File "C:\Users\<>\.pyenv\pyenv-win\versions\3.10.8\lib\site-packages\radian\lineedit\history.py", line 68, in load_history_strings
    backup = f.readlines()
  File "C:\Users\<>\.pyenv\pyenv-win\versions\3.10.8\lib\encodings\cp1252.py", line 23, in decode
    return codecs.charmap_decode(input,self.errors,decoding_table)[0]
UnicodeDecodeError: 'charmap' codec can't decode byte 0x9d in position 2546: character maps to <undefined>

EDIT:

The culprit is here:

radian/radian/lineedit/history.py

Line 67 in adbf612

with open(self.filename, "r+") as f:

Trying to load ~/.radian_history without specifying utf-8 encoding. Deleting my prior history file seems to resolve in this case

[randy3k]

Good catch, we should make sure that the history file is in utf-8.

[psychelzh]

Any clue on why the first character of the package {targets} output is not as expect?

Should be this instead:

[kebuAAA]

where is the history file? I meet the same UnicodeDecodeError while this happens in the radian/rutils.pyfile