{styler} changes unicode characters on Windows

[IndrajeetPatil]

Example:

styler::style_text('suit <- "♠"')
#> suit <- "<U+2660>"

^{Created on 2021-10-04 by the reprex package (v2.0.1)}

Session info

sessioninfo::session_info()
#> - Session info ---------------------------------------------------------------
#>  setting  value                       
#>  version  R version 4.1.1 (2021-08-10)
#>  os       Windows 10 x64              
#>  system   x86_64, mingw32             
#>  ui       RTerm                       
#>  language (EN)                        
#>  collate  English_United Kingdom.1252 
#>  ctype    English_United Kingdom.1252 
#>  tz       Europe/Berlin               
#>  date     2021-10-04                  
#> 
#> - Packages -------------------------------------------------------------------
#>  package     * version date       lib source        
#>  backports     1.2.1   2020-12-09 [1] CRAN (R 4.1.0)
#>  cli           3.0.1   2021-07-17 [1] CRAN (R 4.1.0)
#>  crayon        1.4.1   2021-02-08 [1] CRAN (R 4.1.1)
#>  digest        0.6.28  2021-09-23 [1] CRAN (R 4.1.1)
#>  ellipsis      0.3.2   2021-04-29 [1] CRAN (R 4.1.0)
#>  evaluate      0.14    2019-05-28 [1] CRAN (R 4.1.1)
#>  fansi         0.5.0   2021-05-25 [1] CRAN (R 4.1.1)
#>  fastmap       1.1.0   2021-01-25 [1] CRAN (R 4.1.1)
#>  fs            1.5.0   2020-07-31 [1] CRAN (R 4.1.1)
#>  glue          1.4.2   2020-08-27 [1] CRAN (R 4.1.1)
#>  highr         0.9     2021-04-16 [1] CRAN (R 4.1.1)
#>  htmltools     0.5.2   2021-08-25 [1] CRAN (R 4.1.1)
#>  knitr         1.36    2021-09-29 [1] CRAN (R 4.1.1)
#>  lifecycle     1.0.1   2021-09-24 [1] CRAN (R 4.1.1)
#>  magrittr      2.0.1   2020-11-17 [1] CRAN (R 4.1.1)
#>  pillar        1.6.3   2021-09-26 [1] CRAN (R 4.1.1)
#>  pkgconfig     2.0.3   2019-09-22 [1] CRAN (R 4.1.1)
#>  prettycode    1.1.0   2019-12-16 [1] CRAN (R 4.1.1)
#>  purrr         0.3.4   2020-04-17 [1] CRAN (R 4.1.1)
#>  R.cache       0.15.0  2021-04-30 [1] CRAN (R 4.1.1)
#>  R.methodsS3   1.8.1   2020-08-26 [1] CRAN (R 4.1.0)
#>  R.oo          1.24.0  2020-08-26 [1] CRAN (R 4.1.0)
#>  R.utils       2.11.0  2021-09-26 [1] CRAN (R 4.1.1)
#>  reprex        2.0.1   2021-08-05 [1] CRAN (R 4.1.1)
#>  rlang         0.4.11  2021-04-30 [1] CRAN (R 4.1.1)
#>  rmarkdown     2.11    2021-09-14 [1] CRAN (R 4.1.1)
#>  rstudioapi    0.13    2020-11-12 [1] CRAN (R 4.1.1)
#>  sessioninfo   1.1.1   2018-11-05 [1] CRAN (R 4.1.1)
#>  stringi       1.7.5   2021-10-04 [1] CRAN (R 4.1.1)
#>  stringr       1.4.0   2019-02-10 [1] CRAN (R 4.1.1)
#>  styler        1.6.2   2021-09-23 [1] CRAN (R 4.1.1)
#>  tibble        3.1.5   2021-09-30 [1] CRAN (R 4.1.1)
#>  utf8          1.2.2   2021-07-24 [1] CRAN (R 4.1.1)
#>  vctrs         0.3.8   2021-04-29 [1] CRAN (R 4.1.1)
#>  withr         2.4.2   2021-04-18 [1] CRAN (R 4.1.1)
#>  xfun          0.26    2021-09-14 [1] CRAN (R 4.1.1)
#>  yaml          2.2.1   2020-02-01 [1] CRAN (R 4.1.0)
#> 
#> [1] C:/Users/IndrajeetPatil/Documents/R/win-library/4.1
#> [2] C:/Program Files/R/R-4.1.1/library

I am working in .Rproject with UTF-8 text encoding:

And I also checked that

> Encoding("♠")
[1] "UTF-8"

Based on #376, I thought this might be unexpected behavior?

[lorenzwalthert]

Thanks @IndrajeetPatil. Indeed smells like trouble 😩. I’ll look into it soon…

[lorenzwalthert]

On macOS and R 4.1.0, I can't reproduce it:

styler::style_text('suit <- "♠"')
#> suit <- "♠"

^{Created on 2021-10-05 by the reprex package (v2.0.1)}

Are you on Windows?

[IndrajeetPatil]

Yes, I am on Windows. There is also session information included in my reprex if you need more details about the system.

[lorenzwalthert]

right... I think I need a Windows machine to debug this. If you want, you can already run utils::getParseData(parse(text = 'suit <- "♠"')) and tell me what it gives you, but I think the problem is the parser, not {styler}.

[IndrajeetPatil]

I see:

> utils::getParseData(parse(text = 'suit <- "♠"'))
  line1 col1 line2 col2 id parent       token terminal       text
7     1    1     1   18  7      0        expr    FALSE           
1     1    1     1    4  1      3      SYMBOL     TRUE       suit
3     1    1     1    4  3      7        expr    FALSE           
2     1    6     1    7  2      7 LEFT_ASSIGN     TRUE         <-
4     1    9     1   18  4      6   STR_CONST     TRUE "<U+2660>"
6     1    9     1   18  6      7        expr    FALSE           

[lorenzwalthert]

Ok, thanks. Not sure this should be considered a bug in the parser. We can also go ahead and replace these escaped characters such that you get "\\U2660". I think I'll contact Thomas Kalibera from R core and ask him about it, he worked on the parser in the past.

[MichaelChirico]

FWIW, it may well be styler. IIRC in potools when this came up, it was because of the function I was using to write to disk. I wrote this comment to myself: https://github.com/MichaelChirico/potools/blob/79b06798823fb1ed211efae5da716b48ed7429fc/R/write_po_file.R#L66

…

On Tue, Oct 5, 2021, 1:00 AM Indrajeet Patil ***@***.***> wrote: I see: > utils::getParseData(parse(text = 'suit <- "♠"')) line1 col1 line2 col2 id parent token terminal text 7 1 1 1 18 7 0 expr FALSE 1 1 1 1 4 1 3 SYMBOL TRUE suit 3 1 1 1 4 3 7 expr FALSE 2 1 6 1 7 2 7 LEFT_ASSIGN TRUE <- 4 1 9 1 18 4 6 STR_CONST TRUE "<U+2660>" 6 1 9 1 18 6 7 expr FALSE — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#847 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AB2BA5L5ELSXLFFNL2BUH63UFKWCRANCNFSM5FJDBTKA> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

[lorenzwalthert]

@MichaelChirico I am not sure I can follow, as the above utils::getParseData(parse(text = 'suit <- "♠"')) is neither {styler} code nor do we write or read from disk (it seems). Maybe we should set the Encoding to UTF-8 when parsing, i.e.

utils::getParseData(parse(text = 'suit <- "♠"', encoding = "UTF-8"))

(If anyone wants to tell me what this returns on Windows)
Also: UTF-8 is already enforced for file reading in styler:::read_utf8_bare(), maybe we should also do that for text when we make assumptions about the encoding in parse().

[MichaelChirico]

I may be wrong, just that the <U+...> looked familiar to me, and potools also needs to read, parse & emit source code. so at a glance suggesting a possible relation. unfortunately I also don't have a windows machine (I used iterative debugging on GitHub actions)