fread() throws correct non-ASCII messages

[shrektan]

Fixes #4747

Before the PR the last line throw warnings with garbage letters. This PR fixes it.

Example code

x <- "fa\xE7ile"
Encoding(x) <- "latin1"
txt <- sprintf("A,B\n%s,%s\n%s", x, x, x)
Encoding(txt) <- "UTF-8"
data.table::fread(text = txt, encoding = 'UTF-8')
txt2 <- iconv(txt, "UTF-8", "latin1")
data.table::fread(text = txt2, encoding = 'Latin-1')

Before

Warning message:
In data.table::fread(text = txt2, encoding = "Latin-1") : Discarded single-line footer: <<fa�ile>>

After

Warning message:
In data.table::fread(text = txt, encoding = "UTF-8") :
  Discarded single-line footer: <<façile>>

[codecov]

Codecov Report

Merging #4751 (7f3ae4f) into master (789dfe4) will not change coverage.
The diff coverage is 100.00%.

@@           Coverage Diff           @@
##           master    #4751   +/-   ##
=======================================
  Coverage   99.47%   99.47%           
=======================================
  Files          73       73           
  Lines       14560    14560           
=======================================
  Hits        14483    14483           
  Misses         77       77           

Impacted Files	Coverage Δ
src/freadR.c	100.00% <ø> (ø)
src/fread.c	99.47% <100.00%> (ø)

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 789dfe4...7f3ae4f. Read the comment docs.

[MichaelChirico]

cc @st-pasha , not sure if there's something to do here on the Python side

[jangorecki]

I understand we need this for translateChar(mkCharCE(s, ienc)), right?
It may be good to ask R core for public API for this functionality.

[mattdowle]

Maybe but I have the impression from WRE that Rinternals.h is already a public API, at least partially. For example, the string "#include <Rinternals.h>" is exhibited 13 times in WRE. We need it even for just SEXP (which is stated too in WRE). I did find one use of the word 'public' in the context of Rinternals.h:

Note 144 in full is "see The R API: note that these are not all part of the API."
I remember seeing some debate on R-devel mailing list on this over the years. Some R-core members believe that the R API is anything mentioned in WRE. But that would exclude many of the exported (i.e. non-hidden) functions that many packages use. I believe that #define USE_RINTERNALS before #include <Rinternals.h> (as data.table used to do in the past) does access truly internal non-API functions/structures. But including Rinternals.h is not necessarily accessing non-public API, in fact it's necessary to include it to access all the public API. Note that data.table.h includes Rinternals.h but that doesn't necessarily mean we are using non-public API.

With that said, translateChar and mkCharCE are both highlighted by name in WRE (section 5.15) and for that reason there would be no disagreement by any R-core member that those are already public API.

[MichaelChirico]

that seems... needlessly confusing!

[mattdowle]

I agree. All those that have thought so have engaged. Then what you discover is that R-core members disagree between themselves. Tangent issues (such as data.table using SET_TRUELENGTH) are quickly raised. I assume what you would like is for R.h to be public and Rinternals.h to be not-public? Underlying that seemingly simple task is i) change to R over time, and ii) in our case of data.table, achievements by using SET_TRUELENGTH that some core members wish was never accessible.
Careful what you wish for.

[jangorecki]

Yes, sorry for fuss. When I read Rinternals I had in mind USE_RINTERNALS switch. So definitely confusing.