Raw Byte strings do not handle \r

[matklad]

Everywhere in the language we treat \r\n as \n, and forbid bare \r. In particular, in raw strings we do translate \r\n to \n, which makes sense, given that line ending depends on git config.

However, for raw byte string literals we don't do this, and this looks like a bug?

16:22:13|~/tmp
λ echo -e "fn main() {\n    let (s1, s2) = (br\"\r\", br\"\r\n\");\n    println!(\"{:?} {:?}\", s1, s2);\n}" > main.rs

16:25:49|~/tmp
λ rustc main.rs

16:25:50|~/tmp
λ ./main
[13] [13, 10]

[matklad]

cc @petrochenkov

[matklad]

I guess the best way to solve this is to move \r\n -> \n logic to the unescape module.

[petrochenkov]

I wanted to create an issue about this as well!

I think translation of \r\n into \n and prohibition of bare \r (*) should be consistently done for all literals.
Perhaps even before tokenization (so that proc macros cannot observe \r\n) (not sure about this, it can potentially mess up byte position in files and in code map or something).

(*) The prohibition of bare \r is mostly a conservative and maybe unnecessary measure. Since \r was used as a newline on some legacy (but not truly ancient) systems, it could be reasonable to treat it as one and translate it to \n, and it is also reasonable to pass \r as is, so the error is mostly a refusal to choose.
(It should be covered by a generic lint for confusing invisible characters if the error is removed.)

As a variation of raw byte string literals, include_bytes!() also doesn't translate \r\n, but it can also contain non UTF-8, so I thinks it's ok to keep it as an escape hatch for entirely untranslated bytes.

[matklad]

Perhaps even before tokenization

+1 here. Having both \n and \r\n in the source is especially painful in the IDE, where you need to insert line breaks during refactorings. Figuring this out on a case-by-case basis is pretty horrible. We have an open issue (rust-lang/rust-analyzer#775) about python-style translation of \r\n <-> \n at the IO boundary (in IDE, the translation needs to be bidirectional)

[petrochenkov]

python-style translation of \r\n <-> \n at the IO boundary

Not only Python, C++ also assumes that the newline translation happens at IO boundary during source file loading.

My only concern is that it would prevent mmaping source files directly, but I don't know how important that may be.

[matklad]

Not an expert in mmap, but my understanding is that, in the context of compilers, it won't bring a lot of benefits (files are small and read sequentially and fully), but would cost a lot of complexity (errors during mmap are delivered via signals). Anyway, that's waaaay beyond the scope of this issue :)

I'll look into fixing this this week.

[Xanewok]

I'm working on it.

[matklad]

closed by #60793