Escape fewer Unicode codepoints in Debug impl of str

[tbu-]

Use the same procedure as Python to determine whether a character is
printable, described in PEP 3138. In particular, this means that the
following character classes are escaped:

• Cc (Other, Control)
• Cf (Other, Format)
• Cs (Other, Surrogate), even though they can't appear in Rust strings
• Co (Other, Private Use)
• Cn (Other, Not Assigned)
• Zl (Separator, Line)
• Zp (Separator, Paragraph)
• Zs (Separator, Space), except for the ASCII space ' ' 0x20

This allows for user-friendly inspection of strings that are not
English (e.g. compare "\u{e9}\u{e8}\u{ea}" to "éèê").

Fixes #34318.
CC #34422.

[rust-highfive]

r? @brson

(rust_highfive has picked a reviewer for you, use r? to override)

[sfackler]

Looks like run-pass/ifmt.rs is failing on travis.

[ollie27]

Is this changing char::escape_default? That would be a breaking change.

[tbu-]

It is changing that function. Why is it a breaking change?

[ollie27]

It's a stable function and this will break people's code which relies on the current behaviour.

[steveklabnik]

/cc @rust-lang/libs and @rust-lang/lang

On Jun 26, 2016, 18:04 -0400, Oliver Middletonnotifications@github.com, wrote:

It's a stable function and this will break people's code which relies on the current behaviour.

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly,view it on GitHub(#34485 (comment)), ormute the thread(https://github.com/notifications/unsubscribe/AABsijjdWpFL-HoADL8BmwCH9hBOw61Qks5qPveAgaJpZM4I-joR).

[ranma42]

The documentation explicitly states that any character that is not in the printable ASCII range 0x20 .. 0x7e will be escaped (some characters with ad-hoc sequences, the other ones with hexadecimal Unicode escapes), so this change would at the very least need to change the documentation of the function to match the new behaviour.
This is probably undesirable as the API is stable and the current specification guarantees that the output is only composed of ASCII characters, while the new implementation has much looser and complex guarantees (for example, it would be hard to provide an accurate specification without listing the Unicode character classes that are not escaped).

[BurntSushi]

@ranma42 Agreed. I don't think we can change this. It's very clearly changing the contract of the function.

[tbu-]

@ranma42 OK, assume for now that we don't change that function, but only the Debug implementation of str.

[eddyb]

Didn't these exact conversations happen before? Was a previous attempt abandoned?

[tbu-]

I don't know, I don't remember any.

[eddyb]

I can't find it. Might have been char instead. Don't mind me.

[ranma42]

@tbu- , yes, I think that should work. We might also want to expose it as a function on char (escape_nonprintable?), but I am unsure if this should be done directly or if we should go for an RFC to discuss the details (which Debug impls should escape non printable chars and which ones should escape everything? should we expose the API on char? under what name? are there other conventions besides the one established by Python that might be worth evaluating?).

[brson]

Not sure what I think of this.

The medium that Debug writes to is a byte sequence with no guarantees that it supports Unicode. Certainly most places the output would end up would speak UTF-8 but definitely not all. I don't recall the motivation for escaping Debug so aggressively, but it seems like the possibility of terminals not understanding it must be one reason.

[tbu-]

Actually, the output of Debug is statically guaranteed to be UTF-8.

I would guess that the reason for this is that the people implementing it didn't need non-ASCII characters, and I mean if you don't need them they're just a nuisance. But if you're implementing a non-English program, then it basically makes the Debug implementation useless (see the linked issue in which @liigo shows the unreadable "file not found" message in Chinese.

If you write to a device that doesn't support UTF-8, you should just escape these characters later, when writing to said device -- like the ascii function in Python. The other way around doesn't work.

[ranma42]

But if you're implementing a non-English program, then it basically makes the Debug implementation useless

A possible objection is that

Debug should format the output in a programmer-facing, debugging context.

which seems to imply that it should not be exposed to the users, but rather to tools or developers.
For example, one could say that formatting errors using Debug is not correct if they are going to be shown to the user.

[tbu-]

@ranma42 The linked example for a programmer, @liigo.

[ranma42]

@tbu- not really, in that case he is the user of the rustc program.

[tbu-]

@ranma42 It's a runtime error provided by the operating system, encountered while programming.

EDIT: Also, you could probably look into the PEP, they also give a longer motivation in there.

[liigo]

Debug should format the output in a programmer-facing, debugging context.

I'd rather they don't do these (mostly useless) format/escape for me (a programmer). They (self-righteous people) thought these format may help programmers in debugging context, but it's not true. In some times, non-English programmers got unreadable formatted output, worse than Display output. I don't use very-old output devices, which maybe have problems to display Unicode characters, which really need escape.

@tbu- Maybe we can't change escape_default for compatible reasons, but changing impl Debug for str is just implementation details IMHO.

[ranma42]

@tbu- It is a runtime error provided by the operating system, encountered by rustc and dumped on the console in a bad way. What I am saying is that it is not obvious to me that the correct way to display it is using Debug instead of Display.

[liigo]

@brson: it seems like the possibility of terminals not understanding it must be one reason.

If this was the reason, we should also implement println! to escape Unicode characters.

[liigo]

@ranma42: to display it is using Debug instead of Display

That'd be a big breaking change. Not all Debug types implement Display.

[ranma42]

@liigo Yes, that would be a major breaking change (it would change the constraints on the Result type so that E: fmt:Display).

[ranma42]

To me, the major advantage of the current implementation of Debug for str (always escaping non-ASCII) is that it allows to distinguish between different strings that have the same visual appearance. This applies in the same way to whitespace ('\t' vs spaces), Unicode combining characters and control characters... and maybe more?

Of course this does not mean that it should be used for everything. Specifically, I would only use Debug when the representation of the data is relevant, not when it just needs to be shown to the user. Basically this reasoning is my understanding of the documentation of Debug and Display (programmer-facing vs user-facing). This can of course be incorrect, but in that case I would also ask for an improvement in the documentation ;)

#34318 shows an example where using Debug looks inappropriate (an OS-provided localised error). A partial solution that would have limited impact on other things would be to avoid escaping only the OS localised errors here. I would not even call this a breaking change, given that the output is already not explicitly stabilised (it obviously depends on the current locale and most likely on the operating system + version).

Even though Rust does not (yet) have its own localised error messages, it would not be hard to imagine the same issue affecting other types of output, so it might be a good idea to think of a more general solution to ensure a way forward in this direction.

[tbu-]

@ranma42 If you want to see the exact code points, why only make an exception for English? That's very English-centric. :) EDIT: Imagine the Debug implementation showed an escape sequence for every latin character instead of the actual character. That would be quite a pain to program with, right? That's what it's like if the program deals with, say Chinese strings.

We should probably provide a function that does the same as Debug today, like Python has: ascii. This basically solves the problem. If you need a Debug string like today, you can just put the Debug output into said function and you receive what you wanted.

[liigo]

To me, the major advantage of the current implementation of Debug for str (always escaping non-ASCII) is that it allows to distinguish between different strings that have the same visual appearance.

This is not advantage for non-ASCII text. It just makes unreadable noise (\u{xxxx}...). If someone do think this is a major advantage, please let println! have this advantage too.

[tbu-]

@brson Could you @bors r-, there's another test failure in the WTF-8 code.

[alexcrichton]

@bors: r-

[tbu-]

@alexcrichton It should be fixed now.

[alexcrichton]

@bors: r+ 3d09b4a

[bors]

⌛ Testing commit 3d09b4a with merge d1df3fe...

[bors]

☀️ Test successful - auto-linux-32-nopt-t, auto-linux-32-opt, auto-linux-64-cargotest, auto-linux-64-cross-freebsd, auto-linux-64-debug-opt, auto-linux-64-nopt-t, auto-linux-64-opt, auto-linux-64-opt-mir, auto-linux-64-opt-rustbuild, auto-linux-64-x-android-t, auto-linux-cross-opt, auto-linux-musl-64-opt, auto-mac-32-opt, auto-mac-64-opt, auto-mac-64-opt-rustbuild, auto-mac-cross-ios-opt, auto-win-gnu-32-opt, auto-win-gnu-32-opt-rustbuild, auto-win-gnu-64-opt, auto-win-msvc-32-opt, auto-win-msvc-64-cargotest, auto-win-msvc-64-opt, auto-win-msvc-64-opt-mir, auto-win-msvc-64-opt-rustbuild
Approved by: alexcrichton
Pushing d1df3fe to master...

[SimonSapin]

I’m very late to say this, but this adds 2102 bytes of static data to libcore, whereas previously all large Unicode tables were in the std_unicode crate that #![no_std] programs could opt not to use. 2 KB may not seem like much, but it’s significant when programming a micro-controller that has 16 KB of flash memory.

[aturon]

@SimonSapin I opened #39492 for this issue, and to propose a general policy.

[ariasuni]

I’d like to know what the code do precisely (what are SINGLETONS0U, SINGLETONS0L, NORMAL0 and check()) so I could try to optimize it, at least in size. Also, this code should to belong to libstd_unicode, right?

[SimonSapin]

@ariasuni The commit message and PR message give a list of Unicode categories of characters that are escape, but yes if it’s not already this list should also be in some doc-comment in the code.

I agree that I’d prefer to have these tables in libstd_unicode, but libcore still needs some impl of Debug for str. There can only be one such impl in a program, I don’t know if it’s possible to make it do something different based on whether libstd_unicode is available.

[tbu-]

@ariasuni The high-level view is that you have to store 0x110000 booleans, one for each Unicode code point, which specify whether the given code point should be printed or not (according to the description in the first post of this PR).

The low-level view of this particular implementation seems to have changed since I have implemented it, you can find some notes on the new one in 44bcd26.

[ariasuni]

@SimonSapin We can split libstd_unicode in unicode-casing and unicode-categories, and use only the latter to implement is_printable(). What do you think about it?

[SimonSapin]

@ariasuni even if we did that, that doesn’t solve the situation that a #[no_std] program needs a Debug for str impl and ideally should also be able to opt out of embedding large Unicode tables. If we move is_printable to a separate crate but make libcore depend on that crate, we’ve moved things around without actually changing anything for Rust users. If we add a Cargo feature to disable these including tables (#39492), it doesn’t change much that it’s a separate crate or not.

[ariasuni]

Sorry, I probably overthought it. The idea is to put the code for Unicode categories elsewhere so that we can use it in is_printable() even when using #[no_std] (right now without waiting for the Cargo feature), and this would reduce the code size for those who don’t. Not sure about the stability implications of moving code around thought.

[SimonSapin]

I don’t understand how this would reduce code size, I may be missing something.

[ariasuni]

If I understand correctly, is_printable() is using Unicode categories data in char_private.rs, instead of the data in libstd_unicode used by is_control() is_alphabetic(), is_numeric(), etc.