Inaccurate {Path,OsStr}::to_string_lossy() documentation

[rjooske]

The documentation of Path::to_string_lossy() and OsStr::to_string_lossy() says the following:

Any non-Unicode sequences are replaced with U+FFFD REPLACEMENT CHARACTER

which didn't immediately make sense to me. ("non-Unicode sequences"?)
Since both to_string_lossy functions eventually become just a call to String::from_utf8_lossy, I believe the documentation meant to say:

Any non-UTF-8 sequences are replaced with U+FFFD REPLACEMENT CHARACTER

This PR corrects this mistake in the documentation.

For the record, a similar quote can be found in the documentation of String::from_utf8_lossy:

... During this conversion, from_utf8_lossy() will replace any invalid UTF-8 sequences with U+FFFD REPLACEMENT CHARACTER, ...

[rustbot]

Thanks for the pull request, and welcome! The Rust team is excited to review your changes, and you should hear from @workingjubilee (or someone else) some time within the next two weeks.

Please see the contribution instructions for more information. Namely, in order to ensure the minimum review times lag, PR authors and assigned reviewers should ensure that the review label (S-waiting-on-review and S-waiting-on-author) stays updated, invoking these commands when appropriate:

• @rustbot author: the review is finished, PR author should check the comments and take action accordingly
• @rustbot review: the author is ready for a review, this PR will be queued again in the reviewer's queue

[workingjubilee]

Thanks!

@bors r+

[bors]

📌 Commit 49a93df has been approved by workingjubilee

It is now in the queue for this repository.

[workingjubilee]

@bors rollup

[ollie27]

On Windows it's checking for a UTF-16 sequence not UTF-8.

[workingjubilee]

...hm.

@bors r-

[workingjubilee]

@ollie27 I'm not sure whether this change isn't more correct though, since we're trying to reencode whatever the path is as UTF-8, and we only do this replacement for things that can't be otherwise encoded as UTF-8.

[workingjubilee]

Perhaps it's still not perfectly correct since it should reference the notion of the possible transcoding. But we're not actually checking for UTF-16, at least not as commonly understood, because we're rejecting invalid surrogate pairings, and everything that says it works with "UTF-16" actually also deals in such nonsense. Referencing Unicode ambiguously confuses this matter.

[ChrisDenton]

There's no re-encoding going on in to_string_lossy. It's essentially the same as to_str on valid UTF-8.

On Windows, OsStr is WTF-8 (a superset of UTF-8). Special handling for surrogates only happens when joining or splitting. As far as to_string_lossy is concerned it's either valid UTF-8 or it's not. Which is why to_str only mentions doing a validity check.

[ollie27]

Actually, yeah, #111544 exposed OsStr as a superset of UTF-8 bytes even on Windows so it does make sense to talk about non-UTF-8 sequences here even though the example shows using invalid UTF-16.

Sorry for the noise.

[workingjubilee]

@ollie27 No problem. :^)

There's no re-encoding going on in to_string_lossy.

Correct, I was talking somewhat imprecisely about the "total journey" of some bytes to an OsString to then to_string_lossy which potentially does.

Anyway, it sounds like we're back to a consensus that we should accept this even if it probably could use an even-better wording that is not leaping right into anyone's brains right now or we'd have suggested it. SO!

@bors r+

[bors]

📌 Commit 49a93df has been approved by workingjubilee

It is now in the queue for this repository.