Fix some UTF-8 issues on Windows.

[nitric1]

This fixes #9418 and #9618, and potential problems related to directory walking.

[alexcrichton]

Can you add some comments explaining what these constants are and why they're defined?

[nitric1]

Sorry for my short description. Some comments added.

On Windows, most of char version of functions treat strings as ANSI, not UTF-8, so code using them should be changed to use wchar_t version (like wcsftime, _wstat, ....)

Related issue: #9822

Closes #9418
Closes #9618

[alexcrichton]

Hm it's unfortunate that this probably means that most of our usage of windows apis are "broken" in the sense that we're not dealing with unicode strings properly. This seems like some that's tricky to get right, so I'm wary of adding this sort of functionality without an ability to prevent it regressing. Could you also add some tests with unicode filenames and such to make sure that these apis properly work? The time zone problem may be tough to test, but it would be great if we were able to get it testable as well.

[alexcrichton]

Also I'm still a bit curious as to why we need to use all the wide versions everywhere. If one still take char* to be a "null-terminated string" then most of utf8 still fits into that description. So when you're calling something like stftime with a unicode locale, what happens? Does windows assert? Do we assert?

[klutzy]

most of winapi/libc functions taking char* just don't use utf8. This is an example I actually met before.
I made a meta-issue #9822 since it looks bizarre for non-windows people and it is problematic everywhere.

[nitric1]

Sorry, "most of char version of functions" was not appropriate. Yes, it's safe to use char-using functions if they just treat it as ASCII (atoi, strtol, ...) or "null-terminated" string (strcpy, strlen, ....) As far as I know locale (setlocale, strptime, ...) or io (fopen, readdir, ...) related functions are problematic.

If a locale related argument (like %A, %B, %X, %Z, etc) is given to strftime on Windows, it can return ANSI string, which Rust cannot accept (UTF-8 assert). On the other hand, non-ASCII UTF-8 string is given to fopen, it will fail.

[alexcrichton]

So just to make sure I understand this correctly, if windows hands us a char*, then it works for all files, it's just not necessarily utf-8, so we use the W versions of functions to hand us utf-16 strings that we can re-encode in utf-8. And then if we call fopen with a string exercising non-ascii characters (but nonzero characters), the function just outright fails? Does it abort? Does it return failure? If it returns failure, what's the strerror reported for it?

[nitric1]

if windows hands us a char*, then it works for all files,

No, it doesn't work for some files which contain non-ANSI[1] character as file name.

so we use the W versions of functions to hand us utf-16 strings that we can re-encode in utf-8.

One more reason is A versions of functions cannot handle some files as I wrote previous sentence.

And then if we call fopen with a string exercising non-ascii characters [...]

fopen returns NULL, and strerror states "No such file or directory".

[1] ANSI is local system encoding except unicode encodings, such as ISO-8859-1, CP949, Shift_JIS. Windows has (almost) fully featured unicode functionality, but ANSI ones remain for backward compatibility.

[alexcrichton]

OK that makes sense to me now, so could you do a few things?

• Add some comments to std::os explaining why different versions are used for windows
• Add some tests as well which create paths with some fun filenames, and then test that these functions succeed on windows?

[nitric1]

Added comments, tests, and rebased.

Force push... :(

[alexcrichton]

I think that this will fail to compile on osx and linux because these symbols aren't defined in rust_builtin.cpp. Feel free to just make them empty functions, that's been the pattern for platform-specific C++ helpers so far.

[alexcrichton]

Nice job, and thanks again! Just two things:

• Would you mind rebasing these into one commit? It's a pretty good change for one decent-sized commit, and the commit can have all the details about what went on.
• Minor comment about stub versions of rust_path_exists_u16 and friends for linux/osx

[nitric1]

Thanks for review! I think it's all done, but if there's something wrong, please let me know.

[nitric1]

I didn't consider linux; fix amended.

[klutzy]

path_is_dir test failed on mac-64-nopt-t, but passed on mac-64-opt. Is it just flaky?

[alexcrichton]

Hm, this is using some test files in the os's tmp directory, but builders can run simultaneously on one machine, so I don't think that this is playing nicely when that's happening. Each test needs to choose a unique location to test its paths/directories. There isn't the convenience of extra's tmpdir interface, but you could work around it but just generating some random strings to prepend on the path instead.

[nitric1]

Temporary path randomized; there will not be race conditions, hopefully.