Add support for UWP targets

[chouquette]

Hi,

This pull request aims at adding support for UWP (Universal Windows Apps) platform.
A few notes:

• This requires a very recent mingw-w64 version (containing this commit and the previous related ones: mirror/mingw-w64@e8c433c#diff-eefdfbfe9cec5f4ebab88c9a64d423a9)
• This was tested using LLVM/clang rather than gcc, and so far it assumes that LLVM/clang will be the native compiler. This is mostly due to the fact that the support for exceptions/stack unwinding for UWP got much more attention in libunwind
• The "uwp" part of the target needs support for it in the cc-rs & backtrace-rs crates. I'll create the MR there right after I submit this one and will link everything together, but I'm not sure what's the correct way of dealing with external dependencies in the context of rust
• Enabling import libraries and copying them across stages requires a change in cargo, for which I'll open a MR right after I submit this one as well
• The i686 stack unwinding is unsupported for now, because LLVM assumes SjLj, while rust seems to assume SEH will be used. I'm unsure how to fix this

Also, this is my first encounter with rust, so please bear with my code, it might not feel so idiomatic or even correct :)

I'm pretty sure there's a way of doing things in a cleaner way when it comes to win/c.rs, maybe having a UWP & desktop specific modules, and import those conditionally? It doesn't feel right to sprinkle #[cfg(...)] all over the place

Off course, I'll gladly update anything you see fit (to the extent of my abilities/knowledge :) )!

Thanks,

[rust-highfive]

Thanks for the pull request, and welcome! The Rust team is excited to review your changes, and you should hear from @cramertj (or someone else) soon.

If any changes to this PR are deemed necessary, please add them as extra commits. This ensures that the reviewer can see what has changed since they last reviewed the code. Due to the way GitHub handles out-of-date commits, this should also make it reasonably obvious what issues have or haven't been addressed. Large or tricky changes may require several passes of review and changes.

Please see the contribution instructions for more information.

[rust-highfive]

The job x86_64-gnu-llvm-6.0 of your PR failed on Travis (raw log). Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log.

travis_time:end:0943f614:start=1556198122940102368,finish=1556198208740101322,duration=85799998954
$ git checkout -qf FETCH_HEAD
travis_fold:end:git.checkout

Encrypted environment variables have been removed for security reasons.
See https://docs.travis-ci.com/user/pull-requests/#pull-requests-and-security-restrictions
$ export SCCACHE_BUCKET=rust-lang-ci-sccache2
$ export SCCACHE_REGION=us-west-1
$ export GCP_CACHE_BUCKET=rust-lang-ci-cache
$ export AWS_ACCESS_KEY_ID=AKIA46X5W6CZEJZ6XT55
---

[00:03:15] travis_fold:start:tidy
travis_time:start:tidy
tidy check
[00:03:15] tidy error: /checkout/src/librustc_codegen_ssa/back/linker.rs:407: trailing whitespace
[00:03:17] some tidy checks failed
[00:03:17] 
[00:03:17] 
[00:03:17] command did not execute successfully: "/checkout/obj/build/x86_64-unknown-linux-gnu/stage0-tools-bin/tidy" "/checkout/src" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage0/bin/cargo" "--no-vendor" "--quiet"
[00:03:17] 
[00:03:17] 
[00:03:17] failed to run: /checkout/obj/build/bootstrap/debug/bootstrap test src/tools/tidy
[00:03:17] Build completed unsuccessfully in 0:00:45
[00:03:17] Build completed unsuccessfully in 0:00:45
[00:03:17] Makefile:67: recipe for target 'tidy' failed
[00:03:17] make: *** [tidy] Error 1
The command "stamp sh -x -c "$RUN_SCRIPT"" exited with 2.
travis_time:start:026c2957
$ date && (curl -fs --head https://google.com | grep ^Date: | sed 's/Date: //g' || true)
Thu Apr 25 13:20:15 UTC 2019
---
travis_time:end:0a7543e8:start=1556198416643725526,finish=1556198416648335453,duration=4609927
travis_fold:end:after_failure.3
travis_fold:start:after_failure.4
travis_time:start:1892e129
$ ln -s . checkout && for CORE in obj/cores/core.*; do EXE=$(echo $CORE | sed 's|obj/cores/core\.[0-9]*\.!checkout!\(.*\)|\1|;y|!|/|'); if [ -f "$EXE" ]; then printf travis_fold":start:crashlog\n\033[31;1m%s\033[0m\n" "$CORE"; gdb --batch -q -c "$CORE" "$EXE" -iex 'set auto-load off' -iex 'dir src/' -iex 'set sysroot .' -ex bt -ex q; echo travis_fold":"end:crashlog; fi; done || true
travis_fold:end:after_failure.4
travis_fold:start:after_failure.5
travis_time:start:0177dd68
travis_time:start:0177dd68
$ cat ./obj/build/x86_64-unknown-linux-gnu/native/asan/build/lib/asan/clang_rt.asan-dynamic-i386.vers || true
cat: ./obj/build/x86_64-unknown-linux-gnu/native/asan/build/lib/asan/clang_rt.asan-dynamic-i386.vers: No such file or directory
travis_fold:end:after_failure.5
travis_fold:start:after_failure.6
travis_time:start:234a7718
$ dmesg | grep -i kill

I'm a bot! I can only do what humans tell me to, so if this was not helpful or you have suggestions for improvements, please ping or otherwise contact @TimNN. (Feature Requests)

[chouquette]

Argh, sorry I missed the If any changes to this PR are deemed necessary, please add them as extra commits. and updated the commit directly to remove the trailing whitespace. I'll push extra commits for future changes.

[retep998]

I'm curious as to why you decided to start with -gnu first and not add support for UWP in -msvc?

[cramertj]

@bors r? @alexcrichton

[chouquette]

I'm curious as to why you decided to start with -gnu first and not add support for UWP in -msvc?

Very fair question :) I guess mostly because I have more experience in cross compiling to uwp than I have with compiling using visual studio.
That being said, most if not all of the libstd commits will also be required there.
A good part of this PR is also fixing some GCC-ism which don't hold when using llvm, those obviously won't be required when adding a MSVC target

[mati865]

I think you will have to setup new mingw-w64 toolchain for it. Currently Rust CI is using 2 year old GCC based toolchain:

rust/appveyor.yml

Line 119 in e8310a7

MINGW_ARCHIVE: x86_64-6.3.0-release-posix-seh-rt_v5-rev2.7z

[chouquette]

I think you will have to setup new mingw-w64 toolchain for it. Currently Rust CI is using 2 year old GCC based toolchain:

What are the options? I'm not sure there are prebuilt version with the required mingw-w64 patches yet. For VLC we have a bunch of docker images which can be used, but I'd understand if this is not an option
I can probably also ask Martin to ship a new tarball of https://github.com/mstorsjo/llvm-mingw/, which would essentially be what's built in the docker images

[mati865]

What are the options?

I also think Martin's toolchain would be the best bet here. I think somebody could build it with docker and upload to https://s3-us-west-1.amazonaws.com/rust-lang-ci2/rust-ci-mirror

[rust-highfive]

The job mingw-check of your PR failed on Travis (raw log). Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log.

travis_time:end:27ee211c:start=1556276263052822885,finish=1556276355746389353,duration=92693566468
$ git checkout -qf FETCH_HEAD
travis_fold:end:git.checkout

Encrypted environment variables have been removed for security reasons.
See https://docs.travis-ci.com/user/pull-requests/#pull-requests-and-security-restrictions
$ export SCCACHE_BUCKET=rust-lang-ci-sccache2
$ export SCCACHE_REGION=us-west-1
$ export GCP_CACHE_BUCKET=rust-lang-ci-cache
$ export AWS_ACCESS_KEY_ID=AKIA46X5W6CZEJZ6XT55
---
[00:07:03] configure: build.locked-deps    := True
[00:07:03] configure: llvm.ccache          := sccache
[00:07:03] configure: build.cargo-native-static := True
[00:07:03] configure: dist.missing-tools   := True
[00:07:03] configure: build.configure-args := ['--enable-sccache', '--disable-manage-submodu ...
[00:07:03] configure: writing `config.toml` in current directory
[00:07:03] configure: 
[00:07:03] configure: run `python /checkout/x.py --help`
[00:07:03] configure: 
---
travis_time:end:33683f54:start=1556276898876733852,finish=1556276898881398951,duration=4665099
travis_fold:end:after_failure.3
travis_fold:start:after_failure.4
travis_time:start:015aa880
$ ln -s . checkout && for CORE in obj/cores/core.*; do EXE=$(echo $CORE | sed 's|obj/cores/core\.[0-9]*\.!checkout!\(.*\)|\1|;y|!|/|'); if [ -f "$EXE" ]; then printf travis_fold":start:crashlog\n\033[31;1m%s\033[0m\n" "$CORE"; gdb --batch -q -c "$CORE" "$EXE" -iex 'set auto-load off' -iex 'dir src/' -iex 'set sysroot .' -ex bt -ex q; echo travis_fold":"end:crashlog; fi; done || true
travis_fold:end:after_failure.4
travis_fold:start:after_failure.5
travis_time:start:024c0612
travis_time:start:024c0612
$ cat ./obj/build/x86_64-unknown-linux-gnu/native/asan/build/lib/asan/clang_rt.asan-dynamic-i386.vers || true
cat: ./obj/build/x86_64-unknown-linux-gnu/native/asan/build/lib/asan/clang_rt.asan-dynamic-i386.vers: No such file or directory
travis_fold:end:after_failure.5
travis_fold:start:after_failure.6
travis_time:start:015f999c
$ dmesg | grep -i kill

I'm a bot! I can only do what humans tell me to, so if this was not helpful or you have suggestions for improvements, please ping or otherwise contact @TimNN. (Feature Requests)

[alexcrichton]

Thanks for the PR for this @chouquette! I'll say up front that I know very little about UWP, its ecosystem, and its idioms, so sorry if anything below is naive or seems like it should have an obvious answer!

This naturally seems like it's "kind of a big deal" in the sense that I want to make sure we move on this deliberately, and it's great that it doesn't really look like too much is required to get things working. Could you explain a bit about UWP idioms and background? From some reading it sounds like it's sort of an alternative C library, almost like glibc vs musl, is that right? Are UWP still compiled to native executables and they're just run in a slightly different environment, or are they exclusively just using different APIs internally?

In terms of the patch here the biggest worry on my end is what you've already mentioned which is the #[cfg] being sprinkled sort of "seemingly randomly" throughout. The unix port of the standard library doesn't really fare a ton better in this regard, but it's always best where possible to absolutely minimize #[cfg] usage of leverage if cfg!(...) if you can. It'd be even best if we could just switch the main Windows targets to relying on the same APIs as UWP if we can (I haven't checked the specifics, but if it's good enough for UWP it seems like it would in theory be good enough for the main target as well).

In terms of naming is there any precedent we can draw from? Do clang/MSVC/others already have names for these targets that we should be drawing from? It naively seems like it'd be nice to have the word "windows" somewhere in there.

@retep998 can you speak more to perhaps what this will require on the ecosystem side of things in Rust? Is winapi prepared to start supporting a target like this? Do you have thoughts on how the target should be set up?

Also and finally, the full suite of support options for a new target like unwinding, stack probes, etc, is all fine to defer to later. It's good to get as much working now as you can, but it doesn't have to be a full and complete target right out the gate.

[chouquette]

Hi,

No worries, I'll have probably naive rust questions in my reply, so it's all fair :)

The TL;DR for UWP is that it's the same architecture as a regular Windows, but you indeed get only a subset (and a very small superset) of the usual APIs. It's rather hard to give an overview of what is available or not, because they tend to change with every new versions (though the trend is to make more functions available). The current list is there: https://docs.microsoft.com/en-us/uwp/win32-and-com/win32-apis
The counterpart of UWP, so, your usual win32 application is usually referred to as a "desktop app"
Allegedly, a universal app will work the same on all supported platforms, so regular computers, Xbox One, Hololens, and more or less on Windows Mobile which is not supported anymore.
If you want your application to be available through the Windows Store, you need to validate it using the Windows Application Certification Kit (WACK), which will check that you're not using forbidden functions.
On regular computers, all functions are available in a way, since you're using the same shared libraries as your regular applications (at least this was the case a few years ago, I'm unsure now), but it changes drastically on Xbox one at least, where kernel32.dll isn't available.
Linking to the correct dll is handled by mingw-w64, and everything is dispatched according to the documentation linked above.

I agree with the #[cfg] but here comes the rather naive rust questions! I tried a few times to use if cfg!(...) to avoid duplicating an entire function, but my understanding is that it will replace the predicate by the resulting boolean literal; so the body of the condition might or might not be compiled/linked, depending on the optimizer. So far it was always being built. Does that change when used in the global scope (not sure of the terminology here, but basically out of a function definition)
If so I'll gladly regroup all UWP specific declarations and all desktop specific declarations together!

About naming, since it's the same architecture, it's usually the same one.
When using gcc, we've always used <arch>-w64-mingw32 like when targeting a regular windows. This is forcing us to use specfiles to fiddle with what gets linked by default (https://code.videolan.org/videolan/vlc-winrt/blob/master/libvlc/compile.sh#L107 ; watch your eyes :) )
In all fairness, I don't know if this has changed with more recent versions of GCC

LLVM/clang is a bit smarter, and will understand if we're targeting UWP based on the libraries that we're linking with, so using -lwindowsapp basically triggers the UWP mode (not linking with -lkernel32, -lole32 and various of their friends)

In retrospect, I tend to think that using the same triplet for both desktop versions and UWP versions was a mistake, hence the new one. That being said, I agree that it might be more obvious to mention windows (and as a matter of fact it would probably reduce the size of the patchset, but I didn't check if some places depend on the -pc- of the triplet.)
If you prefer, I can look into switching to <arch>-uwp-windows<gnu/msvc>?

When possible, I used the UWP variants instead of the old ones, but sadly the only function that was both allowed on UWP and available since Windows XP was SHGetFolderPathW (afc92f2)

[mati865]

Answering missed questions:

From some reading it sounds like it's sort of an alternative C library, almost like glibc vs musl, is that right?
[...]
It'd be even best if we could just switch the main Windows targets to relying on the same APIs as UWP if we can

UWP works only on Windows 10+, Xbox One and similar.

What you are asking for is UCRT, it's supposed to replace MSVCRT in the future.
I think late MSVC versions already link to it and mingw-w64 can be built to target it but that would probably break users with MSVCRT mingw-w64. It's available in Windows 7 (with Windows Updates installed) and newer.

[chouquette]

Someone just made me realize that SHGetFolderPathW is not available on UWP... turns out it was building because I included libwinstorecompat... (https://github.com/mirror/mingw-w64/blob/master/mingw-w64-libraries/winstorecompat/src/SHGetFolderPathW.c) but I suppose I'll need to write another function that does something more useful :)

[retep998]

UWP apps run in a different environment from desktop apps. They have access to a different subset of Windows API with some overlap and they are generally much more restricted and sandboxed than desktop apps. The intention of UWP is to have a single code base that can work on any UWP platform such as Windows 10, Xbox One, Windows Mobile, etc. When naming these targets we'd have to think about future targets for UWP on other devices. What would an Xbox One target be called for example?

For winapi I do intend to support the new UWP targets. I'm mainly just blocked on an official decision on what the new targets will be called. Most of the work will be in determining which libraries to link to and refactoring the build script for that.

For crates that use winapi, the work will be similar to std where they will have to cfg various things to call one API for desktop apps and call another API for UWP apps. Pure Rust crates that do not call winapi directly will generally work on UWP without modification as long as their dependencies also work on UWP. However some functionality simply isn't available on UWP and so some crates may never work.

[alexcrichton]

Ok thanks for all the info! @retep998 do you have thoughts/opinions about the target naming? @chouquette's suggestion of $arch-uwp-windows-$env seems not bad, although I'm not sure if we'd also have specific targets for things like Xbox/etc. It's perhaps a good enough starting place and we could base an xbox target off the uwp one if necessary?

@chouquette you're right yeah that cfg! uses a boolean, so it's not great for tweaking linkage since we don't want to rely too much on the optimizer for what symbols to link to. That means it can't be used everywhere, but I think we'd probably want to use it in as many places as possible since it's generally easier to read. I'll comment with a few possible places below.

[chouquette]

What are the options?

I also think Martin's toolchain would be the best bet here. I think somebody could build it with docker and upload to https://s3-us-west-1.amazonaws.com/rust-lang-ci2/rust-ci-mirror

I can easily build the docker but I doubt I can push to your registry, I can also build the latest toolchain and put it somewhere for you to build a docker out of it. Let me know what you want me to do and I'll happily help

[alexcrichton]

For platform support, Rust technically only supports Windows 7+. Historically we've taken patches to make libstd compatible with all-the-way-back-to-XP for projects, but lots of time has passed since that last happened and one of our flagship users of this support, Firefox, no longer supports XP and Windows Vista. In that sense I believe it should be ok to land code that requires Windows 7+.

For building and toolchains, I think this PR should focus on landing support in the compiler, rustbuild, and libstd. We can add it to CI afterwards in a follow-up PR

[rust-highfive]

The job mingw-check of your PR failed on Travis (raw log). Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log.

travis_time:end:1cf60c26:start=1556641649167640406,finish=1556641736659844503,duration=87492204097
$ git checkout -qf FETCH_HEAD
travis_fold:end:git.checkout

Encrypted environment variables have been removed for security reasons.
See https://docs.travis-ci.com/user/pull-requests/#pull-requests-and-security-restrictions
$ export SCCACHE_BUCKET=rust-lang-ci-sccache2
$ export SCCACHE_REGION=us-west-1
$ export GCP_CACHE_BUCKET=rust-lang-ci-cache
$ export AWS_ACCESS_KEY_ID=AKIA46X5W6CZEJZ6XT55
---
[00:07:34] configure: build.locked-deps    := True
[00:07:34] configure: llvm.ccache          := sccache
[00:07:34] configure: build.cargo-native-static := True
[00:07:34] configure: dist.missing-tools   := True
[00:07:34] configure: build.configure-args := ['--enable-sccache', '--disable-manage-submodu ...
[00:07:34] configure: writing `config.toml` in current directory
[00:07:34] configure: 
[00:07:34] configure: run `python /checkout/x.py --help`
[00:07:34] configure: 
---
travis_time:end:00506de5:start=1556642319166596412,finish=1556642319172781229,duration=6184817
travis_fold:end:after_failure.3
travis_fold:start:after_failure.4
travis_time:start:1c786fbf
$ ln -s . checkout && for CORE in obj/cores/core.*; do EXE=$(echo $CORE | sed 's|obj/cores/core\.[0-9]*\.!checkout!\(.*\)|\1|;y|!|/|'); if [ -f "$EXE" ]; then printf travis_fold":start:crashlog\n\033[31;1m%s\033[0m\n" "$CORE"; gdb --batch -q -c "$CORE" "$EXE" -iex 'set auto-load off' -iex 'dir src/' -iex 'set sysroot .' -ex bt -ex q; echo travis_fold":"end:crashlog; fi; done || true
travis_fold:end:after_failure.4
travis_fold:start:after_failure.5
travis_time:start:20a5c120
travis_time:start:20a5c120
$ cat ./obj/build/x86_64-unknown-linux-gnu/native/asan/build/lib/asan/clang_rt.asan-dynamic-i386.vers || true
cat: ./obj/build/x86_64-unknown-linux-gnu/native/asan/build/lib/asan/clang_rt.asan-dynamic-i386.vers: No such file or directory
travis_fold:end:after_failure.5
travis_fold:start:after_failure.6
travis_time:start:0b20c299
$ dmesg | grep -i kill

I'm a bot! I can only do what humans tell me to, so if this was not helpful or you have suggestions for improvements, please ping or otherwise contact @TimNN. (Feature Requests)

[chouquette]

Better now. I removed the debug commits & the MSVC configuration for the time being and I'll work on it again with its original author

[alexcrichton]

@bors: r+

[bors]

📌 Commit 7ed5c36 has been approved by alexcrichton

[bors]

⌛ Testing commit 7ed5c36 with merge 4268e7e...

[bors]

☀️ Test successful - checks-azure
Approved by: alexcrichton
Pushing 4268e7e to master...