Skip to content

Build host toolchain w/o copying it from snapshots #1024

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 15 commits into from
May 20, 2020

Conversation

MaxDesiatov
Copy link

@MaxDesiatov MaxDesiatov commented May 16, 2020

Currently the host toolchain is copied over to the target WASI toolchain from snapshots downloaded from swift.org. This is not ideal as snapshots on swift.org are not always fresh and don't always match what we have in the swiftwasm branch or even in the upstream master branch. This causes issues similar to what can be seen in #1000, where the host toolchain and SDK don't match the target toolchain with this error:

error: compiled module was created by an older version of the compiler; 
rebuild 'Swift' and try again: 
/usr/lib/swift/macosx/Swift.swiftmodule/x86_64-apple-macos.swiftmodule

The solution is to:

  1. build the host toolchain together with the host SDK and SwiftPM as the first part of the CI process
  2. install the host products into the $SOURCE_PATH/host-toolchain-sdk directory
  3. clean up the build directory
  4. build the target SwiftWasm SDK
  5. install the target SwiftWasm SDK into the $SOURCE_PATH/install directory
  6. copy over WASI SDK clang/llvm-ar/wasm-ld into the $SOURCE_PATH/host-toolchain-sdk directory
  7. copy over the target SwiftWasm SDK into the $SOURCE_PATH/host-toolchain-sdk directory
  8. create an archive from the $SOURCE_PATH/host-toolchain-sdk directory

Thus build-swiftpm.sh script is no longer needed as SwiftPM is already compiled as a part of the host toolchain, which also avoids similar issues where SwiftPM from upstream snapshots can also be out of sync from what we build here on CI.

@MaxDesiatov MaxDesiatov marked this pull request as ready for review May 18, 2020 18:01
@MaxDesiatov
Copy link
Author

@kateinoigakukun I've verified that both Linux and macOS distributions produce valid and working binaries with SwiftPM, the PR is now ready for review 🙂

@kateinoigakukun
Copy link
Member

@MaxDesiatov Looks good! Building host toolchain from same source is cleaner than before.
My only concern is CI time, Doesn't this new building toolchain job reach the limit of CI time when there is no cache?

@MaxDesiatov
Copy link
Author

@kateinoigakukun No, it didn't reach the limit even once during my testing. Only the swift build directory is cleaned up after the host toolchain is built, and only because build-script doesn't reconfigure CMake files properly. Thus, only the SDK (stdlib, runtime etc) is rebuilt, and the toolchain is picked up from the local build-cache. I'll bump the cache version after this is merged, which should get the build time closer to 1-2 hours.

@MaxDesiatov
Copy link
Author

MaxDesiatov commented May 20, 2020

Ok, I see that you're asking "when there is no cache", even in that case I don't think the build time would drastically increase. For the same reason, the only additional time spent here is on building the target SDK, the rest is cached locally during the same build.

@MaxDesiatov MaxDesiatov merged commit 88fd32c into swiftwasm May 20, 2020
@MaxDesiatov MaxDesiatov deleted the maxd/build-host-toolchain branch May 20, 2020 09:16
MaxDesiatov added a commit that referenced this pull request May 23, 2020
…hain"

This reverts commit 88fd32c, reversing
changes made to 04c8aba.
MaxDesiatov added a commit that referenced this pull request Jul 7, 2020
Build both host and target toolchains on CI

Both `libPackageDescription.dylib` and `swiftc` used to compile package manifests should come from the same toolchain for SwiftPM to work on macOS. This is resolved here by re-reverting the previous change #1024 to build both host and target toolchain in one go on CI. I'm wary of the disk space/time limit constraints, but I think those should be resolved separately when they appear, otherwise SwiftPM is completely broken on macOS currently. Resolves #1365.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants