From 9aac86adfbf7812809c81ae034dc3d184c217bc7 Mon Sep 17 00:00:00 2001 From: Alex Crichton Date: Sat, 15 Jun 2024 11:17:09 -0700 Subject: [PATCH] Rewrite the build system with CMake This commit is an attempt to provide a concrete path forward on WebAssembly/wasi-sdk#425. I personally think it's pretty important to get the ability to have more architectures here but at the same time I also think it's important to to take this as an opportunity to refactor and improve the build system of this repository. To that end this represents my attempt to improve the status quo. This removes the old `Makefile` and replaces it with a CMake-based system to build all these projects. Overall this is intended to be a "no functional change" intended sort of refactoring. Changing build systems inevitably causes issues, however, so this change additionally has a very high likelihood of needing follow-up fixes. At a high enough level this commit introduces two major changes to how this repository is built: 1. The `make`-based system (the root `Makefile`) is replaced with CMake. This additionally updates tests to use CMake. 2. A single "build" is split into either building a toolchain or building a sysroot. This enables builds to only build one or the other as necessary. The first change, using CMake, is due to the fact that using `make` on Windows basically is not pleasant coupled with the fact that more advanced logic, such as changing flags, compilers, etc, is much easier with a CMake-based system. The second change is intended to cover the use case of #425 in addition to refactoring the current build. Throughout this change I have intentionally not tried to keep a 1:1 correspondance with behaviors in the old `Makefile` because much of this PR is intended to address shortcomings in the old build system. A list of changes, improvements, etc, made here are: * CMake provides a much nicer portability story to Windows than `make`. This is moving towards the direction of not needing `bash`, for example, to build an SDK. Currently `wasi-libc` still requires this, but that's now the only "hard" dependency. * The set of targets built can now be configured for smaller builds and/or debugging just a single target. All WASI targets are still built by default but it's much easier to add/remove them. * Different targets are now able to be built in parallel as opposed to the unconditional serial-nature of the `Makefile`. * Use of `ninja` is no longer required and separate build systems can be used if desired. * The sysroot and the toolchain can now be built with different CMake build profiles. For example the `Makefile` hardcoded `MinSizeRel` and `RelWithDebInfo` and this can now be much more easily customized by the SDK builder. * Tarballs are now more consistently produced and named. For a tarball of the name `foo.tar.gz` it's guaranteed that there's a single folder `foo` created when unpacking the tarball. * The macOS binaries are no longer hybrid x64/arm64 binaries which greatly inflates the size of the SDK. There's now a separate build for each architecture. * CI now produces arm64-linux binaries. The sysroot is not built on the arm64-linux builder and the sysroot from the x86_64-linux builder is used instead. * Tests are almost ready to execute on Windows, there's just a few minor issues related to exit statuses and probably line endings which need to be worked out. Will require someone with a Windows checkout, however. * Tests are now integrated into CMake. This means that the wasm binaries are able to be built in parallel and the tests are additionally executed in parallel with `ctest`. It is possible to build/run a single test. Tests no longer place all of their output in the source tree. * Out-of-tree builds are now possible and the build/installation directories can both be customized. * CI configuration of Windows/macOS/Linux is much more uniform by having everything in one build matrix instead of separate matrices. * Linux builds are exclusively done in docker containers in CI now. CI no longer produces two Linux builds only for one to be discarded when artifacts are published. * Windows 32-bit builds are no longer produced in CI since it's expected that everyone actually wants the 64-bit ones instead. * Use of `ccache` is now automatically enabled if it's detected on the system. * Many preexisting shell scripts are now translated to CMake one way or another. * There's no longer a separate build script for how to build wasi-sdk in docker and outside of docker which needs to be kept in sync, everything funnels through the same script. * The `docker/Dockerfile` build of wasi-sdk now uses the actual toolchain built from CI and additionally doesn't duplicate various CMake-based configuration files. Overall one thing I want to additionally point out is that I'm not CMake expert. I suspect there's lots of little stylistic and such improvements that can be made. --- .github/workflows/main.yml | 240 ++++++++------- CMakeLists.txt | 45 +++ Dockerfile | 39 --- Makefile | 290 ------------------ RELEASING.md | 3 + ci/build.sh | 49 +++ ci/docker-build.sh | 58 ++++ ci/docker/Dockerfile.arm64-linux | 23 ++ ci/docker/Dockerfile.common | 32 ++ ci/docker/Dockerfile.x86_64-linux | 4 + ci/docker/README.md | 10 + ci/merge-artifacts.sh | 67 ++++ cmake/Platform/WASI.cmake | 1 + cmake/wasi-sdk-dist.cmake | 37 +++ cmake/wasi-sdk-enable-ccache.cmake | 17 + cmake/wasi-sdk-sysroot.cmake | 276 +++++++++++++++++ cmake/wasi-sdk-toolchain.cmake | 147 +++++++++ deb_from_installation.sh | 43 --- docker/Dockerfile | 60 +--- docker/wasi-sdk-pthread.cmake | 36 --- docker/wasi-sdk.cmake | 29 -- docker_build.sh | 18 -- strip_symbols.sh | 15 - tar_from_installation.sh | 51 --- tests/CMakeLists.txt | 90 ++++++ tests/cmake/CMakeLists.txt | 34 -- tests/cmake/test_driver.sh | 17 - tests/compile-only/CMakeLists.txt | 9 + tests/general/CMakeLists.txt | 8 + tests/general/abort.c.stderr.expected.filter | 2 +- tests/general/assert-fail.c | 4 + tests/general/assert-fail.c.stderr.expected | 2 +- .../assert-fail.c.stderr.expected.filter | 3 +- tests/general/clocks.c.options | 1 - tests/general/iostream_main.cc.options | 1 - tests/general/mmap.c.options | 1 - .../printf-long-double-enabled.c.options | 1 - tests/general/sigabrt.c.options | 1 - .../general/sigabrt.c.stderr.expected.filter | 2 +- tests/general/signals.c | 7 + tests/general/signals.c.options | 1 - tests/run.sh | 89 ------ tests/testcase.sh | 53 +--- wasi-sdk.cmake | 2 +- wasi-sdk.control | 2 +- 45 files changed, 1045 insertions(+), 875 deletions(-) create mode 100644 CMakeLists.txt delete mode 100644 Dockerfile delete mode 100644 Makefile create mode 100755 ci/build.sh create mode 100755 ci/docker-build.sh create mode 100644 ci/docker/Dockerfile.arm64-linux create mode 100644 ci/docker/Dockerfile.common create mode 100644 ci/docker/Dockerfile.x86_64-linux create mode 100644 ci/docker/README.md create mode 100755 ci/merge-artifacts.sh create mode 100644 cmake/wasi-sdk-dist.cmake create mode 100644 cmake/wasi-sdk-enable-ccache.cmake create mode 100644 cmake/wasi-sdk-sysroot.cmake create mode 100644 cmake/wasi-sdk-toolchain.cmake delete mode 100755 deb_from_installation.sh delete mode 100644 docker/wasi-sdk-pthread.cmake delete mode 100644 docker/wasi-sdk.cmake delete mode 100755 docker_build.sh delete mode 100755 strip_symbols.sh delete mode 100755 tar_from_installation.sh create mode 100644 tests/CMakeLists.txt delete mode 100644 tests/cmake/CMakeLists.txt delete mode 100755 tests/cmake/test_driver.sh create mode 100644 tests/compile-only/CMakeLists.txt create mode 100644 tests/general/CMakeLists.txt delete mode 100644 tests/general/clocks.c.options delete mode 100644 tests/general/iostream_main.cc.options delete mode 100644 tests/general/mmap.c.options delete mode 100644 tests/general/printf-long-double-enabled.c.options delete mode 100644 tests/general/sigabrt.c.options delete mode 100644 tests/general/signals.c.options delete mode 100755 tests/run.sh diff --git a/.github/workflows/main.yml b/.github/workflows/main.yml index 9ccef2620..04df496f8 100644 --- a/.github/workflows/main.yml +++ b/.github/workflows/main.yml @@ -11,37 +11,39 @@ on: jobs: build: - name: Native Build + name: Build ${{ matrix.artifact }} runs-on: ${{ matrix.os }} strategy: fail-fast: false matrix: - os: - - ubuntu-latest - - macos-latest + include: + - artifact: x86_64-linux + os: ubuntu-latest + + - artifact: arm64-linux + os: ubuntu-latest + rust_target: aarch64-unknown-linux-gnu + + - artifact: arm64-macos + os: macos-latest + llvm_cmake_flags: -DCMAKE_OSX_DEPLOYMENT_TARGET=10.12 -DCMAKE_OSX_ARCHITECTURES=arm64 + rust_target: aarch64-apple-darwin + + - artifact: x86_64-macos + os: macos-latest + llvm_cmake_flags: -DCMAKE_OSX_DEPLOYMENT_TARGET=10.12 -DCMAKE_OSX_ARCHITECTURES=x86_64 + rust_target: x86_64-apple-darwin + skip_sysroot: 1 + + - artifact: x86_64-windows + os: windows-latest + # TODO: tests are pretty close to passing on Windows but need some + # final tweaks, namely testing the exit code doesn't work since + # exit codes are different on Windows and the `mmap.c` tests seems + # to have issues probably with line endings. Needs someone with a + # Windows checkout tot test further. + skip_tests: 1 steps: - - uses: actions/cache@v4 - with: - path: ~/.cache/ccache - # Bump the prefix number to evict all previous caches and - # enforce a clean build, in the unlikely case that some - # weird build error occur and ccache becomes a potential - # suspect. - key: 0-cache-ubuntu-latest-${{ github.run_id }} - restore-keys: | - 0-cache-ubuntu-latest - if: matrix.os == 'ubuntu-latest' - - uses: actions/cache@v4 - with: - path: ~/Library/Caches/ccache - key: 0-cache-macos-latest-${{ github.run_id }} - restore-keys: | - 0-cache-macos-latest - if: matrix.os == 'macos-latest' - - name: Setup `wasmtime` for tests - uses: bytecodealliance/actions/wasmtime/setup@v1 - with: - version: "18.0.2" - uses: actions/checkout@v4 with: fetch-depth: 0 @@ -53,108 +55,124 @@ jobs: # Server does not allow request for unadvertised object" in the # future. - run: git submodule update --init --depth 32 --jobs 3 + + # Persist ccache-based caches across builds. This directory is configured + # via the CCACHE_DIR env var below for ccache to use. + # + # Bump the prefix number to evict all previous caches and enforce a clean + # build, in the unlikely case that some weird build error occur and ccache + # becomes a potential suspect. + - uses: actions/cache@v4 + id: cache-restore + with: + path: ${{ runner.tool_cache }}/ccache + key: 0-cache-${{ matrix.artifact }}-${{ github.run_id }} + restore-keys: | + 0-cache-${{ matrix.artifact }}- + - run: | + mkdir -p '${{ runner.tool_cache }}/ccache' + echo 'CCACHE_DIR=${{ runner.tool_cache }}/ccache' >> $GITHUB_ENV + shell: bash + + # Configure CMake flags for `ci/build.sh` as necessary for each + # matrix entry. + - run: echo WASI_SDK_CI_TOOLCHAIN_LLVM_CMAKE_ARGS=${{ matrix.llvm_cmake_flags }} >> $GITHUB_ENV + if: matrix.llvm_cmake_flags != '' + shell: bash + - run: | + cmake_args=-DWASI_SDK_ARTIFACT=${{ matrix.artifact }} + if [ "${{ matrix.rust_target }}" != "" ]; then + rustup target add ${{ matrix.rust_target }} + cmake_args="$cmake_args -DRUST_TARGET=${{ matrix.rust_target }}" + fi + echo WASI_SDK_CI_TOOLCHAIN_CMAKE_ARGS="$cmake_args" >> $GITHUB_ENV + shell: bash + - run: echo WASI_SDK_CI_SKIP_SYSROOT=1 >> $GITHUB_ENV + if: matrix.skip_sysroot != '' + - run: echo WASI_SDK_CI_SKIP_TESTS=1 >> $GITHUB_ENV + if: matrix.skip_tests != '' + + # Add some extra installed software on each runner as necessary. + - name: Setup `wasmtime` for tests + uses: bytecodealliance/actions/wasmtime/setup@v1 + with: + version: "18.0.2" - name: Install ccache, ninja (macOS) run: brew install ccache ninja - if: matrix.os == 'macos-latest' + if: runner.os == 'macOS' + - name: Install ccache, ninja (Windows) + run: choco install ccache ninja + if: runner.os == 'Windows' - name: Install ccache, ninja (Linux) - run: sudo apt install ccache ninja-build - if: matrix.os == 'ubuntu-latest' - - name: Build - run: NINJA_FLAGS=-v make package LLVM_CMAKE_FLAGS=-DLLVM_CCACHE_BUILD=ON + run: sudo apt install ccache + if: runner.os == 'Linux' + + - name: Build and test (macOS) + run: ./ci/build.sh + if: runner.os == 'macOS' + + - name: Build and test (Linux) + run: ./ci/docker-build.sh ${{ matrix.artifact }} + if: runner.os == 'Linux' + + # Use a shorter build directory than the default on Windows to avoid + # hitting path length and command line length limits. + - name: Build and test (Windows) + run: ./ci/build.sh C:/wasi-sdk shell: bash - - name: Run the testsuite - run: NINJA_FLAGS=-v make check RUNTIME=wasmtime + if: runner.os == 'Windows' + + # Upload the `dist` folder from the build as the artifacts for this + # runner. - name: Upload artifacts uses: actions/upload-artifact@v4 with: - # Upload the dist folder. Give it a name according to the OS it was built for. - name: ${{ format( 'dist-{0}', matrix.os) }} - path: dist + name: ${{ format( 'dist-{0}', matrix.artifact) }} + path: build/dist - winbuild: - name: Windows Build - runs-on: windows-latest - strategy: - fail-fast: false - matrix: - include: - - arch: x64 - sys: clang64 - env: clang-x86_64 - - arch: x86 - sys: clang32 - env: clang-i686 - steps: - - uses: actions/cache@v4 - with: - path: ~/AppData/Local/ccache - key: 0-${{ format( 'cache-windows-latest-{0}', matrix.arch) }}-${{ github.run_id }} - restore-keys: | - 0-${{ format( 'cache-windows-latest-{0}', matrix.arch) }} - - uses: msys2/setup-msys2@v2 - with: - install: >- - base-devel - git - mingw-w64-${{ matrix.env }}-ccache - mingw-w64-${{ matrix.env }}-cmake - mingw-w64-${{ matrix.env }}-ninja - mingw-w64-${{ matrix.env }}-toolchain - msystem: ${{ matrix.sys }} - update: true - release: false - path-type: inherit - - uses: actions/checkout@v4 - with: - fetch-depth: 0 - - run: git fetch --tags --force - name: Force-fetch tags to work around actions/checkout#290 - - run: git submodule update --init --depth 32 --jobs 3 - - name: Build - shell: msys2 {0} - run: | - make package LLVM_CMAKE_FLAGS=-DLLVM_CCACHE_BUILD=ON - make check - - name: Does it work sans msys2? - run: | - C:\wasi-sdk\bin\clang.exe --version - C:\wasi-sdk\bin\llvm-ar.exe --version - C:\wasi-sdk\bin\wasm-ld.exe --version - - name: Upload artifacts - uses: actions/upload-artifact@v4 + # Help debug ccache issues by showing what happened. + - if: always() + name: Show ccache statistics + run: ccache --show-stats + + # Always save a cache, even if the build failed. This ensures that if + # live-debugging via CI the build gets to pick up where it left off last + # time instead of having to recreate everything each time a failure + # happens. + - if: always() && steps.cache-restore.outputs.cache-hit != 'true' + uses: actions/cache/save@v4 with: - # Upload the dist folder. Give it a name according to the OS it was built for. - name: ${{ format( 'dist-windows-latest-{0}', matrix.arch) }} - path: dist + path: ${{ runner.tool_cache }}/ccache + key: 0-cache-${{ matrix.artifact }}-${{ github.run_id }} - dockerbuild: - name: Docker Build + # Once all of the above matrix entries have completed this job will run and + # assemble the final `wasi-sdk-*` artifacts by fusing the toolchain/sysroot + # artifacts. + finalize: + name: Finalize wasi-sdk artifacts + needs: build runs-on: ubuntu-latest steps: - - uses: actions/cache@v4 - with: - path: ~/.ccache - key: 0-cache-ubuntu-bionic-${{ github.run_id }} - restore-keys: | - 0-cache-ubuntu-bionic - - uses: actions/checkout@v4 - with: - fetch-depth: 0 - - run: git fetch --tags --force - name: Force-fetch tags to work around actions/checkout#290 - - run: git submodule update --init --depth 32 --jobs 3 + # Download all artifacts from all platforms in `build`, merge them into + # final wasi-sdk-* artifacts, and then upload them. + - uses: actions/download-artifact@v4 + - run: ./ci/merge-artifacts.sh + - uses: actions/upload-artifact@v4 + with: + name: release-artifacts + path: dist + # Use the `wasi-sdk-*` artifacts just created to create a docker image + # with a toolchain pre-installed. - uses: docker/login-action@v2 with: registry: ghcr.io username: ${{ github.actor }} password: ${{ secrets.GITHUB_TOKEN }} - - uses: docker/setup-qemu-action@v2 - uses: docker/setup-buildx-action@v2 - - uses: docker/metadata-action@v4 id: meta with: @@ -165,16 +183,6 @@ jobs: type=ref,event=tag type=ref,event=pr type=sha - - - name: Run docker_build script - run: ./docker_build.sh - - name: Upload artifacts - uses: actions/upload-artifact@v4 - with: - # Upload the dist folder. Give it a name according to the OS it was built for. - name: dist-ubuntu-bionic - path: dist - - name: Build and push wasi-sdk docker image uses: docker/build-push-action@v3 with: diff --git a/CMakeLists.txt b/CMakeLists.txt new file mode 100644 index 000000000..90abca435 --- /dev/null +++ b/CMakeLists.txt @@ -0,0 +1,45 @@ +# Build logic for building both a toolchain and a sysroot for WASI. +# +# This top level `CMakeLists.txt` file can be used either to build a clang +# toolchain or a WASI sysroot. Note that this can't be done at the same time. +# A toolchain build requires a compiler for the target architecture. A +# WASI sysroot build requires this previous compiler and must be runnable on +# the host. + +cmake_minimum_required(VERSION 3.26) +project(wasi-sdk) +include(ExternalProject) + +set(WASI_SDK_TARGETS "wasm32-wasi;wasm32-wasip1;wasm32-wasip2;wasm32-wasip1-threads;wasm32-wasi-threads" + CACHE STRING "List of WASI targets to build") +option(WASI_SDK_BUILD_TOOLCHAIN "Build a toolchain instead of the sysroot" OFF) + +set(llvm_proj_dir ${CMAKE_CURRENT_SOURCE_DIR}/src/llvm-project) +set(wasi_libc ${CMAKE_CURRENT_SOURCE_DIR}/src/wasi-libc) + +list(APPEND CMAKE_MODULE_PATH "${CMAKE_CURRENT_SOURCE_DIR}/cmake") +include(wasi-sdk-enable-ccache) + +find_program(PYTHON python3 python REQUIRED) + +# Set some variables based on the `version.py` script +set(version_script ${CMAKE_CURRENT_SOURCE_DIR}/version.py) +execute_process( + COMMAND ${PYTHON} ${version_script} llvm-major --llvm-dir=${llvm_proj_dir} + OUTPUT_VARIABLE clang_version + OUTPUT_STRIP_TRAILING_WHITESPACE) +execute_process( + COMMAND ${PYTHON} ${version_script} + OUTPUT_VARIABLE wasi_sdk_version + OUTPUT_STRIP_TRAILING_WHITESPACE) + +message(STATUS "wasi-sdk toolchain LLVM version is ${clang_version}") +message(STATUS "wasi-sdk version is ${wasi_sdk_version}") + +# Only include one version of the build logic as pulling in both isn't +# supported at this time. +if(WASI_SDK_BUILD_TOOLCHAIN) +include(wasi-sdk-toolchain) +else() +include(wasi-sdk-sysroot) +endif() diff --git a/Dockerfile b/Dockerfile deleted file mode 100644 index adddd029c..000000000 --- a/Dockerfile +++ /dev/null @@ -1,39 +0,0 @@ -# Use a relatively old/stable distro here to maximize the supported platforms -# and avoid depending on more recent version of, say, libc. -# Here we choose Bionic 18.04. -FROM ubuntu:bionic - -# We want to use the same UID/GID of the external user to avoid permission -# issues. See the user setup at the end of the file. -ARG UID=1000 -ARG GID=1000 - -RUN apt-get update \ - && apt-get install -y --no-install-recommends \ - ccache \ - curl \ - ca-certificates \ - build-essential \ - clang \ - python3 \ - git \ - ninja-build \ - && apt-get clean \ - && rm -rf /var/lib/apt/lists/* - -RUN curl -sSLO https://github.com/Kitware/CMake/releases/download/v3.25.1/cmake-3.25.1-linux-x86_64.tar.gz \ - && tar xf cmake-3.25.1-linux-x86_64.tar.gz \ - && rm cmake-3.25.1-linux-x86_64.tar.gz \ - && mkdir -p /opt \ - && mv cmake-3.25.1-linux-x86_64 /opt/cmake -ENV PATH /opt/cmake/bin:$PATH - -ENV RUSTUP_HOME=/rust/rustup CARGO_HOME=/rust/cargo PATH=$PATH:/rust/cargo/bin -RUN curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | \ - sh -s -- -y --profile=minimal && \ - chmod -R a+w /rust - -RUN groupadd -g ${GID} builder && \ - useradd --create-home --uid ${UID} --gid ${GID} builder -USER builder -WORKDIR /workspace diff --git a/Makefile b/Makefile deleted file mode 100644 index da92c1eab..000000000 --- a/Makefile +++ /dev/null @@ -1,290 +0,0 @@ -# Any copyright is dedicated to the Public Domain. -# http://creativecommons.org/publicdomain/zero/1.0/ - -ROOT_DIR=${CURDIR} -LLVM_PROJ_DIR?=$(ROOT_DIR)/src/llvm-project -VERSION_SCRIPT=python3 ./version.py - -# Windows needs munging -ifeq ($(OS),Windows_NT) - -PREFIX?=c:/wasi-sdk -# we need to explicitly call bash -c for makefile $(shell ...), otherwise we'll try under -# who knows what -BASH=bash -c - -ifeq (x$(MSYSTEM),x) -$(error On Windows, this Makefile only works in MSYS2 environments such as git-bash.) -endif - -# msys needs any /-prefixed arguments, or =/ containing, to turn into // -# because it tries to path-expand the / into the msys root. // escapes this. -ESCAPE_SLASH=/ - -BUILD_PREFIX=$(PREFIX) - -# assuming we're running under msys2 (git-bash), PATH needs /c/foo format directories (because -# it itself is :-delimited) -PATH_PREFIX=$(shell cygpath.exe -u $(BUILD_PREFIX)) - -else - -PREFIX?=/opt/wasi-sdk -DESTDIR=$(abspath build/install) -BUILD_PREFIX=$(DESTDIR)$(PREFIX) -PATH_PREFIX=$(BUILD_PREFIX) -ESCAPE_SLASH?= -BASH= - -endif - -ifeq ($(shell uname),Darwin) -override LLVM_CMAKE_FLAGS += -DCMAKE_OSX_ARCHITECTURES="arm64;x86_64" \ - -DCMAKE_OSX_DEPLOYMENT_TARGET=10.12 -endif - -TARGETS = wasm32-wasi wasm32-wasip1 wasm32-wasip2 wasm32-wasip1-threads wasm32-wasi-threads - -# Only the major version is needed for Clang, see https://reviews.llvm.org/D125860. -CLANG_VERSION=$(shell $(VERSION_SCRIPT) llvm-major --llvm-dir=$(LLVM_PROJ_DIR)) -VERSION:=$(shell $(VERSION_SCRIPT)) -DEBUG_PREFIX_MAP=-fdebug-prefix-map=$(ROOT_DIR)=wasisdk://v$(VERSION) - -# Generate debuginfo by default for wasi-sdk since it's easily strippable and -# otherwise quite useful for debugging. -WASI_SDK_CFLAGS := $(DEBUG_PREFIX_MAP) -g -WASI_SDK_CXXFLAGS := $(WASI_SDK_CFLAGS) - -default: build - @echo "Use -fdebug-prefix-map=$(ROOT_DIR)=wasisdk://v$(VERSION)" - -check: - TARGETS="$(TARGETS)" tests/run.sh "$(BUILD_PREFIX)" "$(RUNTIME)" - -clean: - rm -rf build $(DESTDIR) - -# Default symlinks that clang creates to the `clang` executable -CLANG_LINKS_TO_CREATE = clang++ clang-cl clang-cpp - -# Add target-prefixed versions of `clang` and `clang++` so they can be used -# without `--target` as it's auto-inferred from the executable name by clang. -CLANG_LINKS_TO_CREATE += $(foreach target,$(TARGETS),$(target)-clang) -CLANG_LINKS_TO_CREATE += $(foreach target,$(TARGETS),$(target)-clang++) - -# Small helper to create a `join-with` function that can join elements of a -# list with a defined separator. -noop = -space = $(noop) $(noop) -join-with = $(subst $(space),$1,$(strip $2)) - -build/llvm.BUILT: - mkdir -p build/llvm - cd build/llvm && cmake -G Ninja \ - -DCLANG_LINKS_TO_CREATE="$(call join-with,;,$(CLANG_LINKS_TO_CREATE))" \ - -DCMAKE_BUILD_TYPE=MinSizeRel \ - -DLLVM_ENABLE_TERMINFO=OFF \ - -DLLVM_ENABLE_ZLIB=OFF \ - -DLLVM_ENABLE_ZSTD=OFF \ - -DLLVM_STATIC_LINK_CXX_STDLIB=ON \ - -DCMAKE_INSTALL_PREFIX=$(PREFIX) \ - -DLLVM_INCLUDE_TESTS=OFF \ - -DLLVM_INCLUDE_UTILS=OFF \ - -DLLVM_INCLUDE_BENCHMARKS=OFF \ - -DLLVM_INCLUDE_EXAMPLES=OFF \ - -DLLVM_TARGETS_TO_BUILD=WebAssembly \ - -DLLVM_DEFAULT_TARGET_TRIPLE=wasm32-wasi \ - -DLLVM_ENABLE_PROJECTS="lld;clang;clang-tools-extra" \ - $(if $(patsubst 9,,$(CLANG_VERSION)), \ - $(if $(patsubst 10,,$(CLANG_VERSION)), \ - -DDEFAULT_SYSROOT=../share/wasi-sysroot, \ - -DDEFAULT_SYSROOT=$(PREFIX)/share/wasi-sysroot), \ - -DDEFAULT_SYSROOT=$(PREFIX)/share/wasi-sysroot) \ - -DLLVM_INSTALL_BINUTILS_SYMLINKS=TRUE \ - -DLLVM_ENABLE_LIBXML2=OFF \ - $(LLVM_CMAKE_FLAGS) \ - $(LLVM_PROJ_DIR)/llvm - DESTDIR=$(DESTDIR) ninja $(NINJA_FLAGS) -C build/llvm \ - install-clang \ - install-clang-format \ - install-clang-tidy \ - install-clang-apply-replacements \ - install-lld \ - install-llvm-mc \ - install-llvm-ranlib \ - install-llvm-strip \ - install-llvm-dwarfdump \ - install-clang-resource-headers \ - install-ar \ - install-ranlib \ - install-strip \ - install-nm \ - install-size \ - install-strings \ - install-objdump \ - install-objcopy \ - install-c++filt \ - llvm-config - touch build/llvm.BUILT - -# Build the `wasm-component-ld` linker from source via `cargo install`. This is -# used for the `wasm32-wasip2` target natively by Clang. Note that `--root` -# passed to `cargo install` will place it in the output directory automatically. -build/wasm-component-ld.BUILT: build/llvm.BUILT - cargo install wasm-component-ld@0.5.0 --root $(BUILD_PREFIX) - touch build/wasm-component-ld.BUILT - - -# Flags for running `make` in wasi-libc -# $(1): the target that's being built -WASI_LIBC_MAKEFLAGS = \ - -C $(ROOT_DIR)/src/wasi-libc \ - CC=$(BUILD_PREFIX)/bin/clang \ - AR=$(BUILD_PREFIX)/bin/llvm-ar \ - NM=$(BUILD_PREFIX)/bin/llvm-nm \ - SYSROOT=$(BUILD_PREFIX)/share/wasi-sysroot \ - EXTRA_CFLAGS="$(WASI_SDK_CFLAGS) -O2 -DNDEBUG" \ - TARGET_TRIPLE=$(1) - -build/wasi-libc.BUILT: build/compiler-rt.BUILT build/wasm-component-ld.BUILT - $(MAKE) $(call WASI_LIBC_MAKEFLAGS,wasm32-wasi) default libc_so - $(MAKE) $(call WASI_LIBC_MAKEFLAGS,wasm32-wasip1) default libc_so - $(MAKE) $(call WASI_LIBC_MAKEFLAGS,wasm32-wasip2) WASI_SNAPSHOT=p2 default libc_so - $(MAKE) $(call WASI_LIBC_MAKEFLAGS,wasm32-wasi-threads) THREAD_MODEL=posix - $(MAKE) $(call WASI_LIBC_MAKEFLAGS,wasm32-wasip1-threads) THREAD_MODEL=posix - touch build/wasi-libc.BUILT - -build/compiler-rt.BUILT: build/llvm.BUILT - # Do the build, and install it. - mkdir -p build/compiler-rt - cd build/compiler-rt && cmake -G Ninja \ - -DCMAKE_SYSROOT=$(BUILD_PREFIX)/share/wasi-sysroot \ - -DCMAKE_C_COMPILER_WORKS=ON \ - -DCMAKE_CXX_COMPILER_WORKS=ON \ - -DCMAKE_AR=$(BUILD_PREFIX)/bin/ar \ - -DCMAKE_MODULE_PATH=$(ROOT_DIR)/cmake \ - -DCMAKE_BUILD_TYPE=RelWithDebInfo \ - -DCMAKE_TOOLCHAIN_FILE=$(ROOT_DIR)/wasi-sdk.cmake \ - -DCOMPILER_RT_BAREMETAL_BUILD=On \ - -DCOMPILER_RT_BUILD_XRAY=OFF \ - -DCOMPILER_RT_INCLUDE_TESTS=OFF \ - -DCOMPILER_RT_HAS_FPIC_FLAG=OFF \ - -DCOMPILER_RT_ENABLE_IOS=OFF \ - -DCOMPILER_RT_DEFAULT_TARGET_ONLY=On \ - -DWASI_SDK_PREFIX=$(BUILD_PREFIX) \ - -DCMAKE_C_FLAGS="$(WASI_SDK_CFLAGS)" \ - -DLLVM_CONFIG_PATH=$(ROOT_DIR)/build/llvm/bin/llvm-config \ - -DCOMPILER_RT_OS_DIR=wasi \ - -DCMAKE_INSTALL_PREFIX=$(PREFIX)/lib/clang/$(CLANG_VERSION)/ \ - -DCMAKE_VERBOSE_MAKEFILE:BOOL=ON \ - $(LLVM_PROJ_DIR)/compiler-rt/lib/builtins - DESTDIR=$(DESTDIR) ninja $(NINJA_FLAGS) -C build/compiler-rt install - # Install clang-provided headers. - cp -R $(ROOT_DIR)/build/llvm/lib/clang $(BUILD_PREFIX)/lib/ - cp -R $(BUILD_PREFIX)/lib/clang/$(CLANG_VERSION)/lib/wasi $(BUILD_PREFIX)/lib/clang/$(CLANG_VERSION)/lib/wasip1 - cp -R $(BUILD_PREFIX)/lib/clang/$(CLANG_VERSION)/lib/wasi $(BUILD_PREFIX)/lib/clang/$(CLANG_VERSION)/lib/wasip2 - touch build/compiler-rt.BUILT - -# Flags for libcxx and libcxxabi. -# $(1): pthreads ON or OFF -# $(2): shared libraries ON or OFF -# $(3): the name of the target being built for -# $(4): extra compiler flags to pass -LIBCXX_CMAKE_FLAGS = \ - -DCMAKE_C_COMPILER_WORKS=ON \ - -DCMAKE_CXX_COMPILER_WORKS=ON \ - -DCMAKE_AR=$(BUILD_PREFIX)/bin/ar \ - -DCMAKE_MODULE_PATH=$(ROOT_DIR)/cmake \ - -DCMAKE_TOOLCHAIN_FILE=$(ROOT_DIR)/wasi-sdk.cmake \ - -DCMAKE_STAGING_PREFIX=$(PREFIX)/share/wasi-sysroot \ - -DCMAKE_POSITION_INDEPENDENT_CODE=$(2) \ - -DLLVM_CONFIG_PATH=$(ROOT_DIR)/build/llvm/bin/llvm-config \ - -DCMAKE_VERBOSE_MAKEFILE:BOOL=ON \ - -DCXX_SUPPORTS_CXX11=ON \ - -DLIBCXX_ENABLE_THREADS:BOOL=$(1) \ - -DLIBCXX_HAS_PTHREAD_API:BOOL=$(1) \ - -DLIBCXX_HAS_EXTERNAL_THREAD_API:BOOL=OFF \ - -DLIBCXX_BUILD_EXTERNAL_THREAD_LIBRARY:BOOL=OFF \ - -DLIBCXX_HAS_WIN32_THREAD_API:BOOL=OFF \ - -DLLVM_COMPILER_CHECKED=ON \ - -DCMAKE_BUILD_TYPE=RelWithDebInfo \ - -DLIBCXX_ENABLE_SHARED:BOOL=$(2) \ - -DLIBCXX_ENABLE_EXPERIMENTAL_LIBRARY:BOOL=OFF \ - -DLIBCXX_ENABLE_EXCEPTIONS:BOOL=OFF \ - -DLIBCXX_ENABLE_FILESYSTEM:BOOL=ON \ - -DLIBCXX_ENABLE_ABI_LINKER_SCRIPT:BOOL=OFF \ - -DLIBCXX_CXX_ABI=libcxxabi \ - -DLIBCXX_CXX_ABI_INCLUDE_PATHS=$(LLVM_PROJ_DIR)/libcxxabi/include \ - -DLIBCXX_HAS_MUSL_LIBC:BOOL=ON \ - -DLIBCXX_ABI_VERSION=2 \ - -DLIBCXXABI_ENABLE_EXCEPTIONS:BOOL=OFF \ - -DLIBCXXABI_ENABLE_SHARED:BOOL=$(2) \ - -DLIBCXXABI_SILENT_TERMINATE:BOOL=ON \ - -DLIBCXXABI_ENABLE_THREADS:BOOL=$(1) \ - -DLIBCXXABI_HAS_PTHREAD_API:BOOL=$(1) \ - -DLIBCXXABI_HAS_EXTERNAL_THREAD_API:BOOL=OFF \ - -DLIBCXXABI_BUILD_EXTERNAL_THREAD_LIBRARY:BOOL=OFF \ - -DLIBCXXABI_HAS_WIN32_THREAD_API:BOOL=OFF \ - -DLIBCXXABI_ENABLE_PIC:BOOL=$(2) \ - -DLIBCXXABI_USE_LLVM_UNWINDER:BOOL=OFF \ - -DWASI_SDK_PREFIX=$(BUILD_PREFIX) \ - -DUNIX:BOOL=ON \ - --debug-trycompile \ - -DCMAKE_SYSROOT=$(BUILD_PREFIX)/share/wasi-sysroot \ - -DCMAKE_C_FLAGS="$(WASI_SDK_CFLAGS) $(EXTRA_CFLAGS) $(4) --target=$(3)" \ - -DCMAKE_CXX_FLAGS="$(WASI_SDK_CXXFLAGS) $(EXTRA_CXXFLAGS) $(4) --target=$(3)" \ - -DLIBCXX_LIBDIR_SUFFIX=$(ESCAPE_SLASH)/$(3) \ - -DLIBCXXABI_LIBDIR_SUFFIX=$(ESCAPE_SLASH)/$(3) \ - -DLLVM_ENABLE_RUNTIMES="libcxx;libcxxabi" \ - $(LLVM_PROJ_DIR)/runtimes - -# Rules to build libcxx, factored out here to deduplicate the below -# $(1): pthreads ON or OFF -# $(2): shared libraries ON or OFF -# $(3): the name of the target being built for -define BUILD_LIBCXX - mkdir -p build/libcxx-$(3) - cd build/libcxx-$(3) && cmake -G Ninja $(call LIBCXX_CMAKE_FLAGS,$(1),$(2),$(3),$(4)) - ninja $(NINJA_FLAGS) -C build/libcxx-$(3) - DESTDIR=$(DESTDIR) ninja $(NINJA_FLAGS) -C build/libcxx-$(3) install - rm -rf $(BUILD_PREFIX)/share/wasi-sysroot/include/$(3)/c++ - mv $(BUILD_PREFIX)/share/wasi-sysroot/include/c++ $(BUILD_PREFIX)/share/wasi-sysroot/include/$(3)/ -endef - -build/libcxx.BUILT: build/llvm.BUILT build/wasi-libc.BUILT - $(call BUILD_LIBCXX,OFF,ON,wasm32-wasi) - $(call BUILD_LIBCXX,OFF,ON,wasm32-wasip1) - $(call BUILD_LIBCXX,OFF,ON,wasm32-wasip2) - $(call BUILD_LIBCXX,ON,OFF,wasm32-wasi-threads,-pthread) - $(call BUILD_LIBCXX,ON,OFF,wasm32-wasip1-threads,-pthread) - # As of this writing, `clang++` will ignore the above include dirs unless this one also exists: - mkdir -p $(BUILD_PREFIX)/share/wasi-sysroot/include/c++/v1 - touch build/libcxx.BUILT - -build/config.BUILT: - mkdir -p $(BUILD_PREFIX)/share/misc - cp src/config/config.sub src/config/config.guess $(BUILD_PREFIX)/share/misc - mkdir -p $(BUILD_PREFIX)/share/cmake/Platform - cp wasi-sdk.cmake $(BUILD_PREFIX)/share/cmake - cp wasi-sdk-pthread.cmake $(BUILD_PREFIX)/share/cmake - cp cmake/Platform/WASI.cmake $(BUILD_PREFIX)/share/cmake/Platform - touch build/config.BUILT - -build/version.BUILT: - $(VERSION_SCRIPT) dump > $(BUILD_PREFIX)/VERSION - touch build/version.BUILT - -build: build/llvm.BUILT build/wasi-libc.BUILT build/compiler-rt.BUILT build/libcxx.BUILT build/config.BUILT build/version.BUILT - -strip: build/llvm.BUILT - ./strip_symbols.sh $(BUILD_PREFIX)/bin - -package: build/package.BUILT - -build/package.BUILT: build strip - mkdir -p dist - ./deb_from_installation.sh $(shell pwd)/dist "$(VERSION)" "$(BUILD_PREFIX)" - ./tar_from_installation.sh "$(shell pwd)/dist" "$(VERSION)" "$(BUILD_PREFIX)" - touch build/package.BUILT - -.PHONY: default clean build strip package check diff --git a/RELEASING.md b/RELEASING.md index c684149be..6bc387974 100644 --- a/RELEASING.md +++ b/RELEASING.md @@ -1,5 +1,8 @@ # Release Process +> **Note**: These instructions are out-of-date given the latest refactoring of +> the build system and should get updated before the next release. + To publish a new version of `wasi-sdk` as a GitHub release: 1. Tag a commit with an annotated tag. Note that this must be an annotated tag, diff --git a/ci/build.sh b/ci/build.sh new file mode 100755 index 000000000..f391e8ab4 --- /dev/null +++ b/ci/build.sh @@ -0,0 +1,49 @@ +#!/bin/bash + +# Build logic executed in CI. This is intentionally kept relatively minimal to +# one day not live in bash to have a bash-less build on Windows. For now though +# this will unconditionally build a toolchain and then optionally build a +# sysroot. Builders which can't actually execute the toolchain they produce +# skip the sysroot step below. + +set -ex + +# Optionally allow the first argument to this script to be the install +# location. +if [ "$1" = "" ]; then + install_dir=`pwd`/build/install +else + install_dir="$1" +fi + +cmake -G Ninja -B build/toolchain -S . \ + -DWASI_SDK_BUILD_TOOLCHAIN=ON \ + "-DCMAKE_INSTALL_PREFIX=$install_dir" \ + $WASI_SDK_CI_TOOLCHAIN_CMAKE_ARGS \ + "-DLLVM_CMAKE_FLAGS=$WASI_SDK_CI_TOOLCHAIN_LLVM_CMAKE_ARGS" +ninja -C build/toolchain install dist -v + +mv build/toolchain/dist build/dist + +if [ "$WASI_SDK_CI_SKIP_SYSROOT" = "1" ]; then + exit 0 +fi + +# Use the just-built toolchain and its `CMAKE_TOOLCHAIN_FILE` to build a +# sysroot. +cmake -G Ninja -B build/sysroot -S . \ + "-DCMAKE_TOOLCHAIN_FILE=$install_dir/share/cmake/wasi-sdk.cmake" \ + -DCMAKE_C_COMPILER_WORKS=ON \ + -DCMAKE_CXX_COMPILER_WORKS=ON \ + -DWASI_SDK_INCLUDE_TESTS=ON \ + "-DCMAKE_INSTALL_PREFIX=$install_dir" +ninja -C build/sysroot install dist -v + +mv build/sysroot/dist/* build/dist + +if [ "$WASI_SDK_CI_SKIP_TESTS" = "1" ]; then + exit 0 +fi + +# Run tests to ensure that the sysroot works. +ctest --output-on-failure --parallel 10 --test-dir build/sysroot/tests diff --git a/ci/docker-build.sh b/ci/docker-build.sh new file mode 100755 index 000000000..a695d6f4a --- /dev/null +++ b/ci/docker-build.sh @@ -0,0 +1,58 @@ +#!/bin/sh + +# This is a helper script invoked from CI which will execute the `ci/build.sh` +# script within a docker container. This builds `ci/docker/Dockerfile.common` +# along with the specified `ci/docker/Dockerfile.$x` from the command line. +# This container is then used to execute `ci/build.sh`. + +set -e + +if [ "$1" = "" ]; then + echo "Usage: $0 " + echo "" + echo "example: $0 x86_64-linux" + exit 1 +fi + +set -x + +# Build the base image which the image below can used. +docker build \ + --file ci/docker/Dockerfile.common \ + --tag wasi-sdk-builder-base \ + ci/docker + +# Build the container that is going to be used +docker build \ + --file ci/docker/Dockerfile.$1 \ + --tag wasi-sdk-builder \ + ci/docker + +# Perform the build in `/src`. The current directory is mounted read-write at +# this location as well. To ensure that container-created files are reasonable +# on the host as well the `--user` is passed to configure various permissions. +args="--workdir /src --volume `pwd`:/src:Z" +args="$args --user $(id -u):$(id -g)" + +# Persist the ccache directory on the host to ensure repeated runs/debugging +# of this container don't take forever. Also enables caching in CI. +ccache_dir=$CCACHE_DIR +if [ "$ccache_dir" = "" ]; then + ccache_dir=$HOME/.ccache +fi +args="$args --volume $ccache_dir:/ccache:Z --env CCACHE_DIR=/ccache" + +# Inherit some tools from the host into this container. This ensures that the +# decision made on CI of what versions to use is the canonical source of truth +# for theset ools +args="$args --volume `rustc --print sysroot`:/rustc:ro" +args="$args --volume $(dirname $(which wasmtime)):/wasmtime:ro" + +# Before running `ci/build.sh` set up some rust/PATH related info to use what +# was just mounted above, and then execute the build. +docker run \ + $args \ + --tty \ + --init \ + wasi-sdk-builder \ + bash -c 'CARGO_HOME=/tmp/cargo-home PATH=$PATH:/rustc/bin:/wasmtime exec ci/build.sh' diff --git a/ci/docker/Dockerfile.arm64-linux b/ci/docker/Dockerfile.arm64-linux new file mode 100644 index 000000000..fa9a16ea4 --- /dev/null +++ b/ci/docker/Dockerfile.arm64-linux @@ -0,0 +1,23 @@ +FROM wasi-sdk-builder-base + +RUN apt-get install -y g++-aarch64-linux-gnu + +# Don't build a sysroot for this cross-compiled target since it would require a +# host compiler and the sysroot is otherwise already built on other CI builders. +ENV WASI_SDK_CI_SKIP_SYSROOT 1 + +ENV WASI_SDK_CI_TOOLCHAIN_LLVM_CMAKE_ARGS \ + -DCMAKE_C_COMPILER=aarch64-linux-gnu-gcc \ + -DCMAKE_CXX_COMPILER=aarch64-linux-gnu-g++ \ + -DCMAKE_CROSSCOMPILING=True \ + -DCMAKE_CXX_FLAGS=-march=armv8-a \ + -DCMAKE_SYSTEM_PROCESSOR=arm64 \ + -DCMAKE_SYSTEM_NAME=Linux \ + -DLLVM_HOST_TRIPLE=aarch64-linux-gnu \ + -DRUST_TARGET=aarch64-unknown-linux-gnu + +ENV WASI_SDK_CI_TOOLCHAIN_CMAKE_ARGS \ + -DWASI_SDK_ARTIFACT=arm64-linux \ + -DRUST_TARGET=aarch64-unknown-linux-gnu + +ENV CARGO_TARGET_AARCH64_UNKNOWN_LINUX_GNU_LINKER aarch64-linux-gnu-gcc diff --git a/ci/docker/Dockerfile.common b/ci/docker/Dockerfile.common new file mode 100644 index 000000000..1ab98a61a --- /dev/null +++ b/ci/docker/Dockerfile.common @@ -0,0 +1,32 @@ +# Use a relatively old/stable distro here to maximize the supported platforms +# and avoid depending on more recent version of, say, libc. +# Here we choose Bionic 18.04. + +FROM ubuntu:18.04 + +RUN apt-get update \ + && apt-get install -y --no-install-recommends \ + ccache \ + curl \ + ca-certificates \ + build-essential \ + clang \ + python3 \ + git \ + unzip \ + xz-utils + +RUN curl -sSLO https://github.com/Kitware/CMake/releases/download/v3.29.5/cmake-3.29.5-linux-x86_64.tar.gz \ + && tar xf cmake-3.29.5-linux-x86_64.tar.gz \ + && rm cmake-3.29.5-linux-x86_64.tar.gz \ + && mkdir -p /opt \ + && mv cmake-3.29.5-linux-x86_64 /opt/cmake + +ENV PATH /opt/cmake/bin:$PATH + +RUN curl -sSLO https://github.com/ninja-build/ninja/releases/download/v1.12.1/ninja-linux.zip \ + && unzip ninja-linux.zip \ + && rm *.zip \ + && mv ninja /opt/cmake/bin + +ENV XDG_CACHE_HOME /tmp/cache diff --git a/ci/docker/Dockerfile.x86_64-linux b/ci/docker/Dockerfile.x86_64-linux new file mode 100644 index 000000000..47473e691 --- /dev/null +++ b/ci/docker/Dockerfile.x86_64-linux @@ -0,0 +1,4 @@ +FROM wasi-sdk-builder-base + +ENV WASI_SDK_CI_TOOLCHAIN_CMAKE_ARGS \ + -DWASI_SDK_ARTIFACT=x86_64-linux diff --git a/ci/docker/README.md b/ci/docker/README.md new file mode 100644 index 000000000..09ed7bcf4 --- /dev/null +++ b/ci/docker/README.md @@ -0,0 +1,10 @@ +# About + +This folder contains the docker images that are used in CI to build the wasi-sdk +release toolchains. Docker is used to intentionally use older Linux +distributions to build the toolchain to have a more maximal set of glibc +compatibility. + +These images are intended to be used on an x86\_64 host. Images start from the +`Dockerfile.common` file and then layer on target-specific +toolchains/options/etc as necessary. diff --git a/ci/merge-artifacts.sh b/ci/merge-artifacts.sh new file mode 100755 index 000000000..a778a0e6f --- /dev/null +++ b/ci/merge-artifacts.sh @@ -0,0 +1,67 @@ +#!/bin/sh + +# Helper script executed on CI once all builds have completed. This takes +# `wasi-toolchain-*` artifacts and `wasi-sysroot-*` artifacts and merges +# them together into a single `wasi-sdk-*` artifact. Toolchains which don't +# have a sysroot that they themselves built use a sysroot from the x86_64-linux +# toolchain. + +set -ex + +rm -rf dist +mkdir dist +version=$(./version.py) + +make_deb() { + build=$1 + dir=$2 + + + case $build in + dist-x86_64-linux) + deb_arch=amd64 + ;; + dist-arm64-linux) + deb_arch=arm64 + ;; + *) + echo "unknown build $build" + exit 1 + esac + + mkdir dist/pkg + mkdir dist/pkg/opt + mkdir dist/pkg/DEBIAN + sed s/VERSION/$version/ wasi-sdk.control | \ + sed s/ARCH/$deb_arch/ > dist/pkg/DEBIAN/control + cp -R $dir dist/pkg/opt/wasi-sdk + deb_name=$(echo $(basename $dir) | sed 's/.tar.gz//') + (cd dist && dpkg-deb -b pkg $deb_name.deb) + rm -rf dist/pkg +} + +for build in dist-*; do + toolchain=`ls $build/wasi-toolchain-*` + if [ -f $build/wasi-sysroot-* ]; then + sysroot=`ls $build/wasi-sysroot-*` + else + sysroot=`ls dist-x86_64-linux/wasi-sysroot-*` + fi + + sdk_dir=`basename $toolchain | sed 's/.tar.gz//' | sed s/toolchain/sdk/` + mkdir dist/$sdk_dir + + tar xf $toolchain -C dist/$sdk_dir --strip-components 1 + tar xf $sysroot -C dist/$sdk_dir --strip-components 1 + tar czf dist/$sdk_dir.tar.gz -C dist $sdk_dir + + if echo $build | grep -q linux; then + make_deb $build dist/$sdk_dir + fi + rm -rf dist/$sdk_dir +done + +# In addition to `wasi-sdk-*` also preserve artifacts for just the sysroot +# and just compiler-rt. +cp dist-x86_64-linux/wasi-sysroot-* dist +cp dist-x86_64-linux/libclang_rt* dist diff --git a/cmake/Platform/WASI.cmake b/cmake/Platform/WASI.cmake index b49713f1b..1b67f1010 100644 --- a/cmake/Platform/WASI.cmake +++ b/cmake/Platform/WASI.cmake @@ -1 +1,2 @@ set(WASI 1) +set(CMAKE_EXECUTABLE_SUFFIX ".wasm") diff --git a/cmake/wasi-sdk-dist.cmake b/cmake/wasi-sdk-dist.cmake new file mode 100644 index 000000000..8a9b2b3a1 --- /dev/null +++ b/cmake/wasi-sdk-dist.cmake @@ -0,0 +1,37 @@ +# Helper function to create tarballs for wasi-sdk. +# +# The `target` is the name of the CMake target to create for the creation of +# this tarball. The `tarball` argument is where the final tarball will be +# located. The name of the tarball is also used for the name of the root folder +# in the tarball. The `dir` argument is is the directory that will get packaged +# up within the tarball. +function(wasi_sdk_add_tarball target tarball dir) + cmake_path(GET tarball PARENT_PATH tarball_dir) + + # Run STEM twice to chop of both `.gz` and `.tar` in `.tar.gz` + cmake_path(GET tarball STEM LAST_ONLY tarball_stem) + cmake_path(GET tarball_stem STEM LAST_ONLY tarball_stem) + + if(CMAKE_SYSTEM_NAME MATCHES Windows) + # Copy the contents of symlinks on Windows to avoid dealing with symlink + set(copy_dir ${CMAKE_COMMAND} -E copy_directory ${dir} ${tarball_stem}) + else() + # ... but on non-Windows copy symlinks themselves to cut down on + # distribution size. + set(copy_dir cp -R ${dir} ${tarball_stem}) + endif() + + add_custom_command( + OUTPUT ${tarball} + # First copy the directory under a different name, the filestem of the + # tarball. + COMMAND ${copy_dir} + # Next use CMake to create the tarball itself + COMMAND ${CMAKE_COMMAND} -E tar cfz ${tarball} ${tarball_stem} + # Finally delete the temporary directory created above. + COMMAND ${CMAKE_COMMAND} -E rm -rf ${tarball_stem} + WORKING_DIRECTORY ${tarball_dir} + COMMENT "Creating ${tarball}..." + ) + add_custom_target(${target} DEPENDS ${tarball}) +endfunction() diff --git a/cmake/wasi-sdk-enable-ccache.cmake b/cmake/wasi-sdk-enable-ccache.cmake new file mode 100644 index 000000000..f5e1f304e --- /dev/null +++ b/cmake/wasi-sdk-enable-ccache.cmake @@ -0,0 +1,17 @@ +# Helper module to auto-enable ccache if detected. + +find_program(CCACHE ccache) + +option(WASI_SDK_DISABLE_CCACHE "Force disable ccache even if it's found" OFF) + +if(NOT CMAKE_C_COMPILER_LAUNCHER) + if(NOT WASI_SDK_DISABLE_CCACHE) + if(CCACHE) + set(CMAKE_C_COMPILER_LAUNCHER ccache) + set(CMAKE_CXX_COMPILER_LAUNCHER ccache) + message(STATUS "Auto-enabling ccache") + else() + message(STATUS "Failed to auto-enable ccache, not found on system") + endif() + endif() +endif() diff --git a/cmake/wasi-sdk-sysroot.cmake b/cmake/wasi-sdk-sysroot.cmake new file mode 100644 index 000000000..5c23fe610 --- /dev/null +++ b/cmake/wasi-sdk-sysroot.cmake @@ -0,0 +1,276 @@ +# Build logic for building a sysroot for wasi-sdk which includes compiler-rt, +# wasi-libc, libcxx, and libcxxabi. + +if(NOT CMAKE_BUILD_TYPE) + set(CMAKE_BUILD_TYPE RelWithDebInfo) +endif() + +find_program(MAKE make REQUIRED) + +option(WASI_SDK_DEBUG_PREFIX_MAP "Pass `-fdebug-prefix-map` for built artifacts" ON) +option(WASI_SDK_INCLUDE_TESTS "Whether or not to build tests by default" OFF) + +set(wasi_sysroot ${CMAKE_INSTALL_PREFIX}/share/wasi-sysroot) + +if(WASI_SDK_DEBUG_PREFIX_MAP) + add_compile_options( + -fdebug-prefix-map=${CMAKE_CURRENT_SOURCE_DIR}=wasisdk://v${wasi_sdk_version}) +endif() + + +# Default arguments for builds of cmake projects (mostly LLVM-based) to forward +# along much of our own configuration into these projects. +set(default_cmake_args + -DCMAKE_SYSTEM_NAME=WASI + -DCMAKE_SYSTEM_VERSION=1 + -DCMAKE_SYSTEM_PROCESSOR=wasm32 + -DCMAKE_BUILD_TYPE=${CMAKE_BUILD_TYPE} + -DCMAKE_AR=${CMAKE_AR} + -DCMAKE_C_COMPILER=${CMAKE_C_COMPILER} + -DCMAKE_CXX_COMPILER=${CMAKE_CXX_COMPILER} + -DCMAKE_C_COMPILER_WORKS=ON + -DCMAKE_CXX_COMPILER_WORKS=ON + -DCMAKE_SYSROOT=${wasi_sysroot} + -DCMAKE_MODULE_PATH=${CMAKE_CURRENT_SOURCE_DIR}/cmake) + +if(CMAKE_C_COMPILER_LAUNCHER) + list(APPEND default_cmake_args -DCMAKE_C_COMPILER_LAUNCHER=${CMAKE_C_COMPILER_LAUNCHER}) +endif() +if(CMAKE_CXX_COMPILER_LAUNCHER) + list(APPEND default_cmake_args -DCMAKE_CXX_COMPILER_LAUNCHER=${CMAKE_CXX_COMPILER_LAUNCHER}) +endif() + +# ============================================================================= +# compiler-rt build logic +# ============================================================================= + +set(compiler_rt_dst ${CMAKE_INSTALL_PREFIX}/lib/clang/${clang_version}) +ExternalProject_Add(compiler-rt-build + SOURCE_DIR "${llvm_proj_dir}/compiler-rt" + CMAKE_ARGS + ${default_cmake_args} + -DCOMPILER_RT_BAREMETAL_BUILD=ON + -DCOMPILER_RT_BUILD_XRAY=OFF + -DCOMPILER_RT_INCLUDE_TESTS=OFF + -DCOMPILER_RT_HAS_FPIC_FLAG=OFF + -DCOMPILER_RT_ENABLE_IOS=OFF + -DCOMPILER_RT_DEFAULT_TARGET_ONLY=ON + -DCMAKE_C_COMPILER_TARGET=wasm32-wasi + -DCOMPILER_RT_OS_DIR=wasi + -DCMAKE_INSTALL_PREFIX=${compiler_rt_dst} + EXCLUDE_FROM_ALL ON + USES_TERMINAL_CONFIGURE ON + USES_TERMINAL_BUILD ON + USES_TERMINAL_INSTALL ON +) + +# In addition to the default installation of `compiler-rt` itself also copy +# around some headers and make copies of the `wasi` directory as `wasip1` and +# `wasip2` +execute_process( + COMMAND ${CMAKE_C_COMPILER} -print-runtime-dir + OUTPUT_VARIABLE clang_runtime_dir + OUTPUT_STRIP_TRAILING_WHITESPACE) +cmake_path(GET clang_runtime_dir PARENT_PATH clang_runtime_libdir) # chop off `wasi` +cmake_path(GET clang_runtime_libdir PARENT_PATH clang_sysroot_dir) # chop off `lib` +add_custom_target(compiler-rt-post-build + COMMAND ${CMAKE_COMMAND} -E copy_directory + ${clang_sysroot_dir} ${compiler_rt_dst} + COMMAND ${CMAKE_COMMAND} -E copy_directory + ${compiler_rt_dst}/lib/wasi ${compiler_rt_dst}/lib/wasip1 + COMMAND ${CMAKE_COMMAND} -E copy_directory + ${compiler_rt_dst}/lib/wasi ${compiler_rt_dst}/lib/wasip2 + COMMAND ${CMAKE_COMMAND} -E copy_directory_if_different + ${compiler_rt_dst}/lib ${clang_runtime_libdir} + COMMENT "finalizing compiler-rt installation" +) +add_dependencies(compiler-rt-post-build compiler-rt-build) + +add_custom_target(compiler-rt DEPENDS compiler-rt-build compiler-rt-post-build) + +# ============================================================================= +# wasi-libc build logic +# ============================================================================= + +function(define_wasi_libc target) + set(build_dir ${CMAKE_CURRENT_BINARY_DIR}/wasi-libc-${target}) + + if(${target} MATCHES threads) + set(extra_make_flags THREAD_MODEL=posix) + elseif(${target} MATCHES p2) + set(extra_make_flags WASI_SNAPSHOT=p2 default libc_so) + else() + set(extra_make_flags default libc_so) + endif() + + string(TOUPPER ${CMAKE_BUILD_TYPE} CMAKE_BUILD_TYPE_UPPER) + get_property(directory_cflags DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR} PROPERTY COMPILE_OPTIONS) + set(extra_cflags_list + "${CMAKE_C_FLAGS} ${directory_cflags} ${CMAKE_C_FLAGS_${CMAKE_BUILD_TYPE_UPPER}}") + list(JOIN extra_cflags_list " " extra_cflags) + + ExternalProject_Add(wasi-libc-${target} + # Currently wasi-libc doesn't support out-of-tree builds so feigh a + # "download command" which copies the source tree to a different location + # so out-of-tree builds are supported. + DOWNLOAD_COMMAND + ${CMAKE_COMMAND} -E copy_directory ${wasi_libc} ${build_dir} + SOURCE_DIR "${build_dir}" + CONFIGURE_COMMAND "" + BUILD_COMMAND + ${MAKE} -j8 -C ${build_dir} + CC=${CMAKE_C_COMPILER} + AR=${CMAKE_AR} + NM=${CMAKE_NM} + SYSROOT=${wasi_sysroot} + EXTRA_CFLAGS=${extra_cflags} + TARGET_TRIPLE=${target} + ${extra_make_flags} + INSTALL_COMMAND "" + DEPENDS compiler-rt + EXCLUDE_FROM_ALL ON + USES_TERMINAL_CONFIGURE ON + USES_TERMINAL_BUILD ON + USES_TERMINAL_INSTALL ON + ) +endfunction() + +foreach(target IN LISTS WASI_SDK_TARGETS) + define_wasi_libc(${target}) +endforeach() + +# ============================================================================= +# libcxx build logic +# ============================================================================= + +function(define_libcxx target) + if(${target} MATCHES threads) + set(threads ON) + set(pic OFF) + set(target_flags -pthread) + else() + set(threads OFF) + set(pic ON) + set(target_flags "") + endif() + + set(runtimes "libcxx;libcxxabi") + + get_property(dir_compile_opts DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR} PROPERTY COMPILE_OPTIONS) + get_property(dir_link_opts DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR} PROPERTY LINK_OPTIONS) + set(extra_cflags_list ${CMAKE_C_FLAGS} ${target_flags} --target=${target} ${dir_compile_opts} ${dir_link_opts}) + list(JOIN extra_cflags_list " " extra_cflags) + set(extra_cxxflags_list ${CMAKE_CXX_FLAGS} ${target_flags} --target=${target} ${dir_compile_opts} ${dir_link_opts}) + list(JOIN extra_cxxflags_list " " extra_cxxflags) + + ExternalProject_Add(libcxx-${target}-build + SOURCE_DIR ${llvm_proj_dir}/runtimes + CMAKE_ARGS + ${default_cmake_args} + # Ensure headers are installed in a target-specific path instead of a + # target-generic path. + -DCMAKE_INSTALL_INCLUDEDIR=${wasi_sysroot}/include/${target} + -DCMAKE_STAGING_PREFIX=${wasi_sysroot} + -DCMAKE_POSITION_INDEPENDENT_CODE=${pic} + -DCXX_SUPPORTS_CXX11=ON + -DLIBCXX_ENABLE_THREADS:BOOL=${threads} + -DLIBCXX_HAS_PTHREAD_API:BOOL=${threads} + -DLIBCXX_HAS_EXTERNAL_THREAD_API:BOOL=OFF + -DLIBCXX_BUILD_EXTERNAL_THREAD_LIBRARY:BOOL=OFF + -DLIBCXX_HAS_WIN32_THREAD_API:BOOL=OFF + -DLLVM_COMPILER_CHECKED=ON + -DLIBCXX_ENABLE_SHARED:BOOL=${pic} + -DLIBCXX_ENABLE_EXPERIMENTAL_LIBRARY:BOOL=OFF + -DLIBCXX_ENABLE_EXCEPTIONS:BOOL=OFF + -DLIBCXX_ENABLE_FILESYSTEM:BOOL=ON + -DLIBCXX_ENABLE_ABI_LINKER_SCRIPT:BOOL=OFF + -DLIBCXX_CXX_ABI=libcxxabi + -DLIBCXX_CXX_ABI_INCLUDE_PATHS=${llvm_proj_dir}/libcxxabi/include + -DLIBCXX_HAS_MUSL_LIBC:BOOL=ON + -DLIBCXX_ABI_VERSION=2 + -DLIBCXXABI_ENABLE_EXCEPTIONS:BOOL=OFF + -DLIBCXXABI_ENABLE_SHARED:BOOL=${pic} + -DLIBCXXABI_SILENT_TERMINATE:BOOL=ON + -DLIBCXXABI_ENABLE_THREADS:BOOL=${threads} + -DLIBCXXABI_HAS_PTHREAD_API:BOOL=${threads} + -DLIBCXXABI_HAS_EXTERNAL_THREAD_API:BOOL=OFF + -DLIBCXXABI_BUILD_EXTERNAL_THREAD_LIBRARY:BOOL=OFF + -DLIBCXXABI_HAS_WIN32_THREAD_API:BOOL=OFF + -DLIBCXXABI_ENABLE_PIC:BOOL=${pic} + -DLIBCXXABI_USE_LLVM_UNWINDER:BOOL=OFF + -DUNIX:BOOL=ON + -DCMAKE_C_FLAGS=${extra_cflags} + -DCMAKE_CXX_FLAGS=${extra_cxxflags} + -DLIBCXX_LIBDIR_SUFFIX=/${target} + -DLIBCXXABI_LIBDIR_SUFFIX=/${target} + + # See https://www.scivision.dev/cmake-externalproject-list-arguments/ for + # why this is in `CMAKE_CACHE_ARGS` instead of above + CMAKE_CACHE_ARGS + -DLLVM_ENABLE_RUNTIMES:STRING=${runtimes} + DEPENDS + wasi-libc-${target} + compiler-rt + EXCLUDE_FROM_ALL ON + USES_TERMINAL_CONFIGURE ON + USES_TERMINAL_BUILD ON + USES_TERMINAL_INSTALL ON + ) + + # As of this writing, `clang++` will ignore the target-specific include dirs + # unless this one also exists: + add_custom_target(libcxx-${target}-extra-dir + COMMAND ${CMAKE_COMMAND} -E make_directory ${wasi_sysroot}/include/c++/v1 + COMMENT "creating libcxx-specific header file folder") + add_custom_target(libcxx-${target} + DEPENDS libcxx-${target}-build libcxx-${target}-extra-dir) +endfunction() + +foreach(target IN LISTS WASI_SDK_TARGETS) + define_libcxx(${target}) +endforeach() + +# ============================================================================= +# misc build logic +# ============================================================================= + +# Add a top-level `build` target as well as `build-$target` targets. +add_custom_target(build ALL) +foreach(target IN LISTS WASI_SDK_TARGETS) + add_custom_target(build-${target}) + add_dependencies(build-${target} libcxx-${target} wasi-libc-${target} compiler-rt) + add_dependencies(build build-${target}) +endforeach() + +# Install a `VERSION` file in the output prefix with a dump of version +# information. +set(version_file_tmp ${CMAKE_CURRENT_BINARY_DIR}/VERSION) +execute_process( + COMMAND ${PYTHON} ${version_script} dump + WORKING_DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR} + OUTPUT_FILE ${version_file_tmp}) +install( + FILES ${version_file_tmp} + DESTINATION ${CMAKE_INSTALL_PREFIX}) + +if(WASI_SDK_INCLUDE_TESTS) + add_subdirectory(tests) +endif() + +include(wasi-sdk-dist) + +set(dist_dir ${CMAKE_CURRENT_BINARY_DIR}/dist) + +# Tarball with just `compiler-rt` builtins within it +wasi_sdk_add_tarball(dist-compiler-rt + ${dist_dir}/libclang_rt.builtins-wasm32-wasi-${wasi_sdk_version}.tar.gz + ${CMAKE_INSTALL_PREFIX}/lib/clang/${clang_version}/lib/wasi) +add_dependencies(dist-compiler-rt compiler-rt) + +# Tarball with the whole sysroot +wasi_sdk_add_tarball(dist-sysroot + ${dist_dir}/wasi-sysroot-${wasi_sdk_version}.tar.gz + ${CMAKE_INSTALL_PREFIX}/share/wasi-sysroot) +add_dependencies(dist-sysroot build install) + +add_custom_target(dist DEPENDS dist-compiler-rt dist-sysroot) diff --git a/cmake/wasi-sdk-toolchain.cmake b/cmake/wasi-sdk-toolchain.cmake new file mode 100644 index 000000000..64a830eab --- /dev/null +++ b/cmake/wasi-sdk-toolchain.cmake @@ -0,0 +1,147 @@ +# Build logic and support for building a Clang toolchain that can target +# WebAssembly and build a WASI sysroot. + +set(LLVM_CMAKE_FLAGS "" CACHE STRING "Extra cmake flags to pass to LLVM's build") +set(RUST_TARGET "" CACHE STRING "Target to build Rust code for, if not the host") +set(WASI_SDK_ARTIFACT "" CACHE STRING "Name of the wasi-sdk artifact being produced") + +string(REGEX REPLACE "[ ]+" ";" llvm_cmake_flags_list "${LLVM_CMAKE_FLAGS}") + +if(NOT CMAKE_BUILD_TYPE) + set(CMAKE_BUILD_TYPE MinSizeRel) +endif() + +set(default_cmake_args + -DCMAKE_BUILD_TYPE=${CMAKE_BUILD_TYPE} + -DCMAKE_AR=${CMAKE_AR} + -DCMAKE_C_COMPILER=${CMAKE_C_COMPILER} + -DCMAKE_CXX_COMPILER=${CMAKE_CXX_COMPILER} + -DCMAKE_INSTALL_PREFIX=${CMAKE_INSTALL_PREFIX}) + +if(CMAKE_C_COMPILER_LAUNCHER) + list(APPEND default_cmake_args -DCMAKE_C_COMPILER_LAUNCHER=${CMAKE_C_COMPILER_LAUNCHER}) +endif() +if(CMAKE_CXX_COMPILER_LAUNCHER) + list(APPEND default_cmake_args -DCMAKE_CXX_COMPILER_LAUNCHER=${CMAKE_CXX_COMPILER_LAUNCHER}) +endif() + +set(links_to_create clang-cl clang-cpp clang++) +foreach(target IN LISTS WASI_SDK_TARGETS) + list(APPEND links_to_create ${target}-clang) + list(APPEND links_to_create ${target}-clang++) +endforeach() + +set(projects "lld;clang;clang-tools-extra") + +set(tools + clang + clang-format + clang-tidy + clang-apply-replacements + lld + llvm-mc + llvm-ranlib + llvm-strip + llvm-dwarfdump + clang-resource-headers + ar + ranlib + strip + nm + size + strings + objdump + objcopy + c++filt + llvm-config) + +list(TRANSFORM tools PREPEND --target= OUTPUT_VARIABLE build_targets) +list(TRANSFORM tools PREPEND --target=install- OUTPUT_VARIABLE install_targets) + +ExternalProject_Add(llvm-build + SOURCE_DIR "${llvm_proj_dir}/llvm" + CMAKE_ARGS + ${default_cmake_args} + -DLLVM_ENABLE_TERMINFO=OFF + -DLLVM_ENABLE_ZLIB=OFF + -DLLVM_ENABLE_ZSTD=OFF + -DLLVM_STATIC_LINK_CXX_STDLIB=ON + -DLLVM_INCLUDE_TESTS=OFF + -DLLVM_INCLUDE_UTILS=OFF + -DLLVM_INCLUDE_BENCHMARKS=OFF + -DLLVM_INCLUDE_EXAMPLES=OFF + -DLLVM_TARGETS_TO_BUILD=WebAssembly + -DLLVM_DEFAULT_TARGET_TRIPLE=wasm32-wasi + -DLLVM_INSTALL_BINUTILS_SYMLINKS=TRUE + -DLLVM_ENABLE_LIBXML2=OFF + # Pass `-s` to strip symbols by default and shrink the size of the + # distribution + -DCMAKE_EXE_LINKER_FLAGS=-s + ${llvm_cmake_flags_list} + # See https://www.scivision.dev/cmake-externalproject-list-arguments/ for + # why this is in `CMAKE_CACHE_ARGS` instead of above + CMAKE_CACHE_ARGS + -DLLVM_ENABLE_PROJECTS:STRING=${projects} + -DCLANG_LINKS_TO_CREATE:STRING=${links_to_create} + BUILD_COMMAND + cmake --build . ${build_targets} + INSTALL_COMMAND + cmake --build . ${install_targets} + USES_TERMINAL_CONFIGURE ON + USES_TERMINAL_BUILD ON + USES_TERMINAL_INSTALL ON +) + +# Build logic for `wasm-component-ld` installed from Rust code. +set(wasm_component_ld_root ${CMAKE_CURRENT_BINARY_DIR}/wasm-component-ld) +set(wasm_component_ld ${wasm_component_ld_root}/bin/wasm-component-ld${CMAKE_EXECUTABLE_SUFFIX}) +set(wasm_component_ld_version 0.5.0) +if(RUST_TARGET) + set(rust_target_flag --target=${RUST_TARGET}) +endif() +add_custom_command( + OUTPUT ${wasm_component_ld} + COMMAND + cargo install --root ${wasm_component_ld_root} ${rust_target_flag} + wasm-component-ld@${wasm_component_ld_version} + COMMENT "Building `wasm-component-ld` ...") + +add_custom_target(wasm-component-ld ALL DEPENDS ${wasm_component_ld}) + +install( + PROGRAMS ${wasm_component_ld} + DESTINATION ${CMAKE_INSTALL_PREFIX}/bin) + +# Setup installation logic for CMake support files. +install( + PROGRAMS src/config/config.sub src/config/config.guess + DESTINATION ${CMAKE_INSTALL_PREFIX}/share/misc) +install( + FILES wasi-sdk.cmake wasi-sdk-pthread.cmake + DESTINATION ${CMAKE_INSTALL_PREFIX}/share/cmake) +install( + DIRECTORY cmake/Platform + DESTINATION ${CMAKE_INSTALL_PREFIX}/share/cmake) + +include(wasi-sdk-dist) + +# Figure out the name of the artifact which is either explicitly specified or +# inferred from CMake default variables. +if(WASI_SDK_ARTIFACT) + set(wasi_sdk_artifact ${WASI_SDK_ARTIFACT}) +else() + if(APPLE) + set(wasi_sdk_os macos) + else() + string(TOLOWER ${CMAKE_SYSTEM_NAME} wasi_sdk_os) + endif() + set(wasi_sdk_arch ${CMAKE_SYSTEM_PROCESSOR}) + set(wasi_sdk_artifact ${wasi_sdk_arch}-${wasi_sdk_os}) +endif() + +set(dist_dir ${CMAKE_CURRENT_BINARY_DIR}/dist) +wasi_sdk_add_tarball(dist-toolchain + ${dist_dir}/wasi-toolchain-${wasi_sdk_version}-${wasi_sdk_artifact}.tar.gz + ${CMAKE_INSTALL_PREFIX}) +add_dependencies(dist-toolchain llvm-build install) +add_custom_target(dist DEPENDS dist-toolchain) diff --git a/deb_from_installation.sh b/deb_from_installation.sh deleted file mode 100755 index 2699c819c..000000000 --- a/deb_from_installation.sh +++ /dev/null @@ -1,43 +0,0 @@ -#!/usr/bin/env sh -set -x - -command -v dpkg-deb >/dev/null -if [ $? -ne 0 ]; then - echo "required tool dpkg-deb missing. exiting" - exit 0 -fi - -set -ex - -if [ -n "$1" ]; then - OUTDIR=$1 -else - OUTDIR=$PWD/dist -fi - -if [ -n "$2" ]; then - VERSION="$2" -else - VERSION=`./version.py` -fi - -if [ -n "$3" ]; then - INSTALL_DIR="$3" -else - INSTALL_DIR=/opt/wasi-sdk -fi - -if [ ! -d $INSTALL_DIR ] ; then - echo "Directory $INSTALL_DIR doesn't exist. Nothing to copy from." - exit 1 -fi - -ARCH=$(dpkg --print-architecture) - -rm -rf build/pkg -mkdir -p build/pkg/opt -mkdir -p build/pkg/DEBIAN -sed -e s/VERSION/$VERSION/ wasi-sdk.control > build/pkg/DEBIAN/control -cp -R $INSTALL_DIR build/pkg/opt/ -cd build && dpkg-deb -b pkg wasi-sdk_$VERSION\_$ARCH\.deb && cd .. -mv build/wasi-sdk_$VERSION\_$ARCH\.deb $OUTDIR/ diff --git a/docker/Dockerfile b/docker/Dockerfile index 63c5a3cff..3dc64cfcb 100644 --- a/docker/Dockerfile +++ b/docker/Dockerfile @@ -1,52 +1,22 @@ -# Docker image with a build toolchain and environment variables set to use -# the wasi-sdk sysroot. The SDK distribution must have first been built, -# for example using docker_build.sh - -# Extract built SDK archive to copy out the sysroot. We use an initial build -# stage to do this to make sure it is only the sysroot, not the entire SDK -# with binaries, that is included in the final image since we install those -# separately. -FROM ubuntu:22.04 as dist - -ADD dist/wasi-sdk-*.*-linux.tar.gz / -ADD dist/libclang_rt.builtins-wasm32-wasi-*.*.tar.gz /wasi-sysroot-clang_rt - -# Move versioned folder to unversioned to using bash glob to allow -# this file to be independent of major version number. -RUN mv /wasi-sdk-* /wasi-sdk +# A container which has a number of build tools pre-installed plus a build of +# `wasi-sdk` installed at `/opt/wasi-sdk`. This also has environment variables +# pre-configued to use the installed toolchain. +# +# This container is built as the last step on CI for this repository and +# pre-built versions of this container are pushed as a package to the repository +# as well. FROM ubuntu:22.04 -ENV LLVM_VERSION 18 - -# Install build toolchain including clang, ld, make, autotools, ninja, and cmake RUN apt-get update && \ - # Temporarily install to setup apt repositories - apt-get install -y curl gnupg && \ -\ - curl -sS https://apt.llvm.org/llvm-snapshot.gpg.key | gpg --dearmor > /etc/apt/trusted.gpg.d/llvm.gpg && \ - echo "deb [signed-by=/etc/apt/trusted.gpg.d/llvm.gpg] http://apt.llvm.org/jammy/ llvm-toolchain-jammy-${LLVM_VERSION} main" >> /etc/apt/sources.list.d/llvm.list && \ - echo "deb-src [signed-by=/etc/apt/trusted.gpg.d/llvm.gpg] http://apt.llvm.org/jammy/ llvm-toolchain-jammy-${LLVM_VERSION} main" >> /etc/apt/sources.list.d/llvm.list && \ -\ - apt-get update && \ - apt-get install -y clang-${LLVM_VERSION} lld-${LLVM_VERSION} cmake ninja-build make autoconf autogen automake libtool && \ - apt-get autoremove -y curl gnupg && \ + apt-get install -y cmake ninja-build make autoconf autogen automake libtool && \ rm -rf /var/lib/apt/lists/* -COPY --from=dist /wasi-sdk/share/wasi-sysroot/ /wasi-sysroot/ -COPY --from=dist /wasi-sysroot-clang_rt/lib/wasi /usr/lib/llvm-${LLVM_VERSION}/lib/clang/${LLVM_VERSION}/lib/wasi - -ADD docker/wasi-sdk.cmake /usr/share/cmake/wasi-sdk.cmake -ADD docker/wasi-sdk-pthread.cmake /usr/share/cmake/wasi-sdk-pthread.cmake -ENV CMAKE_TOOLCHAIN_FILE /usr/share/cmake/wasi-sdk.cmake -ADD cmake/Platform/WASI.cmake /usr/share/cmake/Modules/Platform/WASI.cmake - -ENV CC clang-${LLVM_VERSION} -ENV CXX clang++-${LLVM_VERSION} -ENV LD wasm-ld-${LLVM_VERSION} -ENV AR llvm-ar-${LLVM_VERSION} -ENV RANLIB llvm-ranlib-${LLVM_VERSION} +ADD dist/wasi-sdk-*-x86_64-linux.deb . +RUN dpkg -i wasi-sdk-*.deb && rm wasi-sdk-*.deb -ENV CFLAGS --target=wasm32-wasi --sysroot=/wasi-sysroot -ENV CXXFLAGS --target=wasm32-wasi --sysroot=/wasi-sysroot -ENV LDFLAGS --target=wasm32-wasi --sysroot=/wasi-sysroot +ENV CC /opt/wasi-sdk/bin/clang +ENV CXX /opt/wasi-sdk/bin/clang++ +ENV LD /opt/wasi-sdk/bin/wasm-ld +ENV AR /opt/wasi-sdk/bin/llvm-ar +ENV RANLIB /opt/wasi-sdk/bin/llvm-ranlib diff --git a/docker/wasi-sdk-pthread.cmake b/docker/wasi-sdk-pthread.cmake deleted file mode 100644 index 61354451a..000000000 --- a/docker/wasi-sdk-pthread.cmake +++ /dev/null @@ -1,36 +0,0 @@ -# Cmake toolchain description file for the wasi-sdk docker image - -# This is arbitrary, AFAIK, for now. -cmake_minimum_required(VERSION 3.4.0) - -# To make sure it recognizes the WASI platform -list(APPEND CMAKE_MODULE_PATH /usr/share/cmake/Modules) - -set(CMAKE_SYSTEM_NAME WASI) -set(CMAKE_SYSTEM_VERSION 1) -set(CMAKE_SYSTEM_PROCESSOR wasm32) -set(triple wasm32-wasi-threads) -set(CMAKE_C_FLAGS "${CMAKE_C_FLAGS} -pthread") -set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -pthread") -# wasi-threads requires --import-memory. -# wasi requires --export-memory. -# (--export-memory is implicit unless --import-memory is given) -set(CMAKE_EXE_LINKER_FLAGS "${CMAKE_EXE_LINKER_FLAGS} -Wl,--import-memory") -set(CMAKE_EXE_LINKER_FLAGS "${CMAKE_EXE_LINKER_FLAGS} -Wl,--export-memory") - -set(CMAKE_C_COMPILER /usr/bin/clang-$ENV{LLVM_VERSION}) -set(CMAKE_CXX_COMPILER /usr/bin/clang++-$ENV{LLVM_VERSION}) -set(CMAKE_ASM_COMPILER /usr/bin/clang-$ENV{LLVM_VERSION}) -set(CMAKE_AR /usr/bin/llvm-ar-$ENV{LLVM_VERSION}) -set(CMAKE_RANLIB /usr/bin/llvm-ranlib-$ENV{LLVM_VERSION}) -set(CMAKE_C_COMPILER_TARGET ${triple}) -set(CMAKE_CXX_COMPILER_TARGET ${triple}) -set(CMAKE_ASM_COMPILER_TARGET ${triple}) -SET(CMAKE_SYSROOT /wasi-sysroot) - -# Don't look in the sysroot for executables to run during the build -set(CMAKE_FIND_ROOT_PATH_MODE_PROGRAM NEVER) -# Only look in the sysroot (not in the host paths) for the rest -set(CMAKE_FIND_ROOT_PATH_MODE_LIBRARY ONLY) -set(CMAKE_FIND_ROOT_PATH_MODE_INCLUDE ONLY) -set(CMAKE_FIND_ROOT_PATH_MODE_PACKAGE ONLY) diff --git a/docker/wasi-sdk.cmake b/docker/wasi-sdk.cmake deleted file mode 100644 index eabb877ca..000000000 --- a/docker/wasi-sdk.cmake +++ /dev/null @@ -1,29 +0,0 @@ -# Cmake toolchain description file for the wasi-sdk docker image - -# This is arbitrary, AFAIK, for now. -cmake_minimum_required(VERSION 3.4.0) - -# To make sure it recognizes the WASI platform -list(APPEND CMAKE_MODULE_PATH /usr/share/cmake/Modules) - -set(CMAKE_SYSTEM_NAME WASI) -set(CMAKE_SYSTEM_VERSION 1) -set(CMAKE_SYSTEM_PROCESSOR wasm32) -set(triple wasm32-wasi) - -set(CMAKE_C_COMPILER /usr/bin/clang-$ENV{LLVM_VERSION}) -set(CMAKE_CXX_COMPILER /usr/bin/clang++-$ENV{LLVM_VERSION}) -set(CMAKE_ASM_COMPILER /usr/bin/clang-$ENV{LLVM_VERSION}) -set(CMAKE_AR /usr/bin/llvm-ar-$ENV{LLVM_VERSION}) -set(CMAKE_RANLIB /usr/bin/llvm-ranlib-$ENV{LLVM_VERSION}) -set(CMAKE_C_COMPILER_TARGET ${triple}) -set(CMAKE_CXX_COMPILER_TARGET ${triple}) -set(CMAKE_ASM_COMPILER_TARGET ${triple}) -SET(CMAKE_SYSROOT /wasi-sysroot) - -# Don't look in the sysroot for executables to run during the build -set(CMAKE_FIND_ROOT_PATH_MODE_PROGRAM NEVER) -# Only look in the sysroot (not in the host paths) for the rest -set(CMAKE_FIND_ROOT_PATH_MODE_LIBRARY ONLY) -set(CMAKE_FIND_ROOT_PATH_MODE_INCLUDE ONLY) -set(CMAKE_FIND_ROOT_PATH_MODE_PACKAGE ONLY) diff --git a/docker_build.sh b/docker_build.sh deleted file mode 100755 index 050862fac..000000000 --- a/docker_build.sh +++ /dev/null @@ -1,18 +0,0 @@ -#!/bin/sh -set -ex - -echo "Building the docker image" -docker build \ - --build-arg UID=$(id -u) --build-arg GID=$(id -g) \ - -t wasi-sdk-builder:latest . - -echo "Building the package in docker image" -mkdir -p ~/.ccache -docker run --rm \ - --user $(id -u):$(id -g) \ - -v "$PWD":/workspace:Z \ - -v ~/.ccache:/home/builder/.ccache:Z \ - -e NINJA_FLAGS=-v \ - --tmpfs /tmp:exec \ - wasi-sdk-builder:latest \ - make package LLVM_CMAKE_FLAGS=-DLLVM_CCACHE_BUILD=ON diff --git a/strip_symbols.sh b/strip_symbols.sh deleted file mode 100755 index effd30090..000000000 --- a/strip_symbols.sh +++ /dev/null @@ -1,15 +0,0 @@ -#!/usr/bin/env bash -set -e - -DIRECTORY=${1:-/opt/wasi-sdk/bin} -if [[ "$OSTYPE" == "darwin"* ]] || [[ "$OSTYPE" == "freebsd"* ]]; then -# macos and freebsd find do not support -executable so we fall back on -# having a permission bit to execute: -EXECUTABLES=$(find ${DIRECTORY} -type f -perm +111) -else -EXECUTABLES=$(find ${DIRECTORY} -type f -executable) -fi -for e in ${EXECUTABLES}; do - echo "Stripping symbols: ${e}" - strip ${e} || echo "Failed to strip symbols for ${e}; continuing on." -done diff --git a/tar_from_installation.sh b/tar_from_installation.sh deleted file mode 100755 index 7d0943266..000000000 --- a/tar_from_installation.sh +++ /dev/null @@ -1,51 +0,0 @@ -#!/usr/bin/env bash -set -ex - -if [ -n "$1" ]; then - OUTDIR=$1 -else - OUTDIR=$PWD/dist -fi - -if [ -n "$2" ]; then - VERSION="$2" -else - VERSION=`./version.py` -fi - -if [ -n "$3" ]; then - INSTALL_DIR="$3" -else - INSTALL_DIR=/opt/wasi-sdk -fi - -case "$(uname -s)" in - Linux*) MACHINE=linux;; - Darwin*) MACHINE=macos;; - CYGWIN*) MACHINE=cygwin;; - MINGW*) MACHINE=mingw;; - MSYS*) MACHINE=msys;; #MSYS_NT-10.0-19043 - *) MACHINE="UNKNOWN" -esac - -if [ ! -d $INSTALL_DIR ] ; then - echo "Directory $INSTALL_DIR doesn't exist. Nothing to copy from." - exit 1 -fi - -PKGDIR=build/wasi-sdk-$VERSION -rm -rf $PKGDIR -if [ "$MACHINE" == "cygwin" ] || [ "$MACHINE" == "mingw" ] || [ "$MACHINE" == "msys" ]; then - # Copy with -L to avoid trying to create symlinks on Windows. - cp -R -L $INSTALL_DIR $PKGDIR -else - cp -R $INSTALL_DIR $PKGDIR -fi -cd build -tar czf $OUTDIR/wasi-sdk-$VERSION\-$MACHINE.tar.gz wasi-sdk-$VERSION - -# As well as the full SDK package, also create archives of libclang_rt.builtins -# and the sysroot. These are made available for users who have an existing clang -# installation. -tar czf $OUTDIR/libclang_rt.builtins-wasm32-wasi-$VERSION.tar.gz -C compiler-rt lib/wasi -tar czf $OUTDIR/wasi-sysroot-$VERSION.tar.gz -C wasi-sdk-$VERSION/share wasi-sysroot diff --git a/tests/CMakeLists.txt b/tests/CMakeLists.txt new file mode 100644 index 000000000..f32a760f1 --- /dev/null +++ b/tests/CMakeLists.txt @@ -0,0 +1,90 @@ +# Support for running tests in the `tests/{compile-only,general}` folders +cmake_minimum_required(VERSION 3.22) +project(wasi-sdk-test) +include(CTest) +enable_testing() +set(CMAKE_USE_RELATIVE_PATHS ON) + +add_compile_options(--sysroot=${wasi_sysroot}) +add_link_options(--sysroot=${wasi_sysroot}) + +# Sanity check setup +if (NOT ${CMAKE_SYSTEM_NAME} STREQUAL WASI) + message(FATAL_ERROR "Wrong system name (${CMAKE_SYSTEM_NAME}), wrong toolchain file in use?") +endif() + +if(NOT DEFINED WASI) + message(FATAL_ERROR "WASI is not set, platform file likely not loaded") +endif() + +set(WASI_SDK_RUNWASI "wasmtime" CACHE STRING "Runner for tests") + +# Test everything at O0, O2, and O2+LTO +set(opt_flags -O0 -O2 "-O2 -flto") + +# Executes a single `test` specified. +# +# This will compile `test` for all the various targets and with various +# compiler options. If `runwasi` is non-empty then the test will be executed +# in that runner as well. +function(add_testcase runwasi test) + foreach(target IN LISTS WASI_SDK_TARGETS) + foreach(compile_flags IN LISTS opt_flags) + # Mangle the options into something appropriate for a CMake rule name + string(REGEX REPLACE " " "." target_name "${target}.${compile_flags}.${test}") + + # Add a new test executable based on `test` + add_executable(${target_name} ${test}) + + # Configure all the compile options necessary. For example `--sysroot` + # forces usage of the just-built sysroot. Use `--target` here if the + # target doesn't look like it's already in the name of the compiler as + # well. + target_compile_options(${target_name} PRIVATE --sysroot=${wasi_sysroot}) + if(NOT(CMAKE_C_COMPILER MATCHES ${target})) + target_compile_options(${target_name} PRIVATE --target=${target}) + endif() + + # Apply test-specific compile options and link flags. + if(test MATCHES "clocks.c$") + target_compile_options(${target_name} PRIVATE -D_WASI_EMULATED_PROCESS_CLOCKS) + target_link_options(${target_name} PRIVATE -lwasi-emulated-process-clocks) + elseif(test MATCHES "mmap.c$") + target_compile_options(${target_name} PRIVATE -D_WASI_EMULATED_MMAN) + target_link_options(${target_name} PRIVATE -lwasi-emulated-mman) + elseif(test MATCHES "(sigabrt|signals).c$") + target_compile_options(${target_name} PRIVATE -D_WASI_EMULATED_SIGNAL) + target_link_options(${target_name} PRIVATE -lwasi-emulated-signal) + elseif(test MATCHES "printf-long-double-enabled.c$") + target_link_options(${target_name} PRIVATE -lc-printscan-long-double) + endif() + + # Apply language-specific options and dependencies. + if(test MATCHES "cc$") + target_compile_options(${target_name} PRIVATE -fno-exceptions) + add_dependencies(${target_name} libcxx-${target}) + else() + add_dependencies(${target_name} wasi-libc-${target}) + endif() + + # Apply target-specific options. + if(target MATCHES threads) + target_compile_options(${target_name} PRIVATE -pthread) + endif() + + if(runwasi) + add_test( + NAME test-${target_name} + COMMAND + bash ../testcase.sh + "${runwasi}" + ${test} + $ + WORKING_DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR}) + endif() + endforeach() + endforeach() +endfunction() + +add_subdirectory(compile-only) +add_subdirectory(general) diff --git a/tests/cmake/CMakeLists.txt b/tests/cmake/CMakeLists.txt deleted file mode 100644 index 28c9b839b..000000000 --- a/tests/cmake/CMakeLists.txt +++ /dev/null @@ -1,34 +0,0 @@ -cmake_minimum_required(VERSION 3.22) - -project(wasi-sdk-test) - -# Sanity check setup -if (NOT ${CMAKE_SYSTEM_NAME} STREQUAL WASI) - message(FATAL_ERROR "Wrong system name (${CMAKE_SYSTEM_NAME}), wrong toolchain file in use?") -endif() - -if(NOT DEFINED WASI) - message(FATAL_ERROR "WASI is not set, platform file likely not loaded") -endif() - -set(RUNWASI "" CACHE STRING "Path to or name of WASM runner") - -# Test build a C and C++ target respectively -add_executable(void_main_c ../general/void_main.c) -add_executable(void_main_cc ../general/void_main.cc) - -include(CTest) -enable_testing() - -add_test(NAME void_main_c - COMMAND - ${CMAKE_CURRENT_SOURCE_DIR}/test_driver.sh - ${RUNWASI} - $ - ${CMAKE_CURRENT_SOURCE_DIR}/../general/void_main.c.stdout.expected) -add_test(NAME void_main_cc - COMMAND - ${CMAKE_CURRENT_SOURCE_DIR}/test_driver.sh - ${RUNWASI} - $ - ${CMAKE_CURRENT_SOURCE_DIR}/../general/void_main.cc.stdout.expected) diff --git a/tests/cmake/test_driver.sh b/tests/cmake/test_driver.sh deleted file mode 100755 index 6e469a1ae..000000000 --- a/tests/cmake/test_driver.sh +++ /dev/null @@ -1,17 +0,0 @@ -#!/bin/bash -# Simplified runner for cmake - -set -ex - -runwasi="$1" -target="$2" -stdout_expected="$3" -stderr_expected="/dev/null" - -stdout_observed="$target.stdout.observed" -stderr_observed="$target.stderr.observed" - -"$runwasi" "$target" > "$stdout_observed" 2> "$stderr_observed" - -diff -u "$stderr_expected" "$stderr_observed" -diff -u "$stdout_expected" "$stdout_observed" diff --git a/tests/compile-only/CMakeLists.txt b/tests/compile-only/CMakeLists.txt new file mode 100644 index 000000000..484a9cddc --- /dev/null +++ b/tests/compile-only/CMakeLists.txt @@ -0,0 +1,9 @@ +file(GLOB c_compile_tests RELATIVE ${CMAKE_CURRENT_SOURCE_DIR} "*.c") +file(GLOB cxx_compile_tests RELATIVE ${CMAKE_CURRENT_SOURCE_DIR} "*.cc") + +set(compile_tests ${c_compile_tests} ${cxx_compile_tests}) + +foreach(test IN LISTS compile_tests) + add_testcase("" ${test}) +endforeach() + diff --git a/tests/general/CMakeLists.txt b/tests/general/CMakeLists.txt new file mode 100644 index 000000000..ef0a5528d --- /dev/null +++ b/tests/general/CMakeLists.txt @@ -0,0 +1,8 @@ +file(GLOB c_general_tests RELATIVE ${CMAKE_CURRENT_SOURCE_DIR} "*.c") +file(GLOB cxx_general_tests RELATIVE ${CMAKE_CURRENT_SOURCE_DIR} "*.cc") + +set(general_tests ${c_general_tests} ${cxx_general_tests}) + +foreach(test IN LISTS general_tests) + add_testcase(${WASI_SDK_RUNWASI} ${test}) +endforeach() diff --git a/tests/general/abort.c.stderr.expected.filter b/tests/general/abort.c.stderr.expected.filter index ecd16c053..98233a1ee 100755 --- a/tests/general/abort.c.stderr.expected.filter +++ b/tests/general/abort.c.stderr.expected.filter @@ -2,6 +2,6 @@ set -euo pipefail cat \ - | sed -e 's/main module `abort\.c\.[^`]*\.wasm`/main module `abort.c.---.wasm`/' \ + | sed -e 's/main module `.*abort\.c\.wasm`/main module `abort.c.---.wasm`/' \ | sed -e 's/failed to invoke.*/failed to invoke ---/' \ | sed -E '/0x[[:xdigit:]]+/d' diff --git a/tests/general/assert-fail.c b/tests/general/assert-fail.c index 07f455784..54479f9b5 100644 --- a/tests/general/assert-fail.c +++ b/tests/general/assert-fail.c @@ -1,3 +1,7 @@ +#ifdef NDEBUG +#undef NDEBUG +#endif + #include #include diff --git a/tests/general/assert-fail.c.stderr.expected b/tests/general/assert-fail.c.stderr.expected index 9e0f44c5a..fcf1f75ac 100644 --- a/tests/general/assert-fail.c.stderr.expected +++ b/tests/general/assert-fail.c.stderr.expected @@ -1,4 +1,4 @@ -Assertion failed: false (assert-fail.c: main: 5) +Assertion failed: false (assert-fail.c: main: 9) Error: failed to run main module `assert-fail.c.---.wasm` Caused by: diff --git a/tests/general/assert-fail.c.stderr.expected.filter b/tests/general/assert-fail.c.stderr.expected.filter index a1d649de0..77849a1ca 100755 --- a/tests/general/assert-fail.c.stderr.expected.filter +++ b/tests/general/assert-fail.c.stderr.expected.filter @@ -2,6 +2,7 @@ set -euo pipefail cat \ - | sed -e 's/main module `assert-fail\.c\.[^`]*\.wasm`/main module `assert-fail.c.---.wasm`/' \ + | sed -e 's/main module `.*assert-fail\.c\.wasm`/main module `assert-fail.c.---.wasm`/' \ | sed -e 's/failed to invoke.*/failed to invoke ---/' \ + | sed -e 's/Assertion failed: false (.*assert-fail.c/Assertion failed: false (assert-fail.c/' \ | sed -E '/0x[[:xdigit:]]+/d' diff --git a/tests/general/clocks.c.options b/tests/general/clocks.c.options deleted file mode 100644 index 03953a9dc..000000000 --- a/tests/general/clocks.c.options +++ /dev/null @@ -1 +0,0 @@ --D_WASI_EMULATED_PROCESS_CLOCKS -lwasi-emulated-process-clocks diff --git a/tests/general/iostream_main.cc.options b/tests/general/iostream_main.cc.options deleted file mode 100644 index f7f5b39e8..000000000 --- a/tests/general/iostream_main.cc.options +++ /dev/null @@ -1 +0,0 @@ --fno-exceptions diff --git a/tests/general/mmap.c.options b/tests/general/mmap.c.options deleted file mode 100644 index 63236a027..000000000 --- a/tests/general/mmap.c.options +++ /dev/null @@ -1 +0,0 @@ --D_WASI_EMULATED_MMAN -lwasi-emulated-mman diff --git a/tests/general/printf-long-double-enabled.c.options b/tests/general/printf-long-double-enabled.c.options deleted file mode 100644 index 2b2ec8e65..000000000 --- a/tests/general/printf-long-double-enabled.c.options +++ /dev/null @@ -1 +0,0 @@ --lc-printscan-long-double diff --git a/tests/general/sigabrt.c.options b/tests/general/sigabrt.c.options deleted file mode 100644 index 4f07a3ec6..000000000 --- a/tests/general/sigabrt.c.options +++ /dev/null @@ -1 +0,0 @@ --D_WASI_EMULATED_SIGNAL -lwasi-emulated-signal diff --git a/tests/general/sigabrt.c.stderr.expected.filter b/tests/general/sigabrt.c.stderr.expected.filter index 425b060c2..e6b54c2b2 100755 --- a/tests/general/sigabrt.c.stderr.expected.filter +++ b/tests/general/sigabrt.c.stderr.expected.filter @@ -2,6 +2,6 @@ set -euo pipefail cat \ - | sed -e 's/main module `sigabrt\.c\.[^`]*\.wasm`/main module `sigabrt.c.---.wasm`/' \ + | sed -e 's/main module `.*sigabrt\.c\.wasm`/main module `sigabrt.c.---.wasm`/' \ | sed -e 's/source location: @[[:xdigit:]]*$/source location: @----/' \ | head -n 6 diff --git a/tests/general/signals.c b/tests/general/signals.c index 27c8e9eae..be4f54eef 100644 --- a/tests/general/signals.c +++ b/tests/general/signals.c @@ -1,3 +1,7 @@ +#ifdef NDEBUG +#undef NDEBUG +#endif + #include #include #include @@ -6,7 +10,10 @@ #include // Make sure this exists. +#pragma clang diagnostic push +#pragma clang diagnostic ignored "-W#warnings" #include +#pragma clang diagnostic pop volatile sig_atomic_t flag = 0; diff --git a/tests/general/signals.c.options b/tests/general/signals.c.options deleted file mode 100644 index 4f07a3ec6..000000000 --- a/tests/general/signals.c.options +++ /dev/null @@ -1 +0,0 @@ --D_WASI_EMULATED_SIGNAL -lwasi-emulated-signal diff --git a/tests/run.sh b/tests/run.sh deleted file mode 100755 index bc8dd4d0b..000000000 --- a/tests/run.sh +++ /dev/null @@ -1,89 +0,0 @@ -#!/bin/bash -set -ueo pipefail - -# Top-level test runner. Usage is "run.sh " to run tests -# in compile-only mode, or "run.sh " where -# is a WASI-capable runtime to run the tests in full compile and -# execute mode. -# -# The compiler used during testing is loaded from ``. -if [ $# -lt 1 ]; then - echo "Path to WASI SDK is required" - exit 1 -fi - -wasi_sdk="$1" - -# Determine the wasm runtime to use, if one is provided. -if [ $# -gt 1 ]; then - runwasi="$2" -else - runwasi="" -fi - -testdir=$(dirname $0) - -echo "SDK: $wasi_sdk" - -# NB: all tests are run with the default `clang` and `clang++` executables -# but they're also executed with the target-prefixed `clang` executables to -# ensure that those work as well when the `--target` option is omitted. - -for target in $TARGETS; do - echo "===== Testing target $target =====" - cd $testdir/compile-only - for options in -O0 -O2 "-O2 -flto"; do - echo "===== Testing compile-only with $options =====" - for file in *.c; do - echo "Testing compile-only $file..." - ../testcase.sh "$target" "" "$wasi_sdk/bin/clang" "$options --target=$target" "$file" - ../testcase.sh "$target" "" "$wasi_sdk/bin/$target-clang" "$options" "$file" - done - for file in *.cc; do - echo "Testing compile-only $file..." - ../testcase.sh "$target" "" "$wasi_sdk/bin/clang++" "$options --target=$target -fno-exceptions" "$file" - ../testcase.sh "$target" "" "$wasi_sdk/bin/$target-clang++" "$options -fno-exceptions" "$file" - done - done - cd - >/dev/null - - cd $testdir/general - for options in -O0 -O2 "-O2 -flto"; do - echo "===== Testing with $options =====" - for file in *.c; do - echo "Testing $file..." - ../testcase.sh "$target" "$runwasi" "$wasi_sdk/bin/clang" "$options --target=$target" "$file" - ../testcase.sh "$target" "$runwasi" "$wasi_sdk/bin/$target-clang" "$options" "$file" - done - for file in *.cc; do - echo "Testing $file..." - ../testcase.sh "$target" "$runwasi" "$wasi_sdk/bin/clang++" "$options --target=$target -fno-exceptions" "$file" - ../testcase.sh "$target" "$runwasi" "$wasi_sdk/bin/$target-clang++" "$options -fno-exceptions" "$file" - done - done - cd - >/dev/null -done - -# Test cmake build system for wasi-sdk -test_cmake() { - local option - for option in Debug Release; do - rm -rf "$testdir/cmake/build/$option" - mkdir -p "$testdir/cmake/build/$option" - cd "$testdir/cmake/build/$option" - cmake \ - -G "Unix Makefiles" \ - -DCMAKE_BUILD_TYPE="$option" \ - -DRUNWASI="$runwasi" \ - -DWASI_SDK_PREFIX="$wasi_sdk" \ - -DCMAKE_TOOLCHAIN_FILE="$wasi_sdk/share/cmake/wasi-sdk.cmake" \ - ../.. - make - if [[ -n "$runwasi" ]]; then - ctest --output-on-failure - fi - cd - >/dev/null - done -} - -test_cmake diff --git a/tests/testcase.sh b/tests/testcase.sh index 3d8860464..56aebe2ed 100755 --- a/tests/testcase.sh +++ b/tests/testcase.sh @@ -6,39 +6,18 @@ set -ueo pipefail # Command-line parsing; this script is meant to be run from a higher-level # script, so don't do anything fancy. -target="$1" -runwasi="$2" -compiler="$3" -options="$4" -input="$5" +runwasi="$1" +input="$2" +wasm="$3" # Compile names for generated files. -wasm="$input.$options.wasm" -stdout_observed="$input.$options.stdout.observed" -stderr_observed="$input.$options.stderr.observed" -exit_status_observed="$input.$options.exit_status.observed" - -# Optionally load compiler options from a file. -if [ -e "$input.options" ]; then - file_options=$(cat "$input.options") -else - file_options= -fi - -if echo "$target" | grep -q -- '-threads$'; then - pthread_options="-pthread" -else - pthread_options= -fi - -echo "Testing $input..." - -# Compile the testcase. -$compiler $pthread_options $options $file_options "$input" -o "$wasm" +stdout_observed="$wasm.stdout.observed" +stderr_observed="$wasm.stderr.observed" +exit_status_observed="$wasm.exit_status.observed" # If we don't have a runwasi command, we're just doing compile-only testing. if [ "$runwasi" == "" ]; then - exit 0 + exit 1 fi # Determine the input file to write to stdin. @@ -75,11 +54,7 @@ echo $exit_status > "$exit_status_observed" # Determine the reference files to compare with. if [ -e "$input.stdout.expected" ]; then - if [ -e "$input.$target.stdout.expected" ]; then - stdout_expected="$input.$target.stdout.expected" - else - stdout_expected="$input.stdout.expected" - fi + stdout_expected="$input.stdout.expected" # Apply output filters. if [ -e "$input.stdout.expected.filter" ]; then @@ -93,11 +68,7 @@ else fi if [ -e "$input.stderr.expected" ]; then - if [ -e "$input.$target.stderr.expected" ]; then - stderr_expected="$input.$target.stderr.expected" - else - stderr_expected="$input.stderr.expected" - fi + stderr_expected="$input.stderr.expected" # Apply output filters. if [ -e "$input.stderr.expected.filter" ]; then @@ -117,6 +88,6 @@ fi # If there are any differences, diff will return a non-zero exit status, and # since this script uses "set -e", it will return a non-zero exit status too. -diff -u "$stderr_expected" "$stderr_observed" -diff -u "$stdout_expected" "$stdout_observed" -diff -u "$exit_status_expected" "$exit_status_observed" +diff --ignore-space-change -u "$stderr_expected" "$stderr_observed" +diff --ignore-space-change -u "$stdout_expected" "$stdout_observed" +diff --ignore-space-change -u "$exit_status_expected" "$exit_status_observed" diff --git a/wasi-sdk.cmake b/wasi-sdk.cmake index f6a0059c8..430e98f04 100644 --- a/wasi-sdk.cmake +++ b/wasi-sdk.cmake @@ -1,7 +1,7 @@ # Cmake toolchain description file for the Makefile # This is arbitrary, AFAIK, for now. -cmake_minimum_required(VERSION 3.4.0) +cmake_minimum_required(VERSION 3.5.0) # Until Platform/WASI.cmake is upstream we need to inject the path to it # into CMAKE_MODULE_PATH. diff --git a/wasi-sdk.control b/wasi-sdk.control index 0586f37ba..e9e75c40f 100644 --- a/wasi-sdk.control +++ b/wasi-sdk.control @@ -1,6 +1,6 @@ Package: wasi-sdk Version: VERSION -Architecture: amd64 +Architecture: ARCH Priority: optional Description: Clang toolchain with wasm32-wasi default target, and the wasi sysroot Maintainer: Pat Hickey