diff --git a/README.md b/README.md index 6e6ecbfea..e9ab245c8 100644 --- a/README.md +++ b/README.md @@ -38,22 +38,24 @@ The Wasm-sdk's build process needs some packages : * `clang` * `ninja` -Please refer to your OS documentation to install those packages. +Please refer to your OS documentation to install those packages. -## Build +## Build -To build the full package +To build the full package: ```shell script cd wasi-sdk NINJA_FLAGS=-v make package ``` -The built package can be found into `dist` directory. +The built package can be found into `dist` directory. For releasing a new +version of the package on GitHub, see [RELEASING.md](RELEASING.md). ## Install A typical installation from the release binaries might look like the following: + ```shell script export WASI_VERSION=14 export WASI_VERSION_FULL=${WASI_VERSION}.0 @@ -64,13 +66,16 @@ tar xvf wasi-sdk-${WASI_VERSION_FULL}-linux.tar.gz ## Use Use the clang installed in the wasi-sdk directory: + ```shell script export WASI_SDK_PATH=`pwd`/wasi-sdk-${WASI_VERSION_FULL} CC="${WASI_SDK_PATH}/bin/clang --sysroot=${WASI_SDK_PATH}/share/wasi-sysroot" $CC foo.c -o foo.wasm ``` -Note: `${WASI_SDK_PATH}/share/wasi-sysroot` contains the WASI-specific includes/libraries/etc. The `--sysroot=...` option -is not necessary if `WASI_SDK_PATH` is `/opt/wasi-sdk`. + +Note: `${WASI_SDK_PATH}/share/wasi-sysroot` contains the WASI-specific +includes/libraries/etc. The `--sysroot=...` option is not necessary if +`WASI_SDK_PATH` is `/opt/wasi-sdk`. ## Notes for Autoconf @@ -104,7 +109,7 @@ disabled in a configure step before building with wasi-sdk. This repository does not yet support C++ exceptions. C++ code is supported only with -fno-exceptions for now. Similarly, there is not -yet support for setjmp/longjmp. Work on support for [exception handling] +yet support for setjmp/longjmp. Work on support for [exception handling] is underway at the language level which will support both of these features. diff --git a/RELEASING.md b/RELEASING.md new file mode 100644 index 000000000..9f8dcf794 --- /dev/null +++ b/RELEASING.md @@ -0,0 +1,48 @@ +# Release Process + +To publish a new version of `wasi-sdk` as a GitHub release: + +1. Tag a commit with an annotated tag. Note that this must be an annotated tag, + not a lightweight tag, so that `version.sh` can use it for calculating the + package version (use `git show wasi-sdk-...` to show other tag messages). + + ```shell script + TAG=wasi-sdk-1 + git tag -a $TAG + git push origin $TAG + ``` + +2. Find a successful workflow that CI has run for the tag. That successful + workflow run will have build artifacts that need to be attached to the + release. One could search around in the GitHub [actions], but the following + script will list completed workflows for a tag (get a token [here][tokens]): + + ```shell script + ci/get-workflows-for-tag.sh $TAG $GITHUB_TOKEN + ``` + + [actions]: https://github.com/WebAssembly/wasi-sdk/actions + [tokens]: https://github.com/settings/tokens + +3. Download and unzip the workflow artifacts. Note that artifacts with `+m` or + `.m` suffixes indicate that the Git tree was modified. Expect some duplicates + since some of the same artifacts are built on multiple CI runners (e.g., + Windows, MacOS, Linux). The following script does all of this automatically: + + ```shell script + ci/download-workflow-artifacts.sh $TAG $WORKFLOW_RUN_ID $GITHUB_TOKEN + ``` + +4. Draft a new release. This could be done [manually][releases] but the + following script simplifies the uploading of all the files and auto-generates + the release description: + + ```shell script + ci/draft-release.sh $TAG $ARTIFACTS_DIR $GITHUB_TOKEN + ``` + + [releases]: https://github.com/WebAssembly/wasi-sdk/releases + +5. Publish the release; the previous step only creates a draft. Follow the link + in the previous step or navigate to the GitHub [releases] to review the + description, commit, tag, and assets before clicking "Publish" diff --git a/ci/download-workflow-artifacts.sh b/ci/download-workflow-artifacts.sh new file mode 100755 index 000000000..3fccd587e --- /dev/null +++ b/ci/download-workflow-artifacts.sh @@ -0,0 +1,102 @@ +#!/usr/bin/env bash +set -e + +# This script downloads and unzips the artifacts produced in a workflow run. It +# also checks that the workflow commit corresponds to the tag commit that these +# artifacts will be released under. The script has several pre-requisites: +# - some standard Bash tools (curl, unzip) and one slightly more rare one (jq) +# - an already-created tag in the repository (this marks the code to release) +# - the ID of a workflow run that has run successfully--this is where we +# retrieve the artifacts from +# - a GitHub access token, see https://github.com/settings/tokens +# +# Usage: download-workflow-artifacts.sh + +TAG=$1 +WORKFLOW_RUN_ID=$2 +GITHUB_TOKEN=$3 +GITHUB_API_VERSION=2022-11-28 +GITHUB_API_URL=https://api.github.com/repos/WebAssembly/wasi-sdk +TMP_DIR=$(mktemp -d -t wasi-sdk-artifacts.XXXXXXX) + +if [ -z "${TAG}" ] || [ -z "${WORKFLOW_RUN_ID}" ] || [ -z "${GITHUB_TOKEN}" ]; then + >&2 echo "Missing parameter; exiting..." + >&2 echo "Usage: download-worfklow-artifacts.sh " + exit 1 +fi + +# Get the commit SHA for the passed tag. +# See https://docs.github.com/en/rest/commits/commits#get-a-commit +MATCHING_COMMIT=$(curl \ + -H "Accept: application/vnd.github+json" \ + -H "Authorization: Bearer ${GITHUB_TOKEN}" \ + -H "X-GitHub-Api-Version: ${GITHUB_API_VERSION}" \ + "${GITHUB_API_URL}/commits/${TAG}") +COMMIT=$(echo $MATCHING_COMMIT | jq -r '.sha') +>&2 echo "===== Found commit for tag ${TAG}: ${COMMIT} =====" + +# Check that the commit of the workflow run matches the tag commit and that the +# workflow was successful. +# See https://docs.github.com/en/rest/actions/workflow-runs#get-a-workflow-run +WORKFLOW_RUN=$(curl \ + -H "Accept: application/vnd.github+json" \ + -H "Authorization: Bearer ${GITHUB_TOKEN}" \ + -H "X-GitHub-Api-Version: ${GITHUB_API_VERSION}" \ + "${GITHUB_API_URL}/actions/runs/${WORKFLOW_RUN_ID}") +WORKFLOW_COMMIT=$(echo $WORKFLOW_RUN | jq -r '.head_sha') +WORKFLOW_STATUS=$(echo $WORKFLOW_RUN | jq -r '.status') +>&2 echo "===== Found commit for workflow ${WORKFLOW_RUN_ID}: ${WORKFLOW_COMMIT} =====" +if [ "${COMMIT}" != "${WORKFLOW_COMMIT}" ]; then + >&2 echo "Commit at tag ${TAG} did not match the commit for workflow ${WORKFLOW_RUN_ID}, exiting...:" + >&2 echo " ${COMMIT} != ${WORKFLOW_COMMIT}" + exit 1 +fi +if [ "${WORKFLOW_STATUS}" != "completed" ]; then + >&2 echo "Workflow ${WORKFLOW_RUN_ID} did not end successfully, exiting...:" + >&2 echo " status = ${WORKFLOW_STATUS}" + exit 1 +fi + +# List out the artifacts in the given workflow run. +# See https://docs.github.com/en/rest/actions/artifacts#list-workflow-run-artifacts +ARTIFACTS=$(curl \ + -H "Accept: application/vnd.github+json" \ + -H "Authorization: Bearer ${GITHUB_TOKEN}" \ + -H "X-GitHub-Api-Version: ${GITHUB_API_VERSION}" \ + "${GITHUB_API_URL}/actions/runs/${WORKFLOW_RUN_ID}/artifacts" \ + | jq -r '.artifacts[] | [(.id|tostring), .name, .archive_download_url] | join(",")') + +for A in $ARTIFACTS; do + ID=$(echo $A | cut -d ',' -f 1) + NAME=$(echo $A | cut -d ',' -f 2) + URL=$(echo $A | cut -d ',' -f 3) + TO=$TMP_DIR/$NAME.zip + >&2 echo "===== Downloading: ${TO} =====" + + # Download the artifacts to the temporary directory. + # See https://docs.github.com/en/rest/actions/artifacts#download-an-artifact + curl \ + -H "Accept: application/vnd.github+json" \ + -H "Authorization: Bearer ${GITHUB_TOKEN}" \ + -H "X-GitHub-Api-Version: ${GITHUB_API_VERSION}" \ + --location --output "${TO}" \ + "${GITHUB_API_URL}/actions/artifacts/${ID}/zip" +done + +# Unzip the workflow artifacts into a `release` directory. +pushd $TMP_DIR > /dev/null +mkdir release +ls -1 *.zip | xargs -n1 unzip -q -o -d release +# Some explanation: +# -1 prints each file on a separate line +# -n1 runs the command once for each item +# -q means quietly +# -o allows unzip to overwrite existing files (e.g., multiple copies of `libclang_rt.builtins-wasm32-wasi-...`) +# -d tells unzip which directory to place things in +>&2 echo "===== Files to release: ${TMP_DIR}/release =====" +>&2 ls -1 release +popd > /dev/null + +>&2 echo +>&2 echo "Ensure the above artifacts look correct, then run \`draft-release.sh\` with the following directory:" +echo "${TMP_DIR}/release" diff --git a/ci/draft-release.sh b/ci/draft-release.sh new file mode 100755 index 000000000..207da9796 --- /dev/null +++ b/ci/draft-release.sh @@ -0,0 +1,71 @@ +#!/usr/bin/env bash +set -e + +# This script creates a draft pre-release with the artifacts produced in a +# workflow run (see `download-workflow-artifacts.sh`). Note that the pre-release +# is not published publicly--this is kept as a manual step as a safeguard. The +# script has several pre-requisites: +# - some standard Bash tools (curl, unzip) and one slightly more rare one (jq) +# - an already-created tag in the repository (this marks the code to release) +# - a directory containing the artifacts to attach to the release. +# - a GitHub access token, see https://github.com/settings/tokens +# +# Usage: draft-release.sh + +TAG=$1 +ARTIFACTS_DIR=$2 +GITHUB_TOKEN=$3 +GITHUB_API_VERSION=2022-11-28 +GITHUB_API_URL=https://api.github.com/repos/WebAssembly/wasi-sdk +TMP_DIR=$(mktemp -d -t release.sh.XXXXXXX) + +if [ -z "${TAG}" ] || [ -z "${ARTIFACTS_DIR}" ] || [ -z "${GITHUB_TOKEN}" ]; then + >&2 echo "Missing parameter; exiting..." + >&2 echo "Usage: draft-release.sh " + exit 1 +fi + +# Get the commit SHA for the passed tag. +# See https://docs.github.com/en/rest/commits/commits#get-a-commit +MATCHING_COMMIT=$(curl \ + -H "Accept: application/vnd.github+json" \ + -H "Authorization: Bearer ${GITHUB_TOKEN}" \ + -H "X-GitHub-Api-Version: ${GITHUB_API_VERSION}" \ + "${GITHUB_API_URL}/commits/${TAG}") +COMMIT=$(echo $MATCHING_COMMIT | jq -r '.sha') +>&2 echo "===== Found commit for tag ${TAG}: ${COMMIT} =====" + +# Create a draft pre-release for this commit. +# See https://docs.github.com/en/rest/releases/releases#create-a-release +RELEASE_JSON=$(curl \ + -X POST \ + -H "Accept: application/vnd.github+json" \ + -H "Authorization: Bearer ${GITHUB_TOKEN}"\ + -H "X-GitHub-Api-Version: ${GITHUB_API_VERSION}" \ + "${GITHUB_API_URL}/releases" \ + -d '{"tag_name":"'${TAG}'","target_commitish":"'${COMMIT}'","name":"'${TAG}'","draft":true,"prerelease":true,"generate_release_notes":true}') +UPLOAD_URL=$(echo $RELEASE_JSON | jq -r '.upload_url') +# Remove the "helpful" but invalid URL suffix that GitHub adds: +UPLOAD_URL=${UPLOAD_URL/\{?name,label\}} +HTML_URL=$(echo $RELEASE_JSON | jq -r '.html_url') +>&2 echo "===== Created draft release: ${HTML_URL} =====" + +# Upload the unzipped artifact files to the release. +# See https://docs.github.com/en/rest/releases/assets#upload-a-release-asset +for FILE in $(ls "${ARTIFACTS_DIR}"); do + FROM=$ARTIFACTS_DIR/$FILE + >&2 echo "===== Uploading: ${FROM} =====" + UPLOADED=$(curl \ + -X POST \ + -H "Accept: application/vnd.github+json" \ + -H "Authorization: Bearer ${GITHUB_TOKEN}"\ + -H "X-GitHub-Api-Version: ${GITHUB_API_VERSION}" \ + -H "Content-Type: application/octet-stream" \ + "${UPLOAD_URL}?name=${FILE}" \ + --data-binary "@${FROM}") +done + +>&2 echo +>&2 echo "===== Completed =====" +>&2 echo "This created a draft release, do not forget to manually publish it at:" +echo "${HTML_URL}" diff --git a/ci/get-workflows-for-tag.sh b/ci/get-workflows-for-tag.sh new file mode 100755 index 000000000..1b117cb72 --- /dev/null +++ b/ci/get-workflows-for-tag.sh @@ -0,0 +1,43 @@ +#!/usr/bin/env bash +set -e + +# This script retrieves a list of GitHub workflow runs that have successfully completed for a given +# Git tag. The list is printed to stdout (all other output to stderr). It has several +# pre-requisites: +# - some standard Bash tools (curl) and one slightly more rare one (jq) +# - an already-created tag in the repository (this marks the code to release) +# - a GitHub access token, see https://github.com/settings/tokens +# +# Usage: get-workflows-for-tag.sh + +TAG=$1 +GITHUB_TOKEN=$2 +GITHUB_API_VERSION=2022-11-28 +GITHUB_API_URL=https://api.github.com/repos/WebAssembly/wasi-sdk + +if [ -z "${TAG}" ] || [ -z "${GITHUB_TOKEN}" ]; then + >&2 echo "Missing parameter; exiting..." + >&2 echo "Usage: get-workflows-for-tag.sh " + exit 1 +fi + +# Get the commit SHA for the passed tag. +# See https://docs.github.com/en/rest/commits/commits#get-a-commit +MATCHING_COMMIT=$(curl \ + -H "Accept: application/vnd.github+json" \ + -H "Authorization: Bearer ${GITHUB_TOKEN}" \ + -H "X-GitHub-Api-Version: ${GITHUB_API_VERSION}" \ + "${GITHUB_API_URL}/commits/${TAG}") +COMMIT=$(echo $MATCHING_COMMIT | jq -r '.sha') +>&2 echo "===== Found commit for tag ${TAG}: ${COMMIT} =====" + +# Find the workflow runs matching the tag commit. +# See https://docs.github.com/en/rest/actions/workflow-runs#list-workflow-runs-for-a-repository +MATCHING_WORKFLOWS=$(curl \ + -H "Accept: application/vnd.github+json" \ + -H "Authorization: Bearer ${GITHUB_TOKEN}" \ + -H "X-GitHub-Api-Version: ${GITHUB_API_VERSION}" \ + "${GITHUB_API_URL}/actions/runs?head_sha=${COMMIT}&status=success") +WORKFLOW_RUN_IDS=$(echo $MATCHING_WORKFLOWS | jq -r '.workflow_runs[].id') +>&2 echo "===== Matching workflow run IDs: =====" +echo "$WORKFLOW_RUN_IDS"