diff --git a/guix/README.md b/guix/README.md new file mode 100644 index 0000000000000..8ce8cb97a0d7a --- /dev/null +++ b/guix/README.md @@ -0,0 +1,225 @@ +# Bootstrappable Bitcoin Core Builds + +This directory contains the files necessary to perform bootstrappable Bitcoin +Core builds. + +[Bootstrappability][b17e] furthers our binary security guarantees by allowing us +to _audit and reproduce_ our toolchain instead of blindly _trusting_ binary +downloads. + +We achieve bootstrappability by using Guix as a functional package manager. + +## Requirements + +Conservatively, a x86_64 machine with: + +- 4GB of free disk space on the partition that /gnu/store will reside in +- 24GB of free disk space on the partition that the Bitcoin Core git repository + resides in + +> Note: these requirements are slightly less onerous than those of Gitian builds + +## Setup + +### Installing Guix + +If you're just testing this out, you can use the +[Dockerfile][fanquake/guix-docker] for convenience. It automatically speeds up +your builds by [using substitutes](#speeding-up-builds-with-substitute-servers). +If you don't want this behaviour, refer to the [next +section](#choosing-your-security-model). + +Otherwise, follow the [Guix installation guide][guix/bin-install]. + +> Note: For those who like to keep their filesystems clean, Guix is designed to +> be very standalone and _will not_ conflict with your system's package +> manager/existing setup. It _only_ touches `/var/guix`, `/gnu`, and +> `~/.config/guix`. + +### Choosing your security model + +Guix allows us to achieve better binary security by using our CPU time to build +everything from scratch. However, it doesn't sacrifice user choice in pursuit of +this: users can decide whether or not to bootstrap and to use substitutes. + +After installation, you may want to consider [adding substitute +servers](#speeding-up-builds-with-substitute-servers) to speed up your build if +that fits your security model (say, if you're just testing that this works). +This is skippable if you're using the [Dockerfile][fanquake/guix-docker]. + +If you prefer not to use any substitutes, make sure to set +`ADDITIONAL_GUIX_ENVIRONMENT_FLAGS` like the following snippet. The first build +will take a while, but the resulting packages will be cached for future builds. + +```sh +export ADDITIONAL_GUIX_ENVIRONMENT_FLAGS='--no-substitutes' +``` + +Likewise, to perform a bootstrapped build (takes even longer): + +```sh +export ADDITIONAL_GUIX_ENVIRONMENT_FLAGS='--bootstrap --no-substitutes' +``` + +### Using a version of Guix with `guix time-machine` capabilities + +> Note: This entire section can be skipped if you are already using a version of +> Guix that has [the `guix time-machine` command][guix/time-machine]. + +Once Guix is installed, if it doesn't have the `guix time-machine` command, pull +the latest `guix`. + +```sh +guix pull --max-jobs=4 # change number of jobs accordingly +``` + +Make sure that you are using your current profile. (You are prompted to do this +at the end of the `guix pull`) + +```bash +export PATH="${HOME}/.config/guix/current/bin${PATH:+:}$PATH" +``` + +## Usage + +### As a Development Environment + +For a Bitcoin Core depends development environment, simply invoke + +```sh +guix environment --manifest=contrib/guix/manifest.scm +``` + +And you'll land back in your shell with all the build dependencies required for +a `depends` build injected into your environment. + +### As a Tool for Deterministic Builds + +From the top of a clean Bitcoin Core repository: + +```sh +./contrib/guix/guix-build.sh +``` + +After the build finishes successfully (check the status code please), compare +hashes: + +```sh +find output/ -type f -print0 | sort -z | xargs -r0 sha256sum +``` + +#### Recognized environment variables + +* _**HOSTS**_ + + Override the space-separated list of platform triples for which to perform a + bootstrappable build. _(defaults to "x86\_64-linux-gnu + arm-linux-gnueabihf aarch64-linux-gnu riscv64-linux-gnu")_ + + > Windows and OS X platform triplet support are WIP. + +* _**SOURCES_PATH**_ + + Set the depends tree download cache for sources. This is passed through to the + depends tree. Setting this to the same directory across multiple builds of the + depends tree can eliminate unnecessary redownloading of package sources. + +* _**MAX_JOBS**_ + + Override the maximum number of jobs to run simultaneously, you might want to + do so on a memory-limited machine. This may be passed to `make` as in `make + --jobs="$MAX_JOBS"` or `xargs` as in `xargs -P"$MAX_JOBS"`. _(defaults to the + value of `nproc` outside the container)_ + +* _**SOURCE_DATE_EPOCH**_ + + Override the reference UNIX timestamp used for bit-for-bit reproducibility, + the variable name conforms to [standard][r12e/source-date-epoch]. _(defaults + to the output of `$(git log --format=%at -1)`)_ + +* _**V**_ + + If non-empty, will pass `V=1` to all `make` invocations, making `make` output + verbose. + +* _**ADDITIONAL_GUIX_ENVIRONMENT_FLAGS**_ + + Additional flags to be passed to `guix environment`. For a fully-bootstrapped + build, set this to `--bootstrap --no-substitutes` (refer to the [security + model section](#choosing-your-security-model) for more details). Note that a + fully-bootstrapped build will take quite a long time on the first run. + +## Tips and Tricks + +### Speeding up builds with substitute servers + +_This whole section is automatically done in the convenience +[Dockerfiles][fanquake/guix-docker]_ + +For those who are used to life in the fast _(and trustful)_ lane, you can use +[substitute servers][guix/substitutes] to enable binary downloads of packages. + +> For those who only want to use substitutes from the official Guix build farm +> and have authorized the build farm's signing key during Guix's installation, +> you don't need to do anything. + +#### Authorize the signing keys + +For the official Guix build farm at https://ci.guix.gnu.org, run as root: + +``` +guix archive --authorize < ~root/.config/guix/current/share/guix/ci.guix.gnu.org.pub +``` + +For dongcarl's substitute server at https://guix.carldong.io, run as root: + +```sh +wget -qO- 'https://guix.carldong.io/signing-key.pub' | guix archive --authorize +``` + +#### Use the substitute servers + +The official Guix build farm at https://ci.guix.gnu.org is automatically used +unless the `--no-substitutes` flag is supplied. + +This can be overridden for all `guix` invocations by passing the +`--substitute-urls` option to your invocation of `guix-daemon`. This can also be +overridden on a call-by-call basis by passing the same `--substitute-urls` +option to client tools such at `guix environment`. + +To use dongcarl's substitute server for Bitcoin Core builds after having +[authorized his signing key](#authorize-the-signing-keys): + +``` +export ADDITIONAL_GUIX_ENVIRONMENT_FLAGS='--substitute-urls="https://guix.carldong.io https://ci.guix.gnu.org"' +``` + +## FAQ + +### How can I trust the binary installation? + +As mentioned at the bottom of [this manual page][guix/bin-install]: + +> The binary installation tarballs can be (re)produced and verified simply by +> running the following command in the Guix source tree: +> +> make guix-binary.x86_64-linux.tar.xz + +### When will Guix be packaged in debian? + +Vagrant Cascadian has been making good progress on this +[here][debian/guix-package]. We have all the pieces needed to put up an APT +repository and will likely put one up soon. + +[b17e]: http://bootstrappable.org/ +[r12e/source-date-epoch]: https://reproducible-builds.org/docs/source-date-epoch/ + +[guix/install.sh]: https://git.savannah.gnu.org/cgit/guix.git/plain/etc/guix-install.sh +[guix/bin-install]: https://www.gnu.org/software/guix/manual/en/html_node/Binary-Installation.html +[guix/env-setup]: https://www.gnu.org/software/guix/manual/en/html_node/Build-Environment-Setup.html +[guix/substitutes]: https://www.gnu.org/software/guix/manual/en/html_node/Substitutes.html +[guix/substitute-server-auth]: https://www.gnu.org/software/guix/manual/en/html_node/Substitute-Server-Authorization.html +[guix/time-machine]: https://guix.gnu.org/manual/en/html_node/Invoking-guix-time_002dmachine.html + +[debian/guix-package]: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=850644 +[fanquake/guix-docker]: https://github.com/fanquake/core-review/tree/master/guix diff --git a/guix/guix-build.sh b/guix/guix-build.sh new file mode 100755 index 0000000000000..11d2c8b86727e --- /dev/null +++ b/guix/guix-build.sh @@ -0,0 +1,119 @@ +#!/usr/bin/env bash +export LC_ALL=C +set -e -o pipefail + +# Determine the maximum number of jobs to run simultaneously (overridable by +# environment) +MAX_JOBS="${MAX_JOBS:-$(nproc)}" + +# Download the depends sources now as we won't have internet access in the build +# container +make -C "${PWD}/depends" -j"$MAX_JOBS" download ${V:+V=1} ${SOURCES_PATH:+SOURCES_PATH="$SOURCES_PATH"} + +# Determine the reference time used for determinism (overridable by environment) +SOURCE_DATE_EPOCH="${SOURCE_DATE_EPOCH:-$(git log --format=%at -1)}" + +# Execute "$@" in a pinned, possibly older version of Guix, for reproducibility +# across time. +time-machine() { + guix time-machine --url=https://github.com/dongcarl/guix.git \ + --commit=b066c25026f21fb57677aa34692a5034338e7ee3 \ + -- "$@" +} + +# Function to be called when building for host ${1} and the user interrupts the +# build +int_trap() { +cat << EOF +** INT received while building ${1}, you may want to clean up the relevant + output, deploy, and distsrc-* directories before rebuilding + +Hint: To blow everything away, you may want to use: + + $ git clean -xdff --exclude='/depends/SDKs/*' + +Specifically, this will remove all files without an entry in the index, +excluding the SDK directory. Practically speaking, this means that all ignored +and untracked files and directories will be wiped, allowing you to start anew. +EOF +} + +# Deterministically build Bitcoin Core for HOSTs (overridable by environment) +# shellcheck disable=SC2153 +for host in ${HOSTS=x86_64-linux-gnu arm-linux-gnueabihf aarch64-linux-gnu riscv64-linux-gnu x86_64-w64-mingw32}; do + + # Display proper warning when the user interrupts the build + trap 'int_trap ${host}' INT + + ( + # Required for 'contrib/guix/manifest.scm' to output the right manifest + # for the particular $HOST we're building for + export HOST="$host" + + # Run the build script 'contrib/guix/libexec/build.sh' in the build + # container specified by 'contrib/guix/manifest.scm'. + # + # Explanation of `guix environment` flags: + # + # --container run command within an isolated container + # + # Running in an isolated container minimizes build-time differences + # between machines and improves reproducibility + # + # --pure unset existing environment variables + # + # Same rationale as --container + # + # --no-cwd do not share current working directory with an + # isolated container + # + # When --container is specified, the default behavior is to share + # the current working directory with the isolated container at the + # same exact path (e.g. mapping '/home/satoshi/bitcoin/' to + # '/home/satoshi/bitcoin/'). This means that the $PWD inside the + # container becomes a source of irreproducibility. --no-cwd disables + # this behaviour. + # + # --share=SPEC for containers, share writable host file system + # according to SPEC + # + # --share="$PWD"=/bitcoin + # + # maps our current working directory to /bitcoin + # inside the isolated container, which we later cd + # into. + # + # While we don't want to map our current working directory to the + # same exact path (as this introduces irreproducibility), we do want + # it to be at a _fixed_ path _somewhere_ inside the isolated + # container so that we have something to build. '/bitcoin' was + # chosen arbitrarily. + # + # ${SOURCES_PATH:+--share="$SOURCES_PATH"} + # + # make the downloaded depends sources path available + # inside the isolated container + # + # The isolated container has no network access as it's in a + # different network namespace from the main machine, so we have to + # make the downloaded depends sources available to it. The sources + # should have been downloaded prior to this invocation. + # + # shellcheck disable=SC2086 + time-machine environment --manifest="${PWD}/contrib/guix/manifest.scm" \ + --container \ + --pure \ + --no-cwd \ + --share="$PWD"=/bitcoin \ + --expose="$(git rev-parse --git-common-dir)" \ + ${SOURCES_PATH:+--share="$SOURCES_PATH"} \ + ${ADDITIONAL_GUIX_ENVIRONMENT_FLAGS} \ + -- env HOST="$host" \ + MAX_JOBS="$MAX_JOBS" \ + SOURCE_DATE_EPOCH="${SOURCE_DATE_EPOCH:?unable to determine value}" \ + ${V:+V=1} \ + ${SOURCES_PATH:+SOURCES_PATH="$SOURCES_PATH"} \ + bash -c "cd /bitcoin && bash contrib/guix/libexec/build.sh" + ) + +done diff --git a/guix/libexec/build.sh b/guix/libexec/build.sh new file mode 100644 index 0000000000000..62d1f1edc7120 --- /dev/null +++ b/guix/libexec/build.sh @@ -0,0 +1,314 @@ +#!/usr/bin/env bash +export LC_ALL=C +set -e -o pipefail +export TZ=UTC + +# Check that environment variables assumed to be set by the environment are set +echo "Building for platform triple ${HOST:?not set} with reference timestamp ${SOURCE_DATE_EPOCH:?not set}..." +echo "At most ${MAX_JOBS:?not set} jobs will run at once..." + +##################### +# Environment Setup # +##################### + +# The depends folder also serves as a base-prefix for depends packages for +# $HOSTs after successfully building. +BASEPREFIX="${PWD}/depends" + +# Setup an output directory for our build +OUTDIR="${OUTDIR:-${PWD}/output}" +[ -e "$OUTDIR" ] || mkdir -p "$OUTDIR" + +# Setup the directory where our Bitcoin Core build for HOST will occur +DISTSRC="${DISTSRC:-${PWD}/distsrc-${HOST}}" +if [ -e "$DISTSRC" ]; then + echo "DISTSRC directory '${DISTSRC}' exists, probably because of previous builds... Aborting..." + exit 1 +else + mkdir -p "$DISTSRC" +fi + +# Given a package name and an output name, return the path of that output in our +# current guix environment +store_path() { + grep --extended-regexp "/[^-]{32}-${1}-[^-]+${2:+-${2}}" "${GUIX_ENVIRONMENT}/manifest" \ + | head --lines=1 \ + | sed --expression='s|^[[:space:]]*"||' \ + --expression='s|"[[:space:]]*$||' +} + +# Set environment variables to point Guix's cross-toolchain to the right +# includes/libs for $HOST +case "$HOST" in + *mingw*) + # Determine output paths to use in CROSS_* environment variables + CROSS_GLIBC="$(store_path "mingw-w64-x86_64-winpthreads")" + CROSS_GCC="$(store_path "gcc-cross-${HOST}")" + CROSS_GCC_LIBS=( "${CROSS_GCC}/lib/gcc/${HOST}"/* ) # This expands to an array of directories... + CROSS_GCC_LIB="${CROSS_GCC_LIBS[0]}" # ...we just want the first one (there should only be one) + + NATIVE_GCC="$(store_path gcc-glibc-2.27-toolchain)" + export LIBRARY_PATH="${NATIVE_GCC}/lib:${NATIVE_GCC}/lib64" + export CPATH="${NATIVE_GCC}/include" + + export CROSS_C_INCLUDE_PATH="${CROSS_GCC_LIB}/include:${CROSS_GCC_LIB}/include-fixed:${CROSS_GLIBC}/include" + export CROSS_CPLUS_INCLUDE_PATH="${CROSS_GCC}/include/c++:${CROSS_GCC}/include/c++/${HOST}:${CROSS_GCC}/include/c++/backward:${CROSS_C_INCLUDE_PATH}" + export CROSS_LIBRARY_PATH="${CROSS_GCC}/lib:${CROSS_GCC}/${HOST}/lib:${CROSS_GCC_LIB}:${CROSS_GLIBC}/lib" + ;; + *linux*) + CROSS_GLIBC="$(store_path "glibc-cross-${HOST}")" + CROSS_GLIBC_STATIC="$(store_path "glibc-cross-${HOST}" static)" + CROSS_KERNEL="$(store_path "linux-libre-headers-cross-${HOST}")" + CROSS_GCC="$(store_path "gcc-cross-${HOST}")" + CROSS_GCC_LIBS=( "${CROSS_GCC}/lib/gcc/${HOST}"/* ) # This expands to an array of directories... + CROSS_GCC_LIB="${CROSS_GCC_LIBS[0]}" # ...we just want the first one (there should only be one) + + # NOTE: CROSS_C_INCLUDE_PATH is missing ${CROSS_GCC_LIB}/include-fixed, because + # the limits.h in it is missing a '#include_next ' + export CROSS_C_INCLUDE_PATH="${CROSS_GCC_LIB}/include:${CROSS_GLIBC}/include:${CROSS_KERNEL}/include" + export CROSS_CPLUS_INCLUDE_PATH="${CROSS_GCC}/include/c++:${CROSS_GCC}/include/c++/${HOST}:${CROSS_GCC}/include/c++/backward:${CROSS_C_INCLUDE_PATH}" + export CROSS_LIBRARY_PATH="${CROSS_GCC}/lib:${CROSS_GCC}/${HOST}/lib:${CROSS_GCC_LIB}:${CROSS_GLIBC}/lib:${CROSS_GLIBC_STATIC}/lib" + ;; + *) + exit 1 ;; +esac + +# Sanity check CROSS_*_PATH directories +IFS=':' read -ra PATHS <<< "${CROSS_C_INCLUDE_PATH}:${CROSS_CPLUS_INCLUDE_PATH}:${CROSS_LIBRARY_PATH}" +for p in "${PATHS[@]}"; do + if [ ! -d "$p" ]; then + echo "'$p' doesn't exist or isn't a directory... Aborting..." + exit 1 + fi +done + +# Disable Guix ld auto-rpath behavior +export GUIX_LD_WRAPPER_DISABLE_RPATH=yes + +# Make /usr/bin if it doesn't exist +[ -e /usr/bin ] || mkdir -p /usr/bin + +# Symlink file and env to a conventional path +[ -e /usr/bin/file ] || ln -s --no-dereference "$(command -v file)" /usr/bin/file +[ -e /usr/bin/env ] || ln -s --no-dereference "$(command -v env)" /usr/bin/env + +# Determine the correct value for -Wl,--dynamic-linker for the current $HOST +case "$HOST" in + *linux*) + glibc_dynamic_linker=$( + case "$HOST" in + i686-linux-gnu) echo /lib/ld-linux.so.2 ;; + x86_64-linux-gnu) echo /lib64/ld-linux-x86-64.so.2 ;; + arm-linux-gnueabihf) echo /lib/ld-linux-armhf.so.3 ;; + aarch64-linux-gnu) echo /lib/ld-linux-aarch64.so.1 ;; + riscv64-linux-gnu) echo /lib/ld-linux-riscv64-lp64d.so.1 ;; + *) exit 1 ;; + esac + ) + ;; +esac + +# Environment variables for determinism +export QT_RCC_TEST=1 +export QT_RCC_SOURCE_DATE_OVERRIDE=1 +export TAR_OPTIONS="--owner=0 --group=0 --numeric-owner --mtime='@${SOURCE_DATE_EPOCH}' --sort=name" +export TZ="UTC" + +#################### +# Depends Building # +#################### + +# Build the depends tree, overriding variables that assume multilib gcc +make -C depends --jobs="$MAX_JOBS" HOST="$HOST" \ + ${V:+V=1} \ + ${SOURCES_PATH+SOURCES_PATH="$SOURCES_PATH"} \ + i686_linux_CC=i686-linux-gnu-gcc \ + i686_linux_CXX=i686-linux-gnu-g++ \ + i686_linux_AR=i686-linux-gnu-ar \ + i686_linux_RANLIB=i686-linux-gnu-ranlib \ + i686_linux_NM=i686-linux-gnu-nm \ + i686_linux_STRIP=i686-linux-gnu-strip \ + x86_64_linux_CC=x86_64-linux-gnu-gcc \ + x86_64_linux_CXX=x86_64-linux-gnu-g++ \ + x86_64_linux_AR=x86_64-linux-gnu-ar \ + x86_64_linux_RANLIB=x86_64-linux-gnu-ranlib \ + x86_64_linux_NM=x86_64-linux-gnu-nm \ + x86_64_linux_STRIP=x86_64-linux-gnu-strip \ + qt_config_opts_i686_linux='-platform linux-g++ -xplatform bitcoin-linux-g++' + + +########################### +# Source Tarball Building # +########################### + +# Define DISTNAME variable. +# shellcheck source=contrib/gitian-descriptors/assign_DISTNAME +source contrib/gitian-descriptors/assign_DISTNAME + +GIT_ARCHIVE="${OUTDIR}/src/${DISTNAME}.tar.gz" + +# Create the source tarball if not already there +if [ ! -e "$GIT_ARCHIVE" ]; then + mkdir -p "$(dirname "$GIT_ARCHIVE")" + git archive --output="$GIT_ARCHIVE" HEAD +fi + +########################### +# Binary Tarball Building # +########################### + +# CONFIGFLAGS +CONFIGFLAGS="--enable-reduce-exports --disable-bench --disable-gui-tests" +case "$HOST" in + *linux*) CONFIGFLAGS+=" --enable-glibc-back-compat" ;; +esac + +# CFLAGS +HOST_CFLAGS="-O2 -g" +case "$HOST" in + *linux*) HOST_CFLAGS+=" -ffile-prefix-map=${PWD}=." ;; + *mingw*) HOST_CFLAGS+=" -fno-ident" ;; +esac + +# CXXFLAGS +HOST_CXXFLAGS="$HOST_CFLAGS" + +# LDFLAGS +case "$HOST" in + *linux*) HOST_LDFLAGS="-Wl,--as-needed -Wl,--dynamic-linker=$glibc_dynamic_linker -static-libstdc++ -Wl,-O2" ;; + *mingw*) HOST_LDFLAGS="-Wl,--no-insert-timestamp" ;; +esac + +# Make $HOST-specific native binaries from depends available in $PATH +export PATH="${BASEPREFIX}/${HOST}/native/bin:${PATH}" +( + cd "$DISTSRC" + + # Extract the source tarball + tar -xf "${GIT_ARCHIVE}" + + ./autogen.sh + + # Configure this DISTSRC for $HOST + # shellcheck disable=SC2086 + env CONFIG_SITE="${BASEPREFIX}/${HOST}/share/config.site" \ + ./configure --prefix=/ \ + --disable-ccache \ + --disable-maintainer-mode \ + --disable-dependency-tracking \ + ${CONFIGFLAGS} \ + CFLAGS="${HOST_CFLAGS}" \ + CXXFLAGS="${HOST_CXXFLAGS}" \ + ${HOST_LDFLAGS:+LDFLAGS="${HOST_LDFLAGS}"} + + sed -i.old 's/-lstdc++ //g' {./,src/dashbls/,src/secp256k1/}{config.status,libtool} + + + # Build Bitcoin Core + make --jobs="$MAX_JOBS" ${V:+V=1} + + # Perform basic ELF security checks on a series of executables. + make -C src --jobs=1 check-security ${V:+V=1} + + case "$HOST" in + *linux*|*mingw*) + # Check that executables only contain allowed gcc, glibc and libstdc++ + # version symbols for Linux distro back-compatibility. + make -C src --jobs=1 check-symbols ${V:+V=1} + ;; + esac + + # Make the os-specific installers + case "$HOST" in + *mingw*) + make deploy ${V:+V=1} BITCOIN_WIN_INSTALLER="${OUTDIR}/${DISTNAME}-win64-setup-unsigned.exe" + ;; + esac + + # Setup the directory where our Bitcoin Core build for HOST will be + # installed. This directory will also later serve as the input for our + # binary tarballs. + INSTALLPATH="${PWD}/installed/${DISTNAME}" + mkdir -p "${INSTALLPATH}" + # Install built Bitcoin Core to $INSTALLPATH + make install DESTDIR="${INSTALLPATH}" ${V:+V=1} + + ( + cd installed + + case "$HOST" in + *mingw*) + mv --target-directory="$DISTNAME"/lib/ "$DISTNAME"/bin/*.dll + ;; + esac + + # Prune libtool and object archives + find . -name "lib*.la" -delete + find . -name "lib*.a" -delete + + # Prune pkg-config files + rm -r "${DISTNAME}/lib/pkgconfig" + + # Split binaries and libraries from their debug symbols + { + find "${DISTNAME}/bin" -type f -executable -print0 + find "${DISTNAME}/lib" -type f -print0 + } | xargs -0 -n1 -P"$MAX_JOBS" -I{} "${DISTSRC}/contrib/devtools/split-debug.sh" {} {} {}.dbg + + case "$HOST" in + *mingw*) + cp "${DISTSRC}/doc/README_windows.txt" "${DISTNAME}/readme.txt" + ;; + *linux*) + cp "${DISTSRC}/README.md" "${DISTNAME}/" + ;; + esac + + # Finally, deterministically produce {non-,}debug binary tarballs ready + # for release + case "$HOST" in + *mingw*) + find "${DISTNAME}" -not -name "*.dbg" -print0 \ + | xargs -0r touch --no-dereference --date="@${SOURCE_DATE_EPOCH}" + find "${DISTNAME}" -not -name "*.dbg" \ + | sort \ + | zip -X@ "${OUTDIR}/${DISTNAME}-${HOST//x86_64-w64-mingw32/win64}.zip" \ + || ( rm -f "${OUTDIR}/${DISTNAME}-${HOST//x86_64-w64-mingw32/win64}.zip" && exit 1 ) + find "${DISTNAME}" -name "*.dbg" -print0 \ + | xargs -0r touch --no-dereference --date="@${SOURCE_DATE_EPOCH}" + find "${DISTNAME}" -name "*.dbg" \ + | sort \ + | zip -X@ "${OUTDIR}/${DISTNAME}-${HOST//x86_64-w64-mingw32/win64}-debug.zip" \ + || ( rm -f "${OUTDIR}/${DISTNAME}-${HOST//x86_64-w64-mingw32/win64}-debug.zip" && exit 1 ) + ;; + *linux*) + find "${DISTNAME}" -not -name "*.dbg" -print0 \ + | sort --zero-terminated \ + | tar --create --no-recursion --mode='u+rw,go+r-w,a+X' --null --files-from=- \ + | gzip -9n > "${OUTDIR}/${DISTNAME}-${HOST}.tar.gz" \ + || ( rm -f "${OUTDIR}/${DISTNAME}-${HOST}.tar.gz" && exit 1 ) + find "${DISTNAME}" -name "*.dbg" -print0 \ + | sort --zero-terminated \ + | tar --create --no-recursion --mode='u+rw,go+r-w,a+X' --null --files-from=- \ + | gzip -9n > "${OUTDIR}/${DISTNAME}-${HOST}-debug.tar.gz" \ + || ( rm -f "${OUTDIR}/${DISTNAME}-${HOST}-debug.tar.gz" && exit 1 ) + ;; + esac + ) +) + +case "$HOST" in + *mingw*) + cp -rf --target-directory=. contrib/windeploy + ( + cd ./windeploy + mkdir unsigned + cp --target-directory=unsigned/ "${OUTDIR}/${DISTNAME}-win64-setup-unsigned.exe" + find . -print0 \ + | sort --zero-terminated \ + | tar --create --no-recursion --mode='u+rw,go+r-w,a+X' --null --files-from=- \ + | gzip -9n > "${OUTDIR}/${DISTNAME}-win-unsigned.tar.gz" \ + || ( rm -f "${OUTDIR}/${DISTNAME}-win-unsigned.tar.gz" && exit 1 ) + ) + ;; +esac diff --git a/guix/manifest.scm b/guix/manifest.scm new file mode 100644 index 0000000000000..346a0cb0bea57 --- /dev/null +++ b/guix/manifest.scm @@ -0,0 +1,200 @@ +(use-modules (gnu) + (gnu packages) + (gnu packages autotools) + (gnu packages base) + (gnu packages bash) + (gnu packages check) + (gnu packages commencement) + (gnu packages compression) + (gnu packages cross-base) + (gnu packages file) + (gnu packages gawk) + (gnu packages gcc) + (gnu packages installers) + (gnu packages linux) + (gnu packages mingw) + (gnu packages perl) + (gnu packages pkg-config) + (gnu packages python) + (gnu packages shells) + (gnu packages bison) + (gnu packages version-control) + (guix build-system gnu) + (guix build-system trivial) + (guix gexp) + (guix packages) + (guix profiles) + (guix utils)) + +(define (make-ssp-fixed-gcc xgcc) + "Given a XGCC package, return a modified package that uses the SSP function +from glibc instead of from libssp.so. Our `symbol-check' script will complain if +we link against libssp.so, and thus will ensure that this works properly. + +Taken from: +http://www.linuxfromscratch.org/hlfs/view/development/chapter05/gcc-pass1.html" + (package + (inherit xgcc) + (arguments + (substitute-keyword-arguments (package-arguments xgcc) + ((#:make-flags flags) + `(cons "gcc_cv_libc_provides_ssp=yes" ,flags)))))) + +(define (make-gcc-rpath-link xgcc) + "Given a XGCC package, return a modified package that replace each instance of +-rpath in the default system spec that's inserted by Guix with -rpath-link" + (package + (inherit xgcc) + (arguments + (substitute-keyword-arguments (package-arguments xgcc) + ((#:phases phases) + `(modify-phases ,phases + (add-after 'pre-configure 'replace-rpath-with-rpath-link + (lambda _ + (substitute* (cons "gcc/config/rs6000/sysv4.h" + (find-files "gcc/config" + "^gnu-user.*\\.h$")) + (("-rpath=") "-rpath-link=")) + #t)))))))) + +(define (make-cross-toolchain target + base-gcc-for-libc + base-kernel-headers + base-libc + base-gcc) + "Create a cross-compilation toolchain package for TARGET" + (let* ((xbinutils (cross-binutils target)) + ;; 1. Build a cross-compiling gcc without targeting any libc, derived + ;; from BASE-GCC-FOR-LIBC + (xgcc-sans-libc (cross-gcc target + #:xgcc base-gcc-for-libc + #:xbinutils xbinutils)) + ;; 2. Build cross-compiled kernel headers with XGCC-SANS-LIBC, derived + ;; from BASE-KERNEL-HEADERS + (xkernel (cross-kernel-headers target + base-kernel-headers + xgcc-sans-libc + xbinutils)) + ;; 3. Build a cross-compiled libc with XGCC-SANS-LIBC and XKERNEL, + ;; derived from BASE-LIBC + (xlibc (cross-libc target + base-libc + xgcc-sans-libc + xbinutils + xkernel)) + ;; 4. Build a cross-compiling gcc targeting XLIBC, derived from + ;; BASE-GCC + (xgcc (cross-gcc target + #:xgcc base-gcc + #:xbinutils xbinutils + #:libc xlibc))) + ;; Define a meta-package that propagates the resulting XBINUTILS, XLIBC, and + ;; XGCC + (package + (name (string-append target "-toolchain")) + (version (package-version xgcc)) + (source #f) + (build-system trivial-build-system) + (arguments '(#:builder (begin (mkdir %output) #t))) + (propagated-inputs + `(("binutils" ,xbinutils) + ("libc" ,xlibc) + ("libc:static" ,xlibc "static") + ("gcc" ,xgcc))) + (synopsis (string-append "Complete GCC tool chain for " target)) + (description (string-append "This package provides a complete GCC tool +chain for " target " development.")) + (home-page (package-home-page xgcc)) + (license (package-license xgcc))))) + +(define* (make-bitcoin-cross-toolchain target + #:key + (base-gcc-for-libc gcc-5) + (base-kernel-headers linux-libre-headers-4.19) + (base-libc glibc-2.27) + (base-gcc (make-gcc-rpath-link gcc-9))) + "Convenience wrapper around MAKE-CROSS-TOOLCHAIN with default values +desirable for building Bitcoin Core release binaries." + (make-cross-toolchain target + base-gcc-for-libc + base-kernel-headers + base-libc + base-gcc)) + +(define (make-gcc-with-pthreads gcc) + (package-with-extra-configure-variable gcc "--enable-threads" "posix")) + +(define (make-mingw-pthreads-cross-toolchain target) + "Create a cross-compilation toolchain package for TARGET" + (let* ((xbinutils (cross-binutils target)) + (pthreads-xlibc mingw-w64-x86_64-winpthreads) + (pthreads-xgcc (make-gcc-with-pthreads + (cross-gcc target + #:xgcc (make-ssp-fixed-gcc gcc-9) + #:xbinutils xbinutils + #:libc pthreads-xlibc)))) + ;; Define a meta-package that propagates the resulting XBINUTILS, XLIBC, and + ;; XGCC + (package + (name (string-append target "-posix-toolchain")) + (version (package-version pthreads-xgcc)) + (source #f) + (build-system trivial-build-system) + (arguments '(#:builder (begin (mkdir %output) #t))) + (propagated-inputs + `(("binutils" ,xbinutils) + ("libc" ,pthreads-xlibc) + ("gcc" ,pthreads-xgcc))) + (synopsis (string-append "Complete GCC tool chain for " target)) + (description (string-append "This package provides a complete GCC tool +chain for " target " development.")) + (home-page (package-home-page pthreads-xgcc)) + (license (package-license pthreads-xgcc))))) + + +(packages->manifest + (append + (list ;; The Basics + bash-minimal + which + coreutils + util-linux + ;; File(system) inspection + file + grep + diffutils + findutils + ;; File transformation + patch + gawk + sed + ;; Compression and archiving + tar + bzip2 + gzip + xz + zlib + ;; Build tools + gnu-make + libtool + autoconf + automake + pkg-config + bison + ;; Scripting + perl + python-3.7 + ;; Git + git + ;; Native gcc 9 toolchain targeting glibc 2.27 + (make-gcc-toolchain gcc-9 glibc-2.27)) + (let ((target (getenv "HOST"))) + (cond ((string-suffix? "-mingw32" target) + ;; Windows + (list zip (make-mingw-pthreads-cross-toolchain "x86_64-w64-mingw32") nsis-x86_64)) + ((string-contains target "riscv64-linux-") + (list (make-bitcoin-cross-toolchain "riscv64-linux-gnu" + #:base-gcc-for-libc gcc-7))) + ((string-contains target "-linux-") + (list (make-bitcoin-cross-toolchain target))) + (else '())))))