Provide statically linked binary for Linux/AArch64

[benz0li]

The binary for Linux/x86_64 is statically linked

$ stack --version
Version 2.11.1, Git revision c1167a6abc3f4978ccded5ba0246a57387da0e2f x86_64 hpack-0.35.2
$ file /usr/local/bin/stack 
/usr/local/bin/stack: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, stripped

whereas the binary for Linux/AArch64 is not.

$ stack --version
Version 2.11.1, Git revision c1167a6abc3f4978ccded5ba0246a57387da0e2f aarch64 hpack-0.35.2
$ file /usr/local/bin/stack
/usr/local/bin/stack: ELF 64-bit LSB executable, ARM aarch64, version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux-aarch64.so.1, for GNU/Linux 3.7.0, BuildID[xxHash]=3d8f9d614342c40f, stripped

[mpilgrem]

I am a little confused, and this is the source of my confusion:

• Stack's ChangeLog, under Stack 2.3.1 (before my time), states that 'Since we no longer have dynamically linked Linux binaries, we are removing removing the -static suffix from the static Linux binaries.' (The removal of the -static suffix from file names never happened.)
• Stack has a Cabal flag static. It has the effect of adding -static and -pthread to Cabal's ld-options key. The description of the flag states that those options are passed on to GHC when linking. But GHC aleady defaults to -static (see below)?
• the manual for GHC 9.2.7 (which is what is currently used to build Stack) states that -static is GHC's default (https://downloads.haskell.org/~ghc/9.2.7/docs/html/users_guide/phases.html?highlight=static#ghc-flag--static) I can't see that Stack uses a -dynamic flag anywhere.
• as for -pthread, I understand that it is a GCC linker option: from man gcc: "Link with the POSIX threads library. This option is supported on GNU/Linux targets, most other Unix derivatives, and also on x86 Cygwin and MinGW targets. On some targets this option also sets flags for the preprocessor, so it should be used consistently for both compilation and linking."
• GHC's manual also states about -static: "Tell the linker to avoid shared Haskell libraries, if possible." If -static is the default, does the 'if possible' and the AArch64 Stack being 'dynamically linked' imply that a static AArch64 just is not possible?

The AArch64 Stack gets built through three main files:

1. the Dockerfile at etc/dockerfiles/arm64.Dockerfile (which sets up the /home/stack/release executable compiled from release.hs);
2. the GitHub workflow/job at .github/workflows/integration-tests.yml'/'linux-arm64 (which ends with docker run --rm -v $(pwd):/src -w /src stack bash -c "/home/stack/release build"; and
3. the etc/scripts/release.hs script that makes use of the shake package.

I can't see in any of those three key files where -dynamic would come in.

[mpilgrem]

@hasufell is the master of building things for different platforms. Perhaps he can help bring some clarity.

[benz0li]

[...] does the 'if possible' and the AArch64 Stack being 'dynamically linked' imply that a static AArch64 just is not possible?

Pandoc, which is built using glcr.b-data.ch/ghc/ghc-musl, is statically linked for both Linux/x86_64 and Linux/AArch64:

---

Cross references:

• https://github.com/benz0li/ghc-musl
• Can Binary Releases be Done for aarch64 and armv7h jgm/pandoc#5450 (comment)

[hasufell]

@mpilgrem the release for linux is built in an alpine container (which uses musl):

• stack/.github/workflows/integration-tests.yml

  Line 32 in 62f6709

  release-args: "--alpine"

• stack/etc/scripts/release.hs

  Line 102 in 62f6709

  , Option "" [alpineOptName]

The static is passed to ld, not ghc. That means we instruct the system linker to link C libraries statically:

stack/stack.cabal

Line 516 in 62f6709

ld-options: -static -pthread

We could do this theoretically as well for ARM64 Linux. I'm not too sure how the --docker flag works in stack. I've never used it. Do we need a pre-built musl arm64 stack binary outside of the docker container?

A static binary solves two things:

1. problems with portability across glibc based linux distros
2. running on non-glibc distros (like musl)

The second part is moot if there are no corresponding GHC bindists. So my opinion is that we need proper GHC bindists for musl ARM64, so that stack and ghcup can add it to its metadata and everyone can build static ARM64 binaries without docker tricks.

I can raise this with GHC developers in the next meeting. But overall, I'm not sure how big the gain is for all of this.

But yes, you could probably do this with only a specialized docker image.

[benz0li]

[...] So my opinion is that we need proper GHC bindists for musl ARM64, so that stack and ghcup can add it to its metadata and everyone can build static ARM64 binaries without docker tricks.

Couldn't this be bootstrapped from glcr.b-data.ch/ghc/ghc-musl or from current official Alpine releases?:

• Alpine 3.17 has GHC v9.0.2 available for both architectures
• Alpine 3.18 has GHC v9.4.4 available for both architectures

I can raise this with GHC developers in the next meeting. But overall, I'm not sure how big the gain is for all of this.

Thank you. Alpine Linux (musl libc) is made to produce statically linked binaries. Such binaries are relocatable to on any [compatible¹] Linux distribution.
= A single release for all Linux distributions (per architecture). This is a huge gain for all release managers.

Footnotes

1. If the binary is compiled for a specific architecture, it can only be run on the same architecture. ↩

[benz0li]

Proper GHC bindists for Linux (AArch64) Alpine releases would also resolve haskell/docker-haskell#22.

[hasufell]

Couldn't this be bootstrapped from glcr.b-data.ch/ghc/ghc-musl or from current official Alpine releases for some time?

I'm not sure I understand your proposal.

The normal use case of stack, ghcup and all other tooling has nothing to do with docker, but relies on those tools to download and install a bindist that will work on said platform. --docker and --system-ghc are both specialized use cases.

I think we should support the main use case first.

It might not be hard to convince GHC devs to provide another alpine variant, since alpine x86_64 is already supported.

[mpilgrem]

@hasufell, thank you for your explanations. I've fixed Stack's static Cabal flag documentation.

Notes for my own benefit:

• release.hs build --alpine (used for Linux/x86_64) uses stack install stack --flag=stack:static --docker --system-ghc --no-install-ghc (with some other Stack flags as well).

• The --system-ghc flag will try to use a GHC on the PATH (this is the default for --docker, in any event). The --no-install-ghc will stop the download of a GHC, if one on the PATH is not what is needed.

• The --docker flag will enable Stack building inside the specified Docker container, which in this case is (in stack.yaml) is repo: fpco/alpine-haskell-stack:9.2.7, which specifies the Docker image at the default Docker repository https://hub.docker.com/r/.

• The Dockerfile which built that image is at https://github.com/fpco/alpine-haskell-stack. I think that is, ultimately, built on top of alpine:3.17.2.

[mpilgrem]

@benz0li, I think I have caught up.

If I understand correctly, gitlab.b-data.ch/ghc/ghc-musl:9.2.7-linux-arm64v8 is a Docker image built on Linux/AArch64 with MUSL and that provides an 'unofficial' GHC 9.2.7 that works in that environment. If @hasufell is cautious about using 'unofficial' GHC's, then I would be too.

Will the current (non-static) Linux/AArch64 version of Stack 2.11.1 run in that environment? I need a working (recent) version of Stack in order to build Stack.

If yes, then can I simply use 'non-static' Stack to build a 'static' Stack in that environment by passing -static and -pthread to the linker used by GHC?

[hasufell]

If yes, then can I simply use 'non-static' Stack to build a 'static' Stack in that environment by passing -static and -pthread to the linker used by GHC?

--flag=stack:static does that already:

https://github.com/haskell/stack/blob/c3356348c293552e19d964155241a3f7d633bcaa/stack.cabal#L559

[benz0li]

If I understand correctly, gitlab.b-data.ch/ghc/ghc-musl:9.2.7-linux-arm64v8 is a Docker image built on Linux/AArch64 with MUSL and that provides an 'unofficial' GHC 9.2.7 that works in that environment.

Yes, the multi-arch (linux/amd64, linux/arm64/v8) glcr.b-data.ch/ghc/ghc-musl:9.2.7 (Alpine-based, musl libc) image provides an unofficial GHC 9.2.7 (built using Hadrian, from source, without docs).

If @hasufell is cautious about using 'unofficial' GHC's, then I would be too.

🆗

Will the current (non-static) Linux/AArch64 version of Stack 2.11.1 run in that environment?

No, it does not.

On a machine (Linux/AArch64) with docker-ce installed:

$ docker run --rm -ti glcr.b-data.ch/ghc/ghc-musl:9.2.7 bash

Inside the container:

$ curl -sSL https://github.com/commercialhaskell/stack/releases/download/v2.11.1/stack-2.11.1-linux-aarch64-bin -o /usr/bin/stack
$ chmod +x /usr/bin/stack

$ stack --version
bash: /usr/bin/stack: No such file or directory

$ ldd /usr/bin/stack 
	/lib/ld-linux-aarch64.so.1 (0xffffa0122000)
	libpthread.so.0 => /lib/ld-linux-aarch64.so.1 (0xffffa0122000)
	libz.so.1 => /lib/libz.so.1 (0xffffa00fb000)
	librt.so.1 => /lib/ld-linux-aarch64.so.1 (0xffffa0122000)
	libutil.so.1 => /lib/ld-linux-aarch64.so.1 (0xffffa0122000)
	libdl.so.2 => /lib/ld-linux-aarch64.so.1 (0xffffa0122000)
	libgmp.so.10 => /usr/lib/libgmp.so.10 (0xffffa0085000)
	libc.so.6 => /lib/ld-linux-aarch64.so.1 (0xffffa0122000)
	libm.so.6 => /lib/ld-linux-aarch64.so.1 (0xffffa0122000)
Error relocating /usr/bin/stack: fcntl64: symbol not found

I need a working (recent) version of Stack in order to build Stack.

Stack may also be built using Cabal: See #6141
ℹ️ I did not pass -static and -pthread to GHC, though.

[benz0li]

I have updated the instructions at #6141 (comment) to build a statically linked and stripped Stack executable for both Linux/x86_64 and Linux/AArch64 using Cabal and the multi-arch (linux/amd64, linux/arm64/v8) glcr.b-data.ch/ghc/ghc-musl:9.2.7 docker image.

Statically linked because of cabal configure --enable-executable-static --ghc-option=-optl=-pthread or file cabal.project.local containing

ignore-project: False
executable-static: True

program-options
  ghc-options: -optl=-pthread

ℹ️ This only works with cabal build. See issue haskell/cabal#7236 for more information.

Stripped because of strip $(find dist-newstyle -name stack -type f)

[hasufell]

https://gitlab.haskell.org/ghc/ghc/-/issues/23482

[bjin]

I created a repo to build aarch64 statically-linked binary with CircleCI and aarch64 alpine docker: https://github.com/bjin/stack-aarch64-static

[benz0li]

@hasufell I created a multi-arch¹ docker image² containing an unsupported, statically linked binary at /usr/local/bin/stack (v2.11.1).

This binary is now part of the following multi-arch¹ docker images providing unofficial GHC builds (Alpine-based, musl libc; build flavour: perf+llvm+split_sections):

• glcr.b-data.ch/ghc/ghc-musl:9.6.2
• glcr.b-data.ch/ghc/ghc-musl:9.4.6
• glcr.b-data.ch/ghc/ghc-musl:9.2.8

Stack builds fine with e.g. glcr.b-data.ch/ghc/ghc-musl:9.6.2 using stack install stack --flag=stack:static --no-install-ghc --system-ghc --stack-yaml stack-ghc-9.6.2.yaml.
👉 This project could replace fpco/alpine-haskell-stack:9.2.7 with one of those images to build a statically linked binary for both Linux/x86_64 and Linux/AArch64.

Footnotes

1. linux/amd64, linux/arm64/v8 ↩ ↩²

2. glcr.b-data.ch/commercialhaskell/ssi:2.11.1 ↩

[benz0li]

The unsupported binary in the glcr.b-data.ch/ghc/ghc-musl images will be replaced with the official binary release, as soon as it is available statically linked for both Linux/x86_64 and Linux/AArch64.