Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Build static binaries on Linux using musl #2110

Merged
merged 5 commits into from
Nov 4, 2021
Merged

Build static binaries on Linux using musl #2110

merged 5 commits into from
Nov 4, 2021

Conversation

aykevl
Copy link
Member

@aykevl aykevl commented Sep 14, 2021

This PR adds support for musl-libc and uses it by default on Linux. The main benefit of it is that binaries are always statically linked instead of depending on the host libc, even when using CGo.

Advantages:

  • The resulting binaries are always statically linked.
  • No need for any tools on the host OS, like a compiler, linker, or libc in a release build of TinyGo.
  • This also simplifies cross compilation as no cross compiler is needed (it's all built into the TinyGo release build).

Disadvantages:

  • Binary size increases by 5-6 kilobytes if -no-debug is used. Binary size increases by a much larger margin when debugging symbols are included (the default behavior) because musl is built with debugging symbols enabled.
  • Musl does things a bit differently than glibc, and some CGo code might rely on the glibc behavior.
  • The first build takes a bit longer because musl needs to be built.

CGo doesn't work properly yet. Not quite sure how to best fix that without a somewhat big change. However, it's possible to work around that by first building a non-CGo binary and then building a CGo binary (the issue is that the libc is built after processing CGo headers).

@aykevl aykevl force-pushed the musl branch 2 times, most recently from e6a4a4c to f9b1733 Compare September 21, 2021 14:50
@aykevl aykevl force-pushed the musl branch 2 times, most recently from 89856f8 to cebb135 Compare October 3, 2021 17:37
@aykevl aykevl marked this pull request as ready for review October 3, 2021 17:37
@aykevl aykevl force-pushed the musl branch 3 times, most recently from d9fad20 to 33033d8 Compare October 3, 2021 22:10
@aykevl
Copy link
Member Author

aykevl commented Oct 4, 2021

Note to self: this is failing on Windows and MacOS because the files src/exit/_Exit.c and src/unistd/_exit.c conflict with each other on a case-insensitive file system (only the base name is used, hence the conflict).

@aykevl aykevl force-pushed the musl branch 2 times, most recently from d55dacc to 0dddf24 Compare October 5, 2021 15:46
@aykevl
Copy link
Member Author

aykevl commented Oct 7, 2021

This is now ready for review.

@aykevl
Copy link
Member Author

aykevl commented Oct 7, 2021

From Windows (git bash shell):

$ GOOS=linux GOARCH=arm build/tinygo build -o test.elf ./testdata/cgo
$ scp test.elf ayke@raspberrypi:
test.elf                                                                                                                                  100%  147KB 781.8KB/s   00:00
$ ssh ayke@raspberrypi ./test.elf
fortytwo: 42
add: 8
myint: 3 5
myint size: 2
[...etc]

In other words, it's possible to cross-compile Linux binaries from Windows without installing any extra tools, even including CGo support!

@deadprogram
Copy link
Member

Merge conflicts need resolution, please.

@deadprogram
Copy link
Member

This PR has some failing tests now, @aykevl

@aykevl
Copy link
Member Author

aykevl commented Oct 28, 2021

Yeah, there was a non-code conflict with #2135. Should be fixed now.

@deadprogram
Copy link
Member

This seems good to me, but not really having used musl much, I am no authority. Anyone else like to comment on this PR before we merge it?

@deadprogram
Copy link
Member

Oops, another merge conflict @aykevl

This brings a bit more consistency to libc configuration. It seems
better to me to set the header flags all in the same place, instead of
some in Go code and some in JSON target files (depending on the target).
This is really just a preparatory commit for musl support. The idea is
to store not just the archive file (.a) but also an include directory.
This is optional for picolibc but required for musl, so the main purpose
of this commit is the refactor needed for this change.
This is very useful for debugging.
MacOS X 10.14 has a soft limit of 256 open files by default, at least on
CircleCI. So don't keep object files open while writing the ar file to
reduce the number of open files at once.

Context: the musl libc has more than 256 object files in the .a file.
This resulted in the error "too many open files" on MacOS X 10.14 when
running in CircleCI.
This commit adds support for musl-libc and uses it by default on Linux.
The main benefit of it is that binaries are always statically linked
instead of depending on the host libc, even when using CGo.

Advantages:
  - The resulting binaries are always statically linked.
  - No need for any tools on the host OS, like a compiler, linker, or
    libc in a release build of TinyGo.
  - This also simplifies cross compilation as no cross compiler is
    needed (it's all built into the TinyGo release build).

Disadvantages:
  - Binary size increases by 5-6 kilobytes if -no-debug is used. Binary
    size increases by a much larger margin when debugging symbols are
    included (the default behavior) because musl is built with debugging
    symbols enabled.
  - Musl does things a bit differently than glibc, and some CGo code
    might rely on the glibc behavior.
  - The first build takes a bit longer because musl needs to be built.

As an additional bonus, time is now obtained from the system in a way
that fixes the Y2038 problem because musl has been a bit more agressive
in switching to 64-bit time_t.
@aykevl
Copy link
Member Author

aykevl commented Nov 4, 2021

I've rebased the PR again, thanks!

Now I can also see exactly how much space musl takes up:

$ tinygo build -o test.elf -size=full ./testdata/alias.go
   code  rodata    data     bss |   flash     ram | package
------------------------------- | --------------- | -------
     33    2696     108    1106 |    2837    1214 | (unknown)
   4518       8     240     588 |    4766     828 | C musl
    333      24       0       8 |     357       8 | internal/task
     58       6       0       0 |      64       0 | main
   1895     126       8      50 |    2029      58 | runtime
------------------------------- | --------------- | -------
   6837    2860     356    1752 |   10053    2108 | total

(I've already found a quick way to avoid ~1kB in musl, but that's for a separate PR. I'm pretty sure more can be avoided with some optimizations).

@deadprogram
Copy link
Member

For sure a major accomplishment for cross-compiling. Now merging, thanks @aykevl

@deadprogram deadprogram merged commit 403d935 into dev Nov 4, 2021
@deadprogram deadprogram deleted the musl branch November 4, 2021 16:15
@QuLogic
Copy link
Contributor

QuLogic commented Dec 21, 2021

The first build takes a bit longer because musl needs to be built.

Can we pre-build it, like build-library for compiler-rt and picolibc, or does that not make sense?

@aykevl
Copy link
Member Author

aykevl commented Dec 28, 2021

Can we pre-build it, like build-library for compiler-rt and picolibc, or does that not make sense?

Definitely. It's simply not implemented.
Note that the pre-built binaries are currently not working: they build for a configuration that doesn't exist anymore. See #2220. That's a bug that should be fixed.

@QuLogic
Copy link
Contributor

QuLogic commented Jan 11, 2022

Ah, perhaps that explains why even though I built the libraries into $TINYGOROOT, I still see the errors coming from ~/.cache/tinygo/....

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants