What decides between musl and glibc targets for builds in GitHub releases?

[EliahKagan]

The Linux x86-64 build we release targets musl rather than glibc:

gitoxide/.github/workflows/release.yml

Lines 89 to 91 in 29898e3

	- target: x86_64-unknown-linux-musl
	os: ubuntu-latest
	rust: stable

But the Linux armhf build targets glibc:

gitoxide/.github/workflows/release.yml

Lines 92 to 94 in 29898e3

	- target: arm-unknown-linux-gnueabihf
	os: ubuntu-latest
	rust: nightly

This is even though an arm-unknown-linux-musleabihf target does exist.

This raises several closely interrelated questions. I hope it is okay that I am asking them together:

• Is the musleabihf target less well supported, and not used for that reason?
• Should the musleabihf target be used instead of the gnueabihf target, if it is working?
• If it makes sense to continue offering arm-unknown-linux-gnueabihf , should a x86_64-unknown-linux-musl build be released as well?
• With that current glibc target, and future possible ones if any, are there issues of specifying the minimum glibc version, as in #1392?
• Does the current situation reflect an intentional asymmetry that is valued and that I should therefore be aware of when deciding what targets to propose builds for?

[Byron]

Thanks for asking! I think once again this is mostly copied from ripgrep and I assumed it's related to availability at the time.

Generally, I now think that if MUSL can be built, it should be built and provided as artefact.

• Is the musleabihf target less well supported, and not used for that reason?

I don't know.

• Should the musleabihf target be used instead of the gnueabihf target, if it is working?

Admittedly, I don't know the difference. If there is a practical one, maybe both can be provided instead.

• If it makes sense to continue offering arm-unknown-linux-gnueabihf , should a x86_64-unknown-linux-musl build be released as well?

Most definitely, in theory there are no limits to the kind of binaries we provide. Maybe there is a practical subset also based on what cargo binstall would pick on various platforms.

• With that current glibc target, and future possible ones if any, are there issues of specifying the minimum glibc version, as in #1392?

I don't know.

• Does the current situation reflect an intentional asymmetry that is valued and that I should therefore be aware of when deciding what targets to propose builds for?

Not as far as I know. The choice can be made based on any valuesystem.

[NobodyXu]

cargo-binstall fallback to musl if gnu one is not found, or the system does not use glibc

So we actually prefer maintainers to have statically linked musl, since that would work everywhere.

[EliahKagan]

@Byron:

Regarding musleabihf rather than gnueabihf:

Admittedly, I don't know the difference. If there is a practical one, maybe both can be provided instead.

I think the difference is analogous to -musl vs. -gnu for targets that differ that way. The -musl builds use musl rather than glibc and should work on more systems, but may currently not be able to both build and work properly when OpenSSL is used (and I had thought LibreSSL could be a solution, and it may still be, but right now it runs in to the problem described in #1493).

Sorry about the confusion in my wording here:

If it makes sense to continue offering arm-unknown-linux-gnueabihf , should a x86_64-unknown-linux-musl build be released as well?

We already have a x86_64-unknown-linux-musl build, of course. I meant to ask if we should have a x86_64-unknown-linux-gnu build.

Most definitely, in theory there are no limits to the kind of binaries we provide.

If we are going to provide both -gnu (or -gnueabihf for 32-bit ARM) and -musl (or -musleabif for 32-bit ARM) builds, then I think the -gnu (and -gnueabihf) one should cover all features currently built for non-Linux systems, including max, unless some further reason arises that it is not feasible to do so. As far as I know, the limitation to max-pure is solely due to musl-related limitations.

I vaguely recall that there had been a previous explicit decision in gitoxide to offer only -musl and not -gnu for Linux targets. I may well be misremembering (or have misunderstood). I'll see what i can find about that.

@NobodyXu:

So we actually prefer maintainers to have statically linked musl, since that would work everywhere.

I will try to add musl targets for the current platform that doesn't have it (armhf) as well as for any other platforms I add for which such a target exists (whether or not also adding a gnu target for the platform).

Is it a problem to provide more feature builds for a gnu target than for the corresponding musl target, e.g., to have only the max-pure build for -musl but max-pure and several others (including max) for -gnu? Per above, that is how I plan to do things at least initially. But if this will cause problems than I can rethink it.

[NobodyXu]

Is it a problem to provide more feature builds for a gnu target than for the corresponding musl target, e.g., to have only the max-pure build for -musl but max-pure and several others (including max) for -gnu? Per above, that is how I plan to do things at least initially. But if this will cause problems than I can rethink it.

If it's only performance-wise differences, then it might be ok.

If it's difference in functionality, then it's going to be confusing for users.

[Byron]

We already have a x86_64-unknown-linux-musl build, of course. I meant to ask if we should have a x86_64-unknown-linux-gnu build.

I think it would probably be beneficial to have more choice especially if one is as easy to build as the other. MUSL is a bit special, but even if only max-pure is offered, that should be fine.

It seems worthwhile to also make clear the reasoning behind what's on offer, and what is not, and why, or the goals of releases (provide as much variety as feasible and convenient), to drive future changes to the release process and/or the available build artefacts.

I vaguely recall that there had been a previous explicit decision in gitoxide to offer only -musl and not -gnu for Linux targets. I may well be misremembering (or have misunderstood). I'll see what i can find about that.

From what I know, without recalling any prior decision, is that I made available what could readily be built.

If it's difference in functionality, then it's going to be confusing for users.

Currently, max-pure doesn't support most of the curl-related Git configuration, mostly because nobody needed it. So it's definitely not the same - performance wise there are also differences and it should be noticeably slower due to slower ZLIB and SHA1 implementations.

[EliahKagan]

It seems worthwhile to also make clear the reasoning behind what's on offer, and what is not, and why, or the goals of releases (provide as much variety as feasible and convenient), to drive future changes to the release process and/or the available build artefacts.

Instructions for installing with cargo binstall appear in the top-level readme. So should this explanation for which targets we try to make releases for go in that readme? Or as comments in the release workflow? Or elsewhere?

In addition to the reasons you mention, I think having more binary releases will be helpful for #1238, which I think will either use the gix binary or a special binary meant specifically for it. (If the latter, the gix binary, and thus the gitoxide crate that includes it, should be built for the same platforms, or users will be tempted to use the special-purpose binary even when they really should use gix.) This is because:

• Actions can run on self-hosted ("private") runners which may be of any architecture, not limited to the architectures of GitHub-hosted runners, and that is sometimes the reason one uses a self-hosted runner.
• Even using GitHub-hosted ("public") runners, if I understand the semantics of the job-level container key, a step that uses an action will run in in a container when the container key is used. That container can run any Linux-based operating system targeting any architecture the runner natively supports. If QEMU/binfmt_misc can be used together with the container key, which I am not sure about, then it could be any architecture at all.

If #1238 were itself a container action, then it would use whatever build is installed in its own container, and these considerations would not apply. But it cannot be implemented as a container action, for the reasons given in #1492 (reply in thread).

If it's difference in functionality, then it's going to be confusing for users.

Currently, max-pure doesn't support most of the curl-related Git configuration, mostly because nobody needed it. So it's definitely not the same - performance wise there are also differences and it should be noticeably slower due to slower ZLIB and SHA1 implementations.

Given this, I am unsure what to do, because I have not figured out a way to make those features work on -musl builds at this time, yet it would be useful to have builds that provide them.

For now, even if not for long, I think I'll keep the limitation of all Linux builds to max-pure intact while adding more of them. If problems are going to arise from building more architectures, then I hope to discover those problems sooner rather than later. In addition, even with only max-pure, I think supporting more architectures is good, at least for architectures in significant use.

[Byron]

In general, I wouldn't want to 'break new grounds' in the amount of architectures that gitoxide can be built for on CI, as this isn't really the purpose of this project. Ideally, we can use off-the-shelf and well-known methods of building Rust programs on CI, and if some aren't easy or aren't working, so be it.

OpenSSL is a particular problem, and this probably tells us to try to make max-pure a build that rivals max in functionality and performance. zlibng-rs could solve one of the performance problems (eventually), I think, even though I don't know how portable it is.

[EliahKagan]

Something I had missed is that when cargo binstall obtains GitHub releases linked from this project's release page (as has been supported since #1239), it is only ever obtaining max-pure releases, even for platforms for which other features including max are available alongside them:

gitoxide/Cargo.toml

Lines 317 to 319 in ea373e6

	[package.metadata.binstall]
	pkg-url = "{ repo }/releases/download/v{ version }/gitoxide-max-pure-v{ version }-{ target }{ archive-suffix }"
	bin-dir = "gitoxide-max-pure-v{ version }-{ target }/{ bin }{ binary-ext }"

In view of this, it seems to me that it might be fine to add archives for the three other features, for -gnu targets--the confusion would be no greater than already exists for macOS and Windows. The benefit of doing so is less than I had hoped, since cargo binstall won't currently install them. But the benefit is sustained in the situations where they may be most needed: fast builds downloaded in a deliberate way in an environment where it's inconvenient to do the build oneself.

So if there are no objections, I may include both more architectures (within reason) and more -gnu targets with feature builds not limited to max-pure, together in the same PR.

OpenSSL is a particular problem, and this probably tells us to try to make max-pure a build that rivals max in functionality and performance. zlibng-rs could solve one of the performance problems (eventually), I think, even though I don't know how portable it is.

This sounds like a good idea, and like one of the requests in #1242:

• provide a new type of builds which uses zlib-ng for max perf and linked with it statically, uses rustls to avoid external openssl dep.

Though I think that is suggesting that it would be separate from max-pure, rather than max-pure becoming it. I don't know which is better.

[Byron]

So if there are no objections, I may include both more architectures (within reason) and more -gnu targets with feature builds not limited to max-pure, together in the same PR.

Definitely no objections from me :).

BTW, I consider you the expert in 'releases' now, and believe you know best about platforms, compatibilities, and which kind of builds to use where. All I know is that max-pure is most compatible, while max is fastest, while small just uses a single-thread and minimizes binary size. Maybe this makes it the least useful as well and is nothing more than to prove how 'flexible' gitoxide wants to be.