WIP: treewide: Always prefix compilers #44583

Ericson2314 · 2018-08-07T00:02:31Z

Motivation for this change

The goal is to bring the native and cross code paths closer together by always prefixing compilers. This makes it easier to catch programs that hard-code gcc or clang.

This also includes:

Two patches to help with darwin -> linux cross compilation; since they're fine patches and the same override-gcc fix is needed.
Making the wrapper env vars define CC, LD, and friends use full paths instead of names, generally good for robustness.

The whole thing I plan to merge after 18.09, but maybe those 2 bits can go in first.

Fixes #21471

Things done

Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS)
Built on platform(s)
- NixOS
- macOS
- other Linux distributions
Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
Tested execution of all binary files (usually in ./result/bin/)
Determined the impact on package closure size (by running nix path-info -S before and after)
Fits CONTRIBUTING.md.

Then we don't run the risk of stomping over non-autotools.

This isn't needed now that we use env vars.

GNU tools are prefixed the same whether wrapped or not, but unwrapped LLVM tools are not prefixed because they are multi-target.

It's confusing when the linker and assembler appear to come from cc-wrapper rather than bintools-wrapper.

The unwrapped bintools are already propagated dependencies, and therefore on the PATH of downstream builds, cc-wrapper can get the assembler on its own.

This isn't a MUSL thing, but just needed for cross compilation to x86. No one had tried this when all cross compilation was to linux + glibc, hence why no one noticed this until recently.

Carefully fake cc-version and cc-fullversion to avoid needing a compiler for the kernel itself to build the headers.

LnL7 · 2018-08-07T00:22:57Z

I only glanced very quickly over the changes and this wasn't immediately clear, the un-prefixed wrappers are still available and work right?

Mic92 · 2018-08-07T00:24:05Z

pkgs/os-specific/linux/paxctl/default.nix

-    sed "s|--owner 0 --group 0||g" -i Makefile
+    sed -i Makefile \
+      -e 's|--owner 0 --group 0||g' \
+      -e '/CC:=gcc/d'


Do we have any use for paxctl/grsecurity at all? We removed the kernel long time ago.

No I don't. But it's part of bootstrapping; that's why I fixed it.

Ericson2314 · 2018-08-07T02:06:19Z

@LnL7 actually wrapped binaries are always prefixed with this PR. But I could make targetPrefix a parameter. inPrefix is added because GCC will be prefixed on the inside but Clang won't be, but yes they can be varied independently in principle.

dezgeg · 2018-08-07T15:22:19Z

NAK for removing unprefixed gcc, clang from native compilation. stdenv stands for "standard environment", and that's what it should stay, standard.

matthewbauer · 2018-08-07T15:26:22Z

pkgs/os-specific/linux/kernel-headers/default.nix

+
+    # Skip clean on darwin, case-sensitivity issues.
+    buildPhase = lib.optionalString (!buildPlatform.isDarwin) ''
+      make mrproper $makeFlags


Can you use buildFlags & checkFlags here instead?

matthewbauer · 2018-08-08T17:19:08Z

Can we have an unprefixed symlink for the host compiler? Even just something like:

ln -s $CC $out/bin/cc
ln -s $CXX $out/bin/c++

Would cover most use cases. Maybe one for gcc as well because that's often hardcoded.

Ekleog · 2018-10-03T13:13:03Z

I completely agree with the gain offered by always prefixing the compilers in stdenv: it is the removal of another source of impurity (the host system), which is what nix is all about.

However, a cursory read (which can easily be wrong) makes me think nix-shell -p gcc would also only yield prefixed-gcc, which would not be acceptable as gcc is then supposed to be used by the end-user, and not by nix. Did I read correctly?

Ericson2314 · 2018-10-03T14:49:44Z

gcc and stdenv.cc are currently the same thing (for Linux). I didn't change that but we could.

I sort of rather fix that after as cc-wrapper has a bunch of other issues for manual usage too.

mmahut · 2019-08-19T11:28:54Z

Are there any updates on this pull request, please?

cdepillabout · 2020-05-05T05:47:44Z

It looks like there hasn't been any updates on this in a year, so I will close it.

(@Ericson2314 sorry in advance if you're still working on any of these PRs that I've closed. Definitely don't hesitate to re-open. I've been trying to clean up old PRs tagged haskell.)

Ericson2314 · 2020-05-05T14:26:24Z

@cdepillabout It's fine. I'll reopen if/when I get back to them.

Ericson2314 and others added 12 commits August 6, 2018 16:04

stdenv: Make configurePlatforms control env vars instead

64235cc

Then we don't run the risk of stomping over non-autotools.

treewide: Get rid of configurePlatforms = []

2b64fd8

This isn't needed now that we use env vars.

gcc, binutils, gdb: cctools: Always specify the target platform.

d01351f

binutils, compilers, wrappers: Define and use passthru.targetPrefix

b49c85c

GNU tools are prefixed the same whether wrapped or not, but unwrapped LLVM tools are not prefixed because they are multi-target.

gcc: Always prefix derivation name

c7f5d46

cc-wrapper: Use libexec to avoid polluting the path

4bf6b07

It's confusing when the linker and assembler appear to come from cc-wrapper rather than bintools-wrapper.

bintools-wrapper: Don't copy in the assembler

2c5981e

The unwrapped bintools are already propagated dependencies, and therefore on the PATH of downstream builds, cc-wrapper can get the assembler on its own.

gcc: Clarify Disabling libmpx in cross stage static

a23f098

This isn't a MUSL thing, but just needed for cross compilation to x86. No one had tried this when all cross compilation was to linux + glibc, hence why no one noticed this until recently.

linux-headers: fix Darwin cross build

b228510

Carefully fake cc-version and cc-fullversion to avoid needing a compiler for the kernel itself to build the headers.

bintools-wrapper: Define env vars with full path

d591863

cc-wrapper: Define env vars with full path

eb75e80

perl: Respect the prefix of the C compiler

20f938e

Ericson2314 added 2.status: work-in-progress This PR isn't done 6.topic: portability General portability concerns, not specific to cross-compilation or a specific platform labels Aug 7, 2018

Ericson2314 requested review from basvandijk, LnL7, Mic92, peti and ryantm as code owners August 7, 2018 00:02

Ericson2314 requested review from matthewbauer and dtzWill August 7, 2018 00:03

GrahamcOfBorg added 6.topic: haskell 6.topic: rust 6.topic: stdenv Standard environment labels Aug 7, 2018

paxctl: Don't hardcode CC:=gcc

6c60a14

Ericson2314 force-pushed the always-prefix branch from 99d349f to 6c60a14 Compare August 7, 2018 00:16

Mic92 reviewed Aug 7, 2018

View reviewed changes

matthewbauer reviewed Aug 7, 2018

View reviewed changes

matthewbauer mentioned this pull request Sep 12, 2019

v8: fix build on aarch64 #68550

Merged

10 tasks

lopsided98 mentioned this pull request Mar 6, 2020

gcc: place cross-compiled target libraries in lib output #81844

Merged

10 tasks

cdepillabout closed this May 5, 2020

ghost mentioned this pull request Apr 20, 2023

simpler, saner cross-compilation #227327

Open

emilazy mentioned this pull request Jul 28, 2024

wrapCC, wrapBintools: use absolute paths #314920

Draft

13 tasks

AndersonTorres removed the request for review from dtzWill September 7, 2024 22:41

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

WIP: treewide: Always prefix compilers #44583

WIP: treewide: Always prefix compilers #44583

Ericson2314 commented Aug 7, 2018

LnL7 commented Aug 7, 2018

Mic92 Aug 7, 2018 •

edited

Loading

Ericson2314 Aug 7, 2018

Ericson2314 commented Aug 7, 2018 •

edited

Loading

dezgeg commented Aug 7, 2018

matthewbauer Aug 7, 2018

matthewbauer commented Aug 8, 2018

Ekleog commented Oct 3, 2018

Ericson2314 commented Oct 3, 2018 •

edited

Loading

mmahut commented Aug 19, 2019

cdepillabout commented May 5, 2020

Ericson2314 commented May 5, 2020

WIP: treewide: Always prefix compilers #44583

WIP: treewide: Always prefix compilers #44583

Conversation

Ericson2314 commented Aug 7, 2018

Motivation for this change

Things done

LnL7 commented Aug 7, 2018

Mic92 Aug 7, 2018 • edited Loading

Choose a reason for hiding this comment

Ericson2314 Aug 7, 2018

Choose a reason for hiding this comment

Ericson2314 commented Aug 7, 2018 • edited Loading

dezgeg commented Aug 7, 2018

matthewbauer Aug 7, 2018

Choose a reason for hiding this comment

matthewbauer commented Aug 8, 2018

Ekleog commented Oct 3, 2018

Ericson2314 commented Oct 3, 2018 • edited Loading

mmahut commented Aug 19, 2019

cdepillabout commented May 5, 2020

Ericson2314 commented May 5, 2020

Mic92 Aug 7, 2018 •

edited

Loading

Ericson2314 commented Aug 7, 2018 •

edited

Loading

Ericson2314 commented Oct 3, 2018 •

edited

Loading