coreutils: fix musl cross compilation

[tobim]

This should not affect glibc systems, but targeting staging to be safe.
I only tested this with gcc set to version 8 as default, so the CFLAGS workaround might not be strictly necessary.

Motivation for this change

See #59934.

Things done

Backported https://git.savannah.gnu.org/cgit/gnulib.git/commit/?id=453ff940449bbbde9ec00f0bbf82a359c5598fc7.

• Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS)
• Built on platform(s)
  • NixOS
  • macOS
  • other Linux distributions
• Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
• Tested compilation of all pkgs that depend on this change using nix-shell -p nix-review --run "nix-review wip"
• Tested execution of all binary files (usually in ./result/bin/)
• Determined the impact on package closure size (by running nix path-info -S before and after)
• Assured whether relevant documentation is up to date
• Fits CONTRIBUTING.md.

---

[matthewbauer]

Thanks for doing this! But I think we would prefer to use fetchpatch directly to https://git.savannah.gnu.org/cgit/gnulib.git/patch/?id=453ff940449bbbde9ec00f0bbf82a359c5598fc7

[tobim]

The patch as is does not apply to the tarball that is distributed on the mirrors. Partly because the sources changed in between, partly because the stuff that happens during ./boostrap in coreutils is not obvious, i.e, not all files in gnulib are copied to coreutils.

[tobim]

Maybe we could only keep the modifications to the patch in nixpkgs and patch the patch during prePatch?

Edit: In case you are concerned with the size.

[matthewbauer]

Ok I guess that's alright. It's just a pretty big patch, but we should be able to remove it when the next coreutils release happens?

[tobim]

It is already merged upstream, so yes. I didn't find any info related to release planning with a quick search, but recent releases have taken between 3 and 9 months.

On another note: Do you have an idea why this triggers mass-rebuilds for the other libcs?

[veprbl]

why this triggers mass-rebuilds for the other libcs?

Because you set NIX_CFLAGS_COMPILE = []; for all non-musl libc's. One possible workaround is to do

stdenv.mkDerivation ({
  pname = "foo";
  /*
  ...
  */
} // optionalAttrs stdenv.hostPlatform.isMusl {
  NIX_CFLAGS_COMPILE = [ "-Wno-error" ];
})

[tobim]

Because you set NIX_CFLAGS_COMPILE = []; for all non-musl libc's.

That makes sense. I was hoping empty fields would be skipped when calculating the output hash, but I guess that would create problems...

I'll change it according to your suggestion.

[tobim]

Changed target to master since it doesn't cause a huge amount of builds any more.
Thanks to @veprbl!

[nh2]

@matthewbauer @tobim This breaks the build for me, because a test fails:

NIX_PATH=nixpkgs=. nix-build --no-link -E 'with (import <nixpkgs> {}); pkgsMusl.coreutils'

    FAIL: tests/misc/date-debug
    ===========================

    --- exp2        2019-05-12 20:56:26.336030282 +0000
    +++ out2        2019-05-12 20:56:26.339030405 +0000
    @@ -4,7 +4,7 @@
     date: using specified time as starting value: '02:30:00'
     date: error: invalid date/time value:
     date:     user provided time: '(Y-M-D) 2006-04-02 02:30:00'
    -date:        normalized time: '(Y-M-D) 2006-04-02 03:30:00'
    +date:        normalized time: '(Y-M-D) 2006-04-02 01:30:00'
     date:                                             --
     date:      possible reasons:
     date:        non-existing due to daylight-saving time;
    FAIL tests/misc/date-debug.sh (exit status: 1)

...

============================================================================
Testsuite summary for GNU coreutils 8.31
============================================================================
# TOTAL: 616
# PASS:  470
# SKIP:  145
# XFAIL: 0
# FAIL:  1
# XPASS: 0
# ERROR: 0

This is on commit 3c37e94 (the merge of this to master).

Did it build for you?

[nh2]

Before this PR, the relevant test wasn't run:

date-debug.sh: skipped test: Timezones database not found

Still kinda suspicious for it to return the wrong date in the test. We should figure out what's going on.

[nh2]

@dtzWill do you know what might be going on?

Also, what are these date related changes for?

https://github.com/NixOS/nixpkgs/blob/0da7b54a/pkgs/tools/misc/coreutils/default.nix#L53-L56

Could you add some comments there to explain them?

[nh2]

Indeed musl seems to be the difference.

I've compiled coretuils master at commit 6e97d361b47c981966f0bfef03659c572409bcba on Ubuntu, once with normal gcc and once with musl-gcc (tags/v1.1.22, had to silence one warning as shown here for that).

Running src/date --debug -d 'TZ="America/Edmonton" 2006-04-02 02:30:00' with each:

• With glibc it prints date: normalized time: '(Y-M-D) 2006-04-02 03:30:00'
• With musl it prints date: normalized time: '(Y-M-D) 2006-04-02 01:30:00'

[nh2]

I've started debugging a bit:

In coreutils's debug_strfdatetime the passed-in struct tm const *tm is different between musl and glibc.

Printing:

  fprintf(stderr, "tm_hour = %d\n", tm->tm_hour);
  fprintf(stderr, "tm_isdst = %d\n", tm->tm_isdst);

glibc:

date: error: invalid date/time value:
n = 100
m = 27
pc = 0x7ffc2239bb70
pc->zones_seen = 0
tm_hour = 2
tm_isdst = -1
date:     user provided time: '(Y-M-D) 2006-04-02 02:30:00'
n = 100
m = 27
pc = 0x7ffc2239bb70
pc->zones_seen = 0
tm_hour = 3
tm_isdst = 1
date:        normalized time: '(Y-M-D) 2006-04-02 03:30:00'

musl:

date: error: invalid date/time value:
n = 100
m = 27
pc = 0x7fff2808a0b0
pc->zones_seen = 0
tm_hour = 2
tm_isdst = -1
date:     user provided time: '(Y-M-D) 2006-04-02 02:30:00'
n = 100
m = 27
pc = 0x7fff2808a0b0
pc->zones_seen = 0
tm_hour = 1
tm_isdst = 0
date:        normalized time: '(Y-M-D) 2006-04-02 01:30:00'

tm_hour = 1 instead of 3, and tm_isdst = -1 insteead of 0.

[tobim]

I ran your command on 59a733e (don't think the commits that came afterwards would change anything):

============================================================================
Testsuite summary for GNU coreutils 8.31
============================================================================
# TOTAL: 332
# PASS:  297
# SKIP:  35
# XFAIL: 0
# FAIL:  0
# XPASS: 0
# ERROR: 0
============================================================================

The test that fails for you does not show up at all for me.

[nh2]

@tobim That's very odd. You also have only TOTAL: 332 many tests, while I have TOTAL: 616.

I get the same error on your commit.

As per your ticks in the issue description, you're building on NixOS with sandboxing, just like me.

However, I have the suspicion that some impurity made it into my build. I've just built it on another NixOS machine, and the issue did not occur. For the builds on my Ubuntu machine, I use nix's --builder command to build on a remote NixOS machine. Maybe some dependency of coreutils was previously built on Ubuntu without sandboxing and results in this difference. I will try and find out whether I can identify the difference between the two.

This does not explain, however, why I see the same test failure between glibc and musl when building coreutils only on Ubuntu, where nixpkgs isn't involved at all.

[nh2]

@tobim @matthewbauer OK, I think I found at least the NixOS-related answer:

The machine I built on actually had sandbox = false. I've reproduced that this creates the issue on a completely new throwaway NixOS machine. Switching to sandbox = true on that machine makes the build succeed (and significantly reduces the number of tests run, to the TOTAL: 332 that you've posted).

This means that when the sandbox is on, the relevant test simply does not run, because Timezones database not found in the sandbox.

But the problem remains that the test shouldn't fail; if times are parsed incorrectly somehow, we probably want to get that fixed.

I've posted the problem on the coreutils mailing list: https://lists.gnu.org/archive/html/coreutils/2019-05/msg00031.html

[nh2]

I've discussed with dalias on #musl freenode IRC and he confirmed that the line being different in this test failure is benign.

He also replied to the mailing list thread, but it isn't in the archive yet so I can't link it yet.

I think we should disable this test for now somehow, so that coreutils builds fine with musl even without sandbox.

[nh2]

He also replied to the mailing list thread, but it isn't in the archive yet so I can't link it yet.

https://lists.gnu.org/archive/html/coreutils/2019-05/msg00039.html

[nh2]

PR to patch the test at #61471