sequoia fails at checkPhase

[worldofpeace]

Describe the bug
See https://nix-cache.s3.amazonaws.com/log/cdi6ynvq2qh9mnd2inzfij7vswhvyhgw-sequoia-0.10.0.drv

The particular cases that fail are

failures:
    test::add_then_add_different_key
    test::add_then_import_wrong_key
    test::binding_iterator
    test::delete_binding_then_use
    test::delete_binding_twice
    test::delete_store_then_use
    test::delete_store_twice
    test::import_key
    test::key_iterator
    test::key_not_found
    test::stats
    test::store_iterator
    test::store_network_policy_mismatch

test result: FAILED. 0 passed; 13 failed; 0 ignored; 0 measured; 0 filtered out

I believe rust was updated recently (was that in the release?).

It doesn't appear upstream has made a release recently, so perhaps if this report is forwarded to them they can direct us to which patches could fix the test failures etc.

cc @minijackson @doronbehar

[doronbehar]

According to this and this bug reports, this seems like a ulimit issue. An interesting workaround for hydra could be using RUST_TEST_THREADS=1, as suggested in this comment but it's certainly isn't necessary on my machine because I've tested the build and it passed it.

[worldofpeace]

RUST_TEST_THREADS=1 sounds like it would make an already long checkPhase longer 🤣
How about we try ulimit -n 8192?

[doronbehar]

We can try this `ulimit` indeed. The question is, what are the default limits of a build user on hydra? Are we sure this might make a difference?

[doronbehar]

@lheckemann I noticed you've marked sequoia as broken in 5aa4b19#diff-931adf691a3139ba4385037860e7e086 . Even if a package fails to build on hydra butt doesn't fail locally, does it mean we mark it as broken in its meta? Doesn't sound right to me..

[worldofpeace]

@lheckemann I noticed you've marked sequoia as broken in 5aa4b19#diff-931adf691a3139ba4385037860e7e086 . Even if a package fails to build on hydra butt doesn't fail locally, does it mean we mark it as broken in its meta? Doesn't sound right to me..

@doronbehar That's was the motivation for the issue being opened. If we bump ulimit in preCheck and unmark it as broken I think we should be good. ofborg can serve as testing perhaps similar to hydra.

[doronbehar]

The point is, that we haven't even tried playing with these settings. I would have tried doing it my self but since I know nothing nor I have any effect on Hydra's deployment etc., I wouldn't even know how to test it.

What bothers me is the fact it was marked as broken without any attempts by someone with more influence then me to debug it's failure to build only on hydra.

Are we even sure this wasn't a failure that happened only once?

[minijackson]

Should we close this issue with the #73104 PR?

[jonringer]

No, it's still failing https://hydra.nixos.org/build/107132743, at least for hydra, it runs out of file handles

[erictapen]

A bit off-topic maybe but I generated a Nix expression for sequoia using crate2nix, as build times for sequoia where around hours for me. It works now like a charm, as every crate got its own derivation. It can be found here, as a flake: https://github.com/erictapen/sequoia/blob/nix-flake/flake.nix

Maybe repackaging sequoia with crate2nix could be an option?

[jonringer]

This may be an option, but unpleasant as the generated deps file usually is 1-10k+ of generated noise. (for local building, this works great)

[erictapen]

Yeah, that noise is a bummer. Sometimes I dream of having all of crates.io packaged the way we do it with haskellPackages, but I guess that would blow up Nixpkgs indefinitely. But maybe someday when Nix flakes are ready? …

[doronbehar]

Maybe repackaging sequoia with crate2nix could be an option?

@erictapen I'm not sure I have enough time to have a good view on this. What's most important about sequoia specifically, is the fact we use upstream's Makefile to install many other files that should be part of the package's output - files such as .so libraries, ZSH/Bash completion files and headers. So I'm not sure how will this go together with the crate2nix approach.

[erictapen]

If anyone is interested in a sequoia flake which is built incrementally (vs. everything built in a single derivation), I try to maintain one here: https://github.com/erictapen/sequoia

[doronbehar]

@worldofpeace sequoia doesn't fail anymore, but it hasn't tried 0.16.0 yet - #84464 .

[worldofpeace]

Yep, we need to fix it for python 38 though https://hydra.nixos.org/build/116149675