lisp-modules-new: bring over changes from recent nix-cl

[Uthar]

Based on Uthar/nix-cl@60dbe8f

Description of changes

• Applied recurseIntoAttrs to sbclPackages thanks @nagy
• Marked slashy systems as broken thanks @nagy
• Fixed stack overflow when eval'ing sbclPackages thanks @lukego
• Updated imported.nix to Quicklisp dist of 2022-07-08

Added new packages:

• cl-liballegro-nuklear
• lessp
• rollback
• facts

Things done

• Built on platform(s)
  • x86_64-linux
  • aarch64-linux
  • x86_64-darwin
  • aarch64-darwin
• For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
• Tested, as applicable:
  • NixOS test(s) (look inside nixos/tests)
  • and/or package tests
  • or, for functions and "core" functionality, tests in lib/tests or pkgs/test
  • made sure NixOS tests are linked to the relevant packages
• Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
• Tested basic functionality of all binary files (usually in ./result/bin/)
• 22.11 Release Notes (or backporting 22.05 Release notes)
  • (Package updates) Added a release notes entry if the change is major or breaking
  • (Module updates) Added a release notes entry if the change is significant
  • (Module addition) Added a release notes entry if adding a new NixOS module
  • (Release notes changes) Ran nixos/doc/manual/md-to-db.sh to update generated release notes
• Fits CONTRIBUTING.md.

[vcunat]

FYI, at least hundreds of these new builds fail to build on Hydra. EDIT: something like 700 if I look at the three platforms present in the trunk jobset.

[nagy]

https://hydra.nixos.org/eval/1782855?filter=lispPackages_new.&compare=1782832&full=#tabs-still-fail

Here is a list of them. But I dont think it is that bad. Looking through some of them, I see a lot require native libraries, which means they will need to get some overrides.

[vcunat]

The top two "problematic dependencies" here: https://malob.github.io/nix-review-tools-reports/nixpkgs:trunk/nixpkgs_trunk_1782855.html
might be lower-hanging fruit to fix something like a hundred per platform.

[Uthar]

I'll work on the qt-libs stuff. It requires packaging smokegen, smokeqt and commonqt.

[lukego]

I'll try to help improve the situation during the week too.

On Friday I deployed a Hydra and setup a proof-of-concept jobset for testing Lisp packages. Maybe this can help us to improve things independent of pushes to NixOS upstream? I'm hopeful but I haven't really thought through the details yet or discussed it with anyone.

Link: https://hydra.nuddy.co/

[Uthar]

Maybe let's work on the lisp branch here: https://github.com/nuddyco/nixpkgs/
Then it will be auto-discovered by Hydra, correct?

[lukego]

Maybe let's work on the lisp branch here: https://github.com/nuddyco/nixpkgs/

I added the branch you used for this PR to that Hydra so that it is built too by a separate job. See link at
https://hydra.nuddy.co/project/nixpkgs-lisp. I am happy to add more branches as needed, just let me know.

Hydra will poll that branch every 5 minutes for a new build. Currently this setup has one 16-core Zen 3 CPU for builds.

[lukego]

@Uthar I can also have Hydra build a different Nix expression if you like e.g. you could fork the lispnix package and I could point it there. Just now all it does is build lispPackages_new.sbclPackages but it could be anything.

Hydra has support for "declarative jobsets" where all this configuration can be defined in a Git repo and updated there. I just haven't quite gotten around to remembering how all that works so I'm manually setting up the jobs right now.

[Uthar]

Great stuff! I'll be pushing to the PR branch

I'd like to figure out hydra myself some day

[lukego]

I'm moving a discussion from Uthar/nix-cl#9 here because I think it has more appropriate reach:

I'd like to define overlays on the Lisp packages and I suspect that we need to introduce one more layer of indirection to make this work. Specifically I think that we need to define each Lisp package as a function that returns a derivation, like this random python package, so that the callPackage design pattern can be used to regenerate each package with different dependencies.

The issue now is that the only parameter we can pass into the Lisp package collection is which Lisp compiler to use, and then it uses that to generate derivations with build-asdf-system, and once those derivations are defined then it is hard to update them in non-trivial ways (like the overlays use cases.)

So would it possibly make sense to copy the python packages design and generate a callPackage-compatible interface for each Lisp package so that we can do overlays? Or (very likely) do I misunderstand the whole situation?