Nix flake #6650

nothingmuch · 2023-09-07T14:30:30Z

per @chrisguida's request, cc @jurraca, opening a draft PR for a new flake.nix

disabling doCheck = true is enough to build from source using nix build ".?submodules=1", and in nix develop the python interpreter is set up with the poetry dependencies, but that's as far as I got, this does not yet work.

TODO:

defines a default package: - pname is now "cln" because why not - version uses flake input metadata, doesn't match git describe format building requires submodules: nix build ".?submodules=1" the devshell does not, project can be built from checkout with submodules in this devshell.

nothingmuch · 2023-09-08T07:06:43Z

flake.nix

+  description = "A very basic flake";
+
+  inputs = {
+    nixpkgs.url = "github:NixOS/nixpkgs/23.05";


To use the unstable branch, FYI this currently causes an infinite cascade of derivations due to a breaking change in the nixpkgs python stuff that poetry2nix uses, you would need to pin nixpkgs to just before the merging PR mentioned in this thread:

https://discourse.nixos.org/t/psa-poetry2nix-is-currently-broken-with-nixpkgs-unstable/32281

we probably shouldn't use unstable anyway since it would reevaluate packages every time, not a great UX. Pinning it to the latest stable release seems ok.
We could also figure out which packages we want bleeding edge for and cherry pick those from nixpkgs-unstable, like nix-bitcoin does.

jurraca · 2023-09-08T14:49:39Z

thanks for kicking this off!
I think you may be overcomplicating it by reusing the nixpkgs clightning package. Besides, the point of a flake is to be (mostly) self-contained to the repo. I would just reuse the nixpkgs expression in a local default.nix file or something.
Just thinking out loud about goals here:

you can nix build for major systems (x86-64-linux, aarch64-linux, x86_64-darwin, aarch64-darwin)
you can nix develop and have a set up dev environment
you can run tests across the CLN test suite
should handle setting up a local network in regtest, like startup-regtest.sh does.

chrisguida · 2023-09-11T17:22:38Z

Does anyone know of a good repo to look at for an example of what to do here? @jurraca @nothingmuch

The goal is to be able to type nix develop and have a dev env ready to run make.

Even something without python would work, I'm just trying to get my bearings right now.

Will add startup_regtest.sh once I have that working.

Then I'll add the ability to run this from downstream repos, ie plugins that also need the startup_regtest.sh environment.

nothingmuch · 2023-09-16T10:21:30Z

thanks for kicking this off! I think you may be overcomplicating it by reusing the nixpkgs clightning package.

yep, that's a hack, only intended (i should have clarified this) to delay making some decisions until later but should not be merged as such since it spreads out knowledge about what is appropriate to do to build the project in a very inconvenient way, introducing a dep on nixpkgs at the wrong level of abstraction (edit: that is, in addition to the appropriate dep on nixpkgs, namely relying on it for (native) build inputs)

you can nix build for major systems (x86-64-linux, aarch64-linux, x86_64-darwin, aarch64-darwin)

you can nix develop and have a set up dev environment

you can run tests across the CLN test suite

should handle setting up a local network in regtest, like startup-regtest.sh does.

i would only add to this that it's not just CLN proper but plugin development too that should be facilitated, regardless of whether the plugin lives in this repo or outside of it

nothingmuch · 2023-09-16T10:25:56Z

Does anyone know of a good repo to look at for an example of what to do here? @jurraca @nothingmuch

not off the top of my head

The goal is to be able to type nix develop and have a dev env ready to run make.

that's what i was going for in the first commit but it's somewhat incomplete

Even something without python would work, I'm just trying to get my bearings right now.

python is important IMO because of make check

cdecker · 2024-02-16T16:54:01Z

FWIW I was able to side-step the requirement for poetry2nix altogether by adding a shell hook that'd just invoke poetry install --with=dev --no-root in it. While not as nicely integrated into nix I think it could get us unstuck trying to get the first iteration working, and we can later start looking for better integrated solutions, or remove/replace dependencies that are hard to nixify.

What do you think @nothingmuch ?

chrisguida · 2024-02-16T17:37:24Z

@cdecker do you have a link to your flake handy? would love to take a look!

cdecker · 2024-02-16T17:53:52Z

I'm afraid I have it committed in some random branch that I am having trouble finding again 😊 But I will share as soon as I find it.

JosephGoulden · 2024-08-21T11:07:25Z

I just tried this out. Flake loads once I update it to 24.05, but then I get a stack overflow when building. I can't really help with the C/Make/python stuff but I can write crane based rust derivations.

chrisguida · 2024-08-21T19:17:29Z

@JosephGoulden can you post the error? Would love to resurrect this and get it working... I can jump on a call to collaborate!

JosephGoulden · 2024-08-21T20:53:25Z

Lots of logs about instantiating python stuff then a stack overflow.

instantiated 'python3-3.11.9' -> '/nix/store/123855506lv9kabjla4andyqy4an0sih-python3-3.11.9.drv'
performing daemon worker op: 7
instantiated 'wrap-python-hook' -> '/nix/store/18scsq0ah0xdlcp1yq2kdp3y4kn93m27-wrap-python-hook.drv'
performing daemon worker op: 7
instantiated 'python-remove-tests-dir-hook' -> '/nix/store/xp7kphjxcy8p87jjsklr1vyj1k9kqd2l-python-remove-tests-dir-hook.drv'
performing daemon worker op: 7
instantiated 'python-remove-bin-bytecode-hook' -> '/nix/store/8qn2wd6wqgyak7s84rcw8xsg0jw0mz3x-python-remove-bin-bytecode-hook.drv'
evaluating derivation 'git+file:///lightning#packages.x86_64-linux.default'error: stack overflow (possible infinite recursion)

chrisguida · 2024-08-22T17:32:50Z

Hmm I'm not even getting past nix develop on this branch now...

$ nix develop
error:
       … while calling the 'derivationStrict' builtin

         at /builtin/derivation.nix:9:12: (source not available)

       … while evaluating derivation 'cln'
         whose name attribute is located at /nix/store/vhq11h949l5zycaw07acphv53ifq4p2c-source/pkgs/stdenv/generic/make-derivation.nix:303:7

       … while evaluating attribute 'makeFlags' of derivation 'cln'

         at /nix/store/z8ap2lygq7h2hibjfwm5vl610glx4idk-source/flake.nix:37:11:

           36|           version = "${self.lastModifiedDate}-flake-${self.dirtyShortRev}"; # TODO emulate git describe using CHANGELOG.md or fetch git? impure runCommand?
           37|           makeFlags = ["VERSION=${version}" "MTIME=${self.lastModifiedDate}"];
             |           ^
           38|           nativeBuildInputs = [ cln-meta-project ] ++ upstream.nativeBuildInputs; # FIXME why does python3 in callPackage not work? work around by placing cln-meta-project's python first

       error: attribute 'dirtyShortRev' missing

       at /nix/store/z8ap2lygq7h2hibjfwm5vl610glx4idk-source/flake.nix:36:55:

           35|           name = "cln";
           36|           version = "${self.lastModifiedDate}-flake-${self.dirtyShortRev}"; # TODO emulate git describe using CHANGELOG.md or fetch git? impure runCommand?
             |                                                       ^
           37|           makeFlags = ["VERSION=${version}" "MTIME=${self.lastModifiedDate}"];

Did you modify anything to get the dev shell loading?

JosephGoulden · 2024-08-22T20:16:59Z

Yeah I got the same error at first. I updated inputs.nixpkgs.url to 24.05.

Probably worth trying a rebase after that given the branch is nearly a year old.

chrisguida · 2024-08-26T22:30:10Z

Hmm @JosephGoulden I tried your change, but i get this:

Took like 40GB of RAM before erroring out here xD

Wonder what that's about

chrisguida · 2024-08-26T22:34:34Z

Tried just getting rid of the flake.lock file and i get this:

$ nix develop
warning: Git tree '/home/cguida/work/lightning/elementsproject' is dirty
warning: creating lock file '/home/cguida/work/lightning/elementsproject/flake.lock'
warning: Git tree '/home/cguida/work/lightning/elementsproject' is dirty
error:
       … while calling the 'derivationStrict' builtin

         at /builtin/derivation.nix:9:12: (source not available)

       … while evaluating derivation 'cln'
         whose name attribute is located at /nix/store/p9sy5nf9jdwj69gmr3c3n03npzr5kkqi-source/pkgs/stdenv/generic/make-derivation.nix:336:7

       … while evaluating attribute 'nativeBuildInputs' of derivation 'cln'

         at /nix/store/p9sy5nf9jdwj69gmr3c3n03npzr5kkqi-source/pkgs/stdenv/generic/make-derivation.nix:380:7:

          379|       depsBuildBuild              = elemAt (elemAt dependencies 0) 0;
          380|       nativeBuildInputs           = elemAt (elemAt dependencies 0) 1;
             |       ^
          381|       depsBuildTarget             = elemAt (elemAt dependencies 0) 2;

       error: attribute 'legacyPackages' missing

       at /nix/store/3vqkvn1fbdmn3q3phk4h0z52z6xxcbs4-source/flake.nix:24:10:

           23|       inherit
           24|         (poetry2nix.legacyPackages.${system})
             |          ^
           25|         defaultPoetryOverrides

Just kinda shooting in the dark here

JosephGoulden · 2024-08-27T10:49:24Z

If you change (poetry2nix.legacyPackages.${system}) to (poetry2nix.lib.mkPoetry2Nix { inherit pkgs; }) then you can get a little further.

I did a rebase as well then get stuck at No module named 'hatchling'.

chrisguida · 2024-08-29T21:12:49Z

Not sure how you're getting that.

I'm getting this:

error:
       … while calling the 'derivationStrict' builtin

         at /builtin/derivation.nix:9:12: (source not available)

       … while evaluating derivation 'cln'
         whose name attribute is located at /nix/store/fj33459cnk884y8lrvc7ihxgcgsixxpj-source/pkgs/stdenv/generic/make-derivation.nix:336:7

       … while evaluating attribute 'nativeBuildInputs' of derivation 'cln'

         at /nix/store/fj33459cnk884y8lrvc7ihxgcgsixxpj-source/pkgs/stdenv/generic/make-derivation.nix:380:7:

          379|       depsBuildBuild              = elemAt (elemAt dependencies 0) 0;
          380|       nativeBuildInputs           = elemAt (elemAt dependencies 0) 1;
             |       ^
          381|       depsBuildTarget             = elemAt (elemAt dependencies 0) 2;

       error: attribute 'lib' missing

       at /nix/store/l7xkz1nyvk90pgsvwkxgawa2135z3q3d-source/flake.nix:24:3:

           23|       inherit
           24| 	(poetry2nix.lib.mkPoetry2Nix { inherit pkgs; })
             |   ^
           25|         defaultPoetryOverrides

chrisguida · 2024-08-29T21:16:11Z

Here's my git diff

With this diff I get the "infinite recursion" problem described above

JosephGoulden · 2024-09-10T09:53:43Z

I think we should try a slightly different approach. I created a new PR #7656 which is quite similar to the first commit in this PR.

ShahanaFarooqui · 2024-10-21T16:19:50Z

Closing with PR #7656.

nothingmuch force-pushed the wip-flake branch from 1da05ff to 4299e8e Compare September 7, 2023 14:34

[WIP] poetry2nix based environment

cfed4d4

nothingmuch force-pushed the wip-flake branch from 4299e8e to cfed4d4 Compare September 7, 2023 14:35

chrisguida mentioned this pull request Sep 7, 2023

plugins/clnrest: Adding pyproject.toml for Poetry install #6651

Merged

nothingmuch commented Sep 8, 2023

View reviewed changes

cdecker changed the title ~~Wip flake~~ Nix flake Sep 11, 2023

chrisguida mentioned this pull request Sep 15, 2023

Python path dependency crashing on NixOS #6637

Open

vincenzopalazzo mentioned this pull request Oct 2, 2023

build: add a simple nix dev shell #6739

Closed

JosephGoulden mentioned this pull request Aug 20, 2024

plugins: rust clnrest plugin #7541

Closed

JosephGoulden mentioned this pull request Sep 10, 2024

Add Simple Nix Flake #7656

Merged

ShahanaFarooqui closed this Oct 21, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Nix flake #6650

Nix flake #6650

nothingmuch commented Sep 7, 2023 •

edited

Loading

nothingmuch Sep 8, 2023

jurraca Sep 8, 2023

jurraca commented Sep 8, 2023

chrisguida commented Sep 11, 2023 •

edited

Loading

nothingmuch commented Sep 16, 2023 •

edited

Loading

nothingmuch commented Sep 16, 2023

cdecker commented Feb 16, 2024

chrisguida commented Feb 16, 2024

cdecker commented Feb 16, 2024

JosephGoulden commented Aug 21, 2024

chrisguida commented Aug 21, 2024

JosephGoulden commented Aug 21, 2024

chrisguida commented Aug 22, 2024

JosephGoulden commented Aug 22, 2024

chrisguida commented Aug 26, 2024

chrisguida commented Aug 26, 2024

JosephGoulden commented Aug 27, 2024

chrisguida commented Aug 29, 2024

chrisguida commented Aug 29, 2024

JosephGoulden commented Sep 10, 2024

ShahanaFarooqui commented Oct 21, 2024

Nix flake #6650

Nix flake #6650

Conversation

nothingmuch commented Sep 7, 2023 • edited Loading

nothingmuch Sep 8, 2023

Choose a reason for hiding this comment

jurraca Sep 8, 2023

Choose a reason for hiding this comment

jurraca commented Sep 8, 2023

chrisguida commented Sep 11, 2023 • edited Loading

nothingmuch commented Sep 16, 2023 • edited Loading

nothingmuch commented Sep 16, 2023

cdecker commented Feb 16, 2024

chrisguida commented Feb 16, 2024

cdecker commented Feb 16, 2024

JosephGoulden commented Aug 21, 2024

chrisguida commented Aug 21, 2024

JosephGoulden commented Aug 21, 2024

chrisguida commented Aug 22, 2024

JosephGoulden commented Aug 22, 2024

chrisguida commented Aug 26, 2024

chrisguida commented Aug 26, 2024

JosephGoulden commented Aug 27, 2024

chrisguida commented Aug 29, 2024

chrisguida commented Aug 29, 2024

JosephGoulden commented Sep 10, 2024

ShahanaFarooqui commented Oct 21, 2024

nothingmuch commented Sep 7, 2023 •

edited

Loading

chrisguida commented Sep 11, 2023 •

edited

Loading

nothingmuch commented Sep 16, 2023 •

edited

Loading