Materialization with Nix Flakes?

[bitmappergit]

I can't figure out how to set up materialization when using a flake based project.
Is there any documentation on this or is it unsupported?

[cumber]

The key problem seems to be that a flake's source is copied to the store. So instructions like this:

To materialize project.plan-nix for haskell-language-server entirely, pass a writable path as the materialized argument and run the 'updateMaterialized' script in 'passthru'.

Don't work. The path you pass as the materialzed argument ends up being a path in the store, which of course is not writeable.

But if you get a project's plan-nix attribute, it contains a passthru attribute, which contains derivations for a few scripts. calculateMaterializedSha is one, the updateMaterialized that doesn't work is another, but there's also generateMaterialized.

updateMaterialized has the value you set in the materizlied field hardcoded as a path it will try to write to (but it's been converted to a store path that can't be written); generateMaterialized does not, it takes as an argument a path (which is directory it will create if necessary, and write files inside). So you can use that instead.

This isn't documented well (I couldn't even find anything in the docs really telling me what you're supposed to do to "run a script in passthru"; eventually I figured out it meant the passthru attribute of the nix project structures generated by haskell.nix). I'm not sure if this is really what I'm supposed to be doing with a flake-based project, I just figured it out by trial and error.

I have a gcroot "package" to my flake that just tries to store references to everything the development environment needs to run without downloading. I do nix build '.#gcroot' -o gcroot, so that the nix garbage collector won't delete the development environment until I delete the symlink. Since I'm doing that anyway (and the gcroot package is already just a big link farm to things), I just included the calculateMaterializedSha and generateMaterialized scripts for my project and tools into the gcroot linkfarm. That way I've got them handy to run.

[michaelpj]

It's not documented because nobody has been using the flakes support :)

The business about copying the source into the store seems very annoying. I don't know what we do about that, it is very useful to have a script that just does everything including updating the files on disk.

I have a gcroot "package" to my flake that just tries to store references to everything the development environment needs to run without downloading.

You may want to add <output of project>.roots to that!

[shinzui]

@cumber Do you mind sharing a snippet for your flake.nix?

[cumber]

@shinzui: Sure! It's still a bit of a mess, but this is a skeleton I've copied around a few projects. Very little of it needs to change between projects, which is nice.

{
  description = "<DESCRIPTION-HERE>";

  inputs = {
    haskellNix.url = "github:input-output-hk/haskell.nix";
    nixpkgs.follows = "haskellNix/nixpkgs-unstable";

    flake-utils.url = "github:numtide/flake-utils";
  };


  outputs = { self, nixpkgs, haskellNix, flake-utils }:
    flake-utils.lib.eachSystem [ "x86_64-linux" ] (system:
      let projectName = "<PROJECT-NAME-HERE>";
          compiler-nix-name = "ghc8104";
          index-state = "2021-08-05T00:00:00Z";

          mkProject = haskell-nix: haskell-nix.cabalProject' {
            src = ./.;
            inherit index-state compiler-nix-name;

            plan-sha256 = "...";
            materialized = ./materializations + "/${projectName}";
          };

          overlays = [
            haskellNix.overlay
            (self: super: { ${projectName} = mkProject self.haskell-nix; })
          ];

          pkgs = import nixpkgs { inherit system overlays; };
          project = pkgs.${projectName};
          flake = pkgs.${projectName}.flake {};

          tools = {
            haskell-language-server = {
              inherit index-state;
              plan-sha256 = "...";
              materialized = ./materializations/haskell-language-server;
            };

            hoogle = {
              inherit index-state;
              plan-sha256 = "...";
              materialized = ./materializations/hoogle;
            };
          };

          devShell = project.shellFor {
            packages = ps: [ ps.${projectName} ];
            exactDeps = true;
            inherit tools;
          };

      in flake // {
        inherit overlays devShell;
        nixpkgs = pkgs;

        packages = flake.packages // {
          gcroot = pkgs.linkFarmFromDrvs "${projectName}-shell-gcroot" [
            devShell
            devShell.stdenv
            project.plan-nix
            project.roots

            (
              let compose = f: g: x: f (g x);
                  flakePaths = compose pkgs.lib.attrValues (
                    pkgs.lib.mapAttrs
                      (name: flake: { name = name; path = flake.outPath; })
                  );
              in  pkgs.linkFarm "input-flakes" (flakePaths self.inputs)
            )

            (
              let getMaterializers = ( name: project:
                    pkgs.linkFarmFromDrvs "${name}" [
                      project.plan-nix.passthru.calculateMaterializedSha
                      project.plan-nix.passthru.generateMaterialized
                    ]
                  );
              in
                pkgs.linkFarmFromDrvs "materializers" (
                  pkgs.lib.mapAttrsToList getMaterializers (
                      { ${projectName} = project; }
                      // (pkgs.lib.mapAttrs (_: builtins.getAttr "project") (project.tools tools))
                  )
                )
            )
          ];
        };
      }
    );
}

At the moment when I need to update the materialized stuff, I just comment out the plan-sha256 = and materialized = lines, build the gcroot, and then run:

for f in shell.gcroot/materializers/*; do echo "$(basename $f) - $($f/calculateSha)"; $f/generateMaterialized materializations/$(basename $f); done

That prints out all the hashes I need, and updates the materializations, so I can then update and uncomment the lines in flake.nix. I'm hoping to arrange this better so the updates aren't as manual (separate nix files the flake imports to get the properties, so they can easily be removed then added back in and updated by script?).

[cumber]

It's not documented because nobody has been using the flakes support :)

I'm trying! :) I hope I'll be able to contribute some stuff at some point, if I figure out how to make things work better. I'm still really learning haskell.nix infrastructure itself at the moment, though.

The business about copying the source into the store seems very annoying. I don't know what we do about that, it is very useful to have a script that just does everything including updating the files on disk.

It would be great, yes, but I think it's incompatible with the way flakes are supposed to be completely reproducible. I'm not completely certain it can't work, but I haven't found a way.

One pattern I've been using (in other flakes) is putting scripts in the apps. You can then nix run '.#utility-script' args etc. Using relative paths in the bash script rather than having them as nix paths (which end up absolute) sidesteps the problem.

I have a gcroot "package" to my flake that just tries to store references to everything the development environment needs to run without downloading.

You may want to add <output of project>.roots to that!

I found that one! It's very useful.

[michaelpj]

Using relative paths in the bash script rather than having them as nix paths (which end up absolute) sidesteps the problem.

Right, the problem is working out what the relative path should be at the time when we make the script. Whereas we know the absolute path where the materialized stuff should be... when it's not in the store.

[cumber]

As in, you don't know the relative paths because the user may not be running the script from the root directory of the project? Yeah, that's a pain. As a flake user I'd still prefer "run this command from the root of your project to update materialization" to the message that gets printed now though. ;)

You might be able to find the relative path to a flake's root by assuming the user is in some subfolder of the project and searching upward till you find a flake.nix (much like the way git commands work). Still not guaranteed, but would cover a lot of the pain points. It wouldn't be appropriate to put flake-specific logic into the general haskell.nix stuff of course, but potentially in something that gets called from the example flake scaffolding in the docs, and wouldn't bother non-flake users?

Or maybe you could rely on an environment variable set in a shell hook (since people are almost always entering the dev shell from the right directory)?

I'm just throwing out ideas here. If I had a good fleshed out one I'd whip up a pull request.

(Technically the absolute paths at the time the script was made aren't guaranteed either; the user may have moved the project folder since they obtained the script. I'm not sure exactly what the envisioned workflow is with the non-flake setup, but that's a real danger with my hacky gcroot process; I have been known to run stuff in the gcroot without rebuilding it first after changes.)

[shinzui]

@cumber Thanks for sharing. I finally got it to work with your help. I still don't understand what's going on, so I'll spend some time understanding haskell.nix next.

[miuirussia]

As I understood, materialization is just copying the plan-nix result to the materialization folder, so I implemented it like this:

https://github.com/miuirussia/hls-nix/blob/c10318d40fe6c3cc7160388bb892d4bf77d9693f/flake.nix#L100

Maybe it helps someone)

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.