opam monorepo support

[RyanGibb]

This PR adds support for opam-monorepo.

To support building mirage unikernels for cross-compilation targets we need to build them with dune since mirage 4 (in the absence of ocaml/opam cross compilation support).

Dune supports cross compilation with 'build contexts' using a toolchain which will use a target compiler installed in <opam-switch>/default/<toolchain> (as opposed to the host compiler). But not all dependencies are build with the target compiler. Notably ppx requires building with the host compiler. This information in stored in dune files -- e.g. here. In order to statically know how to build dependencies, we would have to parse the dune files to try and find out (which is non-trivial). Potentially a better solution is to encode the dependency build contexts in opam files, but opam doesn't currently support cross-compilation.

Instead, opam-monorep is used to build mirage unikernels. This fetches all the dependencies (filtered with ?monorepo in the opam file) to a local directory duniverse. Dune then builds the whole project. Unfortunately we can't benefit from the dune cache in our nix derivation, so all the dependencies will be rebuilt on every nix build.

This is implemented in the opam-monorepo project. This PR adds support for this workflow in the for of the queryToScopeMonorepo function, along with the lower layer functions mkSrcsScope, deduplicateSrcs, and defsToSrcs. We refactor the source fetching logic into evaluator/default.nix to share it between mkSrc (a derivation fetching a package's source) and builder.

To see an example of this usage see https://github.com/RyanGibb/mirage-hello. A flake template is the works.

[balsoft]

Is having "" as the fallback name for the case when there's no dev-repo the same as what opam-monorepo does?

[RyanGibb]

opam-monorepo filters out 'virtual' packages (without a dev-repo or url) here: https://github.com/tarides/opam-monorepo/blob/9262e7f71d749520b7e046fbd90a4732a43866e9/lib/lockfile.ml#L104

I'm still thinking about the best way of doing this with opam-nix.

This isn't the only filtering opam-monorepo does: it also filters packages that aren't build with dune (which is the reason for opam-overlays). But AFAIK we can't do this with opam admin list. Maybe modifying the repo to remove any packages not built with dune is the best way to go.

I'm not sure if we need to filter packages built with dune if out of this holds true:

Package versions from earlier repositories take precedence
over package versions from later repositories.

But if this could take a higher version from a later repository (if the higher version doesn't have an overlay yet) it would cause trouble.

[balsoft]

if the higher version doesn't have an overlay yet

Hm, yes, good point. The filtering can probably be done by doing listRepo on the opam-overlays and removing those packages from opam-repository entirely, before passing the final repo to opam admin list. This can be quite slow though.

[RyanGibb]

I've implemented this here, but unfortunately it doesn't consider the case when the newer version does build with dune but an older version (which is overlayed) doesn't.

For example, bheap.1.0.0 doesn't build with dune and therefore has an overlay, but bheapversion.2.0.0 does build with dune so doesn't have an overlay.

When building mirage-hello using opamRepositoryModuloOverlay we get the error

       > [ERROR] No solution including optional dependencies for hello.dev:   * Missing dependency:
       >             - bheap >= 2.0.0
       >             no matching version

I think this is why opam-monorepo filters (if it depends on dune) by package version rather than package name.

[RyanGibb]

One option I can see is parsing every package in the repo and filtering out only those that don't depend on dune. But I imagine this would be quite expensive (there are ~24k packages in opam-repository[0]) even if it would only have to be done once.

Another option would be to use opam monorepo lock to resolve dependency versions.

[0]

opam-repository $ find packages -mindepth 2 -maxdepth 2 | wc -l
24767

[balsoft]

Overall a very neat implementation, seems to be minimal and mostly agree with upstream. Amazing job! The comments are mostly nitpicking, feel free to ignore them if you see fit. Please squash to one (or maybe a couple) commits before I merge this.

[RyanGibb]

Thank you! The comments are very useful :-)

I'm actually working on a buildOpamMonorepo to pick up pin-depends (for example in mirage-www) so will convert this to a draft this and a few other tweaks are complete.

[balsoft]

Why is this needed here? If callers of this function want to remove some packages, it's really easy for them to just call remoteAttrs themselves.

[RyanGibb]

My thinking was this allows packages to be removed based on their name (rather than their dev repo name). For example, the unikernel package we're building has a `dummy' dev-repo which we need to remove.

[RyanGibb]

This is quite useful when we want to exclude dependencies from the monorepo that are in the same source as the unikernel (e.g. https://github.com/mirage/mirage-www) in buildOpamMonorepo with:

filterPkgs = ... (attrNames (latestVersions // query)) ...;

[balsoft]

Suggested change

	! elem true (map (overlayPkgPath: hasInfix overlayPkgPath path) overlayPkgPaths);
	! any (overlayPkgPath: hasInfix overlayPkgPath path) overlayPkgPaths;

[balsoft]

Also, hasInfix might be a bit flaky here, and quite slow.

On a more general note, maybe there's a way to just make a custom joinRepos that would overwrite entire package's instead of just package versions. That would remove the slowdown, but will probably require modifying the symlinkJoin from nixpkgs.

[balsoft]

That just made me realise that joinRepos is not quite correct ATM, since it doesn't actually do the version overriding properly. I think I'll fix it and implement this at the same time.

[RyanGibb]

That sounds like a good idea, but I should warn you I think it will suffer from the same issues I described in the other open comment.

Essentially there's two possibilities when there's a package in opam-overlay with a newer in opam-repository without an overlay. One, an overlay hasn't been written yet. Two, the new version now supports dune and an overlay isn't required. In the former we want to remove the new version from opam-repository, in the latter we don't.

I don't think there's a way to differentiate these cases without parsing the new version's opam file to check for a dune dependency which suggests (although I guess doesn't guarantee, but it seems to work fine for opam-monorepo) dune support.

I'm ironing out the bugs on a function to do this now. It's not fast, but I think it'll work. I'll be offline for the next couple of hours but will push what I have now.

[RyanGibb]

Okay it seems to be working! (Although it's still building... I might lose network connection soon so pushing preemptively)

[balsoft]

Ahh, ok, sorry for not reading thoroughly. I still made the joinRepos a bit better (and quite a lot faster actually) by patching lndir (which comes from... Xorg of all places?)

#20

[balsoft]

Why not just use importOpam? (maybe it's a good idea to rewrite it and other functions to use your pkgNameVer thing, though)

[RyanGibb]

Purely aesthetics. It's hard to judge progress when building ~24k derivations with the same name 😄 I can add this to importOpam if you think it's nice!

[balsoft]

No, there's not. IFD is the only way.

[balsoft]

(it might be a bit faster to do all the opam2json-ning in one derivation and then read the combined output, though).

[RyanGibb]

Yeah, I was thinking the same thing. I'll give it a shot.

[RyanGibb]

If this works, I'm wondering if materialization might be a way of speeding up deployments...

[RyanGibb]

An aside: it would be nice if we could use just the version solving functionality of opam without pulling the whole program in. I might look into the feasibility of a refactor of opam to support this.

[RyanGibb]

Okay, I've hacked something together that builds opamRepositoryFiltered in one derivation:

  # remove non-dune dependant packages
  opamRepositoryFiltered =
    let
      keepPackages = [ "ocaml" "dune" "ocaml-solo5" "ocaml-src" "solo5"
      "ocaml-config" "ocaml-system" "ocaml-variants" "base-bytes" ];
    in bootstrapPackages.stdenv.mkDerivation {
      name = "opamRepositoryFiltered";
      src = opamRepository;
      buildInputs = with bootstrapPackages; [ jq ];
      installPhase = ''
        # look at package version directories
        # (e.g. /nix/store/<hash>-opam-repository-modulo-overlay/packages/<name>/<name>.<version>)
        for f in $(find packages -mindepth 2 -maxdepth 2); do
          echo $f
          # special cases as we will filter them before creating monorepo
          # just needed so `opam admin list` doesn't complain
          # TODO parse queries opam files to a list of dependancies with 
          # build=false so only pick up monorepo deps and we remove this workarounds
          # TODO or just use opam monorepo solver...
          if $(echo "$f" | awk 'BEGIN {
              split("${concatStringsSep " " keepPackages}", parts);
              for (i in parts) dict[parts[i]]=""
            } { exit !($1 in dict) }'); then
            # do nothing
            echo
          # if package depends on dune
          # TODO properly recursively parse vals of depends
          elif $(${opam2json}/bin/opam2json "$f/opam" |\
            jq '[ .depends | flatten | .[] | if type=="string" then . else .val end ] |
              if index("dune") == null then halt_error(1) else halt end'); then
              mkdir -p "$out/$f"
              cp -r "$f" "$out/$f"
          fi
        done
      '';
    };

But it's also very slow (waiting on exact timings now). I think using opam monorepo's solver would be a better solution.

[RyanGibb]

Okay, this creates a very IO bound process that takes ~half an hour to complete 😄 I'm going to look at using opam monorepo lock and parsing the lockfile to solver versions.

[balsoft]

That sounds reasonable.

[RyanGibb]

So trying use opam monorepo lock has more problems than I anticipated. First and foremost I can't get it to use local repositories.

However I don't think we actually need this. It's nice to have, but a work around for:

The case when the newer version does build with dune but an older version (which is overlayed) doesn't.

Is just to downgrade opam-repository.

This isn't ideal, but I'm happy to merge this as-is if you are and create a separate PR for this customer solver logic. Alternatively, I can carry on using the fork and let you know when I have more success.

[RyanGibb]

An aside: another use case I found for opam monorepo is picking up or dependencies. A workaround for this is just manually adding them to the query.

[balsoft]

This isn't ideal, but I'm happy to merge this as-is if you are and create a separate PR for this customer solver logic

Yep, I think this is a good-enough solution as it is right now. opam monorepo lock can be added in a later PR.

[RyanGibb]

Great :-)

[balsoft]

Could you clean up the commit history then?

[RyanGibb]

Done!