Support for node >15 and npm>7 attempt #3

.github/workflows/test.yml

@@ -4,7 +4,7 @@ on:
   push:
 jobs:
   tests:
-    runs-on: ubuntu-latest
+    runs-on: ubuntu-20.04
     steps:
       - uses: actions/checkout@v3
       - uses: cachix/install-nix-action@v18

API.md

@@ -2,6 +2,44 @@
 
 The sections below describe the public API of _npmlock2nix_.
 
+Npmlock2nix is currently in a transition phase. The npmlockfile format evolved with the npm v7 release so does the npmlock2nix parser. If you're trying to parse a lockfile v2, generated via npm >= 7, you'll have to use the `v2` top-level prefix.
+
+IE. you'll have to call the following functions with these prefixes:
+
+```nix
+# V1 lockfiles
+
+npmlock2nix.v1.build {
+…
+}
+
+npmlock2nix.v1.node_modules {
+…
+}
+
+npmlock2nix.v1.shell {
+…
+}
+
+# V2 lockfiles
+
+npmlock2nix.v2.build {
+…
+}
+
+npmlock2nix.v2.node_modules {
+…
+}
+
+npmlock2nix.v2.shell {
+…
+}
+```
+
+Aside from these `v1` and `v2` prefixes, the underlying public API stays the same.
+
+If you forget to specify a `v1`/`v2` prefix, a warning containing some migration instructions will be emitted and npmlock2nix will fall back on using the lockfiles `v1` implementation.
+
 ## Functions
 
 ### node_modules
@@ -62,24 +100,26 @@ The `build` function takes an attribute set with the following attributes:
 When _npmlock2nix_ is used in restricted evaluation mode (hydra for example), `node_modules` needs to be provided with the revision and sha256 of all GitHub dependencies via `githubSourceHashMap`:
 
 ```nix
-npmlock2nix.node_modules {
+npmlock2nix.v1.node_modules {
   src = ./.;
   githubSourceHashMap = {
     tmcw.leftpad.db1442a0556c2b133627ffebf455a78a1ced64b9 = "1zyy1nxbby4wcl30rc8fsis1c3f7nafavnwd3qi4bg0x00gxjdnh";
   };
 }
 ```
 
-Please refer to [github-dependency](https://github.com/tweag/npmlock2nix/blob/master/tests/examples-projects/github-dependency/default.nix) for a fully working example.
+Please refer to [github-dependency](https://github.com/nix-community/npmlock2nix/blob/master/tests/tests-v2/examples-projects/github-dependency) for a fully working example.
 
 ### preInstallLinks
 
+*NOTE* In the `v2` API the `preInstallLinks` attribute was removed. Use source derivation overrides `sourceOverrides`.
+
 Sometimes you may want to augment or populate vendored dependencies in npm packages because they either aren't working or they cannot be fetched during the build phase. This can be achieved by passing a `preInstallLinks` attribute set to `node_modules`.
 
 If you wanted to patch the [cwebp-bin](https://www.npmjs.com/package/cwebp-bin) package to contain the `cwebp` binary from nixpkgs under `vendor/cwebp-bin` you would do so as follows:
 
 ```nix
-npmlock2nix.node_modules {
+npmlock2nix.v1.node_modules {
   src = ./.;
   preInstallLinks = {
     "cwebp-bin" = {
@@ -108,7 +148,7 @@ The first can be useful if you are also using `npm` to interactively update or m
 When you actually want to describe a shell environment or a build, but you need to pass attributes to `node_modules` you can do so by passing them via `node_modules_attrs` in both `build` and `shell`:
 
 ```nix
-npmlock2nix.build {
+npmlock2nix.v1.build {
   src = ./.;
   node_modules_attrs = {
     buildInputs = [ pkgs.zlib ];
@@ -123,7 +163,7 @@ npmlock2nix.build {
 The `sourceOverrides` argument expects an attribute set mapping npm package names to a function describing the modifications of that package. Each function receives an attribute set as a first argument, containing either a `version` attribute if the version is known, or a `github = { org, repo, rev, ref }` attribute if the package is fetched from GitHub. These values can be used to have different overrides depending on the version. The function receives another argument which is the derivation of the fetched source, which can be modified using `.overrideAttrs`. The fetched source mainly runs the [patch phase](https://nixos.org/manual/nixpkgs/stable/#ssec-patch-phase), so of particular interest are the `patches` and `postPatch` attributes, in which `patchShebangs` can be called. Note that `patchShebangs` can only patch shebangs to binaries accessible in the derivation, which you can extend with `buildInputs`. For convenience, the correct version of `nodejs` is always included in `buildInputs`.
 
 ```nix
-npmlock2nix.node_modules {
+npmlock2nix.v1.node_modules {
   sourceOverrides = {
     # sourceInfo either contains:
     # - A version attribute
@@ -135,12 +175,57 @@ npmlock2nix.node_modules {
         some script
       '';
       # ...
-    })
-
-    # Example
+    });
+	
+	# Example
     node-pre-gyp = sourceInfo: drv: drv.overrideAttrs (old: {
       postPatch = "patchShebangs bin";
     });
+
+  };
+}
+```
+
+*Note* In the `v2` API the following is the case:
+
+Npm packages binaries interpreter are automatically patched with environment shebangs when an entry exist for the npm package.
+
+To replace `preInstallLinks` functionality. Symlinked vendored binaries are compiled to be added on node modules installation using npm preinstall script, this is required since npm strip symlinks from package archive.
+
+The `sourceOverrides` argument expects an attribute set mapping npm package names to a function describing the modifications of that package. Each function receives an attribute set as a first argument, containing either a `version` attribute if the version is known, or a `github = { org, repo, rev, ref }` attribute if the package is fetched from GitHub. These values can be used to have different overrides depending on the version. The function receives another argument which is the derivation of the fetched source, which can be modified using `.overrideAttrs`. The fetched source mainly runs the [patch phase](https://nixos.org/manual/nixpkgs/stable/#ssec-patch-phase), so of particular interest are the `patches` and `postPatch` attributes, in which `patchShebangs` can be called.
+
+`sourceOverrides` comes with default override mechanism like patching all npm packages binaries dependencies with `buildRequirePatchShebangs` or patching individual npm package binaries with `packageRequirePatchShebangs`.
+
+```nix
+npmlock2nix.v2.node_modules {
+  sourceOverrides = {
+    # sourceInfo either contains:
+    # - A version attribute
+    # - A github = { org, repo, rev, ref } attribute for GitHub sources
+    package-name = sourceInfo: drv: drv.overrideAttrs (old: {
+      buildInputs = [ somePackage ];
+      patches = [ somePatch ];
+      postPatch = ''
+        mkdir -p vendor
+        ln -sf ${vendored_binary} ./vendor/vendored_binary
+      '';
+      # ...
+    })
+
+    # Patch all npm packages
+    buildRequirePatchShebangs = true;
+
+    # Patch individual package
+    node-pre-gyp = npmlock2nix.v2.packageRequirePatchShebangs;
+
+    # Link vendored binaries
+    package-name = sourceInfo: drv: drv.overrideAttrs (old: {
+      postPatch = ''
+        mkdir -p vendor
+        ln -sf ${vendored_binary} ./vendor/vendored_binary
+      '';
+    });
+
   };
 }
 ```

default.nix

@@ -1,18 +1,25 @@
-{ pkgs ? import ./nix { } }:
+{ pkgs ? import ./nix { }, lib ? pkgs.lib }:
 let
-  internal = pkgs.callPackage ./internal.nix { };
+  v1_internal = pkgs.callPackage ./internal-v1.nix { };
+  v2_internal = pkgs.callPackage ./internal-v2.nix { };
+  separatePublicAndInternalAPI = api: extraAttributes: {
+    inherit (api) shell build node_modules;
+
+    # *** WARNING ****
+    # using any of the functions exposed by `internal` is not supported. That
+    # being said, hiding them would only lead to copy&paste and it is also useful
+    # for testing internal building blocks.
+    internal = lib.warn "[npmlock2nix] You are using the unsupported internal API." (
+      api
+    );
+  } // (lib.listToAttrs (map (name: lib.nameValuePair name api.${name}) extraAttributes));
+  v1 = separatePublicAndInternalAPI v1_internal [ ];
+  v2 = separatePublicAndInternalAPI v2_internal [ "packageRequirePatchShebangs" ];
 in
 {
-  inherit (internal) shell build node_modules;
-
+  inherit v1;
+  v2 = lib.mapAttrs (_: lib.warn "[npmlock2nix] You are using the new v2 beta api. The interface isn't stable yet. Please report any issues at https://github.com/nix-community/npmlock2nix/issues") v2;
   tests = pkgs.callPackage ./tests { };
-
-
-  # *** WARNING ****
-  # using any of the functions exposed by `internal` is not supported. That
-  # being said, hiding them would only lead to copy&paste and it is also useful
-  # for testing internal building blocks.
-  internal = builtins.trace "[npmlock2nix] You are using the unsupported internal API." (
-    internal
-  );
-}
+} // (lib.mapAttrs
+  (lib.warn "[npmlock2nix] You are using the unversion prefix for builders. This is fine for now. In the future we will move to a versioned interface (old versions remain as they are). The currently used functions are availabe as `npmlock2nix.v1` for example `npmlock2nix.v1.build`.")
+  v1)

internal.nix → internal-v1.nix

@@ -334,7 +334,6 @@ rec {
           rm -f node_modules
         fi
       ''}
-
       if test -e node_modules; then
         echo '[npmlock2nix] There is already a `node_modules` directory. Not replacing it.' >&2
         exit 1
@@ -433,9 +432,7 @@ rec {
             in
             ''
               #! ${stdenv.shell}
-
               ${preInstallLinkCommands}
-
               if grep -I -q -r '/bin/' .; then
                 source $TMP/preinstall-env
                 patchShebangs .
@@ -494,14 +491,12 @@ rec {
                 # https://docs.npmjs.com/cli/v8/configuring-npm/package-lock-json#packages
                 # https://developer.mozilla.org/en-US/docs/Web/Security/Subresource_Integrity#tools_for_generating_sri_hashes
                 hash="sha512-$(openssl dgst -sha512 -binary ${lib.escapeShellArg file} | openssl base64 -A)"
-
                 # Constructs a simple { path, value } JSON of the given arguments
                 jq -c --argjson path ${lib.escapeShellArg (builtins.toJSON path)} --arg value "$hash" -n '$ARGS.named'
               '') patchedLockfile'.integrityUpdates}
             } | jq -s --slurpfile original ${patchedLockfile'.result} -f "$jqSetIntegrityPath" > package-lock.json
             set +x
           ''}
-
           ln -sf ${patchedPackagefilePath} package.json
         '';
 
@@ -518,7 +513,6 @@ rec {
         '';
         installPhase = ''
           mkdir "$out"
-
           if test -d node_modules; then
             if [ $(ls -1 node_modules | wc -l) -gt 0 ] || [ -e node_modules/.bin ]; then
               mv node_modules $out/