Node2nix 1.11.0 no longer builds the result/bin symlink, nor result/lib/node_modules/.bin symlinks

[CMCDragonkai]

I'm comparing node2nix 1.11.0 to node2nix 1.9.0.

In the 1.9.0 version. The result of a simple JS application with a bin executable looks like this:

./result
├── bin -> lib/node_modules/.bin
└── lib
    └── node_modules
        ├── .bin
        │   └── dosomething -> ../somerandompackage/src/bin/dosomething.js
        └── somerandompackage
            ├── default.nix
            ├── .gitignore
            ├── nix
            │   └── leveldown.patch
            ├── package.json
            ├── .package.json.un~
            ├── package-lock.json
            ├── pkgs.nix
            ├── release.nix
            ├── shell.nix
            ├── src
            │   └── bin
            │       ├── dosomething.js
            │       ├── .dosomething.js.un~
            │       └── .dosomething.ts.un~
            ├── tsconfig.build.json
            ├── tsconfig.json
            ├── utils.nix
            ├── .utils.nix.swp
            └── .utils.nix.un~

8 directories, 18 files

Note the ./result/bin -> ./result/lib/node_modules/.bin symlink AND also the ./result/lib/node_modules/.bin directory containing the executable symlink dosomething.

With the exact same repository, using node2nix 1.11.0 (the master commit of this repository), the output is now:

./result
└── lib
    └── node_modules
        └── somerandompackage
            ├── default.nix
            ├── .gitignore
            ├── nix
            │   └── leveldown.patch
            ├── package.json
            ├── .package.json.un~
            ├── package-lock.json
            ├── pkgs.nix
            ├── release.nix
            ├── shell.nix
            ├── src
            │   └── bin
            │       ├── dosomething.js
            │       ├── .dosomething.js.un~
            │       └── .dosomething.ts.un~
            ├── tsconfig.build.json
            ├── tsconfig.json
            ├── utils.nix
            ├── .utils.nix.swp
            └── .utils.nix.un~

6 directories, 17 files

Notice how both the bin symlink is not there. Nor is the .bin directory under ./result/lib/node_modules.

I believe the bin symlink is only created if the ./result/lib/node_modules/.bin directory gets created.

However it appears the new way of using npm install (and npm has gone from 6.14.13 to 8.5.0), does not appear to create the .bin directory at all.

This means using node2nix on any JS application that has bin executables no longer builds.

[CMCDragonkai]

Because the new version can't produce executables, and I every variation of npm install didn't work for me, I wrote a jq script that and added it to my own default.nix to ultimately make the executable paths:

    # symlink bin executables
    if [ \
      "$(${jq}/bin/jq 'has("bin")' "$out/lib/node_modules/${utils.node2nixDev.packageName}/package.json")" \
      == \
      "true" \
    ]; then
      mkdir -p "$out/bin"
      while IFS= read -r bin_name && IFS= read -r bin_path; do
        # make files executable
        chmod a+x "$out/lib/node_modules/${utils.node2nixDev.packageName}/$bin_path"
        # create the symlink
        ln -s \
          "$out/lib/node_modules/${utils.node2nixDev.packageName}/$bin_path" \
          "$out/bin/$bin_name"
      done < <(
        ${jq}/bin/jq -r 'select(.bin != null) | .bin | to_entries[] | (.key, .value)' \
        "$out/lib/node_modules/${utils.node2nixDev.packageName}/package.json"
      )
    fi

One thing I'm confused about is why npm install is used at all. In my duct-taped solution I just disabled it entirely with dontNpmInstall = true, and everything works fine.

[svanderburg]

Hmm this is a weird situation. I guess something may have changed recently, so that NPM thinks an executable folder has already been created.

I need to analyze the situation to find out the cause or find an alternative solution in which I create this .bin folder myself

[CMCDragonkai]

Yea, my duct-tape solution above allows me to move forward with node2nix 1.11.0.

However if you could shed light on:

One thing I'm confused about is why npm install is used at all. In my duct-taped solution I just disabled it entirely with dontNpmInstall = true, and everything works fine.

Specifically:

node2nix/nix/node-env.nix

Lines 380 to 387 in 8264147

	if [ "''${dontNpmInstall-}" != "1" ]
	then
	# NPM tries to download packages even when they already exist if npm-shrinkwrap is used.
	rm -f npm-shrinkwrap.json
	npm ${forceOfflineFlag} --nodedir=${nodeSources} ${npmFlags} ${lib.optionalString production "--production"} install
	fi
	'';

What exactly is that line expected to do? If the dependencies are already installed via nix, why run npm install again?

I've disabled it entirely and nothing has changed. Are there install hooks that are supposed to run?

[svanderburg]

NPM has two kinds of purposes -- so it does dependency management and build management (e.g. via script directives). It should skip the former aspects (because that conflicts with Nix), but still do the latter aspect.

In most NPM projects, there's no build management, but in some project there are.

I've just noticed the same problem. I guess some of my package-lock.json hacks could have caused this problem in newer versions of NPM.

I can either fix it by finding the conflicting property or manually create bin/ symlinks.

[jtrees]

I just ran into this too. I'm still not that experienced with nix and I'm totally new to node2nix but let me know if I can help somehow.

EDIT: Actually, I may be running into something else. Investigating...

EDIT 2: Sure enough. I'm on node2nix 1.9.0 and npm 8.5.2 and the symlink to lib/node_modules/.bin is missing (tested with the npm package prebuild-install).

[tstelzer]

I've been using node2nix as a way to use global npm executables (mostly language-servers), which seems to be broken now. Reading through the docs, I'm not even sure this is a supported use-case?

[CMCDragonkai]

Just use the solution I created: #293 (comment) for now to work around it.

I've also got a working project in https://github.com/MatrixAI/TypeScript-Demo-Lib-Native demonstrating native addons and https://github.com/MatrixAI/TypeScript-Demo-Lib without native addons. See default.nix.

[andersk]

There was another discussion about this same issue at NixOS/nixpkgs#145432, where I concluded that the problem is a difference between Node 14 and 16, not a difference between node2nix 1.9.0 and 1.11.0.

Some confounding factors that may have made it initially appear otherwise and made it generally more confusing to understand:

• node2nix 1.9.0 silently ignored --nodejs-16 (Reject invalid command-line options #272).
• nixpkgs.nodePackages is defined with .override { nodejs = nodejs-14_x; } even though nixpkgs.nodejs is nodejs-16_x.
• Until recently, nixpkgs.node2nix used the 1.9.0 source even though its version was set to 1.10.0 (nodePackages: use latest node2nix NixOS/nixpkgs#171195).

[CMCDragonkai]

Just a note, make sure to set executable permission flag on any files with shebang #!/usr/bin/env node so that nix rewrites those paths. Encountered that here: MatrixAI/Polykey#379 (comment)

[winterqt]

According to the NPM docs, this folder structure is still in place as of the latest NPM version.

[winterqt]

Hi @svanderburg!

I believe I have figured out this issue, but before I can properly fix it, I need to know your reasoning behind something, if you don't mind: why does buildNodePackage create $out/lib/node_modules (but then include node_modules in $out/lib/node_modules/${packageName}?)

Basically, is this structure required for packages that create $out/bin, or not (ignoring the fact that this is a breaking change)? I ask this since the issue stems from the fact that us running npm install from $out/lib/node_modules/foo and assuming $out/lib/node_modules/.bin/foo will be created is no longer valid with newer versions of npm. (I have no clue if this is intentional on npm's end or not, if you believe it is a bug we should let them know.)

A fix that I've come up with is running npm install $out/lib/node_modules/foo and then symlinking from the created node_modules directory, though this can obviously be made a bit cleaner by steering away from this structure.

[dbaynard]

Thanks @winterqt

I've been toying with a workaround that adds the following after the call to npm rebuild.

    mkdir -p "$out/lib/node_modules/.bin"
    for i in $out/lib/node_modules/pnpm/bin/*; do
      local j="''${i##*/}"
      ln -sv "$i" "$out/lib/node_modules/.bin/''${j%.*}"
    done

This manually creates the folder that you found npm no longer creates.

I implemented this by patching runHook postRebuild into the installPhase (to complement preRebuild); would you consider adding something like that?

[RaitoBezarius]

Hi there, I just encountered this issue, @svanderburg would you be okay with one of the workaround above?

[lilyinstarlight]

@winterqt, digging deeper it seems npm does not install bins anymore for top-level projects unless you are doing a --global install (an undocumented breaking change that I had to dive into the npm code and git blame to find when and where it happened)

Does npm install the top-level project bins for you if you readjust the directory structure?

Could anyone also try and test #302? I believe I've fixed all of the issues (including this one) with Node.js 16+ / NPM v7+ in that PR

[winterqt]

@lilyinstarlight What do you mean by top-level in this case? Is that what the term is for trying to npm install in node_modules/foo and expecting node_modules/.bin? Not sure if I's call that top-level, seems a bit ambiguous.

[lilyinstarlight]

I'm not 100% because there is a lot of npm code, but yeah whatever directory you are running npm install from is what I mean (and what npm seems to call that in the code)