Node: issue with glide-rs dependency

[wozniakm]

Describe the bug

After installing @valkey/valkey-glide and running npm ls -all I'm seeing following error:

$ npm ls -all
test@1.0.0 /mnt/d/Projekty/test
└─┬ @valkey/valkey-glide@1.2.1
  ├── UNMET OPTIONAL DEPENDENCY @valkey/valkey-glide-darwin-arm64@1.2.1
  ├── UNMET OPTIONAL DEPENDENCY @valkey/valkey-glide-linux-arm64@1.2.1
  ├── UNMET OPTIONAL DEPENDENCY @valkey/valkey-glide-linux-musl-arm64@1.2.1
  ├─┬ @valkey/valkey-glide-linux-musl-x64@1.2.1
  │ ├─┬ glide-rs@0.1.0 invalid: "file:rust-client" from node_modules/@valkey/valkey-glide/node_modules/@valkey/valkey-glide-linux-musl-x64
  │ │ └─┬ save@2.9.0
  │ │   ├── async@3.2.6
  │ │   ├─┬ event-stream@4.0.1
  │ │   │ ├── duplexer@0.1.2
  │ │   │ ├── from@0.1.7
  │ │   │ ├── map-stream@0.0.7
  │ │   │ ├─┬ pause-stream@0.0.11
  │ │   │ │ └── through@2.3.8 deduped
  │ │   │ ├─┬ split@1.0.1
  │ │   │ │ └── through@2.3.8 deduped
  │ │   │ ├─┬ stream-combiner@0.2.2
  │ │   │ │ ├── duplexer@0.1.2 deduped
  │ │   │ │ └── through@2.3.8 deduped
  │ │   │ └── through@2.3.8
  │ │   ├── lodash.assign@4.2.0
  │ │   └── mingo@6.5.2
  │ ├── long@5.3.0
  │ ├─┬ npmignore@0.3.1
  │ │ └── minimist@1.2.8
  │ └─┬ protobufjs@7.4.0
  │   ├── @protobufjs/aspromise@1.1.2
  │   ├── @protobufjs/base64@1.1.2
  │   ├── @protobufjs/codegen@2.0.4
  │   ├── @protobufjs/eventemitter@1.1.0
  │   ├─┬ @protobufjs/fetch@1.1.0
  │   │ ├── @protobufjs/aspromise@1.1.2 deduped
  │   │ └── @protobufjs/inquire@1.1.0 deduped
  │   ├── @protobufjs/float@1.0.2
  │   ├── @protobufjs/inquire@1.1.0
  │   ├── @protobufjs/path@1.1.2
  │   ├── @protobufjs/pool@1.1.0
  │   ├── @protobufjs/utf8@1.1.0
  │   ├─┬ @types/node@22.13.1
  │   │ └── undici-types@6.20.0
  │   └── long@5.3.0 deduped
  ├─┬ @valkey/valkey-glide-linux-x64@1.2.1
  │ ├─┬ glide-rs@0.1.0 invalid: "file:rust-client" from node_modules/@valkey/valkey-glide-linux-x64
  │ │ └── save@2.9.0 deduped
  │ ├── long@5.3.0 deduped
  │ ├── npmignore@0.3.1 deduped
  │ └── protobufjs@7.4.0 deduped
  └── detect-libc@2.0.3

npm error code ELSPROBLEMS
npm error invalid: glide-rs@0.1.0 /mnt/d/Projekty/test/node_modules/@valkey/valkey-glide/node_modules/@valkey/valkey-glide-linux-musl-x64/node_modules/glide-rs
npm error invalid: glide-rs@0.1.0 /mnt/d/Projekty/test/node_modules/@valkey/valkey-glide-linux-x64/node_modules/glide-rs
npm error A complete log of this run can be found in: /home/mateusz/.npm/_logs/2025-02-12T17_39_22_313Z-debug-0.log

I'm having similar error when trying to package Serverless Framework project using esbuild, as one of the steps in the process is calling npm ls:

Error: Command failed with exit code 1: npm ls -json --omit=dev -long -all
npm error code ELSPROBLEMS
npm error invalid: glide-rs@0.1.0 /home/runner/work/unzipped/.esbuild/.build/node_modules/@valkey/valkey-glide-linux-musl-x64/node_modules/glide-rs
npm error invalid: glide-rs@0.1.0 /home/runner/work/unzipped/.esbuild/.build/node_modules/@valkey/valkey-glide-linux-x64/node_modules/glide-rs

I have @valkey/valkey-glide added as external.

Expected Behavior

npm ls -all should not report glide-rs@0.1.0 invalid: "file:rust-client" from node_modules/@valkey/valkey-glide/node_modules/@valkey/valkey-glide-linux-musl-x64.

Current Behavior

It errors out with following error:
glide-rs@0.1.0 invalid: "file:rust-client" from node_modules/@valkey/valkey-glide/node_modules/@valkey/valkey-glide-linux-musl-x64

Reproduction Steps

1. Create package.json file:

{
  "name": "test",
  "version": "1.0.0",
  "main": "index.js",
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  "author": "",
  "license": "ISC",
  "description": "",
  "dependencies": {
    "@valkey/valkey-glide": "^1.2.1"
  }
}

2. Install dependencies with $ npm install
3. Run $ npm ls -all

Possible Solution

Additional Information/Context

When I try to use yarn instead of npm, here is what I see with yarn install:

[1/4] Resolving packages...
error Package "glide-rs" refers to a non-existing file '"/mnt/d/Projekty/test/rust-client"'.
info Visit https://yarnpkg.com/en/docs/cli/install for documentation about this command.
Error: Package "glide-rs" refers to a non-existing file '"/mnt/d/Projekty/test/rust-client"'.
    at MessageError.ExtendableBuiltin (/home/mateusz/.yarn/lib/cli.js:721:66)
    at new MessageError (/home/mateusz/.yarn/lib/cli.js:750:123)
    at FileResolver.<anonymous> (/home/mateusz/.yarn/lib/cli.js:50037:15)
    at Generator.next (<anonymous>)
    at step (/home/mateusz/.yarn/lib/cli.js:310:30)
    at /home/mateusz/.yarn/lib/cli.js:321:13

Client version used

v1.2.1

Engine type and version

OS

Ubuntu

Language

TypeScript

Language Version

Node v20.18.3

Cluster information

No response

Logs

No response

Other information

No response

[avifenesh]

Hi @wozniakm, thanks for opening the issue.
About yarn: we are aware and it is under process #2680 @Muhammad-awawdi-amazon

Just for clear understanding, the issue is not with the package working, but just with the ls command failing?

Found the issue - will get a hot-fix ASAP!
Sorry for the inconvenient!

[wozniakm]

Thank you @avifenesh! It's hard to tell, but if I had to guess I would say it's only issue with ls, but as esbuild is running ls before packaging and it's cancelling the process due to error, I cannot confirm that yet.

[avifenesh]

@wozniakm I might be wrong, the issue I found is not about it, I think.
But - Glide is a native package, you can't bundle it for the web, it has to be configured not to be packed and to run server only.
We are addressing the issue I found, but I'm not sure if this is what you face.

I know that in next.js you need to specifically configure it - see #3089
Do you know if Esbuild has such functionality of not packing for web, and running server only? (server only doesnt mean you can't run on lambda etc, it means it cant be bundled for browser env).

[avifenesh]

Meanwhile, I'm reproducing to see if there's a connection between the issue I found to your issue.

[wozniakm]

It is server only project, we use Serverless Framwork to orchestrate API with API Gateway and AWS Lambda. As I mentioned earlier I marked @valkey/valkey-glide as external dependency, so esbuild should not be trying to bundle them.

I saw #3089 before, but there is nothing relevant for me.

If you want to give me some details on what you have found, I'd be more than happy to take a look on that and check that on my end.

[avifenesh]

I found that we had a fault in the CI which released the RC as stable, but it appears that it is not the issue, since you get 1.2.1 so, there's no issue with getting an RC instead of stable.
I will reproduce and check what I see on my side, and ill update soon.

Do you build and run ls on Ubuntu x86?

[avifenesh]

@wozniakm So current finding. npm install bundled dependencies, but not recursively. If you have it on the top level, it stops there.
Which is fine because we don't need it in the depth, but --all try to find it. I'm looking what are the option to solve.
The code will work fine, it's just an issue with the recursion. Apparently, it's a known issue with npm.

[avifenesh]

@wozniakm Can you please try to set as external external: ['@valkey/*']?
Since the structure is that the glide-platform is external to valkey-glide the base. So setting it as external do set them as external.

You can also share the config of the esbuild so I can play with it myself, I'm not familiar with the tool.

[wozniakm]

I went a bit deeper with esbuild and serverless-esbuild and looks like it uses npm ls to understand packages tree and be able to skip bundling external packages. The error I mentioned before coming from npm ls -all is blocking that process.

Looks like I found the reason of npm complaints.

That's the part of original package.json of valkey-glide-linux-x64:

    "dependencies": {
        "glide-rs": "file:rust-client",
        "long": "^5.2.3",
        "npmignore": "^0.3.1",
        "protobufjs": "^7.4.0"
    },

Issues are gone after changing it to this (note glide-rs version):

    "dependencies": {
        "glide-rs": "v0.1.0",
        "long": "^5.2.3",
        "npmignore": "^0.3.1",
        "protobufjs": "^7.4.0"
    }

It is because in node_modules of this package, there is glide-rs folder with package.json file and following contents:

{
    "name": "glide-rs",
    "version": "0.1.0",
(...)

Not sure if there is easy fix for that, but looking at NPM documentation regarding local paths (https://docs.npmjs.com/cli/v11/configuring-npm/package-json#local-paths) they say that: This feature is helpful for local offline development and creating tests that require npm installing where you don't want to hit an external server, but should not be used when publishing your package to the public registry.

Looks like there is bigger effort needed and fixing that problem should also address yarn problem.