Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug] fs functions that use buffer encoding fail #1818

Closed
1 task
remorses opened this issue Sep 8, 2020 · 15 comments · Fixed by #4943
Closed
1 task

[Bug] fs functions that use buffer encoding fail #1818

remorses opened this issue Sep 8, 2020 · 15 comments · Fixed by #4943
Labels
bug Something isn't working unreproducible This issue cannot be reproduced on master upholded Real issues without formal reproduction

Comments

@remorses
Copy link

remorses commented Sep 8, 2020

  • I'd be willing to implement a fix

Describe the bug

Yarn fs reimplementation assumes fs functions only accept strings as input, instead they could be Buffer or URL for example

The error you get is

TypeError: p.match is not a function
    at Function.resolveVirtual (/Users/morse/Documents/GitHub/sterblue1/.pnp.js:125678:21)
    at VirtualFS.mapToBase (/Users/morse/Documents/GitHub/sterblue1/.pnp.js:125715:22)
    at VirtualFS.readdirPromise (/Users/morse/Documents/GitHub/sterblue1/.pnp.js:125588:44)
    at PosixFS.readdirPromise (/Users/morse/Documents/GitHub/sterblue1/.pnp.js:125588:24)

To Reproduce

const fs = require("fs");

require("./.pnp.js").setup();

const bufdir = Buffer.from(".");
fs.readdir(bufdir, { encoding: "buffer" }, x => console.log(x));
Reproduction
// Sherlock reproduction

Screenshots

If applicable, add screenshots to help explain your problem.

Environment if relevant (please complete the following information):

  • OS: [e.g. OSX, Linux, Windows, ...]
  • Node version [e.g. 8.15.0, 10.15.1, ...]
  • Yarn version [e.g. 2.0.0-rc1, ...]

Additional context

#899 is a similar problem

node-dir npm package (2 million monthly downloads) files function fails because of this

@remorses remorses added the bug Something isn't working label Sep 8, 2020
@yarnbot

This comment has been minimized.

@yarnbot yarnbot added the unreproducible This issue cannot be reproduced on master label Sep 8, 2020
@yarnbot

This comment has been minimized.

@paul-soporan paul-soporan added upholded Real issues without formal reproduction and removed unreproducible This issue cannot be reproduced on master labels Sep 18, 2020
@yarnbot yarnbot added the unreproducible This issue cannot be reproduced on master label Sep 18, 2020
@yarnbot
Copy link
Collaborator

yarnbot commented Sep 18, 2020

We couldn't reproduce your issue (all the assertions passed on master).

@wdfinch
Copy link

wdfinch commented Feb 23, 2022

I'm running into this issue when trying to build a project using rollup with node 17.6.0 and yarn berry v3.2.0. Any updates?

@vicary
Copy link

vicary commented Mar 12, 2022

rant My minimum reproducible repo would include `serverless` and `serverless-esbuild` which is sub-optimal, it can be easily dodged by maintainers so I am not providing it here. And I don't bother learning yarn internals enough to pin point the exact place where it does codegen for `.pnp.cjs`. Nobody else can do this as efficiently as current contributors.

Here is a patch instead.

moar rant

Couldn't resist and finally took a look. Now that I'm in the rabbit hole, it's @yarnpkg/fslib who's responsible of the affected code segment.

I just don't know how tf an unexpected buffer input and an expected string input is simply expressed as PortablePath in the parameter.

Come on, guys.

Workaround

Apply the following patch in your .pnp.cjs,

@@ -32543,10 +32543,11 @@ class VirtualFS extends ProxiedFS {
   mapToBase(p) {
+    const pathString = `${p}`;
     if (pathString === ``)
       return p;
-    if (this.pathUtils.isAbsolute(p))
-      return VirtualFS.resolveVirtual(p);
+    if (this.pathUtils.isAbsolute(pathString))
+      return VirtualFS.resolveVirtual(pathString);
     const resolvedRoot = VirtualFS.resolveVirtual(this.baseFs.resolve(PortablePath.dot));
-    const resolvedP = VirtualFS.resolveVirtual(this.baseFs.resolve(p));
+    const resolvedP = VirtualFS.resolveVirtual(this.baseFs.resolve(pathString));
     return ppath.relative(resolvedRoot, resolvedP) || PortablePath.dot;
   }
   mapFromBase(p) {

I expect the file to be re-generated every time after running yarn add/remove, the exact lines to be patched should change.

This is a very temporary solution and this issue requires actual fix to @yarnpkg/fslib.

PR hints

For those who care enough to go for a PR, you may start here.

mapToBase(p: PortablePath): PortablePath {

export type PortablePath = string & { __pathType: PathType.File | PathType.Portable };
export type NativePath = string & { __pathType?: PathType.File | PathType.Native };

@trollkotze
Copy link

I just encountered this issue when doing rmSync(storagePath, { recursive: true }); where the directory in storagePath has a very extensive subdirectory structure. The same error is not thrown for a different, simpler directory structure. In both cases I provide the path argument as a string, so the error thrown was quite puzzling to me:

TypeError [ERR_INVALID_ARG_TYPE]: The "path" argument must be of type string. Received an instance of Buffer
    at new NodeError (node:internal/errors:372:5)
    at validateString (node:internal/validators:120:11)
    at Object.isAbsolute (node:path:1157:5)
    at VirtualFS.mapToBase (/srv/users/brebweb/apps/brebweb/backup/.pnp.cjs:4367:24)
    at VirtualFS.rmdirSync (/srv/users/brebweb/apps/brebweb/backup/.pnp.cjs:4216:39)
    at PosixFS.rmdirSync (/srv/users/brebweb/apps/brebweb/backup/.pnp.cjs:4216:24)
    at URLFS.rmdirSync (/srv/users/brebweb/apps/brebweb/backup/.pnp.cjs:4216:24)
    at _rmdirSync (node:internal/fs/rimraf:260:21)
    at rimrafSync (node:internal/fs/rimraf:193:7)
    at node:internal/fs/rimraf:253:9 {
  code: 'ERR_INVALID_ARG_TYPE'
}

@ar4hc

This comment was marked as duplicate.

@demurgos
Copy link
Contributor

demurgos commented May 10, 2022

@trollkotze I encountered the same issue as you.
When investigating the stack trace, the buffer is introduced by Node where recursing over child items:

https://github.com/nodejs/node/blob/01408a5aa8d4ed4b486bb8e3d9607f3e342c78b4/lib/internal/fs/rimraf.js#L251

Yarn version: 4.0.0-rc.4
Stack trace:

[error] TypeError [ERR_INVALID_ARG_TYPE]: The "path" argument must be of type string. Received an instance of Buffer
    at new NodeError (node:internal/errors:377:5)
    at validateString (node:internal/validators:119:11)
    at Object.isAbsolute (node:path:1157:5)
    at VirtualFS.mapToBase (/data/projects/eternalfest/eternalfest.net/.pnp.cjs:23673:24)
    at VirtualFS.rmdirSync (/data/projects/eternalfest/eternalfest.net/.pnp.cjs:23522:39)
    at PosixFS.rmdirSync (/data/projects/eternalfest/eternalfest.net/.pnp.cjs:23522:24)
    at URLFS.rmdirSync (/data/projects/eternalfest/eternalfest.net/.pnp.cjs:23522:24)
    at _rmdirSync (node:internal/fs/rimraf:260:21)
    at rimrafSync (node:internal/fs/rimraf:193:7)
    at node:internal/fs/rimraf:253:9

@osg74
Copy link

osg74 commented May 29, 2022

Still reproducible with the latest yarn 4.0.0-rc.6 from sources and node 18.2.0:

➜  web git:(main) ✗ yarn --version
4.0.0-rc.6.git.20220525.hash-d1961ee
➜  web git:(main) ✗ node
Welcome to Node.js v18.2.0.
Type ".help" for more information.
> const fs = require('fs');
> require('./.pnp.cjs').setup();
> const bufdir = Buffer.from('.');
> fs.readdir(bufdir, { encoding: 'buffer' }, x => console.log(x));
> TypeError [ERR_INVALID_ARG_TYPE]: The "path" argument must be of type string. Received an instance of Buffer
    at new NodeError                      (node:internal/errors:377:5)
    at validateString                     (node:internal/validators:119:11)
    at Object.isAbsolute                  (node:path:1157:5)
    at VirtualFS.mapToBase                (.../web/.pnp.cjs:8397:24)
    at VirtualFS.readdirPromise           (.../web/.pnp.cjs:8279:44)
    at PosixFS.readdirPromise             (.../web/.pnp.cjs:8279:24)
    at URLFS.readdirPromise               (.../web/.pnp.cjs:8279:24)
    at                                     .../web/.pnp.cjs:9388:20
    at process.processTicksAndRejections  (node:internal/process/task_queues:77:11) {
  code: 'ERR_INVALID_ARG_TYPE'
}

Using the workaround from @vicary, as above, in the meantime (many thanks!)

Not sure why this issue is labelled as unreproducible though...?

@IgnusG
Copy link

IgnusG commented Jul 3, 2022

This issue likely causes this strange bug vitejs/vite#8897 on node 18> in vite

@troncali
Copy link

troncali commented Jul 9, 2022

Also have the same issue as @IgnusG with Nx where every other (uncached) build and serve fails with the same TypeError for path. Plain yarn nx reset also fails.

Environment:

  • Nx 14.4.2
  • Node 18.5.0
  • Yarn 3.2.1 & 4.0.0-rc.11 (pnpMode: strict, nodeLinker: pnp)
  • macOS 12.4 (arm)

My workaround is to revert to Node 16.16.0, which doesn't trigger the same errors, since editing .pnp.cjs doesn't persist. (Though @vicary's suggestion did work, thanks!)

`yarn nx reset` Log
 yarn nx reset

 >  NX   Resetting the Nx workspace cache and stopping the Nx Daemon.

   This might take a few minutes.


 >  NX   Daemon Server - Stopped

nx.js reset

Clears all the cached Nx artifacts and metadata about the workspace and shuts down the Nx Daemon.

Options:
  --help     Show help                                                                                                                                        [boolean]
  --version  Show version number                                                                                                                              [boolean]

TypeError [ERR_INVALID_ARG_TYPE]: The "path" argument must be of type string. Received an instance of Buffer
    at new NodeError (node:internal/errors:388:5)
    at validateString (node:internal/validators:114:11)
    at Object.isAbsolute (node:path:1157:5)
    at VirtualFS.mapToBase (.../Code/nest-vue/.pnp.cjs:31524:24)
    at VirtualFS.rmdirSync (.../Code/nest-vue/.pnp.cjs:31367:39)
    at PosixFS.rmdirSync (.../Code/nest-vue/.pnp.cjs:31367:24)
    at URLFS.rmdirSync (.../Code/nest-vue/.pnp.cjs:31367:24)
    at _rmdirSync (node:internal/fs/rimraf:260:21)
    at rimrafSync (node:internal/fs/rimraf:193:7)
    at node:internal/fs/rimraf:253:9 {
  code: 'ERR_INVALID_ARG_TYPE'
}
`yarn nx build backend` Log
yarn nx build backend

   ✔    3/3 dependent project tasks succeeded [0 read from cache]

   Hint: you can run the command with --verbose to see the full dependent project outputs

 —————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————


> nx run backend:build --verbose=true

asset main.js 2.96 MiB [emitted] [big] (name: main) 1 related asset
asset seed.js 2.94 MiB [emitted] [big] (name: seed) 1 related asset
asset migrate.js 2.94 MiB [emitted] [big] (name: migrate) 1 related asset
asset static/.gitkeep 0 bytes [emitted] [from: apps/backend/src/static/.gitkeep] [copied]
  
webpack 5.73.0 compiled successfully (70ff4521f3265f7b)

 —————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————

 >  NX   Successfully ran target build for project backend and 3 task(s) it depends on (14s)
 
         With additional flags:
           --verbose=true
 
yarn nx build backend --skip-nx-cache --verbose

   ✔    3/3 dependent project tasks succeeded [0 read from cache]

   Hint: you can run the command with --verbose to see the full dependent project outputs

 —————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————


> nx run backend:build --verbose=true


 >  NX   The "path" argument must be of type string. Received an instance of Buffer


TypeError [ERR_INVALID_ARG_TYPE]: The "path" argument must be of type string. Received an instance of Buffer
    at new NodeError (node:internal/errors:388:5)
    at validateString (node:internal/validators:114:11)
    at Object.isAbsolute (node:path:1157:5)
    at VirtualFS.mapToBase (.../Code/nest-vue/.pnp.cjs:31524:24)
    at VirtualFS.rmdirSync (.../Code/nest-vue/.pnp.cjs:31367:39)
    at PosixFS.rmdirSync (.../Code/nest-vue/.pnp.cjs:31367:24)
    at URLFS.rmdirSync (.../Code/nest-vue/.pnp.cjs:31367:24)
    at _rmdirSync (node:internal/fs/rimraf:260:21)
    at rimrafSync (node:internal/fs/rimraf:193:7)
    at node:internal/fs/rimraf:253:9

 —————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————

 >  NX   Ran target build for project backend and 3 task(s) it depends on (9s)
 
         With additional flags:
           --verbose=true
 
    ✖    1/4 failed
    ✔    3/4 succeeded [0 read from cache]
 
`yarn nx serve backend` Log
> nx run backend:serve --verbose=true

asset main.js 2.96 MiB [emitted] [big] (name: main) 1 related asset
asset seed.js 2.94 MiB [emitted] [big] (name: seed) 1 related asset
asset migrate.js 2.94 MiB [emitted] [big] (name: migrate) 1 related asset
asset static/.gitkeep 0 bytes [emitted] [from: apps/backend/src/static/.gitkeep] [copied]
  
webpack 5.73.0 compiled successfully (70ff4521f3265f7b)
Debugger listening on ws://localhost:9229/89fa1453-d1b0-42d3-b2da-07078286c1d7
Debugger listening on ws://localhost:9229/89fa1453-d1b0-42d3-b2da-07078286c1d7
For help, see: https://nodejs.org/en/docs/inspector
[Nest] 40927  - 07/09/2022, 11:49:08 AM     LOG [NestFactory] Starting Nest application...
[Nest] 40927  - 07/09/2022, 11:49:09 AM     LOG [InstanceLoader] DatabaseProvider dependencies initialized +60ms
[Nest] 40927  - 07/09/2022, 11:49:09 AM     LOG [InstanceLoader] TypeOrmModule dependencies initialized +0ms
[Nest] 40927  - 07/09/2022, 11:49:09 AM     LOG [InstanceLoader] GraphQLProvider dependencies initialized +0ms
[Nest] 40927  - 07/09/2022, 11:49:09 AM     LOG [InstanceLoader] PassportModule dependencies initialized +0ms
[Nest] 40927  - 07/09/2022, 11:49:09 AM     LOG [InstanceLoader] ConfigHostModule dependencies initialized +1ms
[Nest] 40927  - 07/09/2022, 11:49:09 AM     LOG [InstanceLoader] AppModule dependencies initialized +0ms
[Nest] 40927  - 07/09/2022, 11:49:09 AM     LOG [InstanceLoader] ConfigModule dependencies initialized +0ms
[Nest] 40927  - 07/09/2022, 11:49:09 AM     LOG [InstanceLoader] ConfigModule dependencies initialized +0ms
[Nest] 40927  - 07/09/2022, 11:49:09 AM     LOG [InstanceLoader] ConfigModule dependencies initialized +0ms
[Nest] 40927  - 07/09/2022, 11:49:09 AM     LOG [InstanceLoader] DbConfigModule dependencies initialized +0ms
[Nest] 40927  - 07/09/2022, 11:49:09 AM     LOG [InstanceLoader] GqlConfigModule dependencies initialized +0ms
[Nest] 40927  - 07/09/2022, 11:49:09 AM     LOG [InstanceLoader] AppConfigModule dependencies initialized +0ms
[Nest] 40927  - 07/09/2022, 11:49:09 AM     LOG [InstanceLoader] GraphQLSchemaBuilderModule dependencies initialized +17ms
[Nest] 40927  - 07/09/2022, 11:49:09 AM     LOG [InstanceLoader] GraphQLModule dependencies initialized +1ms
[Nest] 40927  - 07/09/2022, 11:49:09 AM     LOG [InstanceLoader] TypeOrmCoreModule dependencies initialized +75ms
[Nest] 40927  - 07/09/2022, 11:49:09 AM     LOG [InstanceLoader] TypeOrmModule dependencies initialized +0ms
[Nest] 40927  - 07/09/2022, 11:49:09 AM     LOG [InstanceLoader] TypeOrmModule dependencies initialized +0ms
[Nest] 40927  - 07/09/2022, 11:49:09 AM     LOG [InstanceLoader] SessionModule dependencies initialized +1ms
[Nest] 40927  - 07/09/2022, 11:49:09 AM     LOG [InstanceLoader] UserModule dependencies initialized +0ms
[Nest] 40927  - 07/09/2022, 11:49:09 AM     LOG [InstanceLoader] AuthModule dependencies initialized +0ms
[Nest] 40927  - 07/09/2022, 11:49:09 AM     LOG [RoutesResolver] AppController {/api}: +6ms
[Nest] 40927  - 07/09/2022, 11:49:09 AM     LOG [RouterExplorer] Mapped {/api, GET} route +1ms
[Nest] 40927  - 07/09/2022, 11:49:09 AM     LOG [RoutesResolver] AuthController {/api} (version: Neutral): +1ms
[Nest] 40927  - 07/09/2022, 11:49:09 AM     LOG [RouterExplorer] Mapped {/api/login, POST} (version: Neutral) route +0ms
[Nest] 40927  - 07/09/2022, 11:49:09 AM     LOG [RouterExplorer] Mapped {/api/protected, GET} (version: Neutral) route +1ms
[Nest] 40927  - 07/09/2022, 11:49:09 AM     LOG [RouterExplorer] Mapped {/api/logout, GET} (version: Neutral) route +0ms
[Nest] 40927  - 07/09/2022, 11:49:09 AM     LOG [RoutesResolver] UserController {/api/user}: +0ms
[Nest] 40927  - 07/09/2022, 11:49:09 AM     LOG [RouterExplorer] Mapped {/api/user/:id, GET} (version: 1) route +0ms
[Nest] 40927  - 07/09/2022, 11:49:09 AM     LOG [RouterExplorer] Mapped {/api/user/create, POST} (version: 1) route +1ms
[Nest] 40927  - 07/09/2022, 11:49:09 AM     LOG [GraphQLModule] Mapped {/api/graphql, POST} route +27ms
[Nest] 40927  - 07/09/2022, 11:49:09 AM     LOG [NestApplication] Nest application successfully started +41ms


yarn start backend --verbose

> nx run backend:serve --verbose=true


 >  NX   The "path" argument must be of type string. Received an instance of Buffer


TypeError [ERR_INVALID_ARG_TYPE]: The "path" argument must be of type string. Received an instance of Buffer
    at new NodeError (node:internal/errors:388:5)
    at validateString (node:internal/validators:114:11)
    at Object.isAbsolute (node:path:1157:5)
    at VirtualFS.mapToBase (.../Code/nest-vue/.pnp.cjs:31524:24)
    at VirtualFS.rmdirSync (.../Code/nest-vue/.pnp.cjs:31367:39)
    at PosixFS.rmdirSync (.../Code/nest-vue/.pnp.cjs:31367:24)
    at URLFS.rmdirSync (.../Code/nest-vue/.pnp.cjs:31367:24)
    at _rmdirSync (node:internal/fs/rimraf:260:21)
    at rimrafSync (node:internal/fs/rimraf:193:7)
    at node:internal/fs/rimraf:253:9

 —————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————

 >  NX   Running target "backend:serve" failed

   Failed tasks:
   
   - backend:serve
   
   Hint: run the command with --verbose for more details.

@troncali
Copy link

I forked version 3.2.1 with @vicary's patch that persists the change to .pnp.cjs: troncali/yarn-3.2.1-pnp-patch.

Yarn runs from a compiled release, so the patch has to be compiled into the release. You can do this by running:

yarn set version from sources --repository https://github.com/troncali/yarn-3.2.1-pnp-patch

Happy to create a PR for this, but unsure if this is the best way to fix the bug. If one of the maintainers wants to provide some feedback or direction on this issue I can work on something more permanent.

@IgnusG
Copy link

IgnusG commented Nov 7, 2022

I'm still having this issue on the latest yarn 3.2.4 on nodejs 19.0.0 using PnP. @vicary's change fixed it but it does require editing .pnp.cjs.

I'm running fs.rmSync using -rf on a directory and it fails if that directory has another directory in it.

@merceyz
Copy link
Member

merceyz commented Nov 7, 2022

I'm still having this issue on the latest yarn 3.2.4

This fix isn't in v3.2.4, it hasn't been released in a stable version yet. We'll make a new stable release with the fix soon but until then you can use a canary build

yarn set version canary

or build it from source

yarn set version --branch "merceyz/release/3.3.0"

@IgnusG
Copy link

IgnusG commented Nov 7, 2022

Perfect, thanks @merceyz! ❤️ Works well on canary

I was looking for release information on the PR but couldn't find any.
It might be a nice addition to have something automated to comment on merged PRs in which versions they are available (at least I've seen something like this in some projects). Just an idea

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working unreproducible This issue cannot be reproduced on master upholded Real issues without formal reproduction
Projects
None yet