fs: move `ToNamespacedPath` to c++ #52135

anonrig · 2024-03-18T15:03:52Z

This is a step forward to move all node:fs implementation to C++. Since we now have support for win32.pathResolve() in C++, we can move forward with moving ToNamespacedPath functions to C++.

In a follow up pull-request, I'll move FromNamespacedPath() from node_url.cc to path.cc

cc @nodejs/fs @RafaelGSS

nodejs-github-bot · 2024-03-18T15:03:57Z

Review requested:

@nodejs/url

anonrig · 2024-03-18T16:00:02Z

cc @nodejs/platform-windows Any idea where this error/warning is thrown from?

src/path.cc

LiviaMedeiros · 2024-03-18T17:03:25Z

AFAICT the changes are implying that getValidatedPath() should do path.ToNamespacedPath() internally, but there's no changes to getValidatedPath() itself in this PR. Does it happen implicitly? If yes, on which step?

src/path.cc

src/path.h

src/path.cc

anonrig · 2024-03-18T17:29:12Z

AFAICT the changes are implying that getValidatedPath() should do path.ToNamespacedPath() internally, but there's no changes to getValidatedPath() itself in this PR. Does it happen implicitly? If yes, on which step?

getValidatedPath() result is called with toNamespacedPath() to ensure Windows long paths are handled correctly. I didn't quite understand your comment, but this pull request basically replaces toNamespacedPath(getValidatedPath(input)) with getValidatedPath(input) and calls toNamespacedPath from C++

LiviaMedeiros · 2024-03-18T18:35:04Z

getValidatedPath() result is called with toNamespacedPath() to ensure Windows long paths are handled correctly. I didn't quite understand your comment, but this pull request basically replaces toNamespacedPath(getValidatedPath(input)) with getValidatedPath(input) and calls toNamespacedPath from C++

Sorry, I read the changes wrong. It doesn't affect js-side toNamespacedPath() itself but adds cpp-side version that is used in cpp implementations for each fs method.
We might need to doublecheck that every method uses this. Of course, it's non-blocking concern: I'm just pointing out that after this change we'll have to check cpp side instead of js.
Right now I'm only seeing that Mkdtemp lacks it here:

node/src/node_file.cc

Lines 2820 to 2823 in d93f610

    
           THROW_IF_INSUFFICIENT_PERMISSIONS( 
        
               env, permission::PermissionScope::kFileSystemWrite, tmpl.ToStringView()); 
        
           const enum encoding encoding = ParseEncoding(isolate, args[1], UTF8);

Which mirrors current (might be bugged on Windows if called with long prefix) js implementation.

nodejs-github-bot · 2024-03-18T20:51:55Z

CI: https://ci.nodejs.org/job/node-test-pull-request/57821/

RafaelGSS · 2024-03-19T11:30:55Z

This should land first #52148.

RafaelGSS

As @joyeecheung suggested on my initial PR for node::pathResolve, I suggest you do the same for path.toNamespacePath. Change the bindings in a different PR and run the tests to see if it works.

Example: #51295

anonrig · 2024-03-19T14:06:54Z

@RafaelGSS the function signature is not 1 to 1. We're always using BufferValue in node:fs.

joyeecheung · 2024-03-19T18:32:16Z

I think this is different from #51295 in that, the other PR only used the C++ version in a not very well-tested branch (only when permission is enabled) so if it's broken, we may not be able to know due to the low test coverage. This PR however changed most fs bindings so if it's broken, it's probably more likely that some tests would fail on Windows.

That said I think we already don't have enough test cases to check the kind of paths that are need to be handled specially by toNamespacedPath(). Perhaps it would be useful to check the cases in #51097 too

anonrig · 2024-06-12T19:48:37Z

hey @huseyinacacak-janea, sorry for the late response. I've updated the pull-request with your recommendations.

nodejs-github-bot · 2024-06-12T19:54:46Z

CI: https://ci.nodejs.org/job/node-test-pull-request/59741/

nodejs-github-bot · 2024-06-12T23:07:30Z

CI: https://ci.nodejs.org/job/node-test-pull-request/59742/

nodejs-github-bot · 2024-06-17T15:51:42Z

CI: https://ci.nodejs.org/job/node-test-pull-request/59844/

Co-Authored-By: Daniel Lemire <daniel@lemire.me>

anonrig · 2024-06-17T23:49:50Z

@RafaelGSS can you look into the permission model changes? I removed an unnecessary test since we don't use path.toNamespacedPath anymore in JS world, and updated several paths and surrounded them with path.toNamespacedPath()

nodejs-github-bot · 2024-06-17T23:52:45Z

CI: https://ci.nodejs.org/job/node-test-pull-request/59851/

RafaelGSS · 2024-06-18T02:09:50Z

test/fixtures/permission/fs-traversal.js

-  }));
-}
-
-// Monkey-patching path module should also not allow path traversal.


Regardless if we are moving the implementation to C++ I believe path.toNamespacedPath will always exist through an API right? If so, I don't think we should remove this test, it won't fail anyway.

I think its not needed but on top of that It fails because win32 path resolves return "." for cwd, but previously it didn't.

RafaelGSS · 2024-06-18T02:13:19Z

test/parallel/test-fs-mkdir.js

@@ -316,7 +316,7 @@ if (common.isMainThread && (common.isLinux || common.isOSX)) {
  const p = fs.mkdirSync(pathname, common.mustNotMutateObjectDeep({ recursive: true }));
  assert.strictEqual(fs.existsSync(pathname), true);
  assert.strictEqual(fs.statSync(pathname).isDirectory(), true);
-  assert.strictEqual(p, firstPathCreated);
+  assert.strictEqual(p, path.toNamespacedPath(firstPathCreated));


Can you explain why we are changing the comparison to include the path.toNamespacedPath? Did tmpdir.resolve API change somehow? If so, isn't it a breaking change?

Technically it isn't since both paths are the same and from Windows perspective having a path with namespaced value does not really have any difference with without the prefix.

PS: Namespaced paths are used to avoid "path is too long" errors on Windows. If you look into this file, you'll see that we already check the value surrounded by toNamespacedPath in several places already. I feel like this is more complete.

src/node_file.cc

nodejs-github-bot · 2024-06-18T12:35:27Z

CI: https://ci.nodejs.org/job/node-test-pull-request/59859/

nodejs-github-bot · 2024-06-19T22:49:03Z

CI: https://ci.nodejs.org/job/node-test-pull-request/59900/

nodejs-github-bot · 2024-06-20T02:11:12Z

Landed in 399eb33

Co-Authored-By: Daniel Lemire <daniel@lemire.me> PR-URL: #52135 Reviewed-By: Geoffrey Booth <webadmin@geoffreybooth.com> Reviewed-By: Vinícius Lourenço Claro Cardoso <contact@viniciusl.com.br> Reviewed-By: Daniel Lemire <daniel@lemire.me> Reviewed-By: Rafael Gonzaga <rafael.nunu@hotmail.com> Reviewed-By: James M Snell <jasnell@gmail.com>

Co-Authored-By: Daniel Lemire <daniel@lemire.me> PR-URL: nodejs#52135 Reviewed-By: Geoffrey Booth <webadmin@geoffreybooth.com> Reviewed-By: Vinícius Lourenço Claro Cardoso <contact@viniciusl.com.br> Reviewed-By: Daniel Lemire <daniel@lemire.me> Reviewed-By: Rafael Gonzaga <rafael.nunu@hotmail.com> Reviewed-By: James M Snell <jasnell@gmail.com>

tniessen · 2024-08-05T06:58:16Z

@anonrig This PR might have broken a few things, PTAL at #54161.

nodejs-github-bot added c++ Issues and PRs that require attention from people who are familiar with C++. lib / src Issues and PRs related to general changes in the lib or src directory. needs-ci PRs that need a full CI run. labels Mar 18, 2024

anonrig force-pushed the simplify-fs branch 2 times, most recently from 003590b to 2d120ca Compare March 18, 2024 15:16

richardlau added fs Issues and PRs related to the fs subsystem / file system. windows Issues and PRs related to the Windows platform. labels Mar 18, 2024