[feat] Replace internal walker with nft

[robertsLando]

AFAIK walker is actually used to get all the files needed by the runtime to run, including node_modules, this is exaclty what nft does.

This could fix some issues that we are having like:

#1130
#1135
#1072

/cc @jesec @leerob @erossignon @hipstersmoothie thoughts?

We could also consider something with ncc

[skaneprime]

No response yet?

[robertsLando]

Nope

[jesec]

Sounds like a good idea. I am not really able to comment much before we actually did it, though.

[robertsLando]

I think that who knows walker the most here is @erossignon as he has worked on it and also bootstrap in his last changes, I would like to know what he thinks about this.

[erossignon]

In fact , we could keep the walker as-is and run ncc as a prebuild step , the result of the ncc step could be incorporated to the packaged exececutable;

IMO, this is already possible today to combine @vercel/ncc and @vercel/pkg ... why do we need do something extra ?

[Tilican]

Hey,
I use pkg to make my project as an .exe and it loads at runtime .js in a 'plugins' folder that have been previously processed by ncc !
But the project that is processed by ncc could have native dependencies ! :)

Example :

.
├── conf.toml
├── board.exe
├── plugins
│ ├── twitch
│ │ ├── index.js
│ │ ├── iohook.node

Is this a possible use case with nft ?

[robertsLando]

But the project that is processed by ncc could have native dependencies

In that case native .node addons needs to be specified as assets in pkg config

[skaneprime]

But the project that is processed by ncc could have native dependencies

In that case native .node addons needs to be specified as assets in pkg config

I used ncc and found out that canvas.node didn't dropped in build folder as a dependency

Can it be that it is trouble with canvas, not the pkg or ncc it self? I can't quite catch but all issues which happens follow ups to canvas module

[robertsLando]

@skaneprime Did you try to use pkg with --debug flag to see what's going on? Also did you try to add canvas node as an asset?

[skaneprime]

@robertsLando there was a lot of logs. Is there a way to store debug into file?
Nonetheless, here the part when it comes to native modules

  [debug] Targets:
  [
  {
    "nodeRange": "node14",
    "platform": "win",
    "arch": "x64",
    "output": "D:\\WORK\\bothub\\discord-base-bot\\bin\\discord-base-bot.exe",
    "forceBuild": false,
    "fabricator": {
      "nodeRange": "node14",
      "platform": "win",
      "arch": "x64",
      "forceBuild": false,
      "binaryPath": "C:\\Users\\ASUS\\.pkg-cache\\v2.6\\fetched-v14.4.0-win-x64"
    },
    "binaryPath": "C:\\Users\\ASUS\\.pkg-cache\\v2.6\\fetched-v14.4.0-win-x64"
  }
]
prebuild-install WARN install No prebuilt binaries found (target=v14.4.0 runtime=node arch=x64 libc= platform=win32)
> [debug] prebuild-install failed[D:\WORK\bothub\discord-base-bot\build\canvas.node]:
  Error: Command failed: C:\Users\ASUS\AppData\Roaming\npm\node_modules\pkg\node_modules\.bin\prebuild-install -t v14.4.0 --platform win32 --arch x64
prebuild-install WARN install No prebuilt binaries found (target=v14.4.0 runtime=node arch=x64 libc= platform=win32)

prebuild-install WARN install No prebuilt binaries found (target=v14.4.0 runtime=node arch=x64 libc= platform=win32)
> [debug] prebuild-install failed[D:\WORK\bothub\discord-base-bot\build\fsevents.node]:
  Error: Command failed: C:\Users\ASUS\AppData\Roaming\npm\node_modules\pkg\node_modules\.bin\prebuild-install -t v14.4.0 --platform win32 --arch x64
prebuild-install WARN install No prebuilt binaries found (target=v14.4.0 runtime=node arch=x64 libc= platform=win32)

prebuild-install WARN install No prebuilt binaries found (target=v14.4.0 runtime=node arch=x64 libc= platform=win32)
> [debug] prebuild-install failed[D:\WORK\bothub\discord-base-bot\build\prebuilds\win32-x64\node.napi.node]:
  Error: Command failed: C:\Users\ASUS\AppData\Roaming\npm\node_modules\pkg\node_modules\.bin\prebuild-install -t v14.4.0 --platform win32 --arch x64
prebuild-install WARN install No prebuilt binaries found (target=v14.4.0 runtime=node arch=x64 libc= platform=win32)

prebuild-install WARN install No prebuilt binaries found (target=v14.4.0 runtime=node arch=x64 libc= platform=win32)
> [debug] prebuild-install failed[D:\WORK\bothub\discord-base-bot\build\prebuilds\win32-x64\node.napi1.node]:
  Error: Command failed: C:\Users\ASUS\AppData\Roaming\npm\node_modules\pkg\node_modules\.bin\prebuild-install -t v14.4.0 --platform win32 --arch x64
prebuild-install WARN install No prebuilt binaries found (target=v14.4.0 runtime=node arch=x64 libc= platform=win32)

prebuild-install WARN install No prebuilt binaries found (target=v14.4.0 runtime=node arch=x64 libc= platform=win32)
> [debug] prebuild-install failed[D:\WORK\bothub\discord-base-bot\node_modules\canvas\build\Release\canvas.node]:
  Error: Command failed: C:\Users\ASUS\AppData\Roaming\npm\node_modules\pkg\node_modules\.bin\prebuild-install -t v14.4.0 --platform win32 --arch x64
prebuild-install WARN install No prebuilt binaries found (target=v14.4.0 runtime=node arch=x64 libc= platform=win32)

prebuild-install WARN install No prebuilt binaries found (target=v14.4.0 runtime=node arch=x64 libc= platform=win32)
> [debug] prebuild-install failed[D:\WORK\bothub\discord-base-bot\node_modules\fsevents\fsevents.node]:
  Error: Command failed: C:\Users\ASUS\AppData\Roaming\npm\node_modules\pkg\node_modules\.bin\prebuild-install -t v14.4.0 --platform win32 --arch x64
prebuild-install WARN install No prebuilt binaries found (target=v14.4.0 runtime=node arch=x64 libc= platform=win32)

[skaneprime]

@skaneprime Did you try to use pkg with --debug flag to see what's going on? Also did you try to add canvas node as an asset?

Forgot to mention. Yes, I added canvas.node as an asset in every way possible. Played with paths (moved it to src folder from node_modules, added the whole canvas module as an asset etc) but no result.

BUT I noticed something. I have another native module. It is FSEvents ( dunno from which module. I guess from chokodiar ) and it works well.

[robertsLando]

Based on your logs seems that this canvas.node isn't installed correctly, that could be the reason

[erossignon]

see #1135 and PR #1143 that fixes .node files that require external dlls on windows

[Tilican]

But the project that is processed by ncc could have native dependencies

In that case native .node addons needs to be specified as assets in pkg config

I can't know in advance what .node dependencies will be, because users can add modules themselves ...

It's like plugin system, user can add or remove plugin at runtime

[github-actions]

This issue is stale because it has been open 90 days with no activity. Remove the stale label or comment or this will be closed in 5 days. To ignore this issue entirely you can add the no-stale label

[github-actions]

This issue is now closed due to inactivity, you can of course reopen or reference this issue if you see fit.