Initiative: Single Executable Application

[jesec]

Continuation of nodejs/node#42334 (comment).

Single Executable Application is one of the technical priorities:

Node.js often loses out to other runtimes/languages in cases where being able to package a single, executable application simplifies distribution and management of what needs to be delivered. While there are components/approaches for doing this, they need to be better documented and evangelized so that this is not seen as a barrier for using Node.js in these situations. This is important to support the expansion of where/when Node.js is used in building solutions.

Prior discussions:

• Next-10 Mini-Summit 18 Nov 2021
• Technical Memo on pkg
• Draft: src,doc: Experimental support for SEA (#42334)

Items (core):

• Make it possible for the embedded script to overlay/override the fs.*, including for the internal modules that use them (e.g. loaders).
  • 3 commits at the moment, and they shouldn't affect functionality in any way.
• Support parsing the header, loading the bake-in Node options, and loading the embedded script.
  • Prefer more discussions, especially the parts, when changed later, could be semver-major (e.g. header format).
  • We also have to figure out the info to pass to the embedded script (e.g. scriptPos, scriptSize), so the script can use the info to locate the VFS payload. and, how to pass them.

Items:

• Source-less vm.Script. (and vm.Module?)
  • Involves changes to V8. Need to find an elegant way to bypass bytecode sanity checks in V8 for this case (pkg currently simply removes such checks unconditionally).
  • It is a public API, and extra consideration may be necessary. Proposed addition: sourceless: <boolean> = false in options.
• More fine-grained way to load a native module from an offset in a file (the executable in our case). Some potential methods have to be implemented in the native (C++) land, so it might be better for Node.js to take over via something like process.dlopen(...[, offset]).
  • See previous comments in src,doc: Experimental support for SEA node#42334, and dlopen dylib executables from memory for Android/Linuxish platforms, not disk iree-org/iree#3845.
• Allow post-build swap of intl variants (full-icu, small-icu, system-icu, no-intl).
  • No implementation yet, but definitely good-to-have since intl variant has a significant impact on binary size.
• Fully static linuxstatic binaries for use cases that require extensive portability (e.g. embedded, industrial).

Out-of-scope, per discussions:

• Virtual file system (VFS)
  • The embedded script should do this, and call the user's entrypoint. See pkg.
• Dependency tracing
• Bundling
  • Use webpack or others
• Creation of the executable?
  • node compile is not planned, yet.

Edge cases:

• Should the SEA behave as the bundled application in all scenarios? Can the user use it as the Node.js runtime if they want to?
  • Error: Command failed: Executing pkg cli from the child process of pkg cli getting failed. vercel/pkg#897

---

Proof of concept (nodejs/node#42334 (comment)):

[btsimonh]

"More fine-grained way to load a native module from an offset in a file (the executable in our case)."
In my use case, my VFS is compressed and encrypted. So load from a buffer would be better.
On windows, I looked into this in detail some time back, and it's as close to impossible, that it is certainly impractical, relying on undocumented and possibly fragile APIs (look up 'MS detours').

"Should the SEA behave as the bundled application in all scenarios" - In my nexe based implementation, I disable the command line except in debug builds. e.g. I don't want my customers to be able to run the node debugger on my somewhat-hidden code (especially the decrypt code!).

I also compress and encrypt the code inserted into Node (vfs and launch code). One issue with baking a pseudo-security facility into OSS is that there is no true code protection, so we have to rely on obscurity and difficult to debug code. If it was 'standard' code, that all disappears. nexe's code patching system facilitates the patching of the node c++ to achieve this, but the patches themselves should probably remain unique to users if they want this kind of facility?

[github-actions]

There has been no activity on this feature request for 5 months and it is unlikely to be implemented. It will be closed 6 months after the last non-automated comment.

For more information on how the project manages feature requests, please consult the feature request management document.

[ovflowd]

@RaisinTen can we remove the stale label from here?

[github-actions]

There has been no activity on this feature request for 5 months and it is unlikely to be implemented. It will be closed 6 months after the last non-automated comment.

For more information on how the project manages feature requests, please consult the feature request management document.

[piranna]

This is not stale... or is it?

[RaisinTen]

I'll transfer this to the single-executable repo, where it won't keep getting marked as stale.