Support ES6 Modules

[opensas]

I'm trying to package a node app as an exe using pkg, and I'd like to use ES6 imports.

I have something like this in my src/app.js:

import express from 'express'
const app = express()

const eco = (req, res) => {
  const { method, url, headers, query } = req
  res.json({ method, url, headers, query })
}

app.all('/', eco)

app.listen(3000, () => console.log(`listening on http://localhost:3000`))

in my package.json I have:

{
  "name": "pkg-test",
  "version": "1.0.0",
  "description": "",
  "main": "src/app.js",
  "type": "module",
  "type": "module",
  "scripts": {
"build": "pkg --targets=node12-win-x64 --output=iisnode-pkg.exe --options experimental-modules src/app.js",
    "start": "node --experimental-modules src/app.js",
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  "author": "",
  "license": "ISC",
  "dependencies": {
    "express": "^4.17.1"
  },
  "devDependencies": {
    "pkg": "^4.4.0"
  }

}

npm start works fine

$ npm start

> iisnode-pkg@1.0.0 start C:\data\devel\apps\tmp\iisnode-pkg
> node --experimental-modules src/app.js

(node:10668) ExperimentalWarning: The ESM module loader is experimental.
welcome to iisnode-pkg
iisnode-pkg listening on http://localhost:3000

but npm run build gives a waning and then running the exe throws an error:

>npm run build

> iisnode-pkg@1.0.0 build C:\data\devel\apps\tmp\iisnode-pkg
> pkg --targets=node12-win-x64 --output=iisnode-pkg.exe src/app.js --config package.json

> pkg@4.4.0
> Warning Failed to make bytecode node12-x64 for file C:\snapshot\iisnode-pkg\src\app.js

>iisnode-pkg.exe
C:\snapshot\iisnode-pkg\src\app.js:2
import express from 'express'
       ^^^^^^^

SyntaxError: Unexpected identifier
    at Module._compile (internal/modules/cjs/loader.js:701:23)
    at Module._compile (pkg/prelude/bootstrap.js:1268:32)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:768:10)
    at Module.load (internal/modules/cjs/loader.js:626:32)
    at Function.Module._load (internal/modules/cjs/loader.js:553:12)
    at Function.Module.runMain (pkg/prelude/bootstrap.js:1316:12)
    at internal/main/run_main_module.js:17:11

It seems like the --options experimental-modules parameter from the build script in package.json is not beign taken into account.

Any idea how can I use ES6 module imports from a node app packaged with pkg?

PS: also asked at SO in case anybody wants to raise their reputation

[soryy708]

Make sure you're running a pretty recent version of Node.
The implementation of ESM in Node.js changed in version 12.0.0. "type": "module" wasn't supported in 10.x.

You need to add options to the pkg configuration.
So, in package.json:

"pkg": {
    "options": ["experimental-modules"]
}

Node documentation: https://nodejs.org/api/esm.html

[gt-Lee]

I suggest you package your backend service with webpack before using Pkg

[soryy708]

@gt-Lee Transpilation may not be neccesary with very modern node versions, which support ecmascript modules experimentally. Looks like he's trying to set that up.

[gautaz]

Hello,

What is the current status of this issue?

Using pkg@5.1.0 with NodeJS@14.16.1 still ends up with Failed to make bytecode messages:

Targeting node14.16.1-linux-x64
> pkg@5.1.0
> Warning Failed to make bytecode node14.16.1-x64 for file /snapshot/project/index.js
> Warning Failed to make bytecode node14.16.1-x64 for file /snapshot/project/asset-scrubber.js

The experimental-modules flag is not to be used anymore as NodeJS@14.16.1 supports ESM natively.

Using webpack seems a bit overkill.
What is the best course of action to work around this issue if ESM are not yet supported?

[robertsLando]

@erossignon any clue?

[lkesteloot]

I found a temporary work-around for this: Package everything into a single non-module JavaScript file using webpack. This is my webpack.config.js file:

import path from "path";
import { URL } from 'url';

// Replace __dirname from non-module node. May not work well if there are
// spaces in the path (will show up as %20).
const __dirname = new URL('.', import.meta.url).pathname;

const exports = {
    mode: "production",
    entry: "./dist/index.js",
    target: "node",
    output: {
        path: path.resolve(__dirname, "binaries"),
        chunkFormat: "commonjs",
    },
};

export default exports;

This will start at the root of your source tree (dist/index.js, for me generated by TypeScript) and generate a single binaries/main.js file. I can then run:

pkg --out-path binaries binaries/main.js

to create the three binaries in the same directory.

[gautaz]

Hello @lkesteloot, I used esbuild with an additional plugin to manage native modules extracted from this esbuild issue.

Still, using an additional bundler seems a bit cumbersome as pkg is also a bundler of some sort...

[jesec]

Check out https://github.com/vercel/ncc . It is quite easy to use.

[taylorjdawson]

What would be nice would be for pkg to a) provide an option to transpile while packaging, b) provide access to the internals (plugin system) so that we could apply a transform to the file before it gets packaged.

The root issue here is that tools like babel, swc, and ncc don't follow imports like pkg does. We just need a way to merge the two.

[rightaway]

If your target Node version already supports modules then shouldn't pkg already work with it? Or does pkg reimplement these features somehow? Can anyone who knows how this works give some info.

[gautaz]

@rightaway FWIU pkg tries to find what dependencies should be included in the output by parsing the source code for require.

To support ES6 modules, it has to do the same for import.
Currently this seems to rely on Babel but I have no insight of the implications of this.

[rightaway]

@gautaz If it relies on babel maybe there could be some way to customize the options passed to babel so we can tell if we're using ES6 modules?

@jesec @igorklopov Is it in the eventual roadmap to offer support for ES6 modules or will it not be in the scope of pkg?

[ngustavo]

I tried both "type": "module" and .mjs but it throws SyntaxError: Cannot use import statement outside a module. I think they can be added since Node.js is already using them in the stable build. I'm not an expert on how pkg parses dependencies but my help is here if needed.

[vsnthdev]

I was just using an external library from npm which was using ESM modules 🤷‍♂️ and now I can't compile my CLI using pkg 😐

It's been almost one and half years since the issue was opened and no packagers currently support bundling native ESM modules. Tried nexe, nodec and pkg.

[rightaway]

@vasanthdeveloper Which library is it?

[vsnthdev]

It's itivrutaha, and I fixed this issue by migrating to esbuild, and I'll add a workaround to this in a few minutes here.
And my project is https://github.com/serverfiles/serverfiles. I've already fixed the issue here 😊

[konsumer]

@vasanthdeveloper how did you run it? I am still getting errors with this:

esbuild --bundle --target=node10 --outdir=./dist --platform=node ./crossaudio.js && pkg --public --out-path dist/ ./dist/crossaudio.js

When I try to run the result on mac or linux:

node:internal/url:1372
    throw new ERR_INVALID_ARG_TYPE('path', ['string', 'URL'], path);
    ^

TypeError [ERR_INVALID_ARG_TYPE]: The "path" argument must be of type string or an instance of URL. Received undefined
    at new NodeError (node:internal/errors:363:5)
    at fileURLToPath (node:internal/url:1372:11)
    at Object.<anonymous> (/snapshot/dist/crossaudio.js:14277:49)
    at Module._compile (pkg/prelude/bootstrap.js:1751:22)
    at Object.Module._extensions..js (node:internal/modules/cjs/loader:1138:10)
    at Module.load (node:internal/modules/cjs/loader:989:32)
    at Function.Module._load (node:internal/modules/cjs/loader:829:14)
    at Function.runMain (pkg/prelude/bootstrap.js:1804:12)
    at node:internal/main/run_main_module:17:47 {
  code: 'ERR_INVALID_ARG_TYPE'
}

My project is here

Even when I try to run the resulting bundle:

node ./dist/crossaudio.js

file:///home/konsumer/Documents/otherdev/crossaudio/packages/cli/dist/crossaudio.js:12766
var import_assert = __toModule(require("assert"));
                    ^

ReferenceError: require is not defined in ES module scope, you can use import instead
This file is being treated as an ES module because it has a '.js' file extension and '/home/konsumer/Documents/otherdev/crossaudio/packages/cli/package.json' contains "type": "module". To treat it as a CommonJS script, rename it to use the '.cjs' file extension.
    at file:///home/konsumer/Documents/otherdev/crossaudio/packages/cli/dist/crossaudio.js:12766:21
    at ModuleJob.run (node:internal/modules/esm/module_job:175:25)
    at async Loader.import (node:internal/modules/esm/loader:178:24)
    at async Object.loadESM (node:internal/process/esm_loader:68:5)

But the source is fine:

node crossaudio.js

Usage: crossaudio.js <synthfile> [options]

Positionals:
  synthfile  The crossaudio script that defines your synth

Options:
  --help     Show help                                                 [boolean]
  --version  Show version number                                       [boolean]

Examples:
  crossaudio.js file.js --cutoff=74 --reso  Setup midi CC to control synth param
  nance=71                                  s "cutoff" and "resonance".
  crossaudio.js file.js --note=note         Send midi-message for note on/off

If I do this to force cjs extension:

esbuild --bundle --target=node10 --outdir=./dist --platform=node --out-extension:.js=.cjs ./crossaudio.js && pkg --public --out-path dist/ ./dist/crossaudio.cj

I get same error when I run the build product:

node dist/crossaudio.cjs

node:internal/url:1372
    throw new ERR_INVALID_ARG_TYPE('path', ['string', 'URL'], path);
    ^

TypeError [ERR_INVALID_ARG_TYPE]: The "path" argument must be of type string or an instance of URL. Received undefined
    at new NodeError (node:internal/errors:363:5)
    at fileURLToPath (node:internal/url:1372:11)
    at Object.<anonymous> (/home/konsumer/Documents/otherdev/crossaudio/packages/cli/dist/crossaudio.cjs:14277:49)
    at Module._compile (node:internal/modules/cjs/loader:1108:14)
    at Object.Module._extensions..js (node:internal/modules/cjs/loader:1137:10)
    at Module.load (node:internal/modules/cjs/loader:988:32)
    at Function.Module._load (node:internal/modules/cjs/loader:828:14)
    at Function.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:76:12)
    at node:internal/main/run_main_module:17:47 {
  code: 'ERR_INVALID_ARG_TYPE'
}

[azerpas]

For those using Typescript, set

    "target": "ES2020",                               
    "module": "commonjs",

inside your tsconfig.json

[ryanblock]

Any word on this? ESM has been in the works for years and is now fully supported in active and current Node versions – transpilation should not be a necessary step to using ESM in 2021. This issue in particular has been open for two years.

Related: is this project being actively improved, or is it in bug fix / maintenance mode? Checking in on key issues / offers for PRs have gone unresponded to; I see that occasional maintenance releases are still being occasionally cut, but as far as maintainer participation it seems like crickets here in the issue tracker.

It's totally cool if this project is no longer being actively improved or if PRs just aren't welcomed, but some guidance from the maintainers would be helpful in setting expectations.

[leerob]

It's totally cool if this project is no longer being actively improved or if PRs just aren't welcomed, but some guidance from the maintainers would be helpful in setting expectations.

Feel free to open PRs, we can check them out 😄 As long as there's proper testing, I see no issue adding new features!

[gr2m]

@ryanblock please let us know if find a way to make it work :)

[Oloompa]

If it can help others, i moved to caxa which handle ESM out-of-the-box.

[gr2m]

Worked for me, cheers:
https://github.com/octokit/fixtures-server/blob/3b6d1dfa6d250388fddab9cf3d3ab4fcee3cdb9b/.github/workflows/release.yml

[rightaway]

@ryanblock I don't expect much to happen with pkg.

https://github.com/leafac/caxa looks very good. Responsive maintainer and works with ES6 modules and native modules.

[ryanblock]

@leerob given the conversation that's ensued, thoughts on the other part of my question (is this project being actively improved, or is it in bug fix / maintenance mode?)? PRs are welcomed (great!), but are y'all just maintaining, or actively investing in improving pkg?

[jesec]

It is straightforward to implement. We just have to support tracing of dependencies for ES6 "import", like what we currently did for CommonJS (see walker.ts).

However, I would rather not reinvent the wheel, since nft does just that. #1138 would be the prerequisite for ES6 module support.

At the moment, I recommend users to use a bundler like esbuild or webpack to transpile to CommonJS style import. If you use TypeScript, simply specify "module": "commonjs".

[ryanblock]

Thanks for the context, @jesec! Imo bundling esm to cjs is totally out of the question. Call me crazy, but I like usable stack traces!

[Nisthar]

@ryanblock I don't expect much to happen with pkg.

https://github.com/leafac/caxa looks very good. Responsive maintainer and works with ES6 modules and native modules.

caxa doesn't hide the source code in any way Right? its a big no for me

[rightaway]

@Nisthar it will leafac/caxa#21