Investigate integration/awareness of N-API with node pre-gyp

[mhdawson]

One of the wins for using N-API is not needing a binary for every version of Node. pre-gyp current assumes there is a different one so some investigation/change are needed to optimize for an N-API world.

[jschlight]

The module in question is mapbox/node-pre-gyp. There is a nice background piece, Cross-platform addons with node-pre-gyp.

node-pre-gyp essentially enhances nodejs/node-gyp with the ability to automatically upload binaries as part of the build process. These pre-built binaries can then be downloaded when the native module is deployed.

I am researching the current interoperability between node-pre-gyp and N-API with an eye towards understanding steps that can be taken to make the interaction seamless.

[jschlight]

I've updated my N-API-based module to use node-pre-gyp instead of node-gyp. I'm able to build and upload binaries to S3 and have these binaries detected and automatically downloaded at deployment.

There may be a straightforward solution to using node-pre-gyp with N-API that I'd like to discuss tomorrow during the call.

[addaleax]

Okay, so what I’m thinking based off the discussion we’re having right now:

• We should add process.versions.napi that exports NAPI_MODULE_VERSION (Michael)
• We should consider adding {node_abi_napi} field to node-pre-gyp that yields:
  • If N-API is available, node_abi_napi resolves to napi
  • If N-API is not available, node_abi_napi resolves to the same as node_abi (e.g. 49) to accommodate users of node-addon-api which use the built-in version of N-API
• We should consider adding a {napi_version} field to node-pre-gyp that yields NAPI_MODULE_VERSION
  • Along with it, a new package.json field for the node-pre-gyp options that could be called e.g. "napi-versions": which would contain an array of N-API versions that the author is building for, and that consequently are available for download

[jschlight]

Considering that the process.versions.napi value is just now being introduced, what is the best way to reliably determine, from JavaScript, if the currently running Node instance supports N-API?

node-pre-gyp maintains an abi_crosswalk.json file that maps Node versions to versions for the Node ABI and V8. They look like this:

  "8.9.0": {
    "node_abi": 57,
    "v8": "6.1"
  },
  "9.0.0": {
    "node_abi": 59,
    "v8": "6.2"
  }

We could piggyback on this file to include N-API version numbers. But I need to know which Node versions support which N-API version.

[mhdawson]

Right now there are only 2 N-API versions so it should be easy to figure that out, more complicated is that we were only planning to update once we exited experimental which may complicate things.

[mhdawson]

Once option may be to consider that only the first version of 8.X, 6.X etc where N-API is non-experimental is supported but I'm not sure how practical that might be.

[jschlight]

The value of {node_abi_napi} described above, can be determined directly from the value of {napi_version}. I'm thinking we have a couple of options for the task of determining {napi_version}:

OPTION 1

If the process.versions.napi value is present, then the N-API version is directly determined.

If the process.versions.napi value is not present, N-API is not supported.

This approach essentially says node-pre-gyp is supported only for those versions of Node with a non-experimental N-API implementation.

OPTION 2

If the process.versions.napi value is present, then the N-API version is directly determined as it is for option 1.

If the process.versions.napi value is not present, the N-API version is determined heuristically from the Node version. (Think hard-coded table.)

In this option, the heuristic needs to be determined only once and should be accurate indefinitely. As deployments of Node supporting process.versions.napi become more prevalent, the heuristic code will be relied upon less and less.

[jschlight]

Note: This proposal was last updated 2018-01-18. It's been superseded by the actual Implementation description found as a comment farther down.

I'm posting this proposed implementation for everyone's review. I'm anxious to receive any feedback you may have.

Proposed implementation

New path substitution variables

Add two new substitution variables for the binary properties module_path, remote_path, and package_name.

1. {napi_version} If N-API is supported, this value is the N-API version number supported by Node. If N-API is not supported, this value is an empty string.

2. {node_abi_napi} If the value returned for {napi_version} is non empty, this value is 'napi'. If the value returned for {napi_version} is empty, this value is the value returned for {node_abi}.

New package.json variable

Add an optional new package.json property under node-pre-gyp's binary section.

1. napi-versions An array of N-API versions that the author is building for, and that consequently are available for download.

Changes to publishing

Add code to support the napi-versions package.json property.

1. Inspect the binary property found on the package.json file. If the napi-versions property is present and the {napi_version} substitution string is present in any of module_path, remote_path, and package_name properties, inspect the values found in the napi-versions property. For each value found in the array, use node-gyp to compile a binary for that N-API version and upload the resulting binary using the remote_path, and package_name properties.

   If either the napi-versions property or the {napi_version} substitution string is absent, compile and upload as is currently implemented.

Changes to installing

Add code to support the napi-versions package.json property.

1. Inspect the binary property found on the package.json file.

   If the napi-versions property is present and the {napi_version} substitution string is present in any of module_path, remote_path, and package_name properties, inspect the values found in the napi-versions property. From the array, determine the largest version greater than or equal to the Node N-API version of the installing machine. Use this value for the {napi_version} substitution string and download the binary file using the current implementation. If no value is found in the array matching the criteria, the module does not support the Node N-API version of the installing machine.

   If the {napi_version} substitution string is present, and the napi-versions property is absent, for the purposes of the download, set the {napi_version} value to the Node N-API version of the installing machine, and attempt to download the binary with that N-API version. If the download fails, decrement N-API version by one and retry the download. The download fails when the N-API version reaches zero.

   If both the napi-versions property and the {napi_version} substitution string are absent, download the binary as is currently implemented.

Concerns

1. There can be an issue in the case where the napi-versions property is absent and the {napi_version} substitution string is present. For example, the module developer may be developing with a version of Node that supports N-API version 3. In the case where the napi-versions property is absent, the published binary is tagged for N-API version 3 whether it requires the features of N-API version 3 or not. A user of the module might be running a version of Node that supports N-API version 2. In this case, the user would not be able to download the binary because it appears to require N-API version 3.

   This issue could be resolved if we require the napi-versions property to be present if the {napi_version} substitution string is used. It would also greatly simplify the node-pre-gyp installation code by eliminating the need for sequential download requests at installation time.

[mhdawson]

The case that I think the array could handle (not with the current description though since it only uses the lowest) is:

• Module maintainer publishes 2 versions of their module. Version 1 (supports N-API A) and Version 2 (supports N-API B, where B >A). Version 2 is much faster because it uses some new functions only available in the later version of N-API
• Module is used with Node.js version that only supports N-API A. Version 1 of the module is downloaded and everything runs ok.
• Module is used with Node.js version that supports N-API B. Version 2 is downloaded and it runs much faster than if it had used Version 1.

If package.json just had the lowest N-API version supported, at install time how would we know Version 2 was available for use ?

[jschlight]

I think I'm getting greater clarity here.

If a module specifies "napi-versions":[3,4,5] it is declaring that the code will work under any of those three versions. How does it achieve this? Is the C/C++ code using conditional compilation?

If so, the node-pre-gyp publisher will need to fire off three builds and uploads, one targeting each of the three supported versions. Then the correct binary can be located and downloaded at installation time.

In the example shown here, the node-pre-gyp publishing system will need to be running a version of Node supporting N-API version 5 or later. I'm assuming that a system running a version of Node supporting N-API version 5 can compile C/C++ code targeting N-API version 4 and lower? Correct?

[mhdawson]

If a module specifies "napi-versions":[3,4,5] it is declaring that the code will work under any of those three versions. How does it achieve this? Is the C/C++ code using conditional compilation?

The module owner would have to provide 3 different binaries. Conditional compilation would be one way they could structure their code to support that.

In the example shown here, the node-pre-gyp publishing system will need to be running a version of Node supporting N-API version 5 or later. I'm assuming that a system running a version of Node supporting N-API version 5 can compile C/C++ code targeting N-API version 4 and lower? Correct?

It can compile code by avoiding functions that are not available in the earlier versions. In addition Kyle has an issue about adding #define that can be used to specify the level of functions you want to be visible (ie if you defined NAPI_API_VERION_4 ( I know I have the name wrong but you get the idea) then only the functions in version 4 and lower would be available when compiling.

[jschlight]

I've updated the proposed implementation shown above. The changes are:

1. When publishing, one binary is compiled and uploaded for each version found in the napi-versions array.
2. When installing, the best matching version from the napi-versions array is downloaded.
3. The concern about the napi-versions array was removed.

[mhdawson]

Overall looks reasonable to me.

In terms of

This issue could be resolved if we require the napi-versions property to be present if the {napi_version} substitution string is used. It would also greatly simplify the node-pre-gyp installation code by eliminating the need for sequential download requests at installation time.

are there any downsides of requiring the napi-versions property to be present?

[jschlight]

This is an update on the PR I've been working for node-pre-gyp. At this point I have all the basic functionality working including configure, build, package, publish, install, and clean. I'm now working on the ancillary commands like rebuild, reinstall, and a handful of others. I'll then test everything on Mac, Windows, and ARM. I'm hoping to have the PR completed by tomorrow evening.

[mhdawson]

@jschlight sounds like you are making good progress :)

[jschlight]

Note: This implementation note was last updated 2018-01-31. It represents the current implementation of the impending node-pre-gyp pull request.

Implementation

New variables available to binding.gyp and package.json

Three new substitution variables are available to the binding.gyp file, and by extension, to the binary properties module_path, remote_path, and package_name.

1. napi_version If N-API is supported by the currently executing Node instance, this value is the N-API version number supported by Node. If N-API is not supported, this value is an empty string.

   napi_version is determined first from the process.versions.napi value. If process.versions.napi is absent, the Node version is inspected. If the Node version is 9.3.0 or greater, napi_version is set to 2. Otherwise, if the Node version is 8.0.0 or greater, napi_version is to 1. Otherwise, napi_version is set to the empty string.

2. node_abi_napi If the value returned for napi_version is non empty, this value is 'napi'. If the value returned for napi_version is empty, this value is the value returned for node_abi.

3. napi-build-version When the package.json napi-versions property is specified (see below), node-pre-gyp will fire off one separate build for each N-API version found in this array. This value is set for each build.

binding.gyp variables

All three variables above are available to the binding.gyp file.

The binding.gyp file can be configured to communicate the napi-build-version value to the C/C++ code being compiled. For example:

"defines": [
  "NAPI_BUILD_VERSION=<(napi_build_version)",
],

This causes the C/C++ NAPI_BUILD_VERSION value to be set to the N-API version for which the build is being requested.

package.json variables

The string substitution variables {napi_version}, {node_abi_napi}, and {napi-build-version} are available the binary properties module_path, remote_path, and package_name on the package.json file.

New package.json property

An optional new package.json property under node-pre-gyp's binary section is supported.

1. napi-versions An array of N-API versions that the author is building for, and that consequently are available for download.

Note that if the napi-versions property is specified, then the text substitution string {napi-build-version} must appear in the binary module_path value and in either the remote_path or package_name value on the package.json file.

Changes to building and publishing

New code supports the napi-versions package.json property.

1. Inspect the binary property found on the package.json file. If the napi-versions property is present and the {napi_build_version} substitution string is present in the module_path property and either the remote_path or package_name property, inspect the values found in the napi-versions property. For each value found in the array, use node-gyp to compile a binary for that N-API version and upload each resulting binary using the remote_path and package_name properties.

   If the napi-versions property is absent, compile and upload as is currently implemented.

Changes to installing

New code supports the napi-versions package.json property.

1. Inspect the binary property found on the package.json file.

   If the napi-versions property is present and the {napi_build_version} substitution string is present in the module_path property and either the remote_path or package_name property, inspect the values found in the napi-versions property. From the array, determine the largest version greater than or equal to the Node N-API version of the installing machine. Use this value for the {napi_build_version} substitution string and download (or build, if necessary) the binary file using the current implementation. If no value is found in the array matching the criteria, the module does not support the Node N-API version of the installing machine.

   If both the napi-versions property is absent, download (or build, if necessary) the binary as is currently implemented.

[jschlight]

I've completed all the coding for the node-pre-gyp PR and the code works for my test module. I've committed the code here in case working group members are interested in taking look.

https://github.com/inspiredware/node-pre-gyp/tree/napi-support

Although I could submit a PR now, there are still a handful of things I'd like to cleanup before I make a formal PR to the node-pre-gyp maintainers. On my to-do list:

• Create a working test example to include in the node-pre-gyp repository so the maintainers have something to work from.

• Update the node-pre-gyp documentation for the changes I've made.

• Test it on Windows and ARM platforms.

These tasks will probably take through the end of this week.

[jschlight]

I've completed all of the objectives in the previous comment. However, 3 of the 28 automated tests are failing. The failing tests are related to looking out on the file system for specific files in specific locations. For N-API builds, these files are in different locations.

I'd like to see if I can address these three failures before I put in the pull request in on Monday.

[jschlight]

I've just made the PR to node-pre-gyp mapbox/node-pre-gyp#345

Fingers crossed.

[mhdawson]

Closing as PR has landed.