Removing NAN-based native modules from the Ethereum ecosystem

[alcuadrado]

I'm creating this issue to describe a project we've been working on at Nomic Labs, and what's needed for it to be completed

The problem

Ethereum-related dependencies and applications on npm normally have direct or indirect dependencies on native Node.js modules. Most of them are based on the ABI-unstable Native Abstractions for Node.js API, or NAN, and need to be recompiled every time you install them.

This has multiple problems:

1. It makes installing Ethereum packages slow, degrading the development experience.

2. Many of those native modules break whenever a new Major Node.js version is released. This has lead to versions of Node.js being unusable for Ethereum development for months in the past.

3. To compile the native modules, a C++ compiler, Python, and node-gyp need to be installed. This setup can be tricky, slow, and normally requires downloading gigabytes of data, making it a blocker for many new developers. This is a recurrent issue in workshops and other educational contexts, where time and bandwidth constraints make setting up the environment almost impossible.

The solution

We analyzed the major javascript Ethereum tools and libraries and identified that most of the dependencies requiring recompilation are cryptographic primitives implementations. The only exceptions being websocket and fsevents@^1.0.0.

To remove these dependencies from the Ethereum ecosystem, we should replace them with pure JavaScript or N-API based native dependencies. The latter is a new Node.js interface to create native modules that don't require recompilation.

The obvious way of doing this is choosing a new set of dependencies, and modifying every project to use those. This is a major effort, and can easily get outdated by:

1. Projects accidentally reintroducing native dependencies.

2. The chosen dependencies deciding to start using native modules.

3. New projects reintroducing native dependencies into the ecosystem.

To avoid this, we decided to create ethereum-cryptography, a package with all the cryptographic primitives that Ethereum requires. This way the risk of (1) and (3) is reduced, as this library can be the obvious choice whenever a cryptographic function is needed, and if (2) were to happen, a single package should be updated to remove the native dependency from the entire ecosystem.

This approach introduces its own risks:

1. The chosen cryptographic dependencies may be less tested and have uncovered security vulnerabilities. To reduce this risk we chose the same dependencies that were already being used, whenever their implementation and interfaces made it possible.

2. The new code in the middleware package may add new vulnerabilities or expose the cryptographic functions in a dangerous way. To reduce this risk, we hired Trail Of Bits to conduct a security assessment. Its results can be seen here.

While creating ethereum-cryptography, we also made sure that it works well when targeting the web. This is done by choosing light dependencies that work well with browser bundlers, rebundling them when that's not the case, and testing the library with Webpack, Rollup, Parcel, and Browserify.

Deployment strategy

A naive approach to deploying these changes would be to simply release a new patch/minor of the latest versions of all the packages that depend on native libraries. The problem with it is that users stuck with older major versions of different tools won't benefit from this project until they upgrade, which may not be possible. Note that some of the libraries using native dependencies can be very deep in the dependency tree, which makes an uncoordinated upgrade unlikely.

To avoid this situation, we'll take a different approach. Instead of starting from the libraries, we'd do it from common development setups, analyze which libraries and versions they depend on, and release new minor/patch versions of those.

For example, a project A may depend on the library B, which in turn has a native dependency. If project B's latest version is 2.0.0, and the library A depends con B@^1.0.0, two versions of B need to be released, one in the 2.x branch, and another one in the 1.x. This way, both direct and indirect consumers of B benefit from this project.

The same approach will be used for non-cryptographic dependencies.

Current status

The analysis of de different setups and the plan to implement ethereum-cryptography through the ecosystem can be found here.

This final phase of the project is in charge of the Tenderly team, who will be submitting pull requests to the necessary projects. The only work required from the other projects is reviewing the changes and publishing new versions.

For comparison, ethereum-cryptography has been used in Buidler for a month without any inconvenience. The migration was trivial to implement and review.