Precompiled Binary Hosting

[reconbot]

As of today, the state of the art of distributing binary packages is done using node-pre-gyp or prebuild. They require pre-compiling all the platforms and ABI versions you want to support and uploading them to github or s3.

While it's technically possible for users to compile modules themselves it's usually a large burden for users and a primary source of support issues in my case. Additionally while it's technically possible to ship all precompiled binaries in the npm package itself, it requires a bit of operations work as you have to somehow consolidate all builds onto a publishing machine.

So prebuilding the binaries works but it requires external web hosting (sually s3 or github releases) to host the binary files. With github releases I've had org and repo changes break the download for old modules. And I've seen projects loose their aws account and now old versions wont install.

I'd like to propose that NPM host the binary files and allow them to be added to a package at any time post publishing. This would allow standardizing hosting of modules alongside the javascript. I think whatever the future holds for seamless binary support for nodejs, this is a good first step.

[ljharb]

Allowing them to be added at any time seems like it’d be a risk that a compromised author could add malicious binaries to existing versions.

[reconbot]

Maybe don't let files be replaced? Coordinating the upload before publish is a leap as it has to be built on at least 3 different platforms and sometimes takes some time. Also as new versions of node and electron are released it would be nice to be able to publish binaries to support without having to force an upgrade of the package.

[ljharb]

Right, but if there's no custom build for someone's platform, and a bad actor suddenly adds one, then suddenly those users would be running malicious code on install.

[reconbot]

Fair enough, I’d like some allowances for uploading after the fact as it would incredibly improve the workflow and match all available tooling behavior today.

[MylesBorins]

Hey @reconbot thanks for opening this discussion!

Definitely an interesting problem space. I'm going to spend some cycles thinking about solutions to this. One thing I'm wondering, GitHub does allow you to upload artifacts to a release... I wonder if there is a way we could use a release triggered action to upload these resources to the GitHub release, which should be a canonical URL.

I recognize this is not an integrated solution, but i'm thinking it might be sufficient. Thoughts?

[reconbot]

That’s pretty much how a lot of people are dealing with it today and is what I’m referring to when I say they’re uploaded to GitHub. These do break over time and aren’t linked to the package itself.

[MarshallOfSound]

Using GitHub releases is tricky because they are by-design mutable. A malicious actor can upload a different binary to a github release at any time. If the package-lock or yarn.lock file contains a checksum for these binary files that might be a good option though

[reconbot]

It’s quite difficult to orchestrate building on so many systems (I’m currently over 100 builds a release across 4 platforms) and then getting the files to once place and to publish them together. Which would be required for checksums. If it was easy I’d put all the builds inside the tarball and skip external hosting all together.

The upload to a release and be happy about what makes it there has been a godsend. I’d want to look for a developer experience that makes us both happy.

[mcollina]

@MylesBorins check out the approach used in https://github.com/mcollina/native-hdr-histogram. Note that we bundle all binaries in the same npm package. This provides a good dev experience but it results in unnecessary bytes being moved through the network. Providing a way to publish binaries together with the module would be great.

[MylesBorins]

Thanks for the input y'all. Definitely recognize that mutability of GitHub does make using Releases for this a challenge. Going to follow up internally with some folks and see what options we have to explore here.