Integrate a Node native module like utp-native or leveljs in order to demonstrate vercel/pkg

[CMCDragonkai]

The vercel/pkg has successfully been implemented and is now producing pre-release packages in our release page.

However we have yet to test how it works when dealing with native modules like utp-native.

We need to add in utp-native and leveljs as demonstrations of building a CLI executable with native code.

This will be essential for js-polykey to be capable of producing release executables.

1. Integrate leveldb into TypeScript-Demo-Lib
2. Integrate utp-native into TypeScript-Demo-Lib
3. Perform a hello world equivalent of using these 2 libraries, for leveldb that's CRUD on the database, for utp-native we would need to start a server, and close the server.
4. Then use vercel/pkg to package up the program
5. Sanity check the release
6. Fix up any native module configuration required for vercel/pkg

Checklist:

• I've done some reading on native modules and vercel/pkg.
• cloned the TypeScript-Demo-Lib, built and ran it to see it working.
• built TypeScript-Demo-Lib with pkg and ran it in linux to check it is working.
• Written a hello world test for level.
• written a hello world test for utp-native.
• pkg with level and test it works.
• pkg with utp-native and test it works.
• create a test for worker threads.
• pkg with worker threads and test it works

---

Relevant upstream PR vercel/pkg#1273

[CMCDragonkai]

The vercel/pkg has been integrated into release.nix.

To see how to use it, see how the CI/CD uses it: https://github.com/MatrixAI/TypeScript-Demo-Lib/blob/master/.gitlab-ci.yml

[CMCDragonkai]

This has been merged upstream NixOS/nixpkgs#128967

So please also update the pkgs.nix to point to the latest master pin.

[CMCDragonkai]

Another library that is point of concern is js-encryptedfs.

The EFS uses threads.js. Which has a dependency on loading a file from its current directory.

See how in src/workers/WorkerManager it uses:

    this.pool = Pool(() => spawn(new Worker('./efsWorker')), coreCount);

This then has to load a file ./efsWorker that is relative to the script that contains that statement. Which is src/workers/efsWorker.ts.

That script will need to exist in the vercel/pkg virtual file system when it is packaged up.

We need to test js-encryptedfs inside TypeScript-Demo-Lib and see if vercel/pkg can properly package this up.

Note that there are notes about this problem in relation to webpack which has a similar problem:

• https://threads.js.org/usage#typescript-workers-in-nodejs
• https://threads.js.org/getting-started#build-with-webpack

We might need to do something similar for vercel/pkg.

[CMCDragonkai]

Beware of https://github.com/vercel/pkg#snapshot-filesystem and https://github.com/vercel/pkg#scripts

This might require some digging into the vercel/pkg source code.

Use it directly from npm by doing npm install --save-dev pkg. And then just use pkg by itself.

I had previously cloned the repo entirely, and was changing its source code and then calling it from that repo. That's also possible, never assume that upstream dependencies are immutable, we can always fork them and make changes to them.

[tegefaulkes]

Progress:

• I've done some reading on native modules and vercel/pkg.
• cloned the TypeScript-Demo-Lib, built and ran it to see it working.
• built TypeScript-Demo-Lib with pkg and ran it in linux to check it is working.
• Written a hello world test for level.
• written a hello world test for utp-native.

Still do to:

• pkg with level and test it works.
• pkg with utp-native and test it works.
• create a test for worker threads.
• pkg with worker threads and test it works

[tegefaulkes]

I was able to get level to work. But you need to include the leveldown/prebuilds folder with the executable.
Folder structure looks like

• linux-x64 //Executable.
• prebuilds
  • linux-x64

[CMCDragonkai]

Can you integrate this into our CI/CD and Nix release builds? What changes would be required?

…

On 7/20/21 1:35 PM, Brian Botha wrote: I was able to get level to work. But you need to include the |leveldown/prebuilds| folder with the executable. Folder structure looks like * linux-x64 //Executable. * prebuilds o linux-x64 — You are receiving this because you were assigned. Reply to this email directly, view it on GitHub <#27 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAE4OHN5GAIXJQLX6OFV6CLTYTVJ5ANCNFSM47Y2CL5Q>.

[tegefaulkes]

utp-native works if I add the following lines to package.json:

{
    //...
    "pkg": {
        "assets": "node_modules/utp-native/**/*"
    },
}

[tegefaulkes]

Can you integrate this into our CI/CD and Nix release builds? What changes would be required?
…
On 7/20/21 1:35 PM, Brian Botha wrote: I was able to get level to work. But you need to include the |leveldown/prebuilds| folder with the executable. Folder structure looks like * linux-x64 //Executable. * prebuilds o linux-x64 — You are receiving this because you were assigned. Reply to this email directly, view it on GitHub <#27 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAE4OHN5GAIXJQLX6OFV6CLTYTVJ5ANCNFSM47Y2CL5Q.

Unless we can find a way to include this inside the executable. We will have to copy the prebuilds over a possibly create a zip archive.