Packaging up js-quic for Linux, Windows, MacOS

[CMCDragonkai]

Specification

As js-quic is a native package, it has much more complicated packaging procedure making it similar to js-db. However this is different because it uses Rust instead of C++, therefore the entire toolchain is different from js-db.

Rather than tsc, node, node-gyp, napi-macros and node-gyp-build. This is just tsc, node and napi-rs.

What needs to be done is:

1. Remove initial experimental scaffolding that was using neon. For example the neon-cli package is no longer needed and cargo-cp-artifact is also not required.
2. Remove initial experimental scaffolding that was using napi-macros and node-addon-api and node-gyp. All of these packages and related configuration is no longer needed. This includes removing node-gyp-build too.
3. We are only using napi-rs and the related rust and cargo tools are brought in from the shell.nix.
4. Unrelated and useless packages should be removed like threads.
5. The npm run napi-build and npm run napi-build-prod should be integrated into the npm run prebuild script located in src/scripts/prebuild.js.
6. The entire prebuild.js was ported from js-db which uses node-gyp. This does not use node-gyp at all, and instead needs to use napi build command instead. All relevant flags and options should be adapted to napi build which comes from the @napi-rs/cli package.
7. The purpose of prebuild.js is be able to build binaries and put them into a prebuild directory. This directory is packaged as part of the npm distribution, and so binary compiled artifacts are "prebuilt" before they are downloaded by downstream users. We should maintain this structure.
8. There is some work on native packages that spread their binary artifacts into different NPM pacakges, and for the npm to install the ones that are required, thus reducing the final closure size. Haven't experimented with this yet, but I see some other native packages making use of this pattern.
9. Currently the napi build ends up putting the artifact into quic.linux-x64-gnu.node on the project root. This should be put into the equivalent prebuild directories that js-db does.
10. If we do use the same prebuild directories... we might recover the ability to use node-gyp-build to look it up. But right now all that logic is written into src/native/quiche.ts. Alternatively we give up on node-gyp-build entirely and just use our own script to find the right bindings.

Note that on MacOS and Windows we are using homebrew and chocolatey respectively. This means the rust toolchain would come from those 2 areas instead of Nix. No way to use Nix on MacOS just yet, so it's just a good idea to ensure that we are in fact building with the same Rust version.

Additional context

• Clean up superfluous compilation flags js-db#43 - superfluous compilation flags are no longer a thing with the Rust toolchain, alot more efficient
• Cross-Compilation for linux-arm64 and win-arm64 TypeScript-Demo-Lib-Native#3
• MacOS Runner Nix Integration for Platform-Specific Build TypeScript-Demo-Lib-Native#1
• Follow how js-db looks and feels at the end, just knowing that rust toolchain is more streamlined
• Darwin universal architecture napi-rs/napi-rs#1397 - How to build universal binaries with napi.

Tasks

• 1. Remove initial experimental scaffolding that was using neon. For example the neon-cli package is no longer needed and cargo-cp-artifact is also not required.
• 2. Remove initial experimental scaffolding that was using napi-macros and node-addon-api and node-gyp. All of these packages and related configuration is no longer needed. This includes removing node-gyp-build too.
• 3. We are only using napi-rs and the related rust and cargo tools are brought in from the shell.nix.
• 4. Unrelated and useless packages should be removed like threads.
• 6. The npm run napi-build and npm run napi-build-prod should be integrated into the npm run prebuild script located in src/scripts/prebuild.js.
• 7. The entire prebuild.js was ported from js-db which uses node-gyp. This does not use node-gyp at all, and instead needs to use napi build command instead. All relevant flags and options should be adapted to napi build which comes from the @napi-rs/cli package.
• 8. The purpose of prebuild.js is be able to build binaries and put them into a prebuild directory. This directory is packaged as part of the npm distribution, and so binary compiled artifacts are "prebuilt" before they are downloaded by downstream users. We should maintain this structure.
• 9. There is some work on native packages that spread their binary artifacts into different NPM pacakges, and for the npm to install the ones that are required, thus reducing the final closure size. Haven't experimented with this yet, but I see some other native packages making use of this pattern.
• 10. Currently the napi build ends up putting the artifact into quic.linux-x64-gnu.node on the project root. This should be put into the equivalent prebuild directories that js-db does.
• 11. If we do use the same prebuild directories... we might recover the ability to use node-gyp-build to look it up. But right now all that logic is written into src/native/quiche.ts. Alternatively we give up on node-gyp-build entirely and just use our own script to find the right bindings.
• Test on CI/CD

[CMCDragonkai]

Nothing on mobile just yet... but that will be important in the future. Mobile usage is complicated by the fact that the same code will need to compile to target the mobile platforms, and the bindings won't be napi-rs since it's not an Node API, instead native modules like in React would need to be used.

[CMCDragonkai]

Branch and tag protections have been applied to this repo now.

[CMCDragonkai]

Will temporarily rebase and squash the master branch later in order to clean up the history too.

[CMCDragonkai]

@tegefaulkes to track this.

[CMCDragonkai]

The codesee maps is integrated into this https://app.codesee.io/maps/c108c990-f46b-11ed-ad69-ede85191d1b1.

Gitlab pull mirroring isn't working, so this requires manual pushes atm.

git remote add cicd git@gitlab.com:MatrixAI/open-source/js-quic.git
git push cicd
git push cicd <branch>

[CMCDragonkai]

History has been squashed... and reset. Both staging and master is now on the same commits.

@tegefaulkes can you try to rebuild/re-run any tests to see I didn't screw it up. Just clone the repo to a new directory before trying anything, then if it is all good you can delete both repos and just clone from the new repo.

[CMCDragonkai]

Currently CI/CD is failing... need to investigate why. I haven't reviewed any of the tests or whether the code is linted or not, and I don't know if mac/windows build scripts will work. These will need to be debugged on our local systems in office.

[tegefaulkes]

just ran the tests, I'm getting the following.

[nix-shell:~/matixWorkspace/gitRepos/js-quic]$ npm run test

> @matrixai/quic@0.0.1 test
> jest

Determining test suites to run...
GLOBAL SETUP
 PASS  tests/utils.test.ts (6.647 s)
  utils
    ✓ detect IPv4 mapped IPv6 addresses (24 ms)
    ✓ detect IPv4 mapped IPv6 Dec addresses (9 ms)
    ✓ detect IPv4 mapped IPv6 Hex addresses (12 ms)
    ✓ to IPv4 mapped IPv6 addresses Dec (1 ms)
    ✓ to IPv4 mapped IPv6 addresses Hex (3 ms)
    ✓ from IPv4 mapped IPv6 addresses (3 ms)
    ✓ to canonical IP address (4 ms)
    ✓ resolves zero IP to local IP (4 ms)

 FAIL  tests/QUICSocket.test.ts
  ● Test suite failed to run

    Failed requiring possible native bindings: quic-linux-x64.node,@matrixai/quic-linux-x64

      73 |     }
      74 |   }
    > 75 |   throw new Error(
         |         ^
      76 |     `Failed requiring possible native bindings: ${prebuildTargets.concat(npmTargets)}`
      77 |   );
      78 | }

      at requireBinding (src/native/quiche.ts:75:9)
      at Object.<anonymous> (src/native/quiche.ts:117:25)
      at Object.<anonymous> (src/native/index.ts:1:1)

 FAIL  tests/QUICClient.test.ts
  ● Test suite failed to run

    Failed requiring possible native bindings: quic-linux-x64.node,@matrixai/quic-linux-x64

      73 |     }
      74 |   }
    > 75 |   throw new Error(
         |         ^
      76 |     `Failed requiring possible native bindings: ${prebuildTargets.concat(npmTargets)}`
      77 |   );
      78 | }

      at requireBinding (src/native/quiche.ts:75:9)
      at Object.<anonymous> (src/native/quiche.ts:117:25)
      at Object.<anonymous> (src/native/index.ts:1:1)

 FAIL  tests/QUICStream.test.ts
  ● Test suite failed to run

    Failed requiring possible native bindings: quic-linux-x64.node,@matrixai/quic-linux-x64

      73 |     }
      74 |   }
    > 75 |   throw new Error(
         |         ^
      76 |     `Failed requiring possible native bindings: ${prebuildTargets.concat(npmTargets)}`
      77 |   );
      78 | }

      at requireBinding (src/native/quiche.ts:75:9)
      at Object.<anonymous> (src/native/quiche.ts:117:25)
      at Object.<anonymous> (src/native/index.ts:1:1)

 FAIL  tests/concurrency.test.ts
  ● Test suite failed to run

    Failed requiring possible native bindings: quic-linux-x64.node,@matrixai/quic-linux-x64

      73 |     }
      74 |   }
    > 75 |   throw new Error(
         |         ^
      76 |     `Failed requiring possible native bindings: ${prebuildTargets.concat(npmTargets)}`
      77 |   );
      78 | }

      at requireBinding (src/native/quiche.ts:75:9)
      at Object.<anonymous> (src/native/quiche.ts:117:25)
      at Object.<anonymous> (src/native/index.ts:1:1)

Test Suites: 4 failed, 1 passed, 5 total
Tests:       8 passed, 8 total
Snapshots:   0 total
Time:        7.267 s, estimated 35 s
Ran all test suites.
GLOBAL TEARDOWN

[CMCDragonkai]

That's cause you need to build it now npm run prebuild. We don't use the one that is being put into the project root anymore.

[CMCDragonkai]

Some of your stream tests are really slow btw. We need to review all tests and make them pass.

I've just run them too and I get:

[nix-shell:~/Projects/js-quic]$ npm run test

> @matrixai/quic@0.0.1 test
> jest

Determining test suites to run...
GLOBAL SETUP
ERROR:QUICClient Test.QUICClient.QUICConnection afa13aa3:Connection timed out
ERROR:QUICClient Test.QUICClient:ErrorQUICConnectionTimeout
ERROR:QUICClient Test.QUICClient.QUICConnection 9257e992:recv error TlsFail
ERROR:QUICClient Test.QUICClient:ErrorQUICConnectionTLSFailure
ERROR:QUICClient Test.QUICClient.QUICConnection b9f45a27:recv error TlsFail
ERROR:QUICClient Test.QUICClient:ErrorQUICConnectionTLSFailure
ERROR:QUICClient Test.QUICClient.QUICConnection 191abda0:recv error TlsFail
ERROR:QUICClient Test.QUICClient:ErrorQUICConnectionTLSFailure
ERROR:QUICClient Test.QUICClient.QUICConnection 44d016bf:recv error TlsFail
ERROR:QUICClient Test.QUICClient:ErrorQUICConnectionTLSFailure
ERROR:QUICClient Test.QUICClient.QUICConnection ae8b5bb4:recv error TlsFail
ERROR:QUICClient Test.QUICClient:ErrorQUICConnectionTLSFailure
ERROR:QUICClient Test.QUICClient.QUICConnection 18af7ff4:recv error TlsFail
ERROR:QUICClient Test.QUICClient:ErrorQUICConnectionTLSFailure
ERROR:QUICClient Test.QUICClient.QUICConnection 0831c291:Connection timed out
ERROR:QUICClient Test.QUICServer.QUICConnection 71583001-0831c291:Connection timed out
ERROR:QUICClient Test.QUICClient.QUICConnection 9c27faeb:Connection timed out
ERROR:QUICClient Test.QUICServer.QUICConnection 7bae4ed5-9c27faeb:Connection timed out
ERROR:QUICClient Test.QUICClient.QUICConnection c9ab4e7e:Connection timed out
ERROR:QUICClient Test.QUICClient:ErrorQUICConnectionTimeout
 PASS  tests/QUICClient.test.ts (25.664 s)
  QUICClient
    ✓ times out when there is no server (3042 ms)
    ✎ todo client times out after connection stops responding
    ✎ todo server times out after connection stops responding
    ✎ todo server handles socket error
    ✎ todo client handles socket error
    dual stack client
      ✓ to ipv4 server succeeds (with seed=1978936421) (429 ms)
      ✓ to ipv6 server succeeds (with seed=1161311110) (385 ms)
      ✓ to dual stack server succeeds (with seed=-114001202) (333 ms)
    TLS rotation
      ✓ existing connections config is unchanged and still function (with seed=-302456834) (331 ms)
      ✓ new connections use new config (with seed=-815516438) (584 ms)
    graceful tls handshake
      ✓ server verification succeeds (with seed=565140315) (324 ms)
      ✓ client and server verification succeeds (with seed=1464301031) (421 ms)
      ✓ graceful failure verifying server (with seed=1245969200) (6061 ms)
      ✓ graceful failure verifying client and server (with seed=681974782) (3078 ms)
      ○ skipped client verification succeeds (with seed=731304151)
      ○ skipped graceful failure verifying client (with seed=850795266)
    UDP nat punching
      ✓ server can send init packets (2002 ms)
      ✓ init ends when connection establishes (1038 ms)
      ✓ init returns with existing connections (36 ms)
    handles random packets
      ✓ client handles random noise from server (with seed=1188545511) (119 ms)
      ✓ client handles random noise from external (with seed=-887653356) (108 ms)
      ✓ server handles random noise from client (with seed=-1570099967) (82 ms)
      ✓ server handles random noise from external (with seed=1386981673) (119 ms)
    keepalive
      ✓ connection can time out on client (210 ms)
      ✓ connection can time out on server (210 ms)
      ✓ keep alive prevents timeout on client (336 ms)
      ✓ keep alive prevents timeout on server (340 ms)
      ✓ client keep alive prevents timeout on server (339 ms)
      ✓ Keep alive does not prevent connection timeout (3000 ms)

 PASS  tests/QUICSocket.test.ts
  QUICSocket
    ✓ enabling ipv6 only prevents binding to ipv4 hosts (13 ms)
    ✓ disabling ipv6 only does not prevent binding to ipv6 hosts (1 ms)
    ✓ ipv4 wildcard to ipv4 succeeds (1 ms)
    ✓ ipv6 wildcard to ipv6 succeeds (1 ms)
    ✓ socket should throw if stopped with active connections (17 ms)
    ✓ socket should stop when forced with active connections (1 ms)
    ipv4
      ✓ type will be `ipv4` (2 ms)
      ✓ to ipv4 succeeds (3 ms)
      ✓ to ipv6 fails (11 ms)
      ✓ to dual stack succeeds (1 ms)
    ipv6
      ✓ type will be `ipv6` (1 ms)
      ✓ to ipv6 succeeds (1 ms)
      ✓ to ipv4 fails (2 ms)
      ✓ to dual stack succeeds (1 ms)
    ipv6 only when using `::`
      ✓ type will be `ipv6` (1 ms)
      ✓ to ipv4 fails (1 ms)
      ✓ to ipv6 succeeds (1 ms)
      ✓ to dual stack succeds (1 ms)
    dual stack
      ✓ type will be `ipv4&ipv6` (1 ms)
      ✓ to ipv4 succeeds (1 ms)
      ✓ to ipv6 succeeds (1 ms)
      ✓ to dual stack succeeds (2 ms)
    ipv4 mapped ipv6 - dotted decimal variant
      ✓ type will be `ipv4`
      ✓ to ipv4 fails (1 ms)
      ✓ to ipv6 fails (1 ms)
      ✓ to ipv4 mapped ipv6 succeeds (2 ms)
    ipv4 mapped ipv6 - hex variant
      ✓ type will be `ipv4`
      ✓ to ipv4 fails (1 ms)
      ✓ to ipv6 fails (1 ms)
      ✓ to ipv4 mapped ipv6 succeeds (1 ms)

 PASS  tests/QUICStream.test.ts (29.541 s)
  QUICStream
    ✓ should create streams (with seed=-1671438937) (16000 ms)
    ✓ destroying stream should clean up on both ends while streams are used (with seed=899455710) (4186 ms)
    ✓ should send data over stream (with seed=-246057904) (1243 ms)
    ✓ should propagate errors over stream for writable (with seed=1029918994) (2079 ms)
    ✓ should clean up streams when connection ends (with seed=-2141210820) (4527 ms)
    ✓ streams should contain metadata (with seed=-874283823) (57 ms)
    ✓ streams can be cancelled (with seed=-420901957) (852 ms)
    ○ skipped should propagate errors over stream for readable (with seed=1406844894)

 PASS  tests/concurrency.test.ts (11.345 s)
  Concurrency tests
    ✓ Multiple clients connecting to a server (with seed=-1095171874) (3469 ms)
    ✓ Multiple clients sharing a socket (with seed=1573595033) (6069 ms)
    ✓ Multiple clients sharing a socket with a server (with seed=348828160) (1299 ms)

 PASS  tests/utils.test.ts
  utils
    ✓ detect IPv4 mapped IPv6 addresses (4 ms)
    ✓ detect IPv4 mapped IPv6 Dec addresses (1 ms)
    ✓ detect IPv4 mapped IPv6 Hex addresses (2 ms)
    ✓ to IPv4 mapped IPv6 addresses Dec
    ✓ to IPv4 mapped IPv6 addresses Hex (2 ms)
    ✓ from IPv4 mapped IPv6 addresses (4 ms)
    ✓ to canonical IP address (2 ms)
    ✓ resolves zero IP to local IP (3 ms)

Test Suites: 5 passed, 5 total
Tests:       3 skipped, 4 todo, 71 passed, 78 total
Snapshots:   0 total
Time:        67.265 s
Ran all test suites.
GLOBAL TEARDOWN

[CMCDragonkai]

One of the problems with using brew is that it doesn't always have exact versions. In the case of rust we only have just stable. Long term solution is to move to our own CI/CD system using our own machines with nix on them including macos. Because we cannot afford time to setup nix on gitlab's macos machines.