Cross-Platform Standardisation

[CMCDragonkai]

Specification

We want PK to be truly cross-platform. But PK is very complex, involving many components are not truly cross-platform. These components include:

• Crypto code - this code often relies on native code to generate random numbers, to be performant and to be secure
• Networking - this code often relies on native or platform-specific network functionality
• Database - this code often relies on native or platform-specific filesystem behaviour
• IPC Locking - this code often relies on operating system POSIX features
• JS runtime - the runtime itself is platform-specific as it needs to run our javascript
• GUI - this code is often platform-specific due to the GUI chrome and components

Over the development process of PK, we've found many roadblocks to achieve a truly cross-platform PK system. This issue tracks all of these areas, and the progress towards more cross-platform deployment necessitating multiple new solutions in multiple areas.

• crypto - webcrypto or WASM can help
  • webcrypto provides a common foundation
  • wasm can provide additional crypto utilities that isn't part of webcrypto
• networking - using http3/quic may help
  • quic & http3 is the most likely future standard
  • different platforms may require native access to use
• database - indexed db, this may still require native code no matter what
  • leveldb - C/C++ codebase
  • rocksdb - more advanced than leveldb, can help implement block-level encryption
• ipc locking - this may still require native code no matter what because platforms define their own understanding of what inter-process means
• js runtime - understanding v8 can help because v8 is embeddable and jitless mode works on ios
  • deno is still very green, but it's interesting, it has v8, but will not support nodejs native addons
  • nodejs itself is embeddable, and uses v8
  • jitless v8 works on ios
• GUI - GUI is likely to be platform-specific no matter what, a number of solutions can bridge some platforms however, focusing on HTML/CSS standards should help
  • electron - for desktops only, results in huge packages
  • tauri - rust backend, much smaller packages (10 MiB vs 100 MiB), but can bundle nodejs https://tauri.studio/guides/building/sidecar/
  • nativescript - does not have all web standards available, may need to use native code or WASM - https://docs.nativescript.org/native-api-access.html
  • sciter - not open source
  • godot - game engine (completely different model https://medium.com/swlh/what-makes-godot-engine-great-for-advance-gui-applications-b1cfb941df3b)

This issue is likely to be a moving target, since cross platform development solutions keeps evolving.

Additional context

• Migrating/Integrating WebCrypto (possibly WASM) instead of Node Forge Crypto #270 - exploring webcrypto and wasm for crypto
• Transport Agnostic RPC #249 - RPC behaviour, moving towards json-rpc plus HTTP2/HTTP3 might be a better idea
• GRPC Web Proxy to enable Browser Integration and Eliminate Electron IPC #235 - grpc web proxy may not be necessary if we move away from GRPC
• Using QUIC/HTTP3 to replace utp-native for the Data Transfer Layer in the networking domain #234 - quic/http3 over utp-native
• Third Party Integration - HTTP API, OAuth2 Provider API, Plugin API, Library vs Framework, Smart Tokens #166 - third party integration should be easier with json-rpc than with grpc
• React Native and Mobile OS (iOS and Android) Compatibility #155 - mobile/nativescript compatibility

Tasks

1. ...
2. ...
3. ...

[CMCDragonkai]

HTTP2 servers are compatible with HTTP1.1 with TLS. The way this works is through ALPN. This is done at the TLS-level. See: https://superuser.com/questions/1659248/how-does-browser-know-which-version-of-http-it-should-use-when-sending-a-request

In Nodejs the HTTP2 module is capable of accepting HTTP1.1 with TLS connections: https://nodejs.org/api/http2.html#alpn-negotiation.

• HTTP1.1 client to HTTP2 server - continues but can optionally upgrade to HTTP2 client
• HTTP2 client to HTTP2 server - continues as normal
• HTTP2 and HTTP1.1 client to HTTP2 server - selects HTTP2 during ALPN

If an HTTP3 server is available, an HTTP2 client would have to be "upgraded" to the HTTP3 protocol. It is not possible for an HTTP2 client to directly contact HTTP3, because it is a UDP port.

That means if we are using tunneling http2 which multiplex our jsonrpc into utp-native. Our server side should still receive requests outside of utp-native as just normal http1.1+TLS or http2 on TCP. In fact this is exactly how our GRPC server does right now, it accepts connections on TCP and through the utp-native proxy tunnel.

[CMCDragonkai]

Moving this to main Polykey repository now.

[CMCDragonkai]

While working on #466, I discovered:

• Is there any plans to add WebCrypto API to React-Native? react-native-community/discussions-and-proposals#83
• Get random values facebook/react-native#20686
• feat: rewrite to C++ with JSI LinusU/react-native-get-random-values#40.

As you can see there are some libraries for webcrypto in react native, however they are not fully supported, and some rely on webview bridges. It seems there are 3 possibilities:

1. Webview bridge - probably the slowest
2. Native modules Java and Objective C
3. C++ using JSI - I like this method the best if we are going to use crypto in particular
4. WASM - this would still need native system methods for getting useful entropy

[CMCDragonkai]

In our latest PR #446 we are making use of webcrypto API. However the actual implementation is backed by @peculiar/webcrypto which is a polyfill. In particular it has a different implementation of the Ed25519 algorithms. This is because neither browser webcrypto nor nodejs have a common implementation of Ed25519 and X25519. Nodejs has its own experimental implementation that is not compatible: PeculiarVentures/webcrypto#55

The polyfill is designed for nodejs, it builds up a webcrypto API based on node's crypto API. This is therefore not going to be useful in the case of react native.

There's an additional library: https://github.com/PeculiarVentures/webcrypto-liner that polyfills browser implementations in the same vein. Again not entirely useful for mobile OS, since we don't work directly in a browser. And electron would use the node polyfill anyway.

The current benchmarks show some significant improvement:

Current performance notes:

1. Symmetric operations is 10x improvement
2. Random bytes generation is 100x improvement
3. Generating root key is 3563x improvement (deterministic is 100x - 400x faster due to pbkdf2 is still slow)
4. Asymmetric encryption didn't change much, but the fact that you can now encrypt arbitrary data sizes is significant.
5. Asymmetric decryption is is 68x faster
6. Asymmetric signing is 300x faster
7. Asymmetric verification didn't change much

At the very least we have a situation where we expect there to be a globalThis.crypto object that facilitates webcrypto API, in TS, this is the Crypto type. This allows all of our libraries that support webcrypto to leverage that instead of using other crypto APIs, this includes all of the noble libraries and JOSE libraries.

Now as I said before, we still have to deal with mobile OS, and we deal with that in still the 4 ways:

1. Webview bridge - probably the slowest
2. Native modules Java and Objective C
3. C++ using JSI - I like this method the best if we are going to use crypto in particular
4. WASM - this would still need native system methods for getting useful entropy

I want to provide some possible avenues of approach for each:

1. https://github.com/webview-crypto/react-native-webview-crypto
2. Native modules
   • https://github.com/lyubo/react-native-sodium
3. C++ using JSI
   • https://github.com/standardnotes/react-native-sodium-jsi
4. WASM
   • https://github.com/tonyg/js-nacl
   • https://github.com/jedisct1/libsodium.js

The webview looks unmaintained but it could be done.

What is of interest is the libsodium library. It is claimed that libsodium is meant for cross-platform usage. Look at all the available bindings: https://doc.libsodium.org/bindings_for_other_languages. The fact that there are multiple implementations and the fact that they have been made WASM capable, as well as native node implementations of sodium should meant that libsodium might be exactly what we've been looking for.

Because our libraries still expect a webcrypto API, we may just need to reimplement the webcrypto API using an underlying libsodium library which then means:

1. Top-level libraries: JOSE, @noble, @Scure... etc
2. Webcrypto interface monkey patched as globalThis.crypto
3. Libsodium implementation of webcrypto - provide both native variant from nodejs, as well as WASM variant or native variant for react native

Given the current benchmark numbers, we now have a way of comparing if any particular choice will maintain decent performance of our required crypto primitives.

[CMCDragonkai]

I'd be curious exactly how fast/slow the WASM variants are compared to native code. We're using webcrypto now which I believe ultimately uses node's openssl code which is native, but how much are we losing by using WASM. This allows us to know if it's feasible to use WASM, especially since on ios there's no JIT.

[CMCDragonkai]

In order to use any of the WASM based libraries, we have to have top level await. This is because the wasm libraries all require asynchronous initialisation.

Without a top level await, we would end up having to export promises.

Now top level await is only available as part of the greater migration towards EcmaScript modules over CJS modules.

The transition to ESM is going to be tricky, but this https://www.typescriptlang.org/docs/handbook/esm-node.html provides more information on how to do this.

We have an issue tracking this in MatrixAI/TypeScript-Demo-Lib#32.

Since we are still inter-operating with old CJS modules, most likely this means we are using NodeNext for both moduleResolution and module.

However our ts-node doesn't yet support ESM yet, so until we also change that over if they finally acquire support for this. See: TypeStrong/ts-node#1007

At any case, since WASM is unlikely to be ready for mobile deployment, we're likely to not bother with the WASM variants of libsodium anyway.

[CMCDragonkai]

I've moved to using libsodium in #446 (comment). However there are 2 places that still require webcrypto. Future work should work on these systems to remove the reliance on webcrypto.

In this case, all crypto related functionality will be on top of libsodium.

[CMCDragonkai]

A few other things:

1. js-quic still failing on windows build
2. js-db needs to be turned into using optional dependencies
3. js-mdns now has some code that is only compiled for linux

Some things to factor out:

1. js-ws - this needs to be factored out of Polykey, as interim we can use just the ws node package, but later convert it to a runtime-less library like quiche

#526 is really important cause lots of things are using TLS, and centralising the TLS would be essential:

• js-quic
• js-ws
• Any HTTP related stuff

If we want to centralise to particular TLS implementation, we might even need a separate http related implementation... or at least redirect its usage of TLS by plugging our own TLS to mock over the TLS used by http/http2 module?

One could potentially override nodejs with boringssl alternative, then force the usage of that in js-quic. Will need to dive deeper into how nodejs is compiled to achieve this.