Create QUIC library that can be exposed to JS and uses the Node `dgram` module

[CMCDragonkai]

Specification

We need QUIC in order to simplify our networking stack in PK.

QUIC is a superior UDP layer that can make use of any UDP socket, and create multiplexed reliable streams. It is also capable of hole punching either just by attempting to send ping frames on the stream, or through the unreliable datagrams.

Our goals is to make use of a QUIC library, something that is compilable to desktop and mobile operating systems, expose its functionality to JS, but have the JS runtime manage the actual sockets.

On NodeJS, it can already manage the underlying UDP sockets, and by relying on NodeJS, it will also ensure that these sockets will mix well with the concurrency/parallelism used by the rest of the NodeJS system due to libuv and thus avoid creating a second IO system running in parallel.

On Mobile runtimes, they may not have a dgram module readily available. In such cases, having an IO runtime to supply the UDP sockets may be required. But it is likely there are already existing libraries that provide this like https://github.com/tradle/react-native-udp.

The underlying QUIC library there is expected to be agnostic to the socket runtime. It will give you data that you need to the UDP socket, and it will take data that comes of the UDP socket.

However it does need 2 major duties:

1. The multiplexing and managing of streams.
2. The encryption/decryption TLS side

Again if we want to stay cross platform, we would not want to bind into Node.js's openssl crypto. It would require instead that the library can take a callback of crypto routines to use. However I've found that this is generally not the case with most existing QUIC libraries. But let's see how we go with this.

Additional context

• Using QUIC/HTTP3 to replace utp-native for the Data Transfer Layer in the networking domain Polykey#234 - This is the main issue talking about how to replace utp-native
• https://github.com/napi-rs/napi-rs/tree/main/examples/napi - examples of how to use napi-rs
• https://nodejs.org/api/dgram.html - dgram module in NodeJS where the QUIC data will be plumbed to
• https://blogs.keysight.com/blogs/tech/nwvs.entry.html/2021/07/17/looking_into_quicpa-pUtF.html - some notes about how QUIC data can be viewed on wireshark
• https://www.sitepoint.com/rust-global-variables/
• https://deepu.tech/memory-management-in-rust/
• Support custom certificate verification cloudflare/quiche#326 (comment) - Disabling TLS verification and doing it custom
• Required options Clarify which configuration options are mandatory cloudflare/quiche#1250
• https://blog.cloudflare.com/head-start-with-quic/
• https://github.com/romain-jacotin/quic/blob/master/doc/QUIC_crypto_protocol.md

QUIC and NAPI-RS

• Why does failing to call a callback result in a panic? napi-rs/napi-rs#1373 - callback usage
• Callback usage in factory method results in macro error napi-rs/napi-rs#1374 - Callback usage in factory methods
• Support custom certificate verification cloudflare/quiche#326 (comment) - custom TLS verification in quiche
• When using `External<...>` the `...` ends up in the `index.d.ts` as an unknown type napi-rs/napi-rs#1409 - how to deal with external object types
• How to create a submodule in the rust code? napi-rs/napi-rs#1379 - napi-rs can't export submodules in rust for now

Sub issues

• Enable TLS Configuration with in-memory PEM strings #2
• TLS Rotation for QUICServer and QUICClient #3
• Allow Bidirectional Hole Punching from QUICClient and QUICServer #4
• Fixed size buffer for QUICStream instead of object stream #5
• Keepalive component after QUIC connection is established #6
• Packaging up js-quic for Linux, Windows, MacOS #7
• Create TLS cert arbitraries for testing #8
• Graceful connection failure for failed TLS handshake #9
• Create tests for QUICStreams #10
• Check if application protocols are required #13
• Test connection multiplexing across multiple servers and clients #14
• Create benchmarks for sending data #15
• Propagate connection info along side stream creation #16
• Server selects certificate based on client's support #17
• Canonical form for IP addresses #18

Tasks

1. Experiment with Neon or Napi-rs
2. Experiment with quiche by cloudflare
3. Create bridge code plumbing UDP sockets and QUIC functions
4. Create self signed TLS certificate during development - Create QUIC library that can be exposed to JS and uses the Node dgram module #1 (comment)
5. Extract out the TLS configuration so that it can be set via in-memory PEM variable and key variable. - 2 day - see Enable TLS Configuration with in-memory PEM strings #2
   • This should be doable with https://docs.rs/quiche/latest/quiche/struct.Config.html#method.with_boring_ssl_ctx
   • This needs to construct a SSL ctx with boring ssl, and pass in PEM string and key string (or Uint8Array) to be processed by the boringssl library.
6. Decide whether application protocols are necessary here, or abstract the quiche Config so the user can decide this (especially since this is not a HTTP3 library). - 0.5 day - see Check if application protocols are required #13
7. Fix the timeout events and ensure that when a timeout occurs, that the connection gets cleaned up, and we are not inadvertently clearing the timeout due to null. Right now when a quiche client connects to the server, even after closing, the server side is keeping the connection alive. - 1 day
8. We need lifecycle events for QUICConnection and QUICStream and QUICServer and QUICSocket. This will allow users to hook into the destruction of the object, and perhaps remove their event listeners. These events must be post-facto events. - 0.5 day
9. [ ] Test the QUICStream and change to BYOB style, so that way there can be a byte buffer for it. Testing code should be able generator functions similar to our RPC handlers. - 1 day - see Fixed size buffer for QUICStream instead of object stream #5.
10. Complete the QUICClient with the shared socket QUICSocket. - 3 day
11. Test the multiplexing/demultipexing of the UDP socket with multiple QUICClient and a single QUICServer. - 1 day - See Test connection multiplexing across multiple servers and clients #14
12. [ ] Test the error handling of the QUIC stream, and life cycle and destruction routines. - 1 day - Create tests for QUICStreams #10
13. Benchmark this system, by sending lots of data through. - 1 day - See Create benchmarks for sending data #15
14. Propagate the rinfo from the UDP datagram into the conn.recv() so that the streams (either during construction or otherwise) can have its rinfo updated. Perhaps we can just "set" the rinfo properties of the connection every time we do a conn.recv(). Or... we just mutate the conn parameters every time we receive a UDP packet. - See Propagate connection info along side stream creation #16
15. Ensure that when a user asks stream.connection they can acquire the remote information and also all the remote peer's certificate chain. - See Propagate connection info along side stream creation #16
16. [ ] Integrate dgram sending and recving for hole punching logic. - see Allow Bidirectional Hole Punching from QUICClient and QUICServer #4
17. [ ] Integrate the napi program into the scripts/prebuild.js so we can actually build the package in a similar to other native packages we are using like js-db - see Packaging up js-quic for Linux, Windows, MacOS #7

[CMCDragonkai]

Ok I have managed to get the ability for quiche to send a packet. Actually I haven't sent the packet to the UDP socket, but I can see how quiche attempts to create an initial packet for me plumb into the UDP socket.

This involved several challenges.

The first is that self.0.send takes in a reference to a mutable buffer. There's a preference to avoid copying in Rust lang similar to C and C++.

This buffer can actually be taken from the outside back in JS land. This might be a better idea, as it minimises the amount of copy pasting going on.

Alternatively we can create the array in Rust, then return as a Buffer. However the send call returns the number of bytes written.

Therefore the buffer is pre-allocated to be MAX_DATAGRAM_SIZE and it is subsequently written to.

This means later the array needs to be sliced according the to the write size and then returned as a buffer.

This means we generally 2 styles of calling the Rust functions. We can call them the way rust expected to be called, by passing in a buffer as a parameter and returning the write length as well. It turns out that socket.send actually easily supports taking a buffer, offset and the length to be used: https://nodejs.org/api/dgram.html#socketsendmsg-offset-length-port-address-callback. In terms of speed, it would make use to re-use the same buffer over and over.

The other way is more idiomatic JS, which would mean that the function should return the buffer to be sent that is properly sliced up.

I think we should preserve the rust style, and then do any JS-specific idiomatic abstractions on the JS side.

At the same time, the send can also return an error which indicates that it is Done. If it is done, then there are no more packets to send. Using a JS style call, we can just return undefined in that situation. However I'm not entirely sure if this is actually possible using napi-rs. It seems you have return a Rust value, and that it automatically gets converted to JS via the napi-rs macros.

With rust style calls, the done is returned as an exception which is kind of strange, but also doable. But this is very much different from how JS should be used, and it means we also have to catch non-done exceptions.

Another thing to realise is that when you have situations where you have a buffer of some fixed size, and a then write length indicating how much of that buffer was used. This will always eventually lead to a slicing operation.

When doing any slicing, the function's stack will end up with a pointer (reference) to the slice, it won't do any copying.

Rust does not have dynamically allocated arrays that sit on the stack, so you have to use vectors instead. Vectors are heap-allocated arrays that can be dynamically changed. String is also heap allocated string relative to &str. At the same time, Rust manages the memory of vectors and knows when to deallocate them automatically.

So if we wanted to slice out from the array and also return it as a Buffer to JS, then it has to be turned into a vector first, and that vector will be dynamically allocated simply through the !vec macro.

Finally another problem is sending structs back to JS as objects. This requires hte usage of #[napi(object)] macro. However this only works on structs have all the fields public.

The send info contains information that is necessary to know how to send the packet data to the UDP socket.

It however ends up containing std::net::SocketAddr and std::time::Instant both which contain private fields. So these types cannot be easily automatically converted to JS object. It could probably be done if I implement some sort of interface for the SocketAddr. Like https://docs.rs/napi/latest/napi/trait.NapiRaw.html and https://docs.rs/napi/latest/napi/trait.NapiValue.html.

Alternatively I created my own Host struct that just contains a String and u16. This results in a conversion from the SocketAddr to Host and wrapping std::time::Instant into External::new() because it's supposed to be opaquely compared at the rust-level.

This led to a few other issues:

1. You cannot declare nested structs in Rust, you have to give each one a separate name.
2. There does not appear to be a way to send Rust tuples to JS, napi-rs does not understand how to deal with tuples. I would have thought that they just become arrays on the JS side... but no.
3. Rust enums cannot be represented either, they become objects on the JS side. This is because Rust enums are actually ADTs. They have a tag, and then the subsequent value/values.

So with this all in mind, I ended up creating another custom struct just so that so I can get the value out of conn.send.

This reveals a structure that always contains the initial packet. This is probably the packet that leads to connection negotiation. If you call it again right now, you get a JS error with Done as the message.

{
  out: <Buffer c3 00 00 00 01 10 98 f4 9d 84 85 7a dd 8e f6 ef fe 25 d7 84 b3 e1 14 d3 46 1a ab 5d d2 25 f2 61 c7 0d b0 d7 83 10 88 f0 b3 f8 eb 00 40 e8 58 ad 8e 1e ... 1150 more bytes>,
  info: {
    from: { hostname: '127.0.0.1', port: 55551 },
    to: { hostname: '127.0.0.1', port: 55552 },
    at: [External: 2d723f0]
  }
}

Now we can just plumb this into the dgram module to send.

[CMCDragonkai]

When should the send method be called? Apparently it should be triggered under an under a number of different conditions:

https://docs.rs/quiche/0.16.0/quiche/struct.Connection.html#method.send

And it should be looped in fact, until it is done.

The conditions are:

The application should call send() multiple times until Done is returned, indicating that there are no more packets to send. It is recommended that send() be called in the following cases:

• When the application receives QUIC packets from the peer (that is, any time recv() is also called).
• When the connection timer expires (that is, any time on_timeout() is also called).
• When the application sends data to the peer (for example, any time stream_send() or stream_shutdown() are called).
• When the application receives data from the peer (for example any time stream_recv() is called).

Once is_draining() returns true, it is no longer necessary to call send() and all calls will return Done.

[CMCDragonkai]

So I can imagine attaching callback to things like on_timeout to trigger "sending", and any time we receive packets, and send data to the peer or receive data to the peer.

We could call this "socketFlushing", it will loop until done. The socket flushing would be triggered every time any of the above things happen... which in turn will flush all the packets to the socket.... will it wait for the socket to have actually sent it by waiting on the callback? I'm not sure if that's necessary...

I guess it depends. On the QUIC stream level, it should be considered independent, you don't wait for the socket to be flushed before returning to the user. But one could await a promise will resolve when they really want to have the entire socket flushed.

[CMCDragonkai]

Ok I can see that there is no support for tuples at all here: https://github.com/napi-rs/napi-rs/tree/main/examples/napi. Also not for tuple-structs which I use in a newtype style pattern.

The only thing closest is to create an array like so:

#[napi]
fn to_js_obj(env: Env) -> napi::Result<JsObject> {
  let mut arr = env.create_array(0)?;
  arr.insert("a string")?;
  arr.insert(42)?;
  arr.coerce_to_object()
}

On the generated TS it looks like a object.

I reckon it would be dynamically deconstructable as an array. Pretty useful here, to avoid having to define one-off structs.

[CMCDragonkai]

I've settled on just returning an Array. It beats having to redefine the output type all the time.

[CMCDragonkai]

The streams in quiche doesn't have its own data type. Instead they are identified by a stream ID. The connection struct provides the methods https://docs.rs/quiche/0.16.0/quiche/struct.Connection.html#method.stream_recv and you have to keep track of the stream IDs somehow.

So we would expect that upon creating an RPC call, it would call into the network to create a new stream for a connection.

It seems creating a new stream is basically performing this call: https://docs.rs/quiche/0.16.0/quiche/struct.Connection.html#method.stream_send

You select a stream ID like 0, and you just keep incrementing that number to ensure you have unique stream IDs. The client-side just increments the number, the server side just receives the streams.

Note that once we have a stream, the stream is fully duplex (not half-duplex), in-order and reliable.

[CMCDragonkai]

The quiche::connect and quiche::accept should be both factory method of creating the Connection class.

However in the case of the server side, you can't create a new connection until you've taken a specific packet from the UDP socket and parsed it first. You have to check the internal packet information to decide whether to accept a new connection, or to simply plumb it in as if you received the data for a given stream.

So the order of operations:

1. Receive packet from UDP socket.
2. Check if the packet is a new connection, if so, create the new connection, otherwise retrieve an existing connection.
3. Call conn.recv() to plumb the packet data into some relevant stream.

This does mean we maintain clients hashmap of ConectionId => Connection.

[CMCDragonkai]

I created a function to randomly generate the connection ID.

It's a random 20 bytes.

Instead of using Rust's ring library to get the random data. I'm just calling into a passed in JS function to acquire the random data. This unifies the random data usage directly from the Node runtime (and reduces how much rust code we are using here).

/// Creates random connection ID
///
/// Relies on the JS runtime to provide the randomness system
#[napi]
pub fn create_connection_id<T: Fn(Buffer) -> Result<()>>(
  get_random_values: T
) -> Result<External<quiche::ConnectionId<'static>>> {
  let scid = [0; quiche::MAX_CONN_ID_LEN].to_vec();
  let scid = Buffer::from(scid);
  get_random_values(scid.clone()).or_else(
    |err| Err(Error::from_reason(err.to_string()))
  )?;
  let scid = quiche::ConnectionId::from_vec(scid.to_vec());
  eprintln!("New connection with scid {:?}", scid);
  return Ok(External::new(scid));
}

However I realised that since it's just a 20 byte buffer. Instead of passing an External<quiche::ConnectionId<'static>> around.

We can just take a Buffer directly that we expect to be of length MAX_CONN_ID_LEN.

Meaning the connection ID is just made in JS side, and then just passed in.

On the rust side, they can use quiche::ConnectionId::from_ref(&scid) which means there's no need to to keep any memory on the Rust side. It's all just on the JS side.

This basically means we generally try to keep as much data on the JS side, and manage minimal amounts of memory on the Rust side.

[CMCDragonkai]

The <'static> is used because it's sort of a dummy lifetime, because the ConnectionId is actually an enum of either an owned vector or a reference to some memory that exists elsewhere. The owned vector doesn't have a lifetime annotation.

[CMCDragonkai]

A better alternative to just Buffer would be our own ConnectionId newtype.

But I can't seem to create good constructor for it that would take a callback. Problem in napi-rs/napi-rs#1374.

[CMCDragonkai]

On the connecting side, there's an option for the server name.

If QUIC uses the standard TLS verification, we could imagine that this will not work for us. I'm not entirely sure.

We need to provide our custom TLS verification logic. I haven't found how to do this yet with QUIC but it should be possible.

---

Found it: cloudflare/quiche#326 (comment)

It is in fact possible.

The standard certificate verification must be disabled, and you have to fetch the certificate itself to verify.