QUIC as a native addon (secondary option)

[jasnell]

Just opening this for discussion purposes. Because of the challenges presented by OpenSSL deciding not to land the QUIC related APIs until at least OpenSSL 3.1 (which likely won't happen until 2021 or 2022), the Node.js TSC has expressed some reluctance on landing the QUIC support in core because of the need to float patches on the OpenSSL 1.1.1 version that we ship. The key issues are that (a) the patches will need to be maintained for every 1.1.1 version bump, (b) anyone using openssl via shared library will not be able to use QUIC, and (c) the QUIC implementation would be forced to remain experimental until OpenSSL did officially support QUIC and could be forced to change should OpenSSL decide to go with a different API design than what BoringSSL has adopted.

Therefore, one of the possible paths forward here is to re-implement the QUIC support as a native add-on that statically links it's own modified OpenSSLrather than using core's.

I have started a separate project moving in that direction but there are a number of key challenges stemming from the number of core APIs we are using that are not available via N-API. These aren't difficult or impossible challenges to overcome, but combined they are non-trivial.

• While we do not use the TLS/SSLWrap implementation from node_crypto, we do make use of SecureContext, which is, of course, backed by the OpenSSL version used by Core. As a standalone native addon with it's own separate linked version of OpenSSL, we shouldn't rely on core's SecureContext and instead should have an equivalent backed by the addons OpenSSL.

• The implementation builds directly on UDPWrap and StreamBase, neither of which have N-API equivalents and both of which are guarded by NODE_WANT_INTERNALS). This is not a challenge if we just simply use NODE_WANT_INTERNALS=1 but then we bind ourselves to Node.js internals and fall into the traditional brittle native addon trap that N-API was meant to solve. We also make use of many of the utility macros, types, and functions that are scattered throughout core's internals (Debug, string encoding, Buffer, CHECK, AllocatedBuffer, AliasedArray, http_common, etc). All of which is guarded by NODE_WANT_INTERNALS and not exposed in any way via N-API.

• This one is obvious... the addon would need it's own linked OpenSSL. This certainly should be possible but presents a number of challenges given how non-trivial the OpenSSL maintenance and build can be. We would end up duplicating much of the build dev that's currently in core to support it.

The native addon approach is certainly possible, and I already have a project scaffolded out in a repo that I can make public if folks want to help move things in that direction. There are certainly advantages to the addon approach in that the implementation would not be forced to be experimental for potentially years and could be made to work on existing LTS versions of Node.js. The plus side of having it as a native addon should not be discounted at all. If we can find good solutions to the above challenges, then I'm happy to move things in that direction.

/cc @mcollina @addaleax @bnoordhuis

[jasnell]

To provide additional context, here's a high-level summary of the current implementation:

There are three essential dependencies:

• OpenSSL patched with the BoringSSL QUIC Apis
• ngtcp2 - provides the core QUIC support
• nghttp3 - provides the http/3 support on top of ngtcp2

From core, we depend on the following:

• UDPWrap and uv_udp_t -- We leverage the existing dgram/UDPWrap support to establish the local UDP port but none of the existing dgram js apis are used. The QUIC implementation completely takes the UDP port over at the UDPWrap level.
• StreamBase, Buffer, and Duplex (for QUICStream)
• SocketAddress (const sockaddr wrapper) -- this is a utility added specifically because of QUIC that makes working with sockaddr a bit easier.
• http_common and node_mem -- nghttp3 and nghttp2 share many common elements that have been extracted out into a shared set of utilities to reduce code duplication. These primarily deal with http headers and memory allocation tracking.
• Debug and CHECK -- these are used extensively through the implementation currently.
• Histogram -- these are used for tracking statistics through the lifetime of a QUIC session.
• AsyncWrap -- For all the typical reasons
• BaseObject and BaseObjectPtr
• String encoding -- Primarily for debug purposes, we use core's internal Hex encoding (connection identifiers are byte strings that need to be hex encoded for debug serialization)
• AliasedArray and AliasedStruct -- For all the typical reasons
• crypto_common -- Because QUIC requires TLS 1.3 but does not use the existing TLSWrap, many of the more generalizable crypto functions have been moved out of node_crypto into node_crypto_common to make them sharable to avoid duplication of code. This includes functions like serialization of certs, handling of ALPN identifiers, and so on.

There's likely more that I'm forgetting and I will update the list as I remember them.

As for the implementation itself... the key elements are:

• QuicEndpoint -- This wraps a single UDPWrap instance at the native layer and handles it's lifecycle.

• QuicSocket -- This is the primary entry point to the implementation and acts as both the QUIC client and server implementation. A single QuicSocket may have multiple QuicEndpoints and may be both a client and a server at the same time.

• QuicSession -- A single QUIC "connection". A single instance represents either the client or server side of the connection. A single QuicSession is bound to at most one QuicSocket at a time but may be migrated from one QuicSocket to another and can have a lifecycle that is independent of the actual UDP port bindings.

• QuicStream -- A single uni- or bi-directional dataflow over a QuicSession. Best to think of this as a simple Duplex that can be half-open. A QuicSession may have theoretically any number of QuicStream instances open in either direction at any time.

Add to this a number of utility and helper classes such as QuicApplication (encapsulates the application protocol layer on top of a QuicSession), QuicCID (used as the identifier of a QuicSession), QuicPacket (encapsulates the serialized QUIC packet data to be sent), QuicBuffer (linked list of outbound application data waiting to be serialized into a QUIC packet), and so forth.

The implementation:

• uses ngtcp2 to provide the core handling of QUIC related protocol state, packet serialization/deserialization, and connection management.
• uses OpenSSL to provide the TLS 1.3 implementation.
• uses UDPWrap/uv_udp_t to handle the network i/o.

The overwhelming majority of the implementation exists at the C++ level, with the JavaScript level focusing only on user facing public API elements. A key design goal of the internals is to make QUIC completely usable at the native layer so that we have the option of introducing new application protocols efficiently (http/3 is an example, a QUIC-based inspector protocol implementation replacing the websockets-based would be another example that I've been toying around with). By integrating these at the native layer the way that nghttp3 and Http3Application do, we allow ourselves to take advantage of key performance and implementation optimizations that simply aren't available at the JavaScript level.

[mcollina]

Good writeup and work @jasnell!

[jasnell]

I have started a native addon project called veloce (Italian for 'fast') that is currently a private repo under my personal github that I'm using to explore the native addon implementation. I'm happy to open that to folks who wish to contribute should we decide that the native addon approach is the way to go. Before that approach is tenable, however, we need to resolve the technical challenges around bundling openssl and use of the various Node.js internals.

[ronag]

I'm not that involved with the addons. But if we think it would be reasonable to be able to implement QUIC in terms of an addon. Then doing the extra work required to sufficiently extend addons might be a big plus for other addons as well, e.g. AsyncWrap might be a good idea to expose regardless of this?

@jasnell I remember you mentioning that you might want to consider having a look at http2 based on experience from the Quic work. Does this affect that in anyway?

Also, once OpenSSL in Node core does catch up sufficiently. Would it be easy to consolidate?

[sam-github]

Doing it as an addon seems a mountain of work, tedious though it is, refactoring the PR3 so that it doesn't touch non-QUIC code (except in obviously correct ways) seems faster. I'd be willing to RSLGTM such a thing, but I don't feel as comfortable doing so for the PR as it is.