go-libp2p v0.24

[marten-seemann]

🗺 What's left for release

Early Muxer Selection: #1789 (tracking issue)

WebTransport:

• webtransport: make it possible to record qlogs (controlled by QLOGDIR env) #1828
• webtransport: use the rcmgr to control flow control window increases #1832
• webtransport / quic: allow listening on the same UDP port #1759
• webtransport: use deterministic TLS certificates #1833
• webtransport: disable support for signed/static TLS certificates #1927
• transport.Listener,quic: Support multiple QUIC versions with the same Listener. Only return a single multiaddr per listener. #1923
• webtransport: add PSK to constructor, and fail if it is used #1929

Misc:

• autorelay: don't discard static relays on disconnections #1782
• routed host: search for new multi addresses upon connect failure #1835
• holepunch: add multiaddress filter #1839
• Nodes Are Unable To Open Outbound Connections Via VPNs #1849
• "panic: runtime error: invalid memory address or nil pointer dereference" in p2p/host/autorelay/relay_finder.go:209 if EnableAutoRelay() #1852
• autorelay: remove the default static relay option #1867
• remove deprecated StaticRelays option #1868
• core/protocol: remove deprecated Negotiator.NegotiateLazy #1869
• quic: advertise both quic-draft29 and quic-v1 addresses #1841
• rcmgr: fix missing service peer default in LimitConfig.Apply #1878
• quic: QUIC listener doesn't respect the QUIC version of the listen address #1904
• security protocol ordering preference broken #1906
• add QUIC v1 addresses to the default listen addresses #1914
• swarm / rcmgr: synchronize the concurrent outbound dials with limits #1898
• autorelay: fix flaky TestReconnectToStaticRelays #1903

Breaking Changes

Constructor options

• removed the deprecated libp2p.DefaultStaticRelays and libp2p.StaticRelays options. Static relays can be configured as an option to libp2p.EnableAutoRelay
• libp2p.Security now requires passing in a constructor (previously, it also allowed passing in a fully constructed security transport)
• libp2p.Muxer now requires passing in a fully constructed muxer (previously, it also allowed passing in a muxer constructor)
• Removed libp2pquic.DisableReuseport and libp2pquic.WithMetrics. These options are now available in the quicreuse package (quicreuse.DisableReuseport and quicreuse.WithMetrics), and can be enabled by using the libp2p.QUICReuse option (example: libp2p.QUICReuse(quicreuse.NewConnManager, quicreuse.DisableReuseport, quicreuse.WithMetrics)).

Under the hood, we now use fx in the construction of the host, which lead to a great simplification of our setup logic (see #1858 for details).

QUIC Versions

When we first rolled out QUIC support in 2020, QUIC wasn't an RFC yet (in fact, we were involved in the standardization process at the IETF!). Back then, we rolled out support for QUIC draft-29. This version is almost identical to RFC 9000, so there was never a good reason to force an update.

Now that rust-libp2p is shipping QUIC support, we decided to finally initiate the update. We do so by introducing a new multiaddress component: /ip4/1.2.3.4/udp/4001/quic now denotes a multiaddr that uses QUIC draft-29 (as it has before, we're just making it explicit now). QUIC v1 would use /ip4/1.2.3.4/udp/4001/quic-v1.
We intend to keep support for QUIC draft-29 for roughly half a year, and disable listening on draft-29 addresses after that.

Depending on how you configure your node, you might need to update the addresses you're listening on (i.e. change quic => quic-v1 in the addresses, or duplicate the QUIC addresses).

🔦 Highlights

WebTransport

We added experimental WebTransport support in our last release. Since then, we're using some magic to allow running QUIC and WebTransport on the same port. This means that users who've already configured port forwarding / firewall rules for QUIC don't need to do anything to allow WebTransport connections to their node.

Optimized Stream Multiplexer Selection

When dialing a TCP connection to another libp2p node, we perform the following steps:

1. TCP 3-way handshake: 1 RTT
2. multistream-select to negotiate the security protocol (TLS 1.3 or Noise): 1 RTT
3. security handshake: 1 RTT
4. multistream-select to negotiate the stream multiplexer (yamux or mplex)

In total, this handshakes takes 4 RTTs. In this release, we optimize this handshake by inlining the stream multiplexer negotiation into the security handshake. On the wire, this looks very different for TLS 1.3 and for Noise, but the result is the same: we know which stream multiplexer to use when the handshake finishes, and can therefore save one roundtrip on every handshake. Head to the specification to learn how this works in detail.

Changelog

...

✅ Release Checklist

• Stage 0 - Finishing Touches
  • Go through relevant libp2p repos looking for unreleased changes that should make it into the release. If you find any, cut releases.
  • Run go get -u ./... to see if there are any out-of-date deps that look important. If there are, bubble them. Try to avoid directly updating indirect deps in go-libp2p's go.mod when possible.
• Stage 1 - Release
  • Publish the release through the GitHub UI, adding the release notes. Some users rely on this to receive notifications of new releases.
  • Announce the release on the discuss.libp2p.io. https://discuss.libp2p.io/t/go-libp2p-v0-24-0-released/1738
• Stage 2 - Update Upstream
  • Update the examples to the final release: chore: update examples to v0.24.0 #1936
  • Update the upstream dependencies to the final release and create PRs.
    • filecoin-project/lotus
    • ipfs/kubo: WIP: update go-libp2p to v0.24.0 ipfs/kubo#9423
• Make required changes to the release process.

[p-shahi]

Proposal to add deprecation warning for insecure WebSockets in this release: #1830 (comment)

[BigLep]

Great writeup on the changelog @marten-seemann ! A few things:

1. WebTransport: there isn't a clear statement on whether it's still experimental or not. I assume it isn't (given checklist at Tracking issue: Finish WebTransport Support #1827 ). Lets be explicit, something along the lines of "WebTransport transport is no longer "experimental" and is enabled by default 🚀."
2. "/ip4/1.2.3.4/udp/4001/quic now denotes a multiaddr that uses QUIC draft-29 (as it has before, we're just making it explicit now). QUIC v1 would use /ip4/1.2.3.4/udp/4001/quic." These multiaddrs are the same.
3. It seems like a there are some breaking changes here. Can we call those out at the top? The reason to do this is to build trust with users that they can rely on us to callout breakages clearly upfront.

Idea (might be a good thing in a retro @p-shahi ): author changelogs in markdown file like https://github.com/ipfs/kubo/blob/master/docs/changelogs (in future). It would allow building up the changelog incrementally as part of a PR for the relevant work and easier to comment on than an issue where can't suggest changes inline.

[marten-seemann]

1. WebTransport: there isn't a clear statement on whether it's still experimental or not. I assume it isn't (given checklist at Tracking issue: Finish WebTransport Support #1827 ). Lets be explicit, something along the lines of "WebTransport transport is no longer "experimental" and is enabled by default 🚀."

It's still experimental in this repo. This isn't to say that we're not confident in the implementation, but we've made a ton of changes to the WebTransport code for this release, so keeping it experimental for a little bit longer seems like the safer way to go.

Regarding terminology: There were two reasons why we called WebTransport experimental:

1. our lack of battle-testing of our implementation and
2. the fact that the protocol itself is still an IETF draft

While we're gaining more confidence regarding (1), the protocol itself is guaranteed to change, which means that at some point browsers will switch from one draft version to the other, in all likelihood without any regards for backwards-compatibility (at least that's what happened for QUIC draft versions). This will immediately break compatibility with the WebTransport already deployed in go-libp2p nodes. We need to be clear in our communication that this is almost guaranteed to happen sooner or later.

2. "/ip4/1.2.3.4/udp/4001/quic now denotes a multiaddr that uses QUIC draft-29 (as it has before, we're just making it explicit now). QUIC v1 would use /ip4/1.2.3.4/udp/4001/quic." These multiaddrs are the same.

That's a very unfortunate typo. Thanks for spotting it. I fixed it.

3. It seems like a there are some breaking changes here. Can we call those out at the top? The reason to do this is to build trust with users that they can rely on us to callout breakages clearly upfront.

I added a "Breaking Changes" section.

Idea (might be a good thing in a retro @p-shahi ): author changelogs in markdown file like https://github.com/ipfs/kubo/blob/master/docs/changelogs (in future).

I have mixed feelings about this, partly due to prior experience with this approach. Let's discuss this in the next triage call?

[BigLep]

WebTransport experimental support

Makes sense - thanks. Given our experimental status is dependent on IETF work/decisisions, we'll need to have a way to talk about it. We can talk during a triage meeting, but maybe talk about having WebTransport-draft support.

Changelog

Sure. Talking about it during a retro or triage meeting sounds good.

[p-shahi]

Is #1915 still a part of this release? It's mentioned in v0.25 as well.

[marten-seemann]

Is #1915 still a part of this release? It's mentioned in v0.25 as well.

No. I just removed it.

[MarcoPolo]

Because of #1904 (comment) I won't be able to make a release this week. Next week I should be able to fix that issue and make a release.

[p-shahi]

I propose we add a bugfix section to release notes (makes it easier than going through the changelog)

🐞 Bugfixes since v0.23.4

• Nodes Are Unable To Open Outbound Connections Via VPNs #1849
• websocket: set the HTTP host header #1834
• routed host: search for new multi addresses upon connect failure #1835
• swarm: return filtered addrs in addrsForDial #1855
• connmgr: Concurrent Map Misuse #1847
• "panic: runtime error: invalid memory address or nil pointer dereference" in p2p/host/autorelay/relay_finder.go:209 if EnableAutoRelay() #1852

[p-shahi]

The two remaining open items for this release are

• WIP: update go-libp2p to v0.24.0 ipfs/kubo#9423
  • which will be merged by @Jorropo
  • we have done our part to help move this along
• Update filecoin-project/lotus
  • which we will not do (let the Lotus team do it in on their own schedule)

Because Kubo 9423 will be closed by Kubo maintainers and we have completed our responsibility, closing this release issue as completed.