feat(AutoTLS): opt-in WSS certs from p2p-forge at libp2p.direct

[aschmahmann]

This PR is WIP but the intention is to allow any publicly reachable node on the network to be reachable via Secure WebSockets and thereby improve reachability from browsers. For anyone who reads this and is unfamiliar with this work, don't worry plenty of documentation to come before we're ready to merge.

TLDR

$ ipfs config --json AutoTLS.Enabled true
$ export GOLOG_LOG_LEVEL="error,autotls=debug,p2p-forge/client=debug" # to see whats happening
$ ipfs daemon

TODO

Some outstanding items here are:

• the libp2p.direct service needs to be ready for production use tracked in Enabling AutoTLS feature by default #10560
• libp2phttp: HTTP Peer ID Authentication libp2p/go-libp2p#2854 lands
• documentation
• need to be happy with the config file syntax
• remove debugging logs associated with acquiring certificates
• we should probably wait until we have confidence in our external addresses before trying to get a certificate
• multiplex WSS on the same port as TCP (e.g. by muxing on the ALPN in the TLS cert) so people don't need to open new ports
  • descoped from this PR, will land once feat(tcpreuse): add options for sharing TCP listeners amongst TCP, WS and WSS transports libp2p/go-libp2p#2984 is ready
  • tracked in Enabling AutoTLS feature by default #10560
• there's currently an issue where the application doesn't realize it's acquired a certificate and that it's ready for use (happens at least on first acquisition)
• figure out what's going on with excessive logs like
  • "http: TLS handshake error from ip:port: EOF"
    • unable to reproduce anymore, assuming fixed
  • ERROR p2p-forge/client.maintenance certmagic@v0.21.4/maintain.go:521 error while checking storage for updated ARI; updating ARI now (from ipfs at collab-cluster-sv15-1)
• remove replace github.com/libp2p/go-libp2p from all three go.mod

[lidel]

Updated this PR, aiming at Kubo 0.32. It uses the latest commit from go-libp2p master branch that includes libp2p/go-libp2p#2854.

[lidel]

Pushed:

• Docs
• Logger cleanup, should be up to release standard.
  • Debug logs can be enabled via export GOLOG_LOG_LEVEL="error,p2p-forge/client=debug" and by default only ERROR level is printed to stdout.

Next steps:

• My plan is to wait for go-libp2p 0.37 release, bump dependency and resolve issues
• merge this PR, and start testing docker image from master branch in our collab cluster. this way we don't need to wait for kubo 0.32-rc1, and get real world feedback faster.

[lidel]

Switched to go-libp2p 0.37 from master branch and resolved conflicts, I'll check on CI tomorrow.

[2color]

As discussed, left some feedback on naming. I can submit more suggestions thorough PR tomorrow.

[lidel]

Found a bug in cluster logs ("error while checking storage for updated ARI; updating ARI now"):

Oct 28 08:44:33 collab-cluster-sv15-1 ipfs[99759]: 2024-10-28T08:44:33.576Z        ERROR        p2p-forge/client.maintenance        certmagic@v0.21.4/maintain.go:521        error while checking storage for updated ARI; updating ARI now        {"identifiers": ["*.k51qzi5uqu5dljgcjt37azmfvj1u9b1v9okfd0vhd4gkbcs75cmy0shqjoml6q.libp2p.direct"], "cert_hash": "ac1b9af9ec6e49e5e445593e02620a25a4f6bf5f8013ac0ed7a0206cb9c5f9bd", "ari_unique_id": "kydGmAOpUWiOmNbEQkjbI79YlNI.A_zqjUpBrD6AJHDUgsoK9qj6", "cert_expiry": "2025-01-23T11:57:29.000Z", "error": "loading cert metadata: open /home/ipfs/.local/share/certmagic/certificates/acme-v02.api.letsencrypt.org-directory/wildcard_.k51qzi5uqu5dljgcjt37azmfvj1u9b1v9okfd0vhd4gkbcs75cmy0shqjoml6q.libp2p.direct/wildcard_.k51qzi5uqu5dljgcjt37azmfvj1u9b1v9okfd0vhd4gkbcs75cmy0shqjoml6q.libp2p.direct.json: no such file or directory"}
Oct 28 08:44:33 collab-cluster-sv15-1 ipfs[99759]: 2024-10-28T08:44:33.576Z        DEBUG        p2p-forge/client.acme_client        acme/ari.go:134        getting renewal info        {"names": ["*.k51qzi5uqu5dljgcjt37azmfvj1u9b1v9okfd0vhd4gkbcs75cmy0shqjoml6q.libp2p.direct"]}
Oct 28 08:44:33 collab-cluster-sv15-1 ipfs[99759]: 2024-10-28T08:44:33.656Z        DEBUG        p2p-forge/client.acme_client        acme/http.go:275        http request        {"method": "GET", "url": "https://acme-v02.api.letsencrypt.org/draft-ietf-acme-ari-03/renewalInfo/kydGmAOpUWiOmNbEQkjbI79YlNI.A_zqjUpBrD6AJHDUgsoK9qj6", "headers": {"User-Agent":["CertMagic acmez (linux; amd64)"]}, "response_headers": {"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["121"],"Content-Type":["application/json"],"Date":["Mon, 28 Oct 2024 08:44:33 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Retry-After":["21600"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]}, "status_code": 200}
Oct 28 08:44:33 collab-cluster-sv15-1 ipfs[99759]: 2024-10-28T08:44:33.656Z        INFO        p2p-forge/client.acme_client        acme/ari.go:215        got renewal info        {"names": ["*.k51qzi5uqu5dljgcjt37azmfvj1u9b1v9okfd0vhd4gkbcs75cmy0shqjoml6q.libp2p.direct"], "window_start": "2024-12-23T12:16:59.333Z", "window_end": "2024-12-25T12:16:59.333Z", "selected_time": "2024-12-24T21:30:45.000Z", "recheck_after": "2024-10-28T14:44:33.656Z", "explanation_url": ""}
Oct 28 08:44:33 collab-cluster-sv15-1 ipfs[99759]: 2024-10-28T08:44:33.656Z        ERROR        p2p-forge/client.maintenance        certmagic@v0.21.4/maintain.go:181        updating ARI        {"error": "got new ARI from acme-v02.api.letsencrypt.org-directory, but failed loading stored certificate metadata: loading cert metadata: open /home/ipfs/.local/share/certmagic/certificates/acme-v02.api.letsencrypt.org-directory/wildcard_.k51qzi5uqu5dljgcjt37azmfvj1u9b1v9okfd0vhd4gkbcs75cmy0shqjoml6q.libp2p.direct/wildcard_.k51qzi5uqu5dljgcjt37azmfvj1u9b1v9okfd0vhd4gkbcs75cmy0shqjoml6q.libp2p.direct.json: no such file or directory"}

I suspect this is another race condition where certmanager things (like location .local/share/certmagic/) are initialized with default config before our config is applied (same problem as the logger).

Will look into this.

[lidel]

Potential fix in e51d907 and ipshipyard/p2p-forge@f599f48.

TLDR is that certmagic.NewDefault should be used only for prototyping, or if one is ok with default storage location and logger. Using dedicated cache and avoiding calling certmagic.NewDefault should remove surface for race condition in setting storage paths AFTER maintenance job is started.

I'm going to deploy to collab cluster, let it run over night and see if ARI error is gone.

Update: so far the fix looks good, renewal check does not produce error anymore:

Oct 29 00:21:19 collab-cluster-sv15-1 ipfs[2893066]: 2024-10-29T00:21:19.893Z        INFO        autotls.acme_client        acme/ari.go:215        got renewal info        {"names": ["*.k51qzi5uqu5dljgcjt37azmfvj1u9b1v9okfd0vhd4gkbcs75cmy0shqjoml6q.libp2p.direct"], "window_start": "2024-12-23T12:16:59.333Z", "window_end": "2024-12-25T12:16:59.333Z", "selected_time": "2024-12-24T14:39:17.000Z", "recheck_after": "2024-10-29T06:21:19.893Z", "explanation_url": ""}
Oct 29 00:21:19 collab-cluster-sv15-1 ipfs[2893066]: 2024-10-29T00:21:19.929Z        INFO        autotls        certmagic@v0.21.4/maintain.go:584        updated ACME renewal information        {"identifiers": ["*.k51qzi5uqu5dljgcjt37azmfvj1u9b1v9okfd0vhd4gkbcs75cmy0shqjoml6q.libp2p.direct"], "cert_hash": "ac1b9af9ec6e49e5e445593e02620a25a4f6bf5f8013ac0ed7a0206cb9c5f9bd", "ari_unique_id": "kydGmAOpUWiOmNbEQkjbI79YlNI.A_zqjUpBrD6AJHDUgsoK9qj6", "cert_expiry": "2025-01-23T11:57:29.000Z", "selected_time": "2024-12-25T07:38:42.000Z", "next_update": "2024-10-29T06:21:19.893Z", "explanation_url": ""}

[lidel]

Ok, the error seems to be gone. Considering this resolved, but keeping this open for visibility.

[2color]

Why is port 4082 used here?

Can you use the same port as the /ws address: /ip4/0.0.0.0/tcp/4002/ws?

[lidel]

Not yet, we need libp2p/go-libp2p#2984 for that.

The idea is that TCP port sharing will land in go-libp2p before we enable AutoTLS by default.
For opt-in, using separate port is fine, just need to remember to remind people that they need to safelist additional port on their firewall.

[lidel]

We've been running this in collab cluster and no new errors were found.
Configuration and docs are in place. This is likely as good as it can be without asking wider community for feedback.

Merging to include it in tomorrow's Kubo 0.32.0-rc1 (#10547)