Release v0.13

[BigLep]

go-ipfs v0.13.0 Release

We're happy to announce go-ipfs 0.13.0, packed full of changes and improvements!

As usual, this release includes important fixes, some of which may be critical for security. Unless the fix addresses a bug being exploited in the wild, the fix will not be called out in the release notes. Please make sure to update ASAP. See our release process for details.

Overview

Below is an outline of all that is in this release, so you get a sense of all that's included.

• 🛠 BREAKING CHANGES
  • ipfs block put command
  • ipfs cid codecs command
  • ipfs cid format command
  • Swarm configuration
  • Circuit Relay V1 is deprecated
  • ls requests for /multistream/1.0.0 are removed
  • Gateway Items
• 🔦 Highlights
  • 🧑‍💼 libp2p Network Resource Manager (Swarm.ResourceMgr)
  • 🔃 Relay V2 client with auto discovery (Swarm.RelayClient)
  • 🌉 HTTP Gateway improvements
    • 🍱 Support for Block and CAR response formats
    • 🐎 Fast listing generation for huge directories
    • 🎫 Improved Etag and If-None-Match for bandwidth savings
    • ⛓️ Added X-Ipfs-Roots for smarter HTTP caches
    • 🌡️ Added metrics per response type
  • 🕵️ OpenTelemetry tracing
    • How to use Jaeger UI for visual tracing?
  • 🩺 Built-in ipfs diag profile to ease debugging
  • 🔑 Support for PEM/PKCS8 for key import/export
  • 🧹 Using standard IPLD codec names across the CLI/HTTP API
  • 🐳 Custom initialization for Docker
  • RPC API docs for experimental and deprecated commands
  • Yamux over Mplex

🛠 BREAKING CHANGES

ipfs block put command

ipfs block put command returns a CIDv1 with raw codec by default now.

• ipfs block put --cid-codec makes block put return CID with alternative codec
  • This impacts only the returned CID; it does not trigger any validation or data transformation.
  • Retrieving a block with a different codec or CID version than it was put with is valid.
  • Codec names are validated against tables from go-multicodec library.
• ipfs block put --format is deprecated. It used incorrect codec names and should be avoided for new deployments. Use it only if you need the old, invalid behavior, namely:
  • ipfs block put --format=v0 will produce CIDv0 (implicit dag-pb)
  • ipfs block put --format=cbor will produce CIDv1 with dag-cbor (!)
  • ipfs block put --format=protobuf will produce CIDv1 with dag-pb (!)

ipfs cid codecs command

• Now lists codecs from go-multicodec library.
• ipfs cid codecs --supported can be passed to only show codecs supported in various go-ipfs commands.

ipfs cid format command

• --codec was removed and replaced with --mc to ensure existing users are aware of the following changes:
  • --mc protobuf now correctly points to code 0x50 (was 0x70, which is dab-pg)
  • --mc cbor now correctly points to code 0x51 (was 0x71, which is dag-cbor)

Swarm configuration

• Daemon will refuse to start if long-deprecated RelayV1 config key Swarm.EnableAutoRelay or Swarm.DisableRelay is set to true.
• If Swarm.Transports.Network.Relay is disabled, then Swarm.RelayService and Swarm.RelayClient are also disabled (unless they have been explicitly enabled).

Circuit Relay V1 is deprecated

• By default, Swarm.RelayClient does not use Circuit Relay V1. Circuit V1 support is only enabled when Swarm.RelayClient.StaticRelays are specified.

ls requests for /multistream/1.0.0 are removed

• go-libp2p 0.19 removed support for undocumented ls command (PR). If you are still using it for internal testing, it is time to refactor (example)

Gateway Behavior

Directory listings returned by the HTTP Gateway won't have size column if the directory is bigger than Gateway.FastDirIndexThreshold config (default is 100).

To understand the wider context why we made these changes, read Highlights below.

🔦 Highlights

🧑‍💼 libp2p Network Resource Manager (Swarm.ResourceMgr)

You can now easily bound how much resource usage libp2p consumes! This aids in protecting nodes from consuming more resources then are available to them.

The libp2p Network Resource Manager is enabled by default, but can be disabled via:

ipfs config --json Swarm.ResourceMgr.Enabled false

When enabled, it applies some safe defaults that can be inspected and adjusted with:

• ipfs swarm stats --help
• ipfs swarm limit --help

User changes persist to config at Swarm.ResourceMgr.

🔃 Relay V2 client with auto discovery (Swarm.RelayClient)

All the pieces are enabled for hole-punching by default, improving connecting with nodes behind NATs and Firewalls!

This release enables Swarm.RelayClient by default, along with circuit v2 relay discovery provided by go-libp2p v0.19.0. This means:

1. go-ipfs will coordinate with the counterparty using a relayed connection, to upgrade to a direct connection through a NAT/firewall whenever possible.
2. go-ipfs daemon will automatically use public relays if it detects that it cannot be reached from the public internet (e.g., it's behind a firewall). This results in a /p2p-circuit address from a public relay.

Notes:

• Swarm.RelayClient does not use Circuit Relay V1 nodes any more. Circuit V1 support is only enabled when static relays are specified in Swarm.RelayClient.StaticRelays.
• One can opt-out via Swarm.EnableHolePunching.

🌉 HTTP Gateway improvements

HTTP Gateway enables seamless interop with the existing Web, clients, user agents, tools, frameworks and libraries.

This release ships the first batch of improvements that enable creation of faster and smarter CDNs, and unblocks creation of light clients for Mobile and IoT.

Details below.

🍱 Support for Block and CAR response formats

Alternative response formats from Gateway can be requested to avoid needing to trust a gateway.

For now, {format} is limited to two options:

• raw – fetching single block
• car – fetching entire DAG behind a CID as a CARv1 stream

When not set, the default UnixFS response is returned.

Why these two formats? Requesting Block or CAR for /ipfs/{cid} allows a client to use gateways in a trustless fashion. These types of gateway responses can be verified locally and rejected if digest inside of requested CID does not match received bytes. This enables creation of "light IPFS clients" which use HTTP Gateways as inexpensive transport for content-addressed data, unlocking use in Mobile and IoT contexts.

Future releases will add support for dag-json and dag-cbor responses.

There are two ways for requesting CID specific response format:

1. HTTP header: Accept: application/vnd.ipld.{format}

• Examples: application/vnd.ipld.car, application/vnd.ipld.raw

2. URL paramerer: ?format=

• Useful for creating "Download CAR" links.

Usage examples:

1. Downloading a single raw Block and manually importing it to the local datastore:

$ curl  -H 'Accept: application/vnd.ipld.raw' "http://127.0.0.1:8080/ipfs/QmZULkCELmmk5XNfCgTnCyFgAVxBRBXyDHGGMVoLFLiXEN" --output block.bin
$ cat block.bin | ipfs block put 
$ ipfs cat QmZULkCELmmk5XNfCgTnCyFgAVxBRBXyDHGGMVoLFLiXEN
hello

2. Downloading entire DAG as a CAR file and importing it:

$ ipfs resolve -r  /ipns/webui.ipfs.io
/ipfs/bafybeiednzu62vskme5wpoj4bjjikeg3xovfpp4t7vxk5ty2jxdi4mv4bu
$ curl  -H 'Accept: application/vnd.ipld.car' "http://127.0.0.1:8080/ipfs/bafybeiednzu62vskme5wpoj4bjjikeg3xovfpp4t7vxk5ty2jxdi4mv4bu" --output webui.car
$ ipfs dag import webui.car
$ ipfs dag stat bafybeiednzu62vskme5wpoj4bjjikeg3xovfpp4t7vxk5ty2jxdi4mv4bu --offline
Size: 27684934, NumBlocks: 394

See also:

• Content Addressable aRchives (CAR / .car) Specifications
• IANA media type definitions: application/vnd.ipld.car, application/vnd.ipld.raw
• ipfs-car - CLI tool for verifying and unpacking CAR files
• go-car, js-car – CAR libraries for GO and JS

🐎 Fast listing generation for huge directories

Added Gateway.FastDirIndexThreshold configuration, which allows for fast listings of big directories, without the linear slowdown caused by reading size metadata from child nodes.

As an example, the CID bafybeiggvykl7skb2ndlmacg2k5modvudocffxjesexlod2pfvg5yhwrqm represents UnixFS directory with over 10k (10100) of files.

Opening it with go-ipfs 0.12 would require fetching size information of each file, which would take a long long time, most likely causing timeout in the browser or CDN, and introducing unnecessary burden on the gateway node.

go-ipfs 0.13 opens it instantly, because the number of items is bigger than the default Gateway.FastDirIndexThreshold and only the root UnixFS node needs to be resolved before the HTML Dir Index is returned to the user.

Notes:

• The default threshold is 100 items.
• Setting to 0 will enable fast listings for all directories.
• CLI users will note that this is equivalent to running ipfs ls -s --size=false --resolve-type=false /ipfs/bafybeiggvykl7skb2ndlmacg2k5modvudocffxjesexlod2pfvg5yhwrqm. Now the same speed is available on the gateways.

🎫 Improved Etag and If-None-Match for bandwidth savings

Every response type has an unique Etag which can be used by the client or CDN to save bandwidth, as a gateway does not need to resend a full response if the content was not changed.

Gateway evaluates Etags sent by a client in If-None-Match and returns status code 304 (Not Modified) on strong or weak match (RFC 7232, 2.3).

⛓️ Added X-Ipfs-Roots for smarter HTTP caches

X-Ipfs-Roots is now returned with every Gateway response. It is a way to indicate all CIDs required for resolving path segments from X-Ipfs-Path. Together, these two headers are meant to improve interop with existing HTTP software (load-balancers, caches, CDNs).

This additional information allows HTTP caches and CDNs to make better decisions around cache invalidation: not just invalidate everything under specific IPNS website when the root changes, but do more fine-grained cache invalidation by detecting when only a specific subdirectory (branch of a DAG) changes.

🌡️ Added metrics per response type

New metrics can be found at /debug/metrics/prometheus on the RPC API port (127.0.0.1:5001 is the default):

• gw_first_content_block_get_latency_seconds – the time until the first content block is received on GET from the gateway (no matter the content or response types)
• gw_unixfs_file_get_duration_seconds – the time to serve an entire UnixFS file from the gateway
• gw_unixfs_gen_dir_listing_get_duration_seconds – the time to serve a generated UnixFS HTML directory listing from the gateway
• gw_car_stream_get_duration_seconds – the time to GET an entire CAR stream from the gateway
• gw_raw_block_get_duration_seconds – The time to GET an entire raw Block from the gateway

🕵️ OpenTelemetry tracing

Opt-in tracing support with many spans for tracing the duration of specific tasks performed by go-ipfs.

See Tracing for details.

We will continue to add tracing instrumentation throughout IPFS subcomponents over time.

How to use Jaeger UI for visual tracing?

One can use the jaegertracing/all-in-one Docker image to run a full Jaeger stack and configure go-ipfs to publish traces to it (here, in an ephemeral container):

$ docker run --rm -it --name jaeger \
    -e COLLECTOR_ZIPKIN_HOST_PORT=:9411 \
    -p 5775:5775/udp \
    -p 6831:6831/udp \
    -p 6832:6832/udp \
    -p 5778:5778 \
    -p 16686:16686 \
    -p 14268:14268 \
    -p 14250:14250 \
    -p 9411:9411 \
    jaegertracing/all-in-one

Then, in other terminal, start go-ipfs with Jaeger tracing enabled:

$ OTEL_TRACES_EXPORTER=jaeger ipfs daemon

Finally, the Jaeger UI is available at http://localhost:16686

Below are examples of visual tracing for Gateway requests. (Note: this a preview how useful this insight is. Details may look different now, as we are constantly improving tracing annotations across the go-ipfs codebase.)

CAR	Block	File	Directory

🩺 Built-in ipfs diag profile to ease debugging

The diag profile command has been expanded to include all information that was previously included in the collect-profiles.sh script, and the script has been removed. Profiles are now collected in parallel, so that profile collection is much faster. Specific profiles can also be selected for targeted debugging.

See ipfs diag profile --help for more details.

For general debugging information, see the debug guide.

🔑 Support for PEM/PKCS8 for key import/export

It is now possible to import or export private keys wrapped in interoperable PEM PKCS8 by passing --format=pem-pkcs8-cleartext to ipfs key import and export commands.

This improved interop allows for key generation outside of the IPFS node:

$ openssl genpkey -algorithm ED25519 > ed25519.pem
$ ipfs key import test-openssl -f pem-pkcs8-cleartext ed25519.pem

Or using external tools like the standard openssl to get a PEM file with the public key:

  $ ipfs key export testkey --format=pem-pkcs8-cleartext -o privkey.pem
  $ openssl pkey -in privkey.pem -pubout > pubkey.pem

🧹 Using standard IPLD codec names across the CLI/HTTP API

This release makes necessary (breaking) changes in effort to use canonical codec names from multicodec/table.csv. We also switched to CIDv1 in block put. The breaking changes are discussed above.

🐳 Custom initialization for Docker

Docker images published at https://hub.docker.com/r/ipfs/go-ipfs/ now support custom initialization by mounting scripts in the /container-init.d directory in the container. Scripts can set custom configuration using ipfs config, or otherwise customize the container before the daemon is started.

Scripts are executed sequentially and in lexicographic order, before the IPFS daemon is started and after ipfs init is run and the swarm keys are copied (if the IPFS repo needs initialization).

For more information, see:

• Documentation: Run IPFS inside Docker
• Examples in ipfs-shipyard/go-ipfs-docker-examples.

RPC API docs for experimental and deprecated commands

https://docs.ipfs.io/reference/http/api/ now includes separate sections for experimental and deprecated commands.

We also display a warning in the command line:

$ ipfs name pubsub state --help
WARNING:   EXPERIMENTAL, command may change in future releases

Yamux over Mplex

The more fully featured yamux stream multiplexer is now prioritized over mplex for outgoing connections.

✅ Release Checklist

For each RC published in each stage:

• version string in version.go has been updated (in the release-vX.Y.Z branch).
• tag commit with vX.Y.Z-rcN
• upload to dist.ipfs.io
  1. Build: https://github.com/ipfs/distributions#usage.
  2. Pin the resulting release.
  3. Make a PR against ipfs/distributions with the updated versions, including the new hash in the PR comment.
  4. Ask the infra team to update the DNSLink record for dist.ipfs.io to point to the new distribution.
• cut a pre-release on github and upload the result of the ipfs/distributions build in the previous step.
• Announce the RC:
  • On Matrix (both #ipfs and #ipfs-dev)
  • To the early testers listed in docs/EARLY_TESTERS.md. Do this by copy/pasting their GitHub usernames and checkboxes as a comment so they get a GitHub notification. (example)

Checklist:

• Stage 0 - Automated Testing
  • Upgrade to the latest patch release of Go that CircleCI has published
    • See the list here: https://hub.docker.com/r/cimg/go/tags
    • ipfs/distributions: bump this version
    • ipfs/go-ipfs: example PR
  • Fork a new branch (release-vX.Y.Z) from master and make any further release related changes to this branch. If any "non-trivial" changes (see the footnotes of docs/releases.md for a definition) get added to the release, uncheck all the checkboxes and return to this stage.
    • Follow the RC release process to cut the first RC.
    • Bump the version in version.go in the master branch to vX.(Y+1).0-dev.
  • Automated Testing (already tested in CI) - Ensure that all tests are passing, this includes:
    • unit, sharness, cross-build, etc (make test)
    • lint (make test_go_lint)
    • interop
    • go-ipfs-api
    • go-ipfs-http-client
    • WebUI
• Stage 1 - Internal Testing
  • CHANGELOG.md has been updated
    • use ./bin/mkreleaselog to generate a nice starter list
  • Infrastructure Testing:
    • Deploy new version to a subset of Bootstrappers
    • Deploy new version to a subset of Gateways
    • Deploy new version to a subset of Preload nodes
    • Collect metrics every day. Work with the Infrastructure team to learn of any hiccup
  • IPFS Application Testing - Run the tests of the following applications:
    • IPFS Desktop
      • Ensure the RC is published to the NPM package (happens automatically, just wait for CI)
      • Upgrade to the RC in ipfs-desktop and push to a branch (example), and open a draft PR to track through the final release (example)
      • Ensure CI tests pass, repeat for new RCs
    • IPFS Companion - @lidel
• Stage 2 - Community Prod Testing
  • Documentation
    • Ensure that CHANGELOG.md is up to date
    • Ensure that README.md is up to date
    • Update docs by merging the auto-created PR in https://github.com/ipfs/ipfs-docs/pulls (they are auto-created every 12 hours)
  • Invite the wider community through (link to the release issue):
    • discuss.ipfs.io
    • Matrix
• Stage 3 - Release
  • Final preparation
    • Verify that version string in version.go has been updated.
    • Merge release-vX.Y.Z into the release branch.
    • Tag this merge commit (on the release branch) with vX.Y.Z.
    • Release published
      • to dist.ipfs.io
      • to npm-go-ipfs
      • to chocolatey
        • Manually run the release workflow
      • to snap
      • to github
        • use the artifacts built in CI for dist.ipfs.io: wget "https://ipfs.io/api/v0/get?arg=/ipns/dist.ipfs.io/go-ipfs/$(curl -s https://dist.ipfs.io/go-ipfs/versions | tail -n 1)"
      • to arch (flag it out of date)
    • Cut a new ipfs-desktop release
  • Submit this form to publish a blog post, linking to the GitHub release notes
  • Broadcasting (link to blog post)
    • Twitter (request in Slack channel #pl-marketing-requests)
    • Matrix
    • Reddit
    • discuss.ipfs.io
    • Announce it on the IPFS Users Mailing List
• Post-Release
  • Merge the release branch back into master, ignoring the changes to version.go (keep the -dev version from master).
  • Create an issue using this release issue template for the next release.
  • Make sure any last-minute changelog updates from the blog post make it back into the CHANGELOG.
  • Mark PR draft created for IPFS Desktop as ready for review.

⁉️ Do you have questions?

The best place to ask your questions about IPFS, how it works and what you can do with it is at discuss.ipfs.io. We are also available at the #ipfs channel on Freenode, which is also accessible through our Matrix bridge.

Release improvements for next time

< Add any release improvements that were observed this cycle here so they can get incorporated into future releases. >

[aschmahmann]

Changelog and Contributors

This release has so many changes that GitHub refuses to put them into a single post. The changes can be seen in https://raw.githubusercontent.com/ipfs/go-ipfs/v0.13.0-rc1/CHANGELOG.md.

[marten-seemann]

@BigLep Is there any updated ETA for this release? The question came up in our discussion about hole punching measurements today.

[BigLep]

Thanks for the ping @marten-seemann . This issue is outdated. Updated. Estimated RC date is 2022-04-22. The risk here is if go-libp2p resource manager work exposes more issues than we estimated.

[guseggert]

We agreed today to release RC1 as soon as we have the UX changes for libp2p resource manager locked down, instead of waiting to test with our infra, to gather community feedback about the RM defaults while we continue to test, with the expectation of releasing an RC2 if we need to make further changes.

[BigLep]

2022-05-03 conversation:

• Update to compile with go 1.18.1 (still targeting 1.17.x onwards)
• We're going to work on doing the RC today

[aschmahmann]

Known issues in RC1:

• Investigate why /ipfs/{invalid cid} returns different Status Code now #8924 (will either be fixed or the change documented), but this will get deferred to 0.14
• ipfs init error on Windows: Error loading seed assets #8943
• AcceleratedDHTClient needs more resources than the ResourceManager has by default #8945
• fix: holepunch in go-libp2p 0.19.2 #8966 – go-libp2p hole punching fix (0.19.2)

Not new but additional bug fixes that will be added:

• TarWriter.WriteFile stuck on select when writing to io.pipe #8957 (@BigLep added this - comment/feedback welcome)