IPIP-0523: Prefer format request param over Accept header

[achingbrain]

Browsers will always send an Accept header so if present the format request param should take priority, not the other way round as the spec currently says.

Things to land before ratifying (merging) this IPIP

• test!(ipip-523): prefer format query param over accept header gateway-conformance#252
  • feat(gateway): IPIP-523 format query over Accept header boxo#1074
    • test: IPIP-523 format query precedence over Accept header kubo#11086 (this is where we use everything end-to-end and want CI to be green – and it is!)

[github-actions]

🚀 Build Preview on IPFS ready

• 🔎 Commit: 06f8cc5
• 🔏 CID bafybeiao2pgy55oqmaepfrloui6rfg45ol7qsfgm3zlusc7ugi4gxsgn54
• 📦 Preview:
  • dweb.link
  • w3s.link
  • inbrowser.link

[lidel]

Thank you for cleaning this up. This is def. a technical debt we asccumulated by adding features over time. Work on Service Worker gateway makes it a good time to do this.

lgtm, added some meat to missing/empty sections

GO implementation is ready – details in ipfs/gateway-conformance#252 (review) (if no concerns, we will merge later this week and ship in Kubo 0.40)

[lidel]

ℹ️ @achingbrain fyi I've expanded the Compatibility section. While formally this is a breaking change, in practice boxo/gateway already had an escape hatch in customResponseFormat which prioritized format if Accept has a wildcard or used something other than raw, car or ipns-record

(imo this is good news, means IPIP is actually cleaning up and formalizing real world behavior in browsers)

[aschmahmann]

Maybe also a bad alternative since there's still a need to handle wildcard complexity, but it seems like it'd be helpful if implementations rejected as invalid conflicting requests (e.g. request for a block in the header, but a CAR in the query parameter).

i.e. the rule becomes:

• If the accept header asks for one of multiple possible options and none of them match the one requested in the query parameter (including wildcards matching) -> error
• Otherwise take the more precise variant (e.g. normally the query parameter except for CARs where you might need to check the accept header for a more precise CAR description)
  • If there's ever ambiguity here just error

[lidel]

It was not included in the IPIP on purpose, for pragmatic reasons (cure causes more problems than it solves):

• The whole point of this IPIP is to simplify: "URL param ?format always wins" means no extra code for conflict handling needed. No surface for bugs no matter how your HTTP stack looks like.

  • The important nuance is that this aligns with how CDNs and HTTP caches actually work in practice: they often ignore or poorly handle Accept headers. This includes defaults in software like Nginx and major CDNs like Cloudflare, which can be... liberal with RFC compliance at the edges.

  • In the real world, a request traverses N hops, each one with different quirks. The URL is the only reliable way to ensure correct namespacing of HTTP caches for different response formats across all hops.

    • This is not a theory. ipfs.io / delegated-ipfs.dev are literally adding explicit ?format= on the fly (internally) at Cloudflare CDN right now to ensure CAR, raw, and deserialized responses are not commingled at their and/or internal Nginx caches. Yes, it is this janky, and yes, very likely everyone else running a gateway either ignores this, or is not aware they (in some permutations of caches, cdns, and default reverse-proxy configurations) may have commingled caches if they don't use explicit ?format= and only depend on Accept.

    • Adding a HTTP 400 requirement on conflicting values would be (IMO) a dead/janky spec. At very least, would make the spec harder to test reliably via conformance:

      • In many deployments, CDNs/reverse-proxies strip or transform Accept headers before they reach the cache or final server. A test expecting HTTP 400 for "conflicting headers" would fail with HTTP 200,
        not because the gateway is broken, but because the CDN never forwarded the Accept header in the first place, or because someone has extra cache commingling protections like ipfs.io does.
      • If we dont include it in spec, but do it in boxo, then it would not pass gateway-conformance, forcing everyone to add -skip for that test (aka dead spec / dead test).

[lidel]

Good catch on flagging this, I realize this rationale is likely hidden tribal knowledge. Added shortened version of the above to IPIP in 06f8cc5