feat: X-Ipfs-Roots for smarter HTTP caches

[lidel]

This PR adds X-Ipfs-Roots to Gateway responses.

X-Ipfs-Roots is a way to indicate all CIDs required for resolving path segments from X-Ipfs-Path – both headers are meant to improve interop with existing HTTP software (load-balancers, caches, CDNs).

It allows HTTP caches and CDNs make better decisions around cache invalidation: not only invalidate everything under specific IPNS website when the root changes, but also do more fine-grained caching and content distribution when CID/block aware solutions are in place.

Rationale

X-Ipfs-Path returned on mutable /ipns/ paths is not enough, and Gateway operator community raised the need for additional metadata to enable smarter cache.

This is a low-hanging fruit from #8717 that can be shipped independently of other improvements.

Demo

Before

curl -Is http://127.0.0.1:8080/ipns/docs.ipfs.io/concepts/glossary/ | grep -iE "X-Ipfs|Etag"
Etag: "DirIndex-512eb789cd905714e03f29d4e04de7549e8c9c3e_CID-QmZcUjyj9VamSpGJ8VkWuZAvRMKinm6kBBUhnGy3omNZJC"
X-Ipfs-Path: /ipns/docs.ipfs.io/concepts/glossary/

After

curl -Is http://127.0.0.1:8080/ipns/docs.ipfs.io/concepts/glossary/ | grep -iE "X-Ipfs|Etag"
Etag: "QmZcUjyj9VamSpGJ8VkWuZAvRMKinm6kBBUhnGy3omNZJC"
X-Ipfs-Path: /ipns/docs.ipfs.io/concepts/glossary/
X-Ipfs-Roots: Qmem46CBFD4XHVBsGtoTbM2u5nuRx1XUiRrSXqUKrBxtpa,QmasfzzoGU8FE5z4w3GDfW3DrJVvGPCAbfgTDrGWarxRso,QmZcUjyj9VamSpGJ8VkWuZAvRMKinm6kBBUhnGy3omNZJC

TODO

• add X-Ipfs-Roots (all path CIDs folded into a single header)
• gather feedback from CDN/gateway operators
• add sharness tests for X-Ipfs-Roots (new) and X-Ipfs-Path (missing) and Etag (missing)
• fix a bug where we return DirIndex- Etag for index.html response (on EVERY WEBSITE EVER 🙈)

[thattommyhall]

I didn't immediately understand why it's plural. In my head $root_cid matches whats currently the _dnslink entry.

The way I think about it is that at any moment docs.ipfs.io has a root-cid in _dnslink and the content at docs.ipfs.io$uri would be the same as ipfs://$root_cid$uri so that when one uri expired and got refreshed we could be sure the the root-cid had not changed and we could keep serving up cached content rather than revalidating for every uri (im thinking the cache key would be $root_cid$uri)

Your strategy here permits more granular cache invalidation, I am unsure if I could make use of it in nginx, but its good its there!

[mathew-cf]

This is a great caching change for paths that can be resolved to a CID by DNS

[guseggert]

LGTM so far

[lidel]

Pushed bunch of tests that document and protect the behavior around X-Ipfs-Roots (added in this PR)
but also existing X-Ipfs-Pathand Etag (the latter had a bug – also fixed in this PR).

This is ready for final reviews / comments. If no concerns, the plan is to merge and ship this in go-ipfs 0.13.