IPIP-412: Signaling Block Order in CARs on HTTP Gateways #412
Conversation
lidel
force-pushed
the
ipip-car-order-signaling
branch
from
250bbe2
to
5818113
Compare
lidel
changed the title
lidel
changed the title
lidel
force-pushed
the
ipip-car-order-signaling
branch
2 times, most recently
from
81f42be
to
63ea4ff
Compare
lidel
changed the title
|
Does identity CID handling belong in here or #402? It feels very closely related to duplicate blocks so might be in scope to be described here. |
There was a problem hiding this comment.
From a clients point of view:
order=dfs; dups=ymeans "make it as easy as possible for me to unpack"order=dfs; dups=nmeans "i will track blocks but verifiably get me to the leaf blocks asap!"
Are there other combos that make sense for a client to set?
Could we encode this as a single param to avoid having combinations tha don't make sense. Something like pack=dfs implying plz dedupe if you can and pack=dfs-stream implying dupes must to be included?
For the server-side, we may struggle to guarantee full dedupe in freeway. A CF worker gets 128MB of ram shared between concurrent requests from that region. In theory we could track ~2 million CIDs, but that number will drop as it gets busy. Stable ordering will eat some more ram too.
Let's have discussion in #412 (comment)
I was already proposing to join dupness and ordering in the previous IPIP but it was apparently really confusing.
Allows the gateway to use stream different parallel downloads and the client can still incrementally and stream the data if you are writing it to an (this is being moved to an other IPIP) |
|
@olizilla in my mind these flags are separate for a reason: If gateway can't guarantee anything about duplicates, but can do DFS, it is better if it returns CAR |
# Goals This adds support for depth first traversals to Dagula. It's designed to be in conformance with ipfs/specs#412. (will update if parameter names change) # Implementation - adds order parameter to options struct for getPath and get - renames breadFirstSearch to blockLinks (it isn't really "bread first search" so much as "all the links in this block"). - handles ordering change inside of `get`, using recursion for depth-first-search - adds tests for both orders. --------- Co-authored-by: Alan Shaw <alan.shaw@protocol.ai>
This PR updates dagula and implements block order signaling via HTTP Accept header as specified in ipfs/specs#412. It also renames the query param `car-scope` to `dag-scope` and changes the scope value `file` to `entity` as these were updated in ipfs/specs#402.
First draft based on various prior art and recent discussions cited in the header front matter.
Co-authored-by: Oli Evans <oli.evans@gmail.com>
First draft based on various prior art and recent discussions cited in the header front matter.
lidel
force-pushed
the
ipip-car-order-signaling
branch
3 times, most recently
from
c9e231c
to
e94a390
Compare
- moving spec details to trustless-gateway - rebasing on top of ipip-402 - fixing linter
lidel
force-pushed
the
ipip-car-order-signaling
branch
from
e94a390
to
02a8465
Compare
There was a problem hiding this comment.
We have conformance tests (ipfs/gateway-conformance#87), reference order=dfs implementation is in boxo/gateway (ipfs/boxo#370) and plan to ship this with Kubo 0.22 (ipfs/kubo#9989).
Thank you to everyone involved, especially asking questions and pointing gaps – it all made the specification better.
I believe this is ready for being ratified and merged after final fixture CIDs are known (comment below)
Will flag this during IPFS-Implementers-Sync-2023-07-20.
There was a problem hiding this comment.
There were no ratification concerns during IPFS Implementers sync, nor during the two weeks after it.
Merging! 🎉
For future reference:
- CAR fixtures listed here are part of ipfs/gateway-conformance, consider using that for compliance testing before rolling your own tests.
- implementation of IPIP-412 lives in
boxo/gatewaylibrary and shipped for the first time with Kubo 0.22.0-rc1.
Thank you to everyone involved in this one ❤️
cc @hannahhoward @aschmahmann @Jorropo @willscott @rvagg @ribasushi @John-LittleBearLabs @fabricedesre
TODO
boxo/gateway: feat(gateway): support for order=, dups= parameters from IPIP-412 boxo#370What
This IPIP is based on prior art from IPIP-330, and recent discussions cited in the header front matter, including ones in IPIP-402.
Why
👉 To keep scope in check, I made a conscious decision to only specify a single opt-in ordering that aims to reduce resource usage on light clients like lighter read-only Brave, IPFS in Chromium, Capyloon, RAPIDE, or even bifrost-gateway.
These are the north star use cases for now.
TLDR How
The proposed solution reuses HTTP content type negotiation from RFC9110 and introduces two new parameters for the content type headers
in HTTP requests and responses:
orderanddups.orderparameter allows the client to indicate its preference for aspecific block order in the CAR response, for now we only specify the Depth-First Search order.
dupsparameter specifieswhether duplicate blocks are allowed in the response
The proposed spec creates room for other ones to be added in future IPIPs.
cc @hannahhoward @aschmahmann @Jorropo @willscott @rvagg @ribasushi