Introduce specification for cascading lookup query parameter #9
Conversation
|
@aschmahmann I would love your review on this spec change please |
|
also probably relevant for @lidel |
IPNI.md
Outdated
| @@ -463,6 +463,11 @@ and uses the multihash portion of the CID only. | |||
| * `cid` - _Required_. The default string representation of the Cid. Currently, Base32 is used for | |||
| CIDv1 as the encoding for the multibase string and Base58 is used for CIDv0. | |||
|
|
|||
| ##### Query Parameters | |||
|
|
|||
| * `cascade` - _Optional_. The comma separated alternative routing systems to which the lookup is cascaded in addition to searching IPNI index records. If unspecified, only IPNI index records are searched. Supported values: | |||
There was a problem hiding this comment.
To me, the term "cascade" implies a sequential order. Does that mean an IPNI indexer would wait for an initial response from the IPNI index, and if no results, go on to the next routing system, or ask all routing systems regardless of the results?
If the latter, it may be more clear to change the terminology to "routes" or "routing networks". Otherwise, let's clarify if order matters, or not.
There was a problem hiding this comment.
Does that mean an IPNI indexer would wait for an initial response from the IPNI index, and if no results, go on to the next routing system, or ask all routing systems regardless of the results?
The specification here is specifically loose on the strategy. It is left to the implementers to decide on performing sequential lookup, only carry on if no result is found, find in parallel, etc. This offers flexibility depending on the implementations and whatever SLOs they strive for.
I understand the word "cascade" implies order. What it really means here is "also look up in". "cascade" was the only single word I could think of that was close enough which was also used in discussions with folks. I also think regardless of order, "cascade" fits. Because the specification states that whether the parameter is specified or not, IPNI is always searched and any found results are included.
In practice results will most likely be streamed back as they are found as long as the response timeout allows it (i.e. ndjson streaming that is planned to be implemented across HTTP delegated routing and cid.contact). Because, lookup in different routing systems has different latency characteristics. If order becomes an issue we can later introduce order query parameter that further narrows down the search strategy preference.
There was a problem hiding this comment.
Added extra paragraphs to spell out the absence of order constraints.
It is increasingly useful for the users to be able to search alternative routing systems through the familiar IPNI endpoints. This greatly simplifies the barrier to simply find providers for a given CID, and avoid the need to learn about explicit routing systems such as the DHT. Further, the results can be cached and reused by others considering the fact that a typical IPNI instance will heavily utilise caching mechanisms regardless of cascading lookups. The cascading lookup must be configurable to avoid doubling load on other routing systems in a case where, having searched other routing systems already, the user exclusively wants to search IPNI. For example, Kubo `v0.18.0` is configured by default to lookup records on the DHT and on IPNI. Therefore, automatic cascading of lookups onto the DHT by IPNI would result in duplication of load and waisted effort since Kubo independently looks up the DHT. The changes here introduce a new query parameter, called `cascade` for the lookup APIs, where the user can specify a list of comma separated alternative routing systems to which the query is cascaded in addition to IPNI. By default no cascading occurs. This allows Kubo to gracefully continue functioning while enabling other clients to optionally cascade lookups onto the IPFS DHT.
Specify protocol to find out what cascade options are available to a client using `OPTIONS` method and a specific HTTP response header. This enables clients to programmatically determine the alternative routing systems to which a lookup may optionally be cascaded. Add extra wording to clarify that there are no constraints on the order by which results are returned at the presence of `cascade` query option.
masih
force-pushed
the
masih/cascade_lookup_spec
branch
from
fa14d37
to
b3060a0
Compare
It is increasingly useful for the users to be able to search alternative routing systems through the familiar IPNI endpoints. This greatly simplifies the barrier to simply find providers for a given CID, and avoid the need to learn about explicit routing systems such as the DHT. Further, the results can be cached and reused by others considering the fact that a typical IPNI instance will heavily utilise caching mechanisms regardless of cascading lookups.
The cascading lookup must be configurable to avoid doubling load on other routing systems in a case where, having searched other routing systems already, the user exclusively wants to search IPNI. For example, Kubo
v0.18.0is configured by default to lookup records on the DHT and on IPNI. Therefore, automatic cascading of lookups onto the DHT by IPNI would result in duplication of load and waisted effort since Kubo independently looks up the DHT.The changes here introduce a new query parameter, called
cascadefor the lookup APIs, where the user can specify a list of comma separated alternative routing systems to which the query is cascaded in addition to IPNI. By default no cascading occurs. This allows Kubo to gracefully continue functioning while enabling other clients to optionally cascade lookups onto the IPFS DHT.