Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Feat: pin add --max-depth (arbitrary depth recursive pins) #5142

Open
wants to merge 3 commits into
base: master
Choose a base branch
from
Open

Feat: pin add --max-depth (arbitrary depth recursive pins) #5142

wants to merge 3 commits into from

Commits on Jun 21, 2018

  1. Feat: Arbitrary-depth recursive pin levels. 

    This implements #5133 introducing an option to limit how deep we fetch and store
the DAG associated to a recursive pin ("--max-depth"). This feature
comes motivated by the need to fetch and pin partial DAGs in order to do
DAG sharding with IPFS Cluster.

This means that, when pinning something to --max-depth, the DAG will be
fetched only to that depth and not more.

In order to get this, the PR introduces new recursive pin types: "recursive1"
means: the given CID is pinned along with its direct children (maxDepth=1)

"recursive2" means: the given CID is pinned along with its direct children
and its grandchildren.

And so on...

This required introducing "maxDepth" limits to all the functions walking down
DAGs (in merkledag, pin, core/commands, core/coreapi, exchange/reprovide modules).

maxDepth == -1 effectively acts as no-limit, and all these functions behave like
they did before.

In order to facilitate the task, a new CID Set type has been added:
thirdparty/recpinset. This set carries the MaxDepth associated to every Cid.
This allows to shortcut exploring already explored branches just like the original
cid.Set does. It also allows to store the Recursive pinset (and replaces cid.Set).
recpinset should be moved outside to a different repo eventually.

TODO: tests
TODO: refs -r with --max-depth

License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
    @hsanjuan
    hsanjuan committed Jun 21, 2018
    Configuration menu
    Copy the full SHA
    2bd401c View commit details
    Browse the repository at this point in the history

  2. Feat: add --max-depth to the "refs" command 

    License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
    @hsanjuan
    hsanjuan committed Jun 21, 2018
    Configuration menu
    Copy the full SHA
    45e9a4c View commit details
    Browse the repository at this point in the history

  3. Fix/GC: Do not recursively descend all internal pins 

    InternalPins() is a pinset composed by:

- Recursive pins CIDs
- Direct pins CIDs
- The empty node CID
- A root CID pointing to all above (and any of the subbuckets that may have been created)

It is only set during Flush/Load operations for the pinner.

Thus recursively exploring internal pins in order to decide which CIDs are safe
from GC only re-explores the recursive DAGs and should not be necessary.

Mind that, previously, the CidSet will correctly prune any already explored branches so
it did not have pernicious effects. But now it does.

License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
    @hsanjuan
    hsanjuan committed Jun 21, 2018
    Configuration menu
    Copy the full SHA
    9abb992 View commit details
    Browse the repository at this point in the history