Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Repo Size Constraints #972

Open
jbenet opened this issue Mar 29, 2015 · 4 comments
Open

Repo Size Constraints #972

jbenet opened this issue Mar 29, 2015 · 4 comments
Labels
topic/repo Topic repo

Comments

@jbenet
Copy link
Member

jbenet commented Mar 29, 2015

It is a common problem already that the space of go-ipfs grows unbounded. We want to have the ability to limit the growth.


Complicating factors:

  • pinned vs unpinned objects
  • capacity should be configurable
  • capacity should incorporate all repo contents
  • adding (pinning) content beyond capacity

I've been thinking of a setup with two thresholds:

  • gc threshold: if after this limit, all commands trigger a repo gc.
  • local storage capacity: if adding an object to repo would increase capacity, command errors. (i.e. never exceed capacity)
@jbenet jbenet added the topic/repo Topic repo label Mar 29, 2015
@jbenet jbenet added this to the Resource Constraints milestone Mar 29, 2015
@cryptix
Copy link
Contributor

cryptix commented Mar 29, 2015

A threshold triggering repo gc sgtm. I'd be interested in ratio stats of pinned vs unpinned objects in general.

@jbenet
Copy link
Member Author

jbenet commented Jul 15, 2015

From #1482, i think we should have one global maximum storage, and trigger GC when we past a watermark (say max - 1/5MB close). ideally we could also configure the GC watermark to bring it lower.

@rht
Copy link
Contributor

rht commented Jul 16, 2015

node.repo.datastore.storage_gc_watermark ~ hard limit on pin size? Otherwise even GC after commands won't bring down the repo size.

@rht
Copy link
Contributor

rht commented Jul 16, 2015

How about swapping the hard limit role?
Instead of storage_gc_watermark + storage_max (hard limit), use pin_max (hard limit) + blurriness? blurriness/standard_dev/swappiness is like the amount of bluriness between local storage and peer storage. Default value is 100 (100% of pin_max).
By now, the only concern for global size cap other than blockstore is leveldb store size.

Also, where to put the trigger check?

  1. every 100 GetBlocks cycle
  2. every 10 mins
  3. every time a command is run

@daviddias daviddias removed the backlog label Jan 2, 2016
@Stebalien Stebalien removed this from the Resource Constraints milestone Apr 29, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
topic/repo Topic repo
Projects
None yet
Development

No branches or pull requests

6 participants