Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Improve Garbage Collection Performance #3333

Closed
whyrusleeping opened this issue Oct 27, 2016 · 5 comments
Closed

Improve Garbage Collection Performance #3333

whyrusleeping opened this issue Oct 27, 2016 · 5 comments
Labels
topic/perf Performance
Milestone
Improve Performance

Comments

@whyrusleeping
Copy link
Member

IPFS currently has two different garbage collectors that run. The manual ipfs repo gc, and the provider cache cleanup routine. Currently, ipfs repo gc is very slow, and the providers garbage collection is very ineficient (causes mars to OOM frequently) due to what appears to be the same issue. The problem i've identified so far is that creating ds.Keys from the individual datastores, then parsing those back into strings for different types is quite expensive.

Lets try and figure out how to make both of these usecases faster.
@whyrusleeping whyrusleeping added the topic/perf Performance label Oct 27, 2016
@whyrusleeping whyrusleeping added this to the Improve Performance milestone Oct 28, 2016
@kevina
Copy link
Contributor

kevina commented Oct 31, 2016

Related issue: ipfs/go-datastore#40

@kevina
Copy link
Contributor

kevina commented Nov 14, 2016

Speeding up Blockstore.AllKeysChan() should help improve ipfs repo gc, see #3376. I have not looked into the overhead of creating ds.Keys specifically yet; in particular I have not looked at the providers garbage collection code at all.

@whyrusleeping
Copy link
Member Author

@kevina i've been looking at the providers GC code recently, i'll push a couple PRs towards improving that soon.

@Kubuxu Kubuxu added the status/deferred Conscious decision to pause or backlog label Nov 28, 2016
@kevina
Copy link
Contributor

kevina commented Dec 5, 2016

Related #3462

@whyrusleeping
Copy link
Member Author

I think @kevina has improved the perf of both of these routines pretty decently. Well enough for me to feel okay closing this issue (for now).

@whyrusleeping whyrusleeping removed the status/deferred Conscious decision to pause or backlog label Mar 18, 2017
@RubenKelevra RubenKelevra mentioned this issue Mar 31, 2020
Open
35 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
topic/perf Performance
Projects
None yet
Milestone
Improve Performance
Development

No branches or pull requests
3 participants
@whyrusleeping @kevina @Kubuxu