Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

MFS/IPFS Pin performance #1672

Open
RubenKelevra opened this issue May 21, 2022 · 3 comments
Open

MFS/IPFS Pin performance #1672

RubenKelevra opened this issue May 21, 2022 · 3 comments
Labels
kind/bug A bug in existing code (including security flaws) need/triage Needs initial labeling and prioritization

Comments

@RubenKelevra
Copy link
Collaborator

RubenKelevra commented May 21, 2022

Additional information:

  • OS: Linux
  • IPFS Cluster version: 1.0.1
  • Installation method: dist.ipfs.io
  • IPFS version:
go-ipfs version: 0.14.0-dev-a72753bad-dirty
Repo version: 12
System version: amd64/linux
Golang version: go1.18.2

Describe the bug:

I'm using the MFS to construct my file/folder structure, which then gets added to the cluster-pinset.

I'm currently dealing with performance issues, as the pinning operations after having the full file/folder-strucuture already inside the MFS takes an s**tload of IO.

It kinda needs to be pinned I guess, as I will modify the files/folders before pinning the next ones (and otherwise the GC run might clean things up inadvertently), but isn't there any way to tell ipfs that this stuff is already fully locally available and not traverse the full ~500GB for every change?

Edit: Ah and yes, I already do ipfs-cluster-ctl pin update --no-status $oldcid $newcid followed by an ipfs-cluster-ctl pin rm --no-status $oldcid.

@RubenKelevra RubenKelevra added kind/bug A bug in existing code (including security flaws) need/triage Needs initial labeling and prioritization labels May 21, 2022
@RubenKelevra
Copy link
Collaborator Author

This is how this looks like in iotop:

Screenshot_20220522_071929

And these are some stats from the machine:

Screenshot_20220522_072956
Screenshot_20220522_072407
Screenshot_20220522_072310
Screenshot_20220522_072211

@RubenKelevra
Copy link
Collaborator Author

Here's the IPFS bugreport for this:

ipfs/kubo#8988

@RubenKelevra
Copy link
Collaborator Author

RubenKelevra commented May 24, 2022

And just for context, that's my setting for pinning concurrences on odin:

[ipfs@odin ~]$ cat .ipfs-cluster/service.json | grep pin_tracker -A5
  "pin_tracker": {
    "stateless": {
      "max_pin_queue_size": 1000000,
      "concurrent_pins": 1
    }
  },

I now tried to reduce the concurrent bitswap tasks, hope this helps (without hurting the performance too much):

[ipfs@odin ~]$ ipfs config show | grep Internal -A7
  "Internal": {
    "Bitswap": {
      "EngineBlockstoreWorkerCount": 4,
      "EngineTaskWorkerCount": 2,
      "TaskWorkerCount": 4
    },
    "UnixFSShardingSizeThreshold": "1MB"
  },

A nice workaround would actually be to not pin the folder at all on this node, as I run the GC on demand anyway, when I don't modify the MFS. So all data should be always available until the new CID is put into the cluster-pinset.

Blocker for this is that I cannot exclude the local peer (or all trusted peers for that matter) from beeing part of the pinset. Blocker is tracked here: #997

@hsanjuan maybe you can have another look at this? I think it's fairly easy to implement #997, as it's just a filter for selecting cluster-members as part of the pinset if not -1, -1 replication is set.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/bug A bug in existing code (including security flaws) need/triage Needs initial labeling and prioritization
Projects
None yet
Development

No branches or pull requests

1 participant