ipfs pin add is very slow / hanging

[Voker57]

Version information:

go-ipfs version: 0.4.4-
Repo version: 4
System version: amd64/linux
Golang version: go1.7

Type:

Bug

Priority:

P3

Description:

I have a directory with 25G files in IPFS, to which I'm adding another file with 4GB. After adding, I'm trying to pin new directory version.

ipfs pin add QmPfmRdxrBL3nV9gChCMfv9uAbQSGmh1mcZ6Y1BfwdTBFd
# Actively churning disk for some time, lots of ipfs daemon threads competing for IO, then process is consuming ~15% CPU time indefinitely. I left it running for 10 hours, then killed
# However...
ipfs refs -r QmPfmRdxrBL3nV9gChCMfv9uAbQSGmh1mcZ6Y1BfwdTBFd > /dev/null
# Done in ~15 minutes.

[Kubuxu]

What happens if try pinning after the refs call?

[Voker57]

Results are inconclusive. One time I got fast pin add ~15 minutes, now it hangs anyway.

[Kubuxu]

If you can, providing following debug info would be quite useful: https://github.com/ipfs/go-ipfs/blob/master/docs/debug-guide.md#beginning

[Kubuxu]

Best when you observe that it is stuck.

[Kubuxu]

Make also sure that garbage collector is disabled, it might interfere is you are adding and not pinning.

[Voker57]

Debug info in "stuck" phase: http://dump.bitcheese.net/files/lysomul/ipfs.debug.tar.gz
GC is disabled.

[Voker57]

In "stuck" phase ipfs daemon is sending 10MB/s somewhere. If command is interrupted, network activity disappears.

[wigy-opensource-developer]

This looks really familiar from my gx publish hassles: #3052 (comment)

[Voker57]

I've ran the same pin add with daemon running in offline mode and got more clear goroutine dump: http://dump.bitcheese.net/files/zapumel/ipfs.stacks

I'm new to go so I'm kind of at loss at this, but it appears that fetchNodes is getting stuck on selecting nodes from channel of GetMany() ? I ran the pin op several times and it gets stuck on different hashes each time. Not sure why no goroutines called from github.com/ipfs/go-ipfs/merkledag.fetchNodes.func2 are in the dump.

[Voker57]

Not sure why but reverting 0c14b41 fixes the problem

[Kubuxu]

Looks like the concurrency limit set over there was too low.

We should probably uncap it, check what is reached during normal operation and go on from there.

[ghost]

Consistently seeing similar behaviour when trying to pin the 33c3 schedule (e.g. /ipfs/QmUFz3URXrbRSAUot3j5hPo3Nw3qcGywZXc4wMMKdAd8Js) on the gateways.

[Voker57]

Problem is not low concurrency limit but possible deadlocking when both output and input of FetchNodes() are full. I think I solved it with deferring input into goroutines: #3550

[kevina]

@Kubuxu do you want to take this. I might be able to but I would need a good test case to do what you suggest when you said:

Looks like the concurrency limit set over there was too low.

We should probably uncap it, check what is reached during normal operation and go on from there.

[Kubuxu]

I would wait, I have to evaluate @Voker57 solution.

[whyrusleeping]

@Voker57 @Kubuxu could one of you check that the code in #3571 addresses the issue? I've had a hard time reproducing the failure in unit tests (though we should make a note to add 'pin a really big thing' as one of our larger scale integration tests)

[Voker57]

Issue is not only in pin add hanging but also being slow. Is pin add verifying integrity of whole stored tree really required?

One way to solve this would be keeping index of DAG links and traverse it when needed, only checking if nodes exists on disk instead of parsing all of them each time. Proper implementation of index imo required adding relational DB to ipfs (sqlite? pluggable backend for postgres/mysql?), can be done quick&dirty with forward and reverse indexes in leveldb though.

[whyrusleeping]

@Voker57 Yeah, we have a 'linkservice' abstraction that is supposed to be for this type of optimization. I'm going to bump this issue from 0.4.5 so we can focus on the speed of things before closing it.

[kevina]

Note: The 'linkservice' abstraction will only have an effect when raw leaves are used.

[Kubuxu]

@kevina the linkservice currently only provides optimization when rew leaves are used, there is nothing stopping us from adding cache/index there for other nodes types.

[ghost]

I have a same problem.

I have IPFS 0.4.4.

I try to duplicate files from one computer on another. The .ipfs catalog size — 42 GB. Has received the pins list on one computer and I do pin add on another. At first went quickly, but now when the extent of base on other computer makes 28 GB, process takes 10-15 seconds on each hash. And they are 30000. What worst of all, after interruption of process and restart everything brakes from the very beginning. Slowly even addition of the hashes which are already available in base works.

Whether there is an opportunity to somehow accelerate this process?

[Voker57]

@balancer in 0.4.4 pin add is broken and can randomly deadlock with a lot of data to pin.
In latest git it's properly working and you're welcome to test my patch #3642 which makes pinning of present data much faster.
Also unrelated, but why do you have 30000 pins? Are these all different directories which have to go to different parts of filesystem?

[whyrusleeping]

@balancer This issue has been resolved in 0.4.5 (ipfs pin add will no longer hang and crash), and will be further improved in 0.4.6.

Adding 30,000 pins however will be very inefficient in 0.4.5 due to the issue that was resolved here: #3640

[whyrusleeping]

Closing this now as the optimized code in #3598 was merged

[JazzTp]

go-ipfs_v0.4.17_linux-amd64

ipfs pin add hangs, it only happened so far with three files, first time today

ipfs add <hashcode> -t seems to be working normally, ipfs refs local however doesn't show those hashcodes, and of course they are not in the output of ipfs pin ls -t recursive either.

(OFFTOPIC I haven't been using this node in a while but I had done quite some experimenting and a bit of script writing, 4.1 GB node, Max is set at 500GB for now but wouldn't mind enlarging, the problem was another, internet service provider not allowing to open ports to receive incoming connections, couldn't find out how to proxify the ipfs daemon, searched+tried with various proxifiers with no success, so finally I didn't bother to buy some VPN-proxy service allowing ports forwarding)

[Stebalien]

@JazzTp please open a new issue. This one has been closed for quite a while.

[JazzTp]

OK, going to do that ASAP!