Too much duplicate data

[Stebalien]

When testing on the live network, I'm getting 76% duplicate data when pinning dist.ipfs.io.

This should have been fixed so something is going very wrong here.

[Stebalien]

Unscientifically, turning off eagerly sending small blocks may help? I'm not sure.

However, it only reduced duplicate data from 76% -> 45%. And it seems like duplicate data becomes more of a problem as the query runs.

[Stebalien]

Scientifically, that option shouldn't make a difference because I only made the change on my own node. But the nodes I'm downloading from should mostly be old.

[Stebalien]

I've tried removing the "don't have timeout" and I;m not seeing any improvement. However, it really looks like this problem gets worse and worse as the download progresses.

NOTE: I may have messed up when disabling the "don't have timeout"s. You should double check these results.

[dirkmc]

This was caused by two issues:

• ipfs pin is using a separate session for each block, which essentially causes every want to be broadcast, so we end up with a lot of duplicates
• when asking for a block from peers running an older version of bitswap, we wait for a short time before assuming they are not going to respond and then move on and ask another peer. However the timing was not conservative enough. Fix: Change timing for DONT_HAVE timeouts to be more conservative #371

[Stebalien]

Ok, I found the bug in pin. We were wrapping the dagService and FetchGraph was doing a type assertion.

[Stebalien]

@dirkmc thanks for investigating this. This looks fixed.