perf: add parallelism to hashing

[rklaehn]

The most expensive thing when adding a file is hashing the content.

This adds parallelism to this hashing by splitting the tree building into two stages. One where the leafs are encoded, and one where the actual tree is being built.

This makes sure that iroh-cli now uses all available cores, and therefore increases perf quite a bit. But we are now limited by the DB.

[rklaehn]

Not sure if there is a nicer way to do this than .map(|x| x.and_then(|x| x)).

[dignifiedquire]

can't you use flatten?

[rklaehn]

Now that you mention it... But apparently not.

[b5]

I'm not qualified to review async code, but in smoke testing I did an iroh add -> iroh get loop on a 350MB file, and the result that came back was missing data from the end of the file, roughly ~75MB worth. I tested it on main without issue. Will play some more to see if the problem is repeatable

1. Any idea what might cause such a thing?
2. We should probably make sure there's a solid roundtrip test if we don't already have one.
3. All of this is totally worth it, the speedup is very nice

[rklaehn]

I'm not qualified to review async code, but in smoke testing I did an iroh add -> iroh get loop on a 350MB file, and the result that came back was missing data from the end of the file, roughly ~75MB worth. I tested it on main without issue. Will play some more to see if the problem is repeatable

Thanks for the test. Trying things out is often more useful than staring at the code. I will check this, and also try to figure out how to test this.

[rklaehn]

I'm not qualified to review async code, but in smoke testing I did an iroh add -> iroh get loop on a 350MB file, and the result that came back was missing data from the end of the file, roughly ~75MB worth. I tested it on main without issue. Will play some more to see if the problem is repeatable

1. Any idea what might cause such a thing?
2. We should probably make sure there's a solid roundtrip test if we don't already have one.
3. All of this is totally worth it, the speedup is very nice

Can you retry once #408 is merged?

[b5]

hrm, still seeing the issue on get when running this branch. hash results are the same compared to main, but this might be an issue with the content not actually being in the blocks?

[rklaehn]

hrm, still seeing the issue on get when running this branch. hash results are the same compared to main, but this might be an issue with the content not actually being in the blocks?

Can you tell me how to reproduce this? I tried with a 300m random file, and am not seeing this.

[b5]

I've tested this a bunch locally, and am good with it. I'd like @dignifiedquire to sign off