hashfile.checkout: use multiprocessing.Pool to checkout concurrently

[skshetry]

No description provided.

[dberenbaum]

Did you try it with threads but found it faster with multiprocessing and batches? It's not obvious to me which would be better here. Otherwise LGTM.

[skshetry]

Did you try it with threads but found it faster with multiprocessing and batches? It's not obvious to me which would be better here. Otherwise LGTM.

On my linux machine, threadpoolexecutor was slower than sequential operation. So I used multiprocessing. But mp has overhead, so I had to resort to use batches to reduce the overhead.

It's interesting to see that M1 has fast computation, but slower filesystem. But for me, filesystem operations are quite fast, but computation is the bottleneck. (Linux prob. has low overhead on system calls, and maybe Linux's direntry cache is helping here).

[dberenbaum]

@skshetry Why close this one?

[skshetry]

It's not clear to me how much improvement this makes, compared to the complexity it introduces.