Add parallelism options to speed up dvc add

[Ykid]

One of the project I did contains directories of more than 1m files. When I do dvc add dir1, it takes quite a while to run and I saw that dvc only utilize one cpu to do so, it would be nice if there's parallelism options that can utilize multiple cores of the machine.

[Ykid]

current workaround: use multiple terminals to do dvc add

update: can't do so because there's a repo level lock.

[efiop]

Hi @Ykid !

current workaround: use multiple terminals to do dvc add

Not sure how that is going to help to parallelize single dvc add dir. Or maybe you meant dvc add dir1 dir2 ...?

Not sure how that is going to help to parallelize single dvc add dir. Or maybe you meant dvc add dir1 dir2 ...?

@efiop , it is dvc add dir_1 dir_2 dir_n (clear it out on the Discord conversation, forgot to update it here)

[efiop]

@Ykid Also, what dvc was showing when you were looking at CPU utilization? We do run things like checksum computation in parallel https://github.com/iterative/dvc/blob/master/dvc/remote/base.py#L178 .

@efiop , from the discord convo again, @Ykid mentioned 1 CPU. Not sure in what part of the process, didn't ask 🙈

[efiop]

Discord: https://discordapp.com/channels/485586884165107732/563406153334128681/667622134100787210

[Ykid]

@efiop this is what I got

ps aux | grep dvc
> some-user    1980 99.2  3.9 3483664 2634612 pts/14 Rl+ 13:57  56:44 python dvc add dir1

[efiop]

@Ykid Yes, but what was it doing at that time? Meaining what it was showing to stdout? Was it saying that it is computing checksums or maybe that it is linking?

[Ykid]

should be copying stuff

[efiop]

For the record: decided to try out hardlinks or symlinks as a workaround.

[efiop]

For the record: linking is still single-threaded in dvc, so need to look into parallelizing it as well.