Task queue blocks on too many wildcard operations

[bartek-siudeja]

Let me start by saying that s5cmd is a big improvement over aws and s3cmd. And wildcard support is by far the best feature of s5cmd.

However, it seems that s5cmd (release v0.6.0) is getting stuck when many wildcard commands are submitted using a file or stdin. As soon as number of wildcard (batch) operations is equal to or larger than numworkers nothing is actually downloaded. It looks like all workers are busy handling wildcards, waiting for actual put/get operations to finish. But there are no workers available to perform put/get operations.

This would not be a problem if number of workers could be set arbitrarily high. And technically it can. But we found out that on say i3.2 AWS instance (8 vCPUs) download speed is significantly reduced with 256 workers, compared to 16 or 32 workers. One can get around this by submitting listing commands in one file, then making a long list of non-batch operations in another file.

[ilkinulas]

Thank you very much for the feedback. We will look into it.

[igungor]

Duplicate #58

Let's keep both open.

[sonmezonur]

Hi @bartek-siudeja. We fixed the bug and released v1.0.0.