s3 cp with max_concurrent_requests

[henriqueribeiro]

Confirm by changing [ ] to [x] below:

• I've gone though the User Guide and the API reference
• I've searched for previous similar issues and didn't find any solution

Issue is about usage on:

• Service API : I want to do X using Y service, what should I do?
• CLI : passing arguments or cli configurations.
• Other/Not sure.

Platform/OS/Hardware/Device
What are you running the cli on?

aws --version
aws-cli/2.1.0 Python/3.7.3 Linux/5.8.0-43-generic exe/x86_64.ubuntu.20

Describe the question
I want to reduce the number of threads used to copy a file from S3. I have been playing with max_concurrent_requests and it's not making sense to me. I'm trying to copy a single file (~41GB) to my local machines and if I set max_concurrent_requests to 1, I still get 5 threads copying the file. If I change it to 2, I get 6 threads. What is the rationale behind this?

Also, looking into the documentation:

The aws s3 transfer commands are multithreaded. At any given time, multiple requests to Amazon S3 are in flight. For example, if you are uploading a directory via aws s3 cp localdir s3://bucket/ --recursive, the AWS CLI could be uploading the local files localdir/file1, localdir/file2, and localdir/file3 in parallel. The max_concurrent_requests specifies the maximum number of transfer commands that are allowed at any given time.

Does this configuration limit only the number of files concurrently copied and not the number of threads used for a single multipart file?

Logs/output
Get full traceback and error logs by adding --debug to the command.

[kdaily]

Hi @henriqueribeiro, I'm going to dig into it further, but I do not think it directly controls the number of threads, only the number of active requests.

[alialqh]

A

[alialqh]

A

[alialqh]

> 

[henriqueribeiro]

thank you @kdaily

[alialqh]

F

[alialqh]

S

[alialqh]

F

[alialqh]

Az123456a

[kdaily]

Apologies for the delay. The max_concurrent_requests does not control directly the number of threads. There is currently a minimum number of threads irrespective of that value, and there isn't a way to adjust it directly. Having that would be a feature request. In some other research I came across this issue which feels the same as the one we were working on with @ttheyer (#5876):

boto/boto3#1670

We have it as a backlog item to look into the memory usage pattern to be sure that it matches expectations, but I don't have an update on when that will be done.

I think we can close this in lieu of #5876 and #1670 since they are related and likely the same underlying issue. I don't think that directly adjusting the number of threads is really what you want - you just want to be sure that proper memory management is handled.