Reenables true multithreaded s3 copy

[dlstadther]

Boto3 update removed multithreaded file copy in favor of multithreaded multipart file copy.

Description

Re-enables true multithreaded file copy. (Logic taken directly from pre-boto3 s3.py)

Motivation and Context

Boto3's TransferConfig appears to enable multithreaded copy capabilities for multipart files, but not for the copying of many small files.

Have you tested this? If so, how?

I've run with test code. Other than that, i'm letting travis test it.

[Tarrasch]

Great, please merge if you've tested!

[Tarrasch]

Please just check that this doesn't have a ContextManager interface maybe?

[dlstadther]

A few minutes of Google searching doesn't return valuable resources towards this desire.

[Tarrasch]

Have you considered adding tests as codecov is mostly red? If this is not suitable for testing feel free to skip.

[dlstadther]

I just realized that the test for copy_dir has been set to be skipped on Travis.

I'll see what i can do in the next couple days to enable some degree of testing for this method - even if not for the specific multiprocessing portion (bc i'm not sure how to test that aside from showing that increasing thread count decreases copy time).

[ouanixi]

@dlstadther tests that are skipped on travis are tests that have shown issues with moto (mocking library for boto), it appears when testing multipart functionality, there is a race condition expressed in same tests passing and failing intermittently.

[dlstadther]

I've spent some time trying to test the true multithreaded s3 copy, but cannot come up with a reasonable test to identify the behavior.

• Confirmed that TestS3Client.test_copy_dir continues to run successfully with this code
• Tried to add test to show that as threads increase, runtime reduces, but for this to be a consistent speed improvement the number and size of files has to increase such to be unreasonable (runtime)

@Tarrasch or @ouanixi , do either of you have any suggestions?

Thanks,

[dlstadther]

bump

[Tarrasch]

I wouldn't bother to much about writing unit-tests for multi-threaded code if it's already working in production for you. I wouldn't let that block me from merging.

[dlstadther]

In this case, I'll merge when I return from vacation. Thanks @Tarrasch !

[dlstadther]

Unable to provide unit-tests for multi-threading within method. Code has been running successfully copying millions of files daily in production for ~2 months.

Merging...