Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

pick up the change for s3 retry slow parts upload #714

Merged
merged 1 commit into from
Nov 9, 2023
Merged

Conversation

TingDaoK
Copy link
Contributor

@TingDaoK TingDaoK commented Nov 9, 2023

Issue #, if available:

aws-c-auth v0.7.5 -> v0.7.6
aws-c-http v0.7.13 -> v0.7.14
aws-c-s3 v0.3.22 -> v0.3.23
aws-lc v1.17.1 -> v1.17.2

Description of changes:
With this change, the 30GB file upload has result:

> java -jar /home/ec2-user/fast_ebs/aws-crt-s3-benchmarks/runners/s3-benchrunner-crt-java/target/s3-benchrunner-crt-java-1.0-SNAPSHOT.jar ../benchmarks/upload-30GiB-1x.run.json test-s3-benchmark us-west-2 100.0
Run:1 Secs:8.959 Gb/s:28.8 Mb/s:28764.8 GiB/s:3.3 MiB/s:3429.0
Run:2 Secs:6.772 Gb/s:38.1 Mb/s:38054.6 GiB/s:4.4 MiB/s:4536.5
Run:3 Secs:6.594 Gb/s:39.1 Mb/s:39078.0 GiB/s:4.5 MiB/s:4658.5
Run:4 Secs:6.539 Gb/s:39.4 Mb/s:39407.8 GiB/s:4.6 MiB/s:4697.8
Run:5 Secs:6.575 Gb/s:39.2 Mb/s:39194.3 GiB/s:4.6 MiB/s:4672.3
Run:6 Secs:6.757 Gb/s:38.1 Mb/s:38138.1 GiB/s:4.4 MiB/s:4546.4
Run:7 Secs:6.455 Gb/s:39.9 Mb/s:39925.1 GiB/s:4.6 MiB/s:4759.4
Run:8 Secs:6.990 Gb/s:36.9 Mb/s:36866.4 GiB/s:4.3 MiB/s:4394.8
Run:9 Secs:6.771 Gb/s:38.1 Mb/s:38060.0 GiB/s:4.4 MiB/s:4537.1
Run:10 Secs:6.946 Gb/s:37.1 Mb/s:37102.3 GiB/s:4.3 MiB/s:4422.9
Overall stats; Throughput Mean:37155.2 Mb/s Throughput Variance:534949.8 Mb/s Duration Mean:6.936 s Duration Variance:0.482 s 

While the current main:

> java -jar /home/ec2-user/fast_ebs/aws-crt-s3-benchmarks/runners/s3-benchrunner-crt-java/target/s3-benchrunner-crt-java-1.0-SNAPSHOT.jar ../benchmarks/upload-30GiB-1x.run.json test-s3-benchmark us-west-2 100.0
Run:1 Secs:12.686 Gb/s:20.3 Mb/s:20314.0 GiB/s:2.4 MiB/s:2421.6
Run:2 Secs:9.951 Gb/s:25.9 Mb/s:25897.4 GiB/s:3.0 MiB/s:3087.2
Run:3 Secs:9.861 Gb/s:26.1 Mb/s:26134.2 GiB/s:3.0 MiB/s:3115.4
Run:4 Secs:10.303 Gb/s:25.0 Mb/s:25011.7 GiB/s:2.9 MiB/s:2981.6
Run:5 Secs:8.379 Gb/s:30.8 Mb/s:30753.6 GiB/s:3.6 MiB/s:3666.1
Run:6 Secs:9.627 Gb/s:26.8 Mb/s:26767.4 GiB/s:3.1 MiB/s:3190.9
Run:7 Secs:8.196 Gb/s:31.4 Mb/s:31440.4 GiB/s:3.7 MiB/s:3748.0
Run:8 Secs:9.715 Gb/s:26.5 Mb/s:26524.8 GiB/s:3.1 MiB/s:3162.0
Run:9 Secs:10.692 Gb/s:24.1 Mb/s:24101.3 GiB/s:2.8 MiB/s:2873.1
Run:10 Secs:10.790 Gb/s:23.9 Mb/s:23883.5 GiB/s:2.8 MiB/s:2847.1
Overall stats; Throughput Mean:25718.2 Mb/s Throughput Variance:177417.7 Mb/s Duration Mean:10.020 s Duration Variance:1.452 s 

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

@TingDaoK TingDaoK merged commit 00c4c16 into main Nov 9, 2023
40 checks passed
@TingDaoK TingDaoK deleted the pickup-s3 branch November 9, 2023 23:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants