Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Disable read-backpressure in the crt-c runner #70

Merged
merged 1 commit into from
Aug 22, 2024
Merged

Conversation

graebm
Copy link
Contributor

@graebm graebm commented Aug 22, 2024

Issue:
The crt-c runner had worse performance than the crt-python runner in single-file-download-to-disk workloads, which is odd. You'd think crt-c would be faster, since it's doing pretty much the same thing, and doesn't involve the Python GIL.

Research:
The difference is: crt-c was using read-backpressure, and crt-python wasn't.

Description of changes:
Disable read-backpressure in the crt-c runner. It can rely solely on the memory-limiter to keep memory usage down, since it always processes data synchronously within the body callback.

I'll keep the code in there, for future testing of backpressure. But it's disabled for now.

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

@graebm
Copy link
Contributor Author

graebm commented Aug 22, 2024

Here's my benchmark data on a c5n.18xlarge with the download-30GiB-1x workload, where the disk is EBS gp3

Mysteries:

  • Why didn't increasing backpressure window help performance? Why did throughput only go up when backpressure was disabled completely?
  • Why does Peak RSS stay at 276MiB in all runs? Shouldn't it go higher as we increase the backpressure window?

--- 30GiB-1x with backpressure=256M ---
Run:1 Secs:59.954798 Gb/s:4.298205
Run:2 Secs:49.115315 Gb/s:5.246796
Run:3 Secs:45.563883 Gb/s:5.655752
Run:4 Secs:45.588114 Gb/s:5.652746
Run:5 Secs:40.193904 Gb/s:6.411371
Run:6 Secs:38.568166 Gb/s:6.681625
Run:7 Secs:47.362805 Gb/s:5.440937
Run:8 Secs:38.807289 Gb/s:6.640454
Run:9 Secs:37.309095 Gb/s:6.907110
Run:10 Secs:37.231331 Gb/s:6.921537
Overall Throughput (Gb/s) Median:6.033562 Mean:5.985654 Min:4.298205 Max:6.921537 Variance:0.672911 StdDev:0.820311
Overall Duration (Secs) Median:42.878894 Mean:43.969470 Min:37.231331 Max:59.954798 Variance:45.852753 StdDev:6.771466
Peak RSS:276.175781 MiB

--- 30GiB-1x with backpressure=1024M ---
Run:1 Secs:68.749111 Gb/s:3.748384
Run:2 Secs:69.340125 Gb/s:3.716435
Run:3 Secs:63.877484 Gb/s:4.034255
Run:4 Secs:53.274673 Gb/s:4.837159
Run:5 Secs:46.438401 Gb/s:5.549244
Run:6 Secs:40.501319 Gb/s:6.362707
Run:7 Secs:35.898072 Gb/s:7.178604
Run:8 Secs:30.033225 Gb/s:8.580432
Run:9 Secs:42.440666 Gb/s:6.071960
Run:10 Secs:38.529039 Gb/s:6.688411
Overall Throughput (Gb/s) Median:5.810602 Mean:5.676759 Min:3.716435 Max:8.580432 Variance:2.331751 StdDev:1.527007
Overall Duration (Secs) Median:44.439534 Mean:48.908212 Min:30.033225 Max:69.340125 Variance:180.613051 StdDev:13.439236
Peak RSS:276.316406 MiB <-- WHY DOESN'T THIS GO ANY HIGHER?

--- 30GiB-1x with backpressure=8192M ---
Run:1 Secs:39.513284 Gb/s:6.521808
Run:2 Secs:39.785690 Gb/s:6.477154
Run:3 Secs:35.091017 Gb/s:7.343704
Run:4 Secs:38.688826 Gb/s:6.660787
Run:5 Secs:36.249432 Gb/s:7.109023
Run:6 Secs:35.145728 Gb/s:7.332272
Run:7 Secs:30.947950 Gb/s:8.326821
Run:8 Secs:48.105302 Gb/s:5.356957
Run:9 Secs:39.553579 Gb/s:6.515164
Run:10 Secs:38.802492 Gb/s:6.641275
Overall Throughput (Gb/s) Median:6.651031 Mean:6.828496 Min:5.356957 Max:8.326821 Variance:0.538718 StdDev:0.733974
Overall Duration (Secs) Median:38.745659 Mean:38.188330 Min:30.947950 Max:48.105302 Variance:18.017819 StdDev:4.244740
Peak RSS:276.226562 MiB

--- 30GiB-1x with backpressure=disabled ---
Run:1 Secs:35.244202 Gb/s:7.311785
Run:2 Secs:36.748632 Gb/s:7.012453
Run:3 Secs:34.638007 Gb/s:7.439748
Run:4 Secs:31.173856 Gb/s:8.266479
Run:5 Secs:28.526440 Gb/s:9.033656
Run:6 Secs:28.807611 Gb/s:8.945485
Run:7 Secs:27.161716 Gb/s:9.487546
Run:8 Secs:26.693972 Gb/s:9.653791
Run:9 Secs:26.813365 Gb/s:9.610806
Run:10 Secs:26.767925 Gb/s:9.627120
Overall Throughput (Gb/s) Median:8.989570 Mean:8.638887 Min:7.012453 Max:9.653791 Variance:0.990444 StdDev:0.995210
Overall Duration (Secs) Median:28.667025 Mean:30.257573 Min:26.693972 Max:36.748632 Variance:13.845109 StdDev:3.720902
Peak RSS:277.230469 MiB

@graebm graebm merged commit e699142 into main Aug 22, 2024
5 checks passed
@graebm graebm deleted the crt-c-backpressure-off branch August 22, 2024 20:50
@TingDaoK TingDaoK mentioned this pull request Sep 5, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants