improve rate control for high level API

[wakabayashik]

The original rate control is too rough for me, so I improved the rate control of the high level API.
Note that transferChunkSize may be changed internally to optimize the low rate limit.

[robinrodricks]

Oh great this seems quite advanced. Can you explain the changes you made and why? And whats the hardcoded value for 1 second (1000 ms) used for?

[wakabayashik]

1. I deleted the hard-coded 1 second. due to this one second, the FTP client sends the data as fast as possible in the first second, and then sleeps for hundreds of seconds to try to keep low rate limit specified. I don't know what that means in this "hard-coded 1 second".

2. I modified to adjust the chunk size according to rate limit value. rate control is strongly concerned with chunk size. large chunk against low rate limit makes many peaks, adjusted chunk makes it smooth. of course this adjustment can be achieved using exposed "TransferChunkSize", but its hard for users.

3. I modified so that the stopwatch (rate control criteria) is not restarted every loop. restarting of the stopwatch makes a slight deviation from the specified rate limit. Its only be done if actual rate behind the specified rate.

[robinrodricks]

Can you explain how this works?

[wakabayashik]

Here are I/O graphs captured by Wireshark showing the transfer throttling before & after the update.

Old version

FluentFtp v27.0.3
As you can see the data is transferred in bursts in order to restrict the transfer speed to the limit that you have set.

1. file size = 210MBytes, UploadRateLimit = 10MBytes/sec, FluentFtp v27.0.3
   

2. file size = 150MBytes, UploadRateLimit = 500kBytes/sec, FluentFtp v27.0.3
   

enlarge around peak of 2.

first peak

second peak

last peak

New version

FluentFtp v27.1.0+
As you can see the data is transferred consistently at a lower speed in order to restrict the transfer speed to the limit that you have set. There are no bursts.

3. file size = 210MBytes, UploadRateLimit = 10MBytes/sec, FluentFtp improved
   

4. file size = 150MBytes, UploadRateLimit = 500kBytes/sec, FluentFtp improved
   

---

FluentFtp v27.0.3 has big peaks that violate the limit. Such peaks can cause quality degradation for systems such as VoIP. On the other hand, the improved version are smooth.

[robinrodricks]

Wow, that's a great improvement. Fantastic work!

…

On Fri, Sep 6, 2019, 2:16 PM wakabayashik ***@***.***> wrote: here are I/O graphs captured by Wireshark. please care to the unit of X and Y axis. 1 and 2 => FluentFtp v27.0.3 3 and 4 => FluentFtp improved 5 and 6 => curl.exe <https://curl.haxx.se/> --limit-rate xxx (for comparison) 1. file size = 210MBytes, UploadRateLimit = 10MBytes/sec, FluentFtp v27.0.3 [image: wireshark_IOGraph_original_10MBps] <https://user-images.githubusercontent.com/28973407/64409743-749b1a00-d0c4-11e9-98b7-6ae0828ec7f9.png> 2. file size = 150MBytes, UploadRateLimit = 500kBytes/sec, FluentFtp v27.0.3 [image: wireshark_IOGraph_original_500kBps] <https://user-images.githubusercontent.com/28973407/64409745-749b1a00-d0c4-11e9-8e19-b4946f916772.png> enlarge around peak of 2. first peak [image: wireshark_IOGraph_original_500kBps_peak1] <https://user-images.githubusercontent.com/28973407/64409746-749b1a00-d0c4-11e9-9a75-147422d656be.png> second peak [image: wireshark_IOGraph_original_500kBps_peak2] <https://user-images.githubusercontent.com/28973407/64409747-7533b080-d0c4-11e9-94a2-2aee2fa6a469.png> last peak [image: wireshark_IOGraph_original_500kBps_peak3] <https://user-images.githubusercontent.com/28973407/64409748-7533b080-d0c4-11e9-872a-69858b5f8eb1.png> 1. file size = 210MBytes, UploadRateLimit = 10MBytes/sec, FluentFtp improved [image: wireshark_IOGraph_improved_10MBps] <https://user-images.githubusercontent.com/28973407/64409749-7533b080-d0c4-11e9-8550-c7e47866bb96.png> 2. file size = 150MBytes, UploadRateLimit = 500kBytes/sec, FluentFtp improved [image: wireshark_IOGraph_improved_500kBps] <https://user-images.githubusercontent.com/28973407/64409750-75cc4700-d0c4-11e9-8590-45a8797ccf04.png> 3. file size = 210MBytes, UploadRateLimit = 10MBytes/sec, curl.exe --limit-rate 10M [image: wireshark_IOGraph_curl_10MBps] <https://user-images.githubusercontent.com/28973407/64413788-1fafd180-d0cd-11e9-8874-5709c50a620c.png> 4. file size = 150MBytes, UploadRateLimit = 500kBytes/sec, curl.exe --limit-rate 500k [[image: wireshark_IOGraph_curl_500kBps] <https://user-images.githubusercontent.com/28973407/64412671-e5453500-d0ca-11e9-97b1-d4ce512ccae3.png> ------------------------------ FluentFtp v27.0.3 has big peaks that violate the limit. Such peaks can cause quality degradation for systems such as VoIP. On the other hand, the improved version and curl.exe are smooth. actually, I'm using FluentFtpExtension class as workaround with FluentFtp v27.0.3. FluentFtpExtension.zip <https://github.com/robinrodricks/FluentFTP/files/3582814/FluentFtpExtension.zip> FluentFtpExtension.UploadFileAsyncEx(client,.. instread of client.UploadFileAsync(... — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#446?email_source=notifications&email_token=ABT3UKTSMLSJZV5VLVW6GI3QIIKFJA5CNFSM4ITCCSM2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD6CGDOI#issuecomment-528769465>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABT3UKVYBN2XN2TCIXOV55LQIIKFJANCNFSM4ITCCSMQ> .

[robinrodricks]

Published to Nuget. Thanks again!

https://www.nuget.org/packages/FluentFTP/27.1.0