retry count does not reset with chunked uploads to S3

[simshaun]

We are debugging an issue our client is seeing in Firefox with the S3 uploader where it sometimes fails with

[Fine Uploader 4.3.1] Received response status 400 with body: <?xml version="1.0" encoding="UTF-8"?>
<Error><Code>RequestTimeout</Code><Message>Your socket connection to the server was not read from or written to within the timeout period. Idle connections will be closed.</Message>

The uploader has chunking enabled because the files being uploaded are typically 2 to 6 GB.

I have no idea as to the cause of that error (ideas?), but with the retry feature enabled, the failed chunk seems to successfully upload after 1 or 2 retries. The problem is the retry counter doesnt appear to reset to 0 if the chunk does upload successfully.

As it stands, it seems we need to set an arbitrary large number for maxAutoAttempts to accommodate a large number of retries on these files that are chunked in to over 1,000 pieces.

[rnicholus]

It sounds like you are focusing on the wrong problem here. I'm hesitant to make any changes to account for fundamental server or network flaws. If uploads are frequently failing, you might want to look into the root cause. If it takes a large number of retries to successfully upload a single large file, surely your users must be inconvenienced already.

[simshaun]

Yes, we are still investigating the root cause. Still, I think the retry count should be reset after each chunk succeeds. Perhaps this is a feature request more than a bug report.

---

Regarding the root cause, we just tested uploading nine separate 6GB files in Chrome and it didn't trigger that error even once. Firefox, however, is exhibiting the error quite frequently.

I'm inclined to think this is an issue with Firefox, but I don't yet know how to tell if it's the browser itself or an issue with the code.

Do you think there may be any part of FineUploader, chunking maybe, that could be leading to this problem only in Firefox?

---

Additional clues:

These errors begin occurring frequently when more than once file is being uploaded at once (multiple browser tabs, each uploading a file).

[rnicholus]

We haven't had any other reports of such an issue. What version of Firefox, and do you have a set of steps to reliably reproduce?

[simshaun]

I've reproduced this error 5/5 times on two different PCs under two different ISPs/networks. Mine at our office, and through a VPN to a PC at our client's office.

• Both PCs are running Win 7/Firefox 28.
• For consistency, we are attempting to upload the same 1 GB file on both PCs.

To reproduce, we:

1. Open an S3 upload page 3 times (3 different tabs). Chunked uploads enabled, Retry enabled (10 times max)
2. Choose the 1 GB file on each page so that all 3 tabs are uploading at the same time.
3. Wait. We sometimes begin seeing retries near the beginning of the upload, sometimes it takes a few minutes. It always happens eventually.

Upload speed does not appear to have any effect. Our office connection is a meager 5 MB/s upload, but it errors. Our client's upload speed is pushing 800 MB/s, it errors as well.

[rnicholus]

What version of Fine Uploader?

On Tuesday, March 25, 2014, Shaun notifications@github.com wrote:

I've reproduced this error 5/5 times on two different PCs under two
different ISPs/networks. Mine at our office, and through a VPN to a PC at
our client's office.

• Both PCs are running Win 7/Firefox 28.
• For consistency, we are attempting to upload the same 1 GB file on
  both PCs.

To reproduce, we:

1. Open an S3 upload page 3 times (3 different tabs). Chunked uploads
   enabled, Retry enabled (10 times max)
2. Choose the 1 GB file on each page so that all 3 tabs are uploading
   at the same time.
3. Wait. We sometimes begin seeing retries near the beginning of the
   upload, sometimes it takes a few minutes. It always happens
   eventually.

Upload speed does not appear to have any effect. Our office connection is
a meager 5 MB/s upload, but it errors. Our client's upload speed is pushing
800 MB/s, it errors as well.

Reply to this email directly or view it on GitHubhttps://github.com//issues/1172#issuecomment-38640517
.

[simshaun]

FineUploader 4.3.1

[rnicholus]

Thanks. We will look into this further tomorrow. This case will be
updated with progress and new info as it becomes available.

On Tuesday, March 25, 2014, Shaun notifications@github.com wrote:

FineUploader 4.3.1

Reply to this email directly or view it on GitHubhttps://github.com//issues/1172#issuecomment-38642078
.

[simshaun]

We are receiving the error intermittently today just trying to upload a single file.

So, I've been researching the 400 RequestTimeout problem, and it seems we're definitely not alone in having this issue.

aws/aws-cli#454
Solution: They just retry upon receiving 400 RequestTimeout.

aws/aws-sdk-php#29
Solution: They just retry upon receiving 400 RequestTimeout.
Additional clue: If you continue to see this issue, please ensure that you are not sending an incorrect Content-Length header in your requests.

https://groups.google.com/forum/#!msg/jets3t-users/_jr_8VXzSWU/3EWPjwrUoaYJ
Additional clues: S3 is pretty unforgiving about pauses during an upload and will return
an error within a few seconds. But the most likely problem is simply an incorrect Content-Length value.

http://www.plupload.com/punbb/viewtopic.php?id=3909
No solutions offered, but does give a little insight in to potential problems with Firefox.

[rnicholus]

Note that Fine Uploader doesn't set the Content-Length header, the browser
does, so that's not likely part of the problem here.

On Wed, Mar 26, 2014 at 12:01 PM, Shaun notifications@github.com wrote:

We are receiving the error intermittently today just trying to upload a
single file.

So, I've been researching the 400 RequestTimeout problem, and it seems
we're definitely not alone in having this issue.

aws/aws-cli#454 aws/aws-cli#454

Solution: They just retry upon receiving 400 RequestTimeout.

aws/aws-sdk-php#29 aws/aws-sdk-php#29
Solution: They just retry upon receiving 400 RequestTimeout.
Additional clue: If you continue to see this issue, please ensure that
you are not sending an incorrect Content-Length header in your requests.

https://groups.google.com/forum/#!msg/jets3t-users/_jr_8VXzSWU/3EWPjwrUoaYJ
Additional clues: S3 is pretty unforgiving about pauses during an
upload and will return
an error within a few seconds. But the most likely problem is simply an
incorrect Content-Length value.

http://www.plupload.com/punbb/viewtopic.php?id=3909
No solutions offered, but does give a little insight in to potential
problems with Firefox.

Reply to this email directly or view it on GitHubhttps://github.com//issues/1172#issuecomment-38710189
.

[rnicholus]

I just read the plupload thread, and didn't find anything useful in there. It's not clear how Firefox is involved. Do you have firebug open, or any other extension installed? I've seen extensions and OS antivirus interfere with browser HTTP traffic on numerous occasions.

[simshaun]

The install of Firefox we are testing on at our client's office is a vanilla install. The only extension installed on it is Firebug.

[rnicholus]

Firebug has been a source of constant pain for me. I've found builds of it to be quite buggy and generally avoid using it. Is firebug open when you reproduce these issues?

[simshaun]

That is fairly interesting. We disabled Firebug and are utilizing Firefox's built-in developer tools now.

Perhaps it's coincidental, but whereas we were getting roughly 20-30 errors per upload (uploading 3 files at once) before, we only had at max 2 errors on the same files with Firefox disabled. We're still testing with additional batches of uploads to find out if this remains true, and I'll update when we find out more.

[simshaun]

I think you've hit the nail on the head about Firebug.

We've uploaded 4 batches of 6 GB files (3 uploads at a time in each batch) with Firebug disabled. The first one is described in the comment above. The next 3 batches all completed without error.

Tried a 5th batch with Firebug re-enabled and the errors began appearing again. Waited until each upload hit 30 errors at only ~20% through and disabled Firebug again. Uploads quickly finished successfully without another error. This is definitely not a FineUploader issue as far as I'm concerned.

---

We are receiving errors in IE9 as well. Log from IE9 console:

LOG: [Fine Uploader 4.3.1] Received 1 files or inputs. 
LOG: [Fine Uploader 4.3.1] onSubmit - waiting for onSubmit promise to be fulfilled for 0 
LOG: [Fine Uploader 4.3.1] onSubmit promise success for 0 
LOG: [Fine Uploader 4.3.1] Submitting S3 signature request for 0 
LOG: [Fine Uploader 4.3.1] Sending POST request for 0 
LOG: [Fine Uploader 4.3.1] Sending upload request for 0 
LOG: [Fine Uploader 4.3.1] Received response for 0_17574c5e-5bee-4e8c-a8f3-8070f9281507 
[Fine Uploader 4.3.1] Error when attempting to access iframe during handling of upload response (Access is denied.)
LOG: [Fine Uploader 4.3.1] iframe loaded 
[Fine Uploader 4.3.1] Amazon likely rejected the upload request 

Using Fiddler2 to inspect the request, we see the response from Amazon:

<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>MalformedPOSTRequest</Code><Message>The body of your POST request is not well-formed multipart/form-data.</Message><RequestId>5363D49F5B3E1150</RequestId><HostId>tyUziamtbIKB30oMbY+AVDeaDg4lL5vkr+r5AGeanX0fme/rWbUJhXRiwGI9lAHI</HostId></Error>

IE9 succesfully uploaded a 1GB file then 2x 1GB files at once. It fails consistently with a 4.5GB file. Should I open a new issue for this?

[rnicholus]

No, this is not something we can overcome. Every browser has some sort of a
maximum request size. A 4.5 GB file is likely exceeding that max, perhaps
resulting in a truncated request, which is causing AWS to complain. We can
easily get around this max request size in modern browsers via chunking, but
chunking is not possible in IE9 and older.

On Wed, Mar 26, 2014 at 5:13 PM, Shaun notifications@github.com wrote:

I think you've hit the nail on the head about Firebug.

We've uploaded 4 batches of 6 GB files (3 uploads at a time in each batch)
with Firebug disabled. The first one is described in the comment above. The
next 3 batches all completed without error.

Tried a 5th batch with Firebug re-enabled and the errors began appearing
again. Waited until each upload hit 30 errors at only ~20% through and
disabled Firebug again. Uploads quickly finished successfully without

another error.

This is definitely not a FineUploader issue as far as I'm concerned.
However, we are receiving errors in IE9 as well. Log from IE9 console:

LOG: [Fine Uploader 4.3.1] Received 1 files or inputs.
LOG: [Fine Uploader 4.3.1] onSubmit - waiting for onSubmit promise to be fulfilled for 0
LOG: [Fine Uploader 4.3.1] onSubmit promise success for 0
LOG: [Fine Uploader 4.3.1] Submitting S3 signature request for 0
LOG: [Fine Uploader 4.3.1] Sending POST request for 0
LOG: [Fine Uploader 4.3.1] Sending upload request for 0
LOG: [Fine Uploader 4.3.1] Received response for 0_17574c5e-5bee-4e8c-a8f3-8070f9281507
[Fine Uploader 4.3.1] Error when attempting to access iframe during handling of upload response (Access is denied.)
LOG: [Fine Uploader 4.3.1] iframe loaded
[Fine Uploader 4.3.1] Amazon likely rejected the upload request

Using Fiddler2 to inspect the request, we see the response from Amazon:

MalformedPOSTRequestThe body of your POST request is not well-formed multipart/form-data.5363D49F5B3E1150tyUziamtbIKB30oMbY+AVDeaDg4lL5vkr+r5AGeanX0fme/rWbUJhXRiwGI9lAHI

IE9 succesfully uploaded a 1GB file then 2x 1GB files at once. It fails
consistently with a 4.5GB file. Should I open a new issue for this?

Reply to this email directly or view it on GitHubhttps://github.com//issues/1172#issuecomment-38747051
.

[simshaun]

Thanks for sticking with me through this.

You are correct. It appears IE9's upload limit is 4 GB per http://blogs.msdn.com/b/ieinternals/archive/2011/03/10/wininet-internet-explorer-file-download-and-upload-maximum-size-limits.aspx

I think information like that would useful in the FineUploader docs, even though it's not really your responsibility. Would you consider it?

[rnicholus]

Yep, I'd have to think about where to put it. Can you open up a case so I remember to take care of this?

I guess we can close this one as well?

[EZWrighter]

Can this be re-opened to actually take care of the original posters issue? Retry count should reset upon success. I would say that is pretty standard upload behavior that I have experienced in the past for uploading. There could be all kinds of reasons for network failure, but once the failures have stopped and you get a successful chunk or file uploaded, the old retry count is irrelevant. You could even add it as a config option but I doubt anybody really wants continuous retry count.

[rnicholus]

Sure. I'll reopen this so we can discuss more internally.

[rnicholus]

This will be part of a future release.

[notken]

Was this ever put in to the codebase and I've just not found the setting? I'm putting in deliberate random failures to test, with a retry count of 3. When it gives up, if I click Retry it only retries once before giving up again. The retry count does need resetting to zero after either a successful chunk, or at the very least after a manual retry. I'm trying version 5.3.2

(Sorry, I should point out I'm not using S3. Just a standard upload to server.)

[rnicholus]

This has not made it into a scheduled release yet. When it does, the case will be updated appropriately.

[rnicholus]

By the way, while I do agree that the retry count should be reset after a successful chunk upload, I do not agree that this should be reset on a manual retry.

[notken]

I guess because it's not resetting after a successful chunk, it not resetting when you manually retry seems illogical. It just seems like it's totally ignoring the settings.

When testing it with built-in random failures, if I click Retry I'd expect it to suffer 3 more random failures before stopping again. But in actuality, the moment it's successfully done one after a failure I'd be quite happy for it to keep recovering from occasional failure until it's completed, and only fail if it really couldn't get through at all.

It just seems odd to ask for an auto-retry count in settings, and then ignore it and go for a developer-decided number when you manually Retry. Maybe it all needs to be in the settings.

[rnicholus]

This will be fixed simply by resetting the count once a chunk has
successfully uploaded.
On Wed, Oct 7, 2015 at 5:40 AM notken notifications@github.com wrote:

I guess because it's not resetting after a successful chunk, it not
resetting when you manually retry seems illogical. It just seems like it's
totally ignoring the settings.

When testing it with built-in random failures, if I click Retry I'd expect
it to suffer 3 more random failures before stopping again. But in
actuality, the moment it's successfully done one after a failure I'd be
quite happy for it to keep recovering from occasional failure until it's
completed, and only fail if it really couldn't get through at all.

It just seems odd to ask for an auto-retry count in settings, and then
ignore it and go for a developer-decided number when you manually Retry.
Maybe it all needs to be in the settings.

—
Reply to this email directly or view it on GitHub
#1172 (comment)
.

[ShaharTal]

I am getting this error as well, not just with Firefox, mainly Chrome actually. Only with large files, directly to S3, what info can I give to better help/investigate?