Multipart Upload reporting errors at end of upload in sdk v3 #669

mwoodring · 2015-07-01T15:24:36Z

I am using the MultipartUploader function in the sdk v3. It appears that it uploads all the parts and at the end it goes through the catch exception code to report that multiple parts had errors during upload. Is there a way to stop the upload as soon as it gets an error so I can use the getState function to re-upload the bad part?

Here is the code I using:

$s3Uploader = new MultipartUploader ($s3Client, $fileHndl, $opt);
try {
    $response = $s3Uploader->upload ();
    if ($response->hasKey ('ObjectURL'))
        $awsURL = $response->get ('ObjectURL');
    else
        var_dump ($response);   // ** testing only
}
catch (MultipartUploadException $e) {
    showError ($e, $currFileName, "Multi upload exception");

    $params = $e->getState ()->getId ();
    $junk = $s3Client->abortMultipartUpload ($params);
    $sendOK = false;
 }

The text was updated successfully, but these errors were encountered:

jeremeamia · 2015-07-01T15:41:32Z

Even if there are unsuccessful UploadPart operations somewhere in the middle of the multipart upload, it's best to let it all finish, because the UploadState tracks exactly which parts failed. When you get the state from the exception at the end, you can create a new MultipartUploader from the state that will only attempt to upload the missing parts. An example of this technique is demonstrated here: http://docs.aws.amazon.com/aws-sdk-php/v3/guide/service/s3-multipart-upload.html#recovering-from-errors

mwoodring · 2015-07-01T16:47:16Z

Jeremy,

I have seen the documentation about recovering from errors before. The documentation does not mention that it will upload all parts then exit with an error. Also not mentioned is that creating a new uploader will only upload the parts with errors. Thank you for giving me that info.

Mark

From: Jeremy Lindblom <notifications@github.com mailto:notifications@github.com>
Reply-To: aws/aws-sdk-php <reply@reply.github.com mailto:reply@reply.github.com>
Date: Wednesday, July 01, 2015 at 10:41
To: aws/aws-sdk-php <aws-sdk-php@noreply.github.com mailto:aws-sdk-php@noreply.github.com>
Cc: Mark Woodring <mark.woodring@walsworth.com mailto:mark.woodring@walsworth.com>
Subject: Re: [aws-sdk-php] Multipart Upload reporting errors at end of upload in sdk v3 (#669)

Even if there are unsuccessful UploadPart operations somewhere in the middle of the multipart upload, it's best to let it all finish, because the UploadState tracks exactly which parts failed. When you get the state from the exception at the end, you can create a new MultipartUploader from the state that will only attempt to upload the missing parts. An example of this technique is demonstrated here: http://docs.aws.amazon.com/aws-sdk-php/v3/guide/service/s3-multipart-upload.html#recovering-from-errors

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/669#issuecomment-117720038.

jeremeamia · 2015-07-01T16:52:44Z

I will add some clarifications to the documentation. Thanks for the feedback.

mwoodring · 2015-07-01T19:36:26Z

Jeremy,

I changed my code around to use the method show in the recovering from errors section of the multipart upload documentation. I am now getting this error:

PHP Fatal error: Uncaught exception 'GuzzleHttp\Promise\RejectionException' with message 'The promise was rejected with reason: Invoking the wait callback did not resolve the promise' in /usr/scripts/vendor/guzzlehttp/promises/src/functions.php:111
Stack trace:
#0 /usr/scripts/vendor/guzzlehttp/promises/src/functions.php(495): GuzzleHttp\Promise\exception_for('Invoking the wa...')
#1 /usr/scripts/vendor/guzzlehttp/promises/src/Promise.php(199): GuzzleHttp\Promise{closure}('Invoking the wa...')
#2 /usr/scripts/vendor/guzzlehttp/promises/src/Promise.php(152): GuzzleHttp\Promise\Promise::callHandler(2, 'Invoking the wa...', Array)
#3 /usr/scripts/vendor/guzzlehttp/promises/src/TaskQueue.php(60): GuzzleHttp\Promise\Promise::GuzzleHttp\Promise{closure}()
#4 /usr/scripts/vendor/guzzlehttp/promises/src/Promise.php(228): GuzzleHttp\Promise\TaskQueue->run()
#5 /usr/scripts/vendor/guzzlehttp/promises/src/Promise.php(261): GuzzleHttp\Promise\Promise->waitIfPending()
#6 /usr/scripts/vendor/guzzlehttp/promises/src/Promise.php(219): Guzz in /usr/scripts/vendor/guzzlehttp/promises/src/functions.php on line 111

Any ideas how to resolve this error?

Thanks,

Mark

From: Jeremy Lindblom <notifications@github.com mailto:notifications@github.com>
Reply-To: aws/aws-sdk-php <reply@reply.github.com mailto:reply@reply.github.com>
Date: Wednesday, July 01, 2015 at 11:52
To: aws/aws-sdk-php <aws-sdk-php@noreply.github.com mailto:aws-sdk-php@noreply.github.com>
Cc: Mark Woodring <mark.woodring@walsworth.com mailto:mark.woodring@walsworth.com>
Subject: Re: [aws-sdk-php] Multipart Upload reporting errors at end of upload in sdk v3 (#669)

I will add some clarifications to the documentation. Thanks for the feedback.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/669#issuecomment-117748094.

jeremeamia · 2015-07-02T14:45:21Z

Can you show us what your code looks like now?

mwoodring · 2015-07-02T15:30:51Z

The code is as follows:

$s3Uploader = new MultipartUploader ($s3Client, $fileHndl, $opt);
do {
try {
$response = $s3Uploader->upload ();
if ($response->hasKey ('ObjectURL'))
$awsURL = $response->get ('ObjectURL');
else
var_dump ($response); // ** testing only
}
catch (MultipartUploadException $e) {
showError ($e, $currFileName, "Multi upload exception");

fwrite (STDERR, " restarting upload on parts with errors" . PHP_EOL);
$s3Uploader = new MultipartUploader ($s3Client, $fileHndl, [
'state' => $e->getState (),
]);

}
}
while (! isset ($response));

The showError function just prints out the errors from the exception.

Thanks,

Mark

From: Jeremy Lindblom <notifications@github.com mailto:notifications@github.com>
Reply-To: aws/aws-sdk-php <reply@reply.github.com mailto:reply@reply.github.com>
Date: Thursday, July 02, 2015 at 09:45
To: aws/aws-sdk-php <aws-sdk-php@noreply.github.com mailto:aws-sdk-php@noreply.github.com>
Cc: Mark Woodring <mark.woodring@walsworth.com mailto:mark.woodring@walsworth.com>
Subject: Re: [aws-sdk-php] Multipart Upload reporting errors at end of upload in sdk v3 (#669)

Can you show us what your code looks like now?

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/669#issuecomment-118055475.

jeremeamia · 2015-07-02T16:10:47Z

Does the "Invoking the wait callback did not resolve the promise" error happen every time, or just some of the time?

Side Note: I added some more documentation to the user guide for multipart uploads as per our earlier discussion. See 2df31b3

mwoodring · 2015-07-02T16:24:08Z

I tried the error recovery method on only one 6GB file. I tried it several times on that job and each time I received the fatal error.

Mark

From: Jeremy Lindblom <notifications@github.com mailto:notifications@github.com>
Reply-To: aws/aws-sdk-php <reply@reply.github.com mailto:reply@reply.github.com>
Date: Thursday, July 02, 2015 at 11:10
To: aws/aws-sdk-php <aws-sdk-php@noreply.github.com mailto:aws-sdk-php@noreply.github.com>
Cc: Mark Woodring <mark.woodring@walsworth.com mailto:mark.woodring@walsworth.com>
Subject: Re: [aws-sdk-php] Multipart Upload reporting errors at end of upload in sdk v3 (#669)

Does the "Invoking the wait callback did not resolve the promise" error happen every time, or just some of the time?

Side Note: I added some more documentation to the user guide for multipart uploads as per our earlier discussion. See 2df31b32df31b3

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/669#issuecomment-118080549.

zexz · 2015-07-11T16:08:11Z

Hello, i'm not sure if it does belong here, but we have issues with multipart upload (just switched to aws v3.1.0), here is the error log:

[11-Jul-2015 05:40:37 America/Chicago] PHP Fatal error:  Uncaught exception 'Aws\Exception\MultipartUploadException' with message 'An exception occurred while uploading parts to a multipart upload. The following parts had errors:
- Part 1: Error executing "UploadPart" on "https://path.to.file.zip?partNumber=1&uploadId=gNqjjR_8MAQRvyI1fT64GnYb7yaR5IOqnSE.o61g1n1rr2fX4fTX4B4bqHbjXhIwZceDdd2x0sNQ.GAh.3LUhpJiAt1UY0HiWgBRim.X48ScEpzWLbJNil4flu6NHUAo"; AWS HTTP error: Client error: 400 XAmzContentSHA256Mismatch (client): The provided 'x-amz-content-sha256' header does not match what was computed. - 
XAmzContentSHA256MismatchThe provided 'x-amz-content-sha256' header does not match what was computed.WFrAfemdE4ksWJYvOd3mzAyQwiNPtYLQAwtpTfgkOOw=585ac07de99d13892c58962f39dde6cc0c90c2234fb582d0030b694df82438ecA1B4825462511096oJgr/fh2NHil/VnAfw9a72kQ1RaR8wi1rzD7GSLvzbQ3vQx8NarAPoL6QwD8hc62AzQmjAtqjAw=
- Part 2: Error executing "UploadPart" on "https://path.to.file.zip?partNumber=2&uploadId=gNqjjR_8MAQRvyI1fT64GnYb7yaR5IOqnSE.o61g1n1rr2fX4fTX4B4bqHbjXhIwZceDdd2x0sNQ.GAh.3LUhpJiAt1UY0HiWgBRim.X48ScEpzWLbJNil4flu6NHUAo"; AWS HTTP error: Client error: 400 XAmzContentSHA256Mismatch (client): The provided 'x-amz-content-sha256' header does not match what was computed. - 
XAmzContentSHA256MismatchThe provided 'x-amz-content-sha256' header does not match what was computed.UKx/sV39DhZ3Hx4fYdi98YZ1VopkZV7IdgbHECEP078=50ac7fb15dfd0e16771f1e1f61d8bdf18675568a64655ec87606c710210fd3bfA44FB9D552743B7C7u5EMrtFNOSTI9K1pyUO2kL0EmAK+VM7Qr+yOdFGcoATA7zQzNoGtti1abJsRbOUIHikeGOkpUk=
- Part 3: Error executing "UploadPart" on "https://path.to.file.zip?partNumber=3&uploadId=gNqjjR_8MAQRvyI1fT64GnYb7yaR5IOqnSE.o61g1n1rr2fX4fTX4B4bqHbjXhIwZceDdd2x0sNQ.GAh.3LUhpJiAt1UY0HiWgBRim.X48ScEpzWLbJNil4flu6NHUAo"; AWS HTTP error: Client error: 400 XAmzContentSHA256Mismatch (client): The provided 'x-amz-content-sha256' header does not match what was computed. - 
XAmzContentSHA256MismatchThe provided 'x-amz-content-sha256' header does not match what was computed.jNKAAI7PNUNtccYs9vZmD2K7rYF5NJxtsPcshEwzxTM=8cd280008ecf35436d71c62cf6f6660f62bbad8179349c6db0f72c844c33c533FD0464B481C0C1A8PPVJdcdVa24A12RFWPf9VJ98cPVex2hLBPFQZQaOOlVd0v18DCUiYQtLDhgzTBb8j1WIRhU23ZM=
- Part 4: Error executing "UploadPart" on "https://path.to.file.zip?partNumber=4&uploadId=gNqjjR_8MAQRvyI1fT64GnYb7yaR5IOqnSE.o61g1n1rr2fX4fTX4B4bqHbjXhIwZceDdd2x0sNQ.GAh.3LUhpJiAt1UY0HiWgBRim.X48ScEpzWLbJNil4flu6NHUAo"; AWS HTTP error: Client error: 400 XAmzContentSHA256Mismatch (client): The provided 'x-amz-content-sha256' header does not match what was computed. - 
XAmzContentSHA256MismatchThe provided 'x-amz-content-sha256' header does not match what was computed.VfZiQ/vvOIAzTW2Iohu8wdjOWI0eFfToA5XRAGGDMi8=55f66243fbef3880334d6d88a21bbcc1d8ce588d1e15f4e80395d1006183322fBDEFC5CD47CB77B1zobhlZ3m0w1lbEFRMfY4Yfvz4R1GeM3P91gL3rrqLp1OOUlOSPQBo2VvP+9f1PqApiWIP8FlTWo=
- Part 5: Error executing "UploadPart" on "https://path.to.file.zip?partNumber=5&uploadId=gNqjjR_8MAQRvyI1fT64GnYb7yaR5IOqnSE.o61g1n1rr2fX4fTX4B4bqHbjXhIwZceDdd2x0sNQ.GAh.3LUhpJiAt1UY0HiWgBRim.X48ScEpzWLbJNil4flu6NHUAo"; AWS HTTP error: Client error: 400 XAmzContentSHA256Mismatch (client): The provided 'x-amz-content-sha256' header does not match what was computed. - 
XAmzContentSHA256MismatchThe provided 'x-amz-content-sha256' header does not match what was computed.iihTifIEEGAVVrNeGA0bn5Uuz6Hx6ZWDipPr+siHpLc=8a285389f20410601556b35e180d1b9f952ecfa1f1e995838a93ebfac887a4b74A998BE462A789C9OOW31pVc09eAmK7ZeZXbCokOf0vUTXvQ7ofCaKoDjCpHSKFCcJkzTEjy6sy3GwC4sCdtm1Zervc=
- Part 7: Error executing "UploadPart" on "https://path.to.file.zip?partNumber=7&uploadId=gNqjjR_8MAQRvyI1fT64GnYb7yaR5IOqnSE.o61g1n1rr2fX4fTX4B4bqHbjXhIwZceDdd2x0sNQ.GAh.3LUhpJiAt1UY0HiWgBRim.X48ScEpzWLbJNil4flu6NHUAo"; AWS HTTP error: Client error: 400 XAmzContentSHA256Mismatch (client): The provided 'x-amz-content-sha256' header does not match what was computed. - 
XAmzContentSHA256MismatchThe provided 'x-amz-content-sha256' header does not match what was computed.bTQC9QW84OKeUuC3JFxsBDwqFeg/89vsECAkGnrIBFg=6d3402f505bce0e29e52e0b7245c6c043c2a15e83ff3dbec1020241a7ac80458A5BB9A5B0D459730djuRmeJ4Xv2Twy0kuGVQkGIgu5sHlLrSJ7qk7Dd7Hr5ohiZ+/YG5yZQiwvudYIXDp7oKzRKnBI8=
- Part 6: Error executing "UploadPart" on "https://path.to.file.zip?partNumber=6&uploadId=gNqjjR_8MAQRvyI1fT64GnYb7yaR5IOqnSE.o61g1n1rr2fX4fTX4B4bqHbjXhIwZceDdd2x0sNQ.GAh.3LUhpJiAt1UY0HiWgBRim.X48ScEpzWLbJNil4flu6NHUAo"; AWS HTTP error: Client error: 400 XAmzContentSHA256Mismatch (client): The provided 'x-amz-content-sha256' header does not match what was computed. - 
XAmzContentSHA256MismatchThe provided 'x-amz-content-sha256' header does not match what was computed.i9nHCbbym5VbnuisiNNoHKefev1LzdrZ8Yq7Zs490D0=8bd9c709b6f29b955b9ee8ac88d3681ca79f7afd4bcddad9f18abb66ce3dd03d694465F5C4A1DA31QLiftIN26TJqjgToq88Xmk06h4gj1niEk5KyNOvdB7mfuWdmWCLs7uE6jItwN7/GjzHERFR5zrw=
- Part 8: Error executing "UploadPart" on "https://path.to.file.zip?partNumber=8&uploadId=gNqjjR_8MAQRvyI1fT64GnYb7yaR5IOqnSE.o61g1n1rr2fX4fTX4B4bqHbjXhIwZceDdd2x0sNQ.GAh.3LUhpJiAt1UY0HiWgBRim.X48ScEpzWLbJNil4flu6NHUAo"; AWS HTTP error: Client error: 400 XAmzContentSHA256Mismatch (client): The provided 'x-amz-content-sha256' header does not match what was computed. - 
XAmzContentSHA256MismatchThe provided 'x-amz-content-sha256' header does not match what was computed.0so55Y7l5hVZTrgb+vMa6fFgPtJrbEa3uZD6orqHF4A=d2ca39e58ee5e615594eb81bfaf31ae9f1603ed26b6c46b7b990faa2ba8717805B8946D6E6FB38D8LKV1SzbaW4V9ldtg2H8mDR9RKPFHdEGd5pCDJXqYp5ZarVCoRomsE9C9hZrhqQZfOZVdJlAmTa0=
- Part 10: Error executing "UploadPart" on "https://path.to.file.zip?partNumber=10&uploadId=gNqjjR_8MAQRvyI1fT64GnYb7yaR5IOqnSE.o61g1n1rr2fX4fTX4B4bqHbjXhIwZceDdd2x0sNQ.GAh.3LUhpJiAt1UY0HiWgBRim.X48ScEpzWLbJNil4flu6NHUAo"; AWS HTTP error: Client error: 400 XAmzContentSHA256Mismatch (client): The provided 'x-amz-content-sha256' header does not match what was computed. - 
XAmzContentSHA256MismatchThe provided 'x-amz-content-sha256' header does not match what was computed.Bmd3ZanEUhSBmuspStuhj7IwWkwBiJuxW2rbOpy8Eog=06677765a9c45214819aeb294adba18fb2305a4c01889bb15b6adb3a9cbc1288765F16E2ABC87A06eUes9BW1WFcU55wGdbhFTiCQF+hkotSrDBhKRX8FnlaQRGmjUppB9JC4H1aLsXF6u5zQdbuNt6Q=
- Part 9: Error executing "UploadPart" on "https://path.to.file.zip?partNumber=9&uploadId=gNqjjR_8MAQRvyI1fT64GnYb7yaR5IOqnSE.o61g1n1rr2fX4fTX4B4bqHbjXhIwZceDdd2x0sNQ.GAh.3LUhpJiAt1UY0HiWgBRim.X48ScEpzWLbJNil4flu6NHUAo"; AWS HTTP error: Client error: 400 XAmzContentSHA256Mismatch (client): The provided 'x-amz-content-sha256' header does not match what was computed. - 
XAmzContentSHA256MismatchThe provided 'x-amz-content-sha256' header does not match what was computed.etOd96Xj4yFl6dzfbvQKrOo7lNkHZ0IcoAqAk7BRDp0=7ad39df7a5e3e32165e9dcdf6ef40aacea3b94d90767421ca00a8093b0510e9dD2883C7E4D7EA9C10EsPq7dKMVlNy6W6VmhCRXD+es+l6bzmnXDO44dX94yrRfeufYwaj+8eiEtmjRoj3dCVSFXjS68=
- Part 11: Error executing "UploadPart" on "https://path.to.file.zip?partNumber=11&uploadId=gNqjjR_8MAQRvyI1fT64GnYb7yaR5IOqnSE.o61g1n1rr2fX4fTX4B4bqHbjXhIwZceDdd2x0sNQ.GAh.3LUhpJiAt1UY0HiWgBRim.X48ScEpzWLbJNil4flu6NHUAo"; AWS HTTP error: Client error: 400 XAmzContentSHA256Mismatch (client): The provided 'x-amz-content-sha256' header does not match what was computed. - 
XAmzContentSHA256MismatchThe provided 'x-amz-content-sha256' header does not match what was computed.Zl4vFt+ifvoItLdUfG2CR8gn2d3DIm7CH/gMSdcTQPU=665e2f16dfa27efa08b4b7547c6d8247c827d9ddc3226ec21ff80c49d71340f594DBBFAE50D54BBCPErpqjm+hzVTXW6YK0DiCsIZFKZxivNnnmj16hax/H0ciHj+kcv3y69wldeBGK2Xk7aMQxpyDZk=
- Part 14: Error executing "UploadPart" on "https://path.to.file.zip?partNumber=14&uploadId=gNqjjR_8MAQRvyI1fT64GnYb7yaR5IOqnSE.o61g1n1rr2fX4fTX4B4bqHbjXhIwZceDdd2x0sNQ.GAh.3LUhpJiAt1UY0HiWgBRim.X48ScEpzWLbJNil4flu6NHUAo"; AWS HTTP error: Client error: 400 XAmzContentSHA256Mismatch (client): The provided 'x-amz-content-sha256' header does not match what was computed. - 
XAmzContentSHA256MismatchThe provided 'x-amz-content-sha256' header does not match what was computed.0wSTMCRb3CQeyQtkEKM49TuRlGt2nK9hfRBlUxAwZPY=d3049330245bdc241ec90b6410a338f53b91946b769caf617d106553103064f66E36484A826E0562+YUMWK53gpMhr4JiLWWIhLnEL/i1e+GRPfFkPpkC/MzaivoV8X/dz3L9WTFs01mIWNYN0BcXqg8=
- Part 13: Error executing "UploadPart" on "https://path.to.file.zip?partNumber=13&uploadId=gNqjjR_8MAQRvyI1fT64GnYb7yaR5IOqnSE.o61g1n1rr2fX4fTX4B4bqHbjXhIwZceDdd2x0sNQ.GAh.3LUhpJiAt1UY0HiWgBRim.X48ScEpzWLbJNil4flu6NHUAo"; AWS HTTP error: Client error: 400 XAmzContentSHA256Mismatch (client): The provided 'x-amz-content-sha256' header does not match what was computed. - 
XAmzContentSHA256MismatchThe provided 'x-amz-content-sha256' header does not match what was computed.TO0RHsjJvm36fuFCSRFhO2UWfvIvG5B9OiQbkYiNFgk=4ced111ec8c9be6dfa7ee1424911613b65167ef22f1b907d3a241b91888d1609F4C7D4220236F615a9CXJ4LEChIPGsVRh8r2GcOf+GSzkgCWsQxk0QaRx9xvMXseXG/DNJ2ds42WHLOY6dolPx/ez2w=
- Part 12: Error executing "UploadPart" on "https://path.to.file.zip?partNumber=12&uploadId=gNqjjR_8MAQRvyI1fT64GnYb7yaR5IOqnSE.o61g1n1rr2fX4fTX4B4bqHbjXhIwZceDdd2x0sNQ.GAh.3LUhpJiAt1UY0HiWgBRim.X48ScEpzWLbJNil4flu6NHUAo"; AWS HTTP error: Client error: 400 XAmzContentSHA256Mismatch (client): The provided 'x-amz-content-sha256' header does not match what was computed. - 
XAmzContentSHA256MismatchThe provided 'x-amz-content-sha256' header does not match what was computed.PzmxWL0SFrDHepD6EdX+UoBKZz60KUPyP9n0ZYC+/LM=3f39b158bd1216b0c77a90fa11d5fe52804a673eb42943f23fd9f46580befcb37590DFD6928545FCqM0eVKmkpxg52AbQL+2evghaXR9CXBnpUK0YcKMhbKC2wJKdOqCTCBjb5r9c17aMcqv0RoGZK7M=
' in /vendor/aws/aws/aws-sdk-php/src/Multipart/AbstractUploader.php:130
Stack trace:
#0 [internal function]: Aws\Multipart\AbstractUploader::Aws\Multipart\{closure}()
#1 /vendor/aws/guzzlehttp/promises/src/functions.php(489): Generator->send(NULL)
#2 /vendor/aws/guzzlehttp/promises/src/Promise.php(199): GuzzleHttp\Promise\{closure}(NULL)
#3 /vendor/aws/guzzlehttp/promises/src/Promise.php(152): GuzzleHttp\Promise\Promise::callHandler(1, NULL, Array)
#4 /vendor/aws/guzzlehttp/promises/src/TaskQueue.php(60): GuzzleHttp\Promise\Promise::GuzzleHttp\Promise\{closure}()
#5 /vendor/aws/guzzlehttp/guzzle/src/Handler/CurlMultiHandler.php(96): GuzzleHttp\Promise\TaskQueue->run()
#6 /vendor/aws/guzzlehttp/guzzle/src/Handler/CurlMultiHandler.php(123): GuzzleHttp\Handler\CurlMultiHandler->tick()
#7 /vendor/aws/guzzlehttp/promises/src/Promise.php(240): GuzzleHttp\Handler\CurlMultiHandler->execute(true)
#8 /vendor/aws/guzzlehttp/promises/src/Promise.php(217): GuzzleHttp\Promise\Promise->invokeWaitFn()
#9 /vendor/aws/guzzlehttp/promises/src/Promise.php(261): GuzzleHttp\Promise\Promise->waitIfPending()
#10 /vendor/aws/guzzlehttp/promises/src/Promise.php(219): GuzzleHttp\Promise\Promise->invokeWaitList()
#11 /vendor/aws/guzzlehttp/promises/src/Promise.php(261): GuzzleHttp\Promise\Promise->waitIfPending()
#12 /vendor/aws/guzzlehttp/promises/src/Promise.php(219): GuzzleHttp\Promise\Promise->invokeWaitList()
#13 /vendor/aws/guzzlehttp/promises/src/Promise.php(62): GuzzleHttp\Promise\Promise->waitIfPending()
#14 /vendor/aws/aws/aws-sdk-php/src/Multipart/AbstractUploader.php(86): GuzzleHttp\Promise\Promise->wait()
#15 /vendor/aws/aws/aws-sdk-php/src/S3/S3Client.php(303): Aws\Multipart\AbstractUploader->upload()
#16 /vendor/aws/league/flysystem-aws-s3-v3/src/AwsS3Adapter.php(509): Aws\S3\S3Client->upload('icpayment', 'project_name...', 'PK\x03\x04\x14\x00\x00\x00\x08\x00{+\xEBF\xB7...', 'private', Array)
#17 /vendor/aws/league/flysystem-aws-s3-v3/src/AwsS3Adapter.php(95): League\Flysystem\AwsS3v3\AwsS3Adapter->upload('project_name...', 'PK\x03\x04\x14\x00\x00\x00\x08\x00{+\xEBF\xB7...', Object(League\Flysystem\Config))
#18 /vendor/aws/league/flysystem/src/Filesystem.php(81): League\Flysystem\AwsS3v3\AwsS3Adapter->write('project_name...', 'PK\x03\x04\x14\x00\x00\x00\x08\x00{+\xEBF\xB7...', Object(League\Flysystem\Config))
#19 /system/lib/FILE.php(100): League\Flysystem\Filesystem->write('project_name...', 'PK\x03\x04\x14\x00\x00\x00\x08\x00{+\xEBF\xB7...')
#20 /system/lib/FILE.php(117): FILE::write('project_name...', 'PK\x03\x04\x14\x00\x00\x00\x08\x00{+\xEBF\xB7...')
#21 /cron/ftp_import.php(67): FILE::upload('/tmp/2015-07-11...', 'project_name...')
#22 {main}
  thrown in /vendor/aws/aws/aws-sdk-php/src/Multipart/AbstractUploader.php on line 130

this looks a bit odd to me - because file size is only 65MB
using latest flysystem awss3v3 adapter with aws 3.1.0

jeremeamia · 2015-07-11T17:19:13Z

@zexz We just noticed this problem as well. It's fixed in d34b5e6.

teseo · 2015-08-25T13:42:37Z

I'm afraid is still happening in (3.3.0)

jeskew · 2015-08-25T16:09:52Z

@teseo Can you post a bit more detail? This ticket is still open because we have been unable to reproduce it.

teseo · 2015-08-25T16:23:19Z

Exception message:

An exception occurred while completing a multipart upload.

Exception Trace:

#0 /path/vendor/guzzlehttp/promises/src/Promise.php(199): Aws\Multipart\AbstractUploader->Aws\Multipart\{closure}(Object(Aws\S3\Exception\S3Exception))
#1 /path/vendor/guzzlehttp/promises/src/Promise.php(152): GuzzleHttp\Promise\Promise::callHandler(2, Object(Aws\S3\Exception\S3Exception), Array)
#2 /path/vendor/guzzlehttp/promises/src/TaskQueue.php(60): GuzzleHttp\Promise\Promise::GuzzleHttp\Promise\{closure}()
#3 /path/vendor/guzzlehttp/guzzle/src/Handler/CurlMultiHandler.php(96): GuzzleHttp\Promise\TaskQueue->run()
#4 /path/vendor/guzzlehttp/guzzle/src/Handler/CurlMultiHandler.php(123): GuzzleHttp\Handler\CurlMultiHandler->tick()
#5 /path/vendor/guzzlehttp/promises/src/Promise.php(240): GuzzleHttp\Handler\CurlMultiHandler->execute(true)
#6 /path/vendor/guzzlehttp/promises/src/Promise.php(217): GuzzleHttp\Promise\Promise->invokeWaitFn()
#7 /path/vendor/guzzlehttp/promises/src/Promise.php(261): GuzzleHttp\Promise\Promise->waitIfPending()
#8 /path/vendor/guzzlehttp/promises/src/Promise.php(219): GuzzleHttp\Promise\Promise->invokeWaitList()
#9 /path/vendor/guzzlehttp/promises/src/Promise.php(261): GuzzleHttp\Promise\Promise->waitIfPending()
#10 /path/vendor/guzzlehttp/promises/src/Promise.php(219): GuzzleHttp\Promise\Promise->invokeWaitList()
#11 /path/vendor/guzzlehttp/promises/src/Promise.php(62): GuzzleHttp\Promise\Promise->waitIfPending()
#12 /path/vendor/aws/aws-sdk-php/src/Multipart/AbstractUploader.php(86): GuzzleHttp\Promise\Promise->wait()
#13 /path/vendor/path/Job/MoveToAwsJob.php(121): Aws\Multipart\AbstractUploader->upload()
#14 /path/vendor/slm/queue-beanstalkd/src/SlmQueueBeanstalkd/Worker/BeanstalkdWorker.php(33): Module\Job\MoveToAwsJob->execute()
#15 /path/vendor/slm/queue/src/SlmQueue/Strategy/ProcessQueueStrategy.php(70): SlmQueueBeanstalkd\Worker\BeanstalkdWorker->processJob(Object(Module\Job\MoveToAwsJob), Object(SlmQueueBeanstalkd\Queue\BeanstalkdQueue))
#16 [internal function]: SlmQueue\Strategy\ProcessQueueStrategy->onJobProcess(Object(SlmQueue\Worker\WorkerEvent))
#17 /path/vendor/zendframework/zend-eventmanager/src/EventManager.php(444): call_user_func(Array, Object(SlmQueue\Worker\WorkerEvent))
#18 /path/vendor/zendframework/zend-eventmanager/src/EventManager.php(205): Zend\EventManager\EventManager->triggerListeners('process.job', Object(SlmQueue\Worker\WorkerEvent), Array)
#19 /path/vendor/slm/queue/src/SlmQueue/Strategy/ProcessQueueStrategy.php(55): Zend\EventManager\EventManager->trigger('process.job', Object(SlmQueue\Worker\WorkerEvent))
#20 [internal function]: SlmQueue\Strategy\ProcessQueueStrategy->onJobPop(Object(SlmQueue\Worker\WorkerEvent))
#21 /path/vendor/zendframework/zend-eventmanager/src/EventManager.php(444): call_user_func(Array, Object(SlmQueue\Worker\WorkerEvent))
#22 /path/vendor/zendframework/zend-eventmanager/src/EventManager.php(205): Zend\EventManager\EventManager->triggerListeners('process.queue', Object(SlmQueue\Worker\WorkerEvent), Array)
#23 /path/vendor/slm/queue/src/SlmQueue/Worker/AbstractWorker.php(46): Zend\EventManager\EventManager->trigger('process.queue', Object(SlmQueue\Worker\WorkerEvent))
#24 /path/vendor/slm/queue/src/SlmQueue/Controller/AbstractWorkerController.php(49): SlmQueue\Worker\AbstractWorker->processQueue(Object(SlmQueueBeanstalkd\Queue\BeanstalkdQueue), Array)
#25 /path/vendor/zendframework/zend-mvc/src/Controller/AbstractActionController.php(82): SlmQueue\Controller\AbstractWorkerController->processAction()
#26 [internal function]: Zend\Mvc\Controller\AbstractActionController->onDispatch(Object(Zend\Mvc\MvcEvent))
#27 /path/vendor/zendframework/zend-eventmanager/src/EventManager.php(444): call_user_func(Array, Object(Zend\Mvc\MvcEvent))
#28 /path/vendor/zendframework/zend-eventmanager/src/EventManager.php(205): Zend\EventManager\EventManager->triggerListeners('dispatch', Object(Zend\Mvc\MvcEvent), Object(Closure))
#29 /path/vendor/zendframework/zend-mvc/src/Controller/AbstractController.php(118): Zend\EventManager\EventManager->trigger('dispatch', Object(Zend\Mvc\MvcEvent), Object(Closure))
#30 /path/vendor/zendframework/zend-mvc/src/DispatchListener.php(93): Zend\Mvc\Controller\AbstractController->dispatch(Object(Zend\Console\Request), Object(Zend\Console\Response))
#31 [internal function]: Zend\Mvc\DispatchListener->onDispatch(Object(Zend\Mvc\MvcEvent))
#32 /path/vendor/zendframework/zend-eventmanager/src/EventManager.php(444): call_user_func(Array, Object(Zend\Mvc\MvcEvent))
#33 /path/vendor/zendframework/zend-eventmanager/src/EventManager.php(205): Zend\EventManager\EventManager->triggerListeners('dispatch', Object(Zend\Mvc\MvcEvent), Object(Closure))
#34 /path/vendor/zendframework/zend-mvc/src/Application.php(314): Zend\EventManager\EventManager->trigger('dispatch', Object(Zend\Mvc\MvcEvent), Object(Closure))
#35 /path/public/index.php(17): Zend\Mvc\Application->run()
#36 {main}"

jeskew · 2015-08-25T16:33:01Z

@teseo please see my comment on issue #674.

mwoodring · 2015-08-27T22:27:08Z

I changed the source parameter from a file resource handle (fopen) to a path. Now when the upload gets an error and goes through the catch function to restart the upload, it works as expected. Every time I used a file resource handle as the source and there were errors, it would crash with the message like I showed on July 1.

jeskew · 2015-08-28T17:57:37Z

@mwoodring Got it. The multipart uploader will create a fresh file handle when you give it a path, but it won't automatically rewind an open handle passed to its constructor. I'll update the docs to show how to use this technique with a stream instead of a path.

… handle. Addresses aws#669

jeskew · 2015-09-01T21:09:16Z

I've updated the documentation to warn about streams not being rewound automatically. Please feel free to reopen if you have any further questions.

jeremeamia added the guidance Question that needs advice or information. label Jul 1, 2015

jeskew added a commit to jeskew/aws-sdk-php that referenced this issue Aug 28, 2015

Update multipart uploader error recovery section to show using a file…

d2f2f0f

… handle. Addresses aws#669

jeskew closed this as completed Sep 1, 2015

freekmurze mentioned this issue Mar 14, 2017

Occasional error when uploading a large backup to S3 spatie/laravel-backup#394

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Multipart Upload reporting errors at end of upload in sdk v3 #669

Multipart Upload reporting errors at end of upload in sdk v3 #669

mwoodring commented Jul 1, 2015

jeremeamia commented Jul 1, 2015

mwoodring commented Jul 1, 2015

jeremeamia commented Jul 1, 2015

mwoodring commented Jul 1, 2015

jeremeamia commented Jul 2, 2015

mwoodring commented Jul 2, 2015

jeremeamia commented Jul 2, 2015

mwoodring commented Jul 2, 2015

zexz commented Jul 11, 2015

jeremeamia commented Jul 11, 2015

teseo commented Aug 25, 2015

jeskew commented Aug 25, 2015

teseo commented Aug 25, 2015

jeskew commented Aug 25, 2015

mwoodring commented Aug 27, 2015

jeskew commented Aug 28, 2015

jeskew commented Sep 1, 2015

Multipart Upload reporting errors at end of upload in sdk v3 #669

Multipart Upload reporting errors at end of upload in sdk v3 #669

Comments

mwoodring commented Jul 1, 2015

jeremeamia commented Jul 1, 2015

mwoodring commented Jul 1, 2015

jeremeamia commented Jul 1, 2015

mwoodring commented Jul 1, 2015

Thanks,

jeremeamia commented Jul 2, 2015

mwoodring commented Jul 2, 2015

Thanks,

jeremeamia commented Jul 2, 2015

mwoodring commented Jul 2, 2015

zexz commented Jul 11, 2015

jeremeamia commented Jul 11, 2015

teseo commented Aug 25, 2015

jeskew commented Aug 25, 2015

teseo commented Aug 25, 2015

jeskew commented Aug 25, 2015

mwoodring commented Aug 27, 2015

jeskew commented Aug 28, 2015

jeskew commented Sep 1, 2015