Skip to content
This repository has been archived by the owner on Jul 26, 2022. It is now read-only.

Fail after lengthy upload: s3 plugin #448

Closed
jgn opened this issue Apr 23, 2015 · 5 comments · Fixed by syncany/syncany-plugin-s3#6
Closed

Fail after lengthy upload: s3 plugin #448

jgn opened this issue Apr 23, 2015 · 5 comments · Fixed by syncany/syncany-plugin-s3#6
Labels
type:bug what:plugin

Comments

@jgn
Copy link

jgn commented Apr 23, 2015

After a lot gets uploaded, I get (with --debug):

23-4-15 16:17:55.615 | S3TransferManag | main           | SEVE : Cannot move temp/temp-MPefe-multichunk-64423d879702a890af83327ccdeaf17b5e635006 to multichunks/multichunk-64423d879702a890af83327ccdeaf17b5e635006
org.jets3t.service.ServiceException: Service Error Message. -- ResponseCode: 404, ResponseStatus: Not Found, XML Error Message: <Error><Code>NoSuchKey</Code><Message>The specified key does not exist.</Message><Key>temp/temp-MPefe-multichunk-64423d879702a890af83327ccdeaf17b5e635006</Key><RequestId>46C9E50F678F206A</RequestId><HostId>6ih/Mi14KMD0NvY+SopKdnWBLwPywn7piiTTjw1XoHeEJAfVJUF3LbqFnZ929xz7WYn2H70mT9c=</HostId></Error>
    at org.jets3t.service.impl.rest.httpclient.RestStorageService.performRequest(RestStorageService.java:401)
    at org.jets3t.service.impl.rest.httpclient.RestStorageService.performRequest(RestStorageService.java:265)
    at org.jets3t.service.impl.rest.httpclient.RestStorageService.performRestPut(RestStorageService.java:1019)
    at org.jets3t.service.impl.rest.httpclient.RestStorageService.copyObjectImpl(RestStorageService.java:1993)
    at org.jets3t.service.StorageService.copyObject(StorageService.java:898)
    at org.jets3t.service.StorageService.copyObject(StorageService.java:943)
    at org.jets3t.service.StorageService.moveObject(StorageService.java:992)
    at org.jets3t.service.StorageService.renameObject(StorageService.java:1038)
    at org.syncany.plugins.s3.S3TransferManager.move(S3TransferManager.java:256)
    at org.syncany.plugins.transfer.RetriableTransferManager$5.execute(RetriableTransferManager.java:105)
    at org.syncany.plugins.transfer.RetriableTransferManager.retryMethod(RetriableTransferManager.java:197)
    at org.syncany.plugins.transfer.RetriableTransferManager.move(RetriableTransferManager.java:102)
    at org.syncany.plugins.transfer.TransactionAwareTransferManager.move(TransactionAwareTransferManager.java:124)
    at org.syncany.plugins.transfer.RemoteTransaction.moveToFinalLocation(RemoteTransaction.java:273)
    at org.syncany.plugins.transfer.RemoteTransaction.commit(RemoteTransaction.java:143)
    at org.syncany.plugins.transfer.RemoteTransaction.commit(RemoteTransaction.java:131)
    at org.syncany.operations.up.UpOperation.execute(UpOperation.java:205)
    at org.syncany.Client.up(Client.java:120)
    at org.syncany.cli.UpCommand.execute(UpCommand.java:57)
    at org.syncany.cli.CommandLineClient.runLocally(CommandLineClient.java:389)
    at org.syncany.cli.CommandLineClient.runCommand(CommandLineClient.java:379)
    at org.syncany.cli.CommandLineClient.start(CommandLineClient.java:195)
    at org.syncany.Syncany.main(Syncany.java:55)
23-4-15 16:17:55.616 | RetriableTransf | main           | INFO : StorageException caused by missing file, not the connection. Not retrying.
23-4-15 16:17:55.616 | UpOperation     | main           | INFO : Persisting status of UpOperation to /Users/jgn/Documents/sync-icebox/.syncany/state ...
23-4-15 16:17:55.899 | RemoteTransacti | main           | INFO : Wrote transaction manifest to temporary file: /Users/jgn/Documents/sync-icebox/.syncany/state/transaction-actions.xml
23-4-15 16:17:58.295 | CommandLineClie | main           | SEVE : Command org.syncany.cli.UpCommand@5010cdd4 FAILED.
org.syncany.plugins.transfer.StorageMoveException: org.jets3t.service.ServiceException: Service Error Message. -- ResponseCode: 404, ResponseStatus: Not Found, XML Error Message: <Error><Code>NoSuchKey</Code><Message>The specified key does not exist.</Message><Key>temp/temp-MPefe-multichunk-64423d879702a890af83327ccdeaf17b5e635006</Key><RequestId>46C9E50F678F206A</RequestId><HostId>6ih/Mi14KMD0NvY+SopKdnWBLwPywn7piiTTjw1XoHeEJAfVJUF3LbqFnZ929xz7WYn2H70mT9c=</HostId></Error>
    at org.syncany.plugins.s3.S3TransferManager.move(S3TransferManager.java:260)
    at org.syncany.plugins.transfer.RetriableTransferManager$5.execute(RetriableTransferManager.java:105)
    at org.syncany.plugins.transfer.RetriableTransferManager.retryMethod(RetriableTransferManager.java:197)
    at org.syncany.plugins.transfer.RetriableTransferManager.move(RetriableTransferManager.java:102)
    at org.syncany.plugins.transfer.TransactionAwareTransferManager.move(TransactionAwareTransferManager.java:124)
    at org.syncany.plugins.transfer.RemoteTransaction.moveToFinalLocation(RemoteTransaction.java:273)
    at org.syncany.plugins.transfer.RemoteTransaction.commit(RemoteTransaction.java:143)
    at org.syncany.plugins.transfer.RemoteTransaction.commit(RemoteTransaction.java:131)
    at org.syncany.operations.up.UpOperation.execute(UpOperation.java:205)
    at org.syncany.Client.up(Client.java:120)
    at org.syncany.cli.UpCommand.execute(UpCommand.java:57)
    at org.syncany.cli.CommandLineClient.runLocally(CommandLineClient.java:389)
    at org.syncany.cli.CommandLineClient.runCommand(CommandLineClient.java:379)
    at org.syncany.cli.CommandLineClient.start(CommandLineClient.java:195)
    at org.syncany.Syncany.main(Syncany.java:55)
Caused by: org.jets3t.service.ServiceException: Service Error Message. -- ResponseCode: 404, ResponseStatus: Not Found, XML Error Message: <Error><Code>NoSuchKey</Code><Message>The specified key does not exist.</Message><Key>temp/temp-MPefe-multichunk-64423d879702a890af83327ccdeaf17b5e635006</Key><RequestId>46C9E50F678F206A</RequestId><HostId>6ih/Mi14KMD0NvY+SopKdnWBLwPywn7piiTTjw1XoHeEJAfVJUF3LbqFnZ929xz7WYn2H70mT9c=</HostId></Error>
    at org.jets3t.service.impl.rest.httpclient.RestStorageService.performRequest(RestStorageService.java:401)
    at org.jets3t.service.impl.rest.httpclient.RestStorageService.performRequest(RestStorageService.java:265)
    at org.jets3t.service.impl.rest.httpclient.RestStorageService.performRestPut(RestStorageService.java:1019)
    at org.jets3t.service.impl.rest.httpclient.RestStorageService.copyObjectImpl(RestStorageService.java:1993)
    at org.jets3t.service.StorageService.copyObject(StorageService.java:898)
    at org.jets3t.service.StorageService.copyObject(StorageService.java:943)
    at org.jets3t.service.StorageService.moveObject(StorageService.java:992)
    at org.jets3t.service.StorageService.renameObject(StorageService.java:1038)
    at org.syncany.plugins.s3.S3TransferManager.move(S3TransferManager.java:256)
    ... 14 more
Error: org.jets3t.service.ServiceException: Service Error Message. -- ResponseCode: 404, ResponseStatus: Not Found, XML Error Message: <Error><Code>NoSuchKey</Code><Message>The specified key does not exist.</Message><Key>temp/temp-MPefe-multichunk-64423d879702a890af83327ccdeaf17b5e635006</Key><RequestId>46C9E50F678F206A</RequestId><HostId>6ih/Mi14KMD0NvY+SopKdnWBLwPywn7piiTTjw1XoHeEJAfVJUF3LbqFnZ929xz7WYn2H70mT9c=</HostId></Error>
       Refer to help page using '--help'.

If I then attempt a fresh sy up I get:

23-4-15 18:36:31.318 | CipherSession   | main           | FINE : - Created NEW WRITE secret key Twofish, and added to cache, with salt af9911f05794cb78b620a09c
23-4-15 18:36:35.703 | UpOperation     | main           | INFO : Adding shutdown hook (to allow resuming the upload) ...
23-4-15 18:36:35.703 | RemoteTransacti | main           | INFO : - Starting to upload data in commit.
23-4-15 18:36:35.705 | RemoteTransacti | main           | INFO : - Moving temp. file RemoteFile[name=temp-MPefe-multichunk-64423d879702a890af83327ccdeaf17b5e635006] to final location RemoteFile[name=multichunk-64423d879702a890af83327ccdeaf17b5e635006] ...
23-4-15 18:36:35.778 | S3TransferManag | main           | SEVE : Cannot move temp/temp-MPefe-multichunk-64423d879702a890af83327ccdeaf17b5e635006 to multichunks/multichunk-64423d879702a890af83327ccdeaf17b5e635006
org.jets3t.service.ServiceException: Service Error Message. -- ResponseCode: 404, ResponseStatus: Not Found, XML Error Message: <Error><Code>NoSuchKey</Code><Message>The specified key does not exist.</Message><Key>temp/temp-MPefe-multichunk-64423d879702a890af83327ccdeaf17b5e635006</Key><RequestId>F15D14BA45AA9C44</RequestId><HostId>4z4nf4Ms6vk8tOIuKPk6hqYkGFnCcY8sGcRHfSI37FG5CoPRaAP8dstiL4P1ZZ4eZRZzh4wm2xo=</HostId></Error>
    at org.jets3t.service.impl.rest.httpclient.RestStorageService.performRequest(RestStorageService.java:401)
    at org.jets3t.service.impl.rest.httpclient.RestStorageService.performRequest(RestStorageService.java:265)
    at org.jets3t.service.impl.rest.httpclient.RestStorageService.performRestPut(RestStorageService.java:1019)
    at org.jets3t.service.impl.rest.httpclient.RestStorageService.copyObjectImpl(RestStorageService.java:1993)
    at org.jets3t.service.StorageService.copyObject(StorageService.java:898)
    at org.jets3t.service.StorageService.copyObject(StorageService.java:943)
    at org.jets3t.service.StorageService.moveObject(StorageService.java:992)
    at org.jets3t.service.StorageService.renameObject(StorageService.java:1038)
    at org.syncany.plugins.s3.S3TransferManager.move(S3TransferManager.java:256)
    at org.syncany.plugins.transfer.RetriableTransferManager$5.execute(RetriableTransferManager.java:105)
    at org.syncany.plugins.transfer.RetriableTransferManager.retryMethod(RetriableTransferManager.java:197)
    at org.syncany.plugins.transfer.RetriableTransferManager.move(RetriableTransferManager.java:102)
    at org.syncany.plugins.transfer.TransactionAwareTransferManager.move(TransactionAwareTransferManager.java:124)
    at org.syncany.plugins.transfer.RemoteTransaction.moveToFinalLocation(RemoteTransaction.java:273)
    at org.syncany.plugins.transfer.RemoteTransaction.commit(RemoteTransaction.java:143)
    at org.syncany.operations.up.UpOperation.execute(UpOperation.java:208)
    at org.syncany.Client.up(Client.java:120)
    at org.syncany.cli.UpCommand.execute(UpCommand.java:57)
    at org.syncany.cli.CommandLineClient.runLocally(CommandLineClient.java:389)
    at org.syncany.cli.CommandLineClient.runCommand(CommandLineClient.java:379)
    at org.syncany.cli.CommandLineClient.start(CommandLineClient.java:195)
    at org.syncany.Syncany.main(Syncany.java:55)
23-4-15 18:36:35.780 | RetriableTransf | main           | INFO : StorageException caused by missing file, not the connection. Not retrying.
23-4-15 18:36:35.780 | UpOperation     | main           | INFO : Persisting status of UpOperation to /Users/jgn/Documents/sync-icebox/.syncany/state ...
23-4-15 18:36:36.359 | RemoteTransacti | main           | INFO : Wrote transaction manifest to temporary file: /Users/jgn/Documents/sync-icebox/.syncany/state/transaction-actions.xml
23-4-15 18:36:38.520 | CommandLineClie | main           | SEVE : Command org.syncany.cli.UpCommand@7946738 FAILED.
org.syncany.plugins.transfer.StorageMoveException: org.jets3t.service.ServiceException: Service Error Message. -- ResponseCode: 404, ResponseStatus: Not Found, XML Error Message: <Error><Code>NoSuchKey</Code><Message>The specified key does not exist.</Message><Key>temp/temp-MPefe-multichunk-64423d879702a890af83327ccdeaf17b5e635006</Key><RequestId>F15D14BA45AA9C44</RequestId><HostId>4z4nf4Ms6vk8tOIuKPk6hqYkGFnCcY8sGcRHfSI37FG5CoPRaAP8dstiL4P1ZZ4eZRZzh4wm2xo=</HostId></Error>
    at org.syncany.plugins.s3.S3TransferManager.move(S3TransferManager.java:260)
    at org.syncany.plugins.transfer.RetriableTransferManager$5.execute(RetriableTransferManager.java:105)
    at org.syncany.plugins.transfer.RetriableTransferManager.retryMethod(RetriableTransferManager.java:197)
    at org.syncany.plugins.transfer.RetriableTransferManager.move(RetriableTransferManager.java:102)
    at org.syncany.plugins.transfer.TransactionAwareTransferManager.move(TransactionAwareTransferManager.java:124)
    at org.syncany.plugins.transfer.RemoteTransaction.moveToFinalLocation(RemoteTransaction.java:273)
    at org.syncany.plugins.transfer.RemoteTransaction.commit(RemoteTransaction.java:143)
    at org.syncany.operations.up.UpOperation.execute(UpOperation.java:208)
    at org.syncany.Client.up(Client.java:120)
    at org.syncany.cli.UpCommand.execute(UpCommand.java:57)
    at org.syncany.cli.CommandLineClient.runLocally(CommandLineClient.java:389)
    at org.syncany.cli.CommandLineClient.runCommand(CommandLineClient.java:379)
    at org.syncany.cli.CommandLineClient.start(CommandLineClient.java:195)
    at org.syncany.Syncany.main(Syncany.java:55)
Caused by: org.jets3t.service.ServiceException: Service Error Message. -- ResponseCode: 404, ResponseStatus: Not Found, XML Error Message: <Error><Code>NoSuchKey</Code><Message>The specified key does not exist.</Message><Key>temp/temp-MPefe-multichunk-64423d879702a890af83327ccdeaf17b5e635006</Key><RequestId>F15D14BA45AA9C44</RequestId><HostId>4z4nf4Ms6vk8tOIuKPk6hqYkGFnCcY8sGcRHfSI37FG5CoPRaAP8dstiL4P1ZZ4eZRZzh4wm2xo=</HostId></Error>
    at org.jets3t.service.impl.rest.httpclient.RestStorageService.performRequest(RestStorageService.java:401)
    at org.jets3t.service.impl.rest.httpclient.RestStorageService.performRequest(RestStorageService.java:265)
    at org.jets3t.service.impl.rest.httpclient.RestStorageService.performRestPut(RestStorageService.java:1019)
    at org.jets3t.service.impl.rest.httpclient.RestStorageService.copyObjectImpl(RestStorageService.java:1993)
    at org.jets3t.service.StorageService.copyObject(StorageService.java:898)
    at org.jets3t.service.StorageService.copyObject(StorageService.java:943)
    at org.jets3t.service.StorageService.moveObject(StorageService.java:992)
    at org.jets3t.service.StorageService.renameObject(StorageService.java:1038)
    at org.syncany.plugins.s3.S3TransferManager.move(S3TransferManager.java:256)
    ... 13 more
Error: org.jets3t.service.ServiceException: Service Error Message. -- ResponseCode: 404, ResponseStatus: Not Found, XML Error Message: <Error><Code>NoSuchKey</Code><Message>The specified key does not exist.</Message><Key>temp/temp-MPefe-multichunk-64423d879702a890af83327ccdeaf17b5e635006</Key><RequestId>F15D14BA45AA9C44</RequestId><HostId>4z4nf4Ms6vk8tOIuKPk6hqYkGFnCcY8sGcRHfSI37FG5CoPRaAP8dstiL4P1ZZ4eZRZzh4wm2xo=</HostId></Error>
       Refer to help page using '--help'.

jgn:sync-icebox jgn$

Note that I have removed everything from the bucket, as well as the entire local .syncany and have tried again -- same issue every time.

Total data destined for the bucket: About 39GB, over about 10K files.
@pimotte
Copy link
Member

pimotte commented Apr 24, 2015

Can you search the logs (.syncany/logs/) for the identifier of the multichunk on which it fails? There should be some mention of it being uploaded to the temp location.

@binwiederhier
Copy link
Member

That is a very unfortunate case. I have encountered similar situations in the past with S3. To the best of my knowledge, S3 does not guarantee that an object that is written is available right away. In this case, a temporary file which may be not be available to be moved.

This is an S3 and Swift problem. Both do not guarantee that. The RetriableTransferManager does not retry, because the error code returned is a 404. If it were anything else, we would retry. We cannot retry on a 404 because in case of a rollback, we may delete many non-existing files (deleting in a rollback is also a move). If we retried all of those, that would take forever.

Thinking out loud. What if we used a different non-retriable TM for the rollbacks and enabled retries for moves of non-existing files? With Christian's new code, that'd be really easy.

Related:

@binwiederhier
Copy link
Member

Also, @jgn Thanks for reporting this. Stuff like this is reeeally helpful in the long run!

@pimotte
Copy link
Member

pimotte commented Apr 29, 2015

The problem here is that upload() is not actually atomic. To ensure that we could block there until we can detect that the file exists. That might mean a performance hit, but we can't continue anyway until the file is unambiguously uploaded.

@cr0
Copy link
Member

cr0 commented Apr 29, 2015

Good idea. If we only annotate some plugins (namely s3 and swift) we won't slow down all other plugins. BTW: This would also solve #276.

This was referenced May 8, 2015
Closed
Merged
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
type:bug what:plugin
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Use ReadAfterWriteConsistent feature syncany/syncany-plugin-s3
4 participants
@jgn @binwiederhier @cr0 @pimotte