Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Google Cloud Storage S3 not working as primary storage #26794

Closed
despens opened this issue Apr 27, 2021 · 2 comments
Closed

Google Cloud Storage S3 not working as primary storage #26794

despens opened this issue Apr 27, 2021 · 2 comments
Labels
0. Needs triage Pending check for reproducibility or if it fits our roadmap bug

Comments

@despens
Copy link

despens commented Apr 27, 2021

Steps to reproduce

  1. Set up Google Cloud Storage and HMAC authentification for S3 interoperability
  2. Set up the required settings for nextcloud
  3. Start using nextcloud

Expected behaviour

A Google bucket should be used as primary storage

Actual behaviour

Bucket cannot be used, Google API responds with HTTP 400, "InvalidArgument"

Server configuration

Operating system: ubuntu 18.04

Web server: Nginx

Database: Postgres

PHP version: From official nextcloud docker image

Nextcloud version: 21.0.0 (docker)

Updated from an older Nextcloud/ownCloud or fresh install: updated

Where did you install Nextcloud from: Docker

Signing status:

Signing status 
No errors have been found.

Relevant part of config:*

$CONFIG = array (

    // initializing object store as primary storage

    'objectstore' => array (
      'class' => '\\OC\\Files\\ObjectStore\\S3',
      'arguments' =>
      array (
        'bucket' => 'mah-bukkit',
        'autocreate' => false,
        'key' => 'KEY,
        'secret' => 'SECRET',
        'hostname' => 'storage.googleapis.com',
        'port' => 443,
        'region' => 'us-east1',
        'use_ssl' => true,
        'use_path_style' => true,
      ),
    ),
);

Are you using encryption: no

Logs

nextcloud error log 
[objectstore] Error: Aws\S3\Exception\S3MultipartUploadException: An exception occurred while initiating a multipart upload: Error executing "CreateMultipartUpload" on "https://storage.googleapis.com/nextcloud-root/urn%3Aoid%3A10096439?uploads"; AWS HTTP error: Client error: `POST https://storage.googleapis.com/nextcloud-root/urn%3Aoid%3A10096439?uploads` resulted in a `400 Bad Request` response:
<?xml version='1.0' encoding='UTF-8'?><Error><Code>InvalidArgument</Code><Message>Invalid argument.</Message></Error>
 InvalidArgument (client): Invalid argument. - <?xml version='1.0' encoding='UTF-8'?><Error><Code>InvalidArgument</Code><Message>Invalid argument.</Message></Error> at <<closure>>

 0. /var/www/html/3rdparty/aws/aws-sdk-php/src/Multipart/AbstractUploadManager.php line 156
    Aws\Multipart\AbstractUploadManager->transformException("*** sensitive parameter replaced ***")
 1. /var/www/html/3rdparty/guzzlehttp/promises/src/Promise.php line 204
    Aws\Multipart\AbstractUploadManager->Aws\Multipart\{closure}("*** sensitive parameters replaced ***")
 2. /var/www/html/3rdparty/guzzlehttp/promises/src/Promise.php line 153
    GuzzleHttp\Promise\Promise::callHandler(2, "*** sensitive parameter replaced ***", null)
 3. /var/www/html/3rdparty/guzzlehttp/promises/src/TaskQueue.php line 48
    GuzzleHttp\Promise\Promise::GuzzleHttp\Promise\{closure}("*** sensitive parameters replaced ***")
 4. /var/www/html/3rdparty/guzzlehttp/guzzle/src/Handler/CurlMultiHandler.php line 158
    GuzzleHttp\Promise\TaskQueue->run()
 5. /var/www/html/3rdparty/guzzlehttp/guzzle/src/Handler/CurlMultiHandler.php line 183
    GuzzleHttp\Handler\CurlMultiHandler->tick()
 6. /var/www/html/3rdparty/guzzlehttp/promises/src/Promise.php line 248
    GuzzleHttp\Handler\CurlMultiHandler->execute("*** sensitive parameter replaced ***")
 7. /var/www/html/3rdparty/guzzlehttp/promises/src/Promise.php line 224
    GuzzleHttp\Promise\Promise->invokeWaitFn()
 8. /var/www/html/3rdparty/guzzlehttp/promises/src/Promise.php line 269
    GuzzleHttp\Promise\Promise->waitIfPending()
 9. /var/www/html/3rdparty/guzzlehttp/promises/src/Promise.php line 226
    GuzzleHttp\Promise\Promise->invokeWaitList()
10. /var/www/html/3rdparty/guzzlehttp/promises/src/Promise.php line 269
    GuzzleHttp\Promise\Promise->waitIfPending()
11. /var/www/html/3rdparty/guzzlehttp/promises/src/Promise.php line 226
    GuzzleHttp\Promise\Promise->invokeWaitList()
12. /var/www/html/3rdparty/guzzlehttp/promises/src/Promise.php line 62
    GuzzleHttp\Promise\Promise->waitIfPending()
13. /var/www/html/3rdparty/guzzlehttp/promises/src/Coroutine.php line 67
    GuzzleHttp\Promise\Promise->wait()
14. /var/www/html/3rdparty/guzzlehttp/promises/src/Promise.php line 248
    GuzzleHttp\Promise\Coroutine->GuzzleHttp\Promise\{closure}("*** sensitive parameters replaced ***")
15. /var/www/html/3rdparty/guzzlehttp/promises/src/Promise.php line 224
    GuzzleHttp\Promise\Promise->invokeWaitFn()
16. /var/www/html/3rdparty/guzzlehttp/promises/src/Promise.php line 269
    GuzzleHttp\Promise\Promise->waitIfPending()
17. /var/www/html/3rdparty/guzzlehttp/promises/src/Promise.php line 226
    GuzzleHttp\Promise\Promise->invokeWaitList()
18. /var/www/html/3rdparty/guzzlehttp/promises/src/Promise.php line 62
    GuzzleHttp\Promise\Promise->waitIfPending()
19. /var/www/html/3rdparty/aws/aws-sdk-php/src/Multipart/AbstractUploadManager.php line 83
    GuzzleHttp\Promise\Promise->wait()
20. /var/www/html/lib/private/Files/ObjectStore/S3ObjectTrait.php line 97
    Aws\Multipart\AbstractUploadManager->upload()
21. /var/www/html/lib/private/Files/ObjectStore/ObjectStoreStorage.php line 493
    OC\Files\ObjectStore\S3->writeObject("urn:oid:10096439", null)
22. /var/www/html/lib/private/Files/ObjectStore/ObjectStoreStorage.php line 423
    OC\Files\ObjectStore\ObjectStoreStorage->writeStream("appdata_octvz01 ... g", null, 367336)
23. /var/www/html/lib/private/Files/ObjectStore/ObjectStoreStorage.php line 336
    OC\Files\ObjectStore\ObjectStoreStorage->writeBack("/tmp/oc_tmp_hwTn4c-.jpg", "appdata_octvz01 ... g")
24. <<closure>>
    OC\Files\ObjectStore\ObjectStoreStorage->OC\Files\ObjectStore\{closure}("*** sensitive parameters replaced ***")
25. /var/www/html/apps/files_external/3rdparty/icewind/streams/src/CallbackWrapper.php line 121
    call_user_func(Closure {})
26. <<closure>>
    Icewind\Streams\CallbackWrapper->stream_close()
27. /var/www/html/lib/private/Files/ObjectStore/ObjectStoreStorage.php line 444
    fclose(null)
28. /var/www/html/lib/private/Files/Storage/Wrapper/Wrapper.php line 259
    OC\Files\ObjectStore\ObjectStoreStorage->file_put_contents("appdata_octvz01 ... g", null)
29. /var/www/html/lib/private/Files/Storage/Wrapper/Availability.php line 275
    OC\Files\Storage\Wrapper\Wrapper->file_put_contents("appdata_octvz01 ... g", null)
30. /var/www/html/lib/private/Files/Storage/Wrapper/Wrapper.php line 259
    OC\Files\Storage\Wrapper\Availability->file_put_contents("appdata_octvz01 ... g", null)
31. /var/www/html/lib/private/Files/View.php line 1166
    OC\Files\Storage\Wrapper\Wrapper->file_put_contents("appdata_octvz01 ... g", null)
32. /var/www/html/lib/private/Files/View.php line 703
    OC\Files\View->basicOperation("file_put_contents", "/appdata_octvz0 ... g", ["create","write"], null)
33. /var/www/html/lib/private/Files/Node/Folder.php line 189
    OC\Files\View->file_put_contents("/appdata_octvz0 ... g", null)
34. /var/www/html/lib/private/Files/SimpleFS/NewSimpleFile.php line 134
    OC\Files\Node\Folder->newFile("1140-2046-max.jpg", null)
35. /var/www/html/lib/private/Preview/Generator.php line 261
    OC\Files\SimpleFS\NewSimpleFile->putContent(null)
36. /var/www/html/lib/private/Preview/Generator.php line 140
    OC\Preview\Generator->getMaxPreview(OC\Files\SimpleFS\SimpleFolder {}, OC\Files\SimpleFS\NewSimpleFile {}, "image/jpeg", "")
37. /var/www/html/lib/private/PreviewManager.php line 205
    OC\Preview\Generator->generatePreviews(OC\Files\Node\File {}, [{width: 32,heig ... "], "image/jpeg")
38. /var/www/html/custom_apps/previewgenerator/lib/Command/PreGenerate.php line 217
    OC\PreviewManager->generatePreviews(OC\Files\Node\File {}, [{width: 32,heig ... "])
39. /var/www/html/custom_apps/previewgenerator/lib/Command/PreGenerate.php line 195
    OCA\PreviewGenerator\Command\PreGenerate->processFile(OC\Files\Node\File {})
40. /var/www/html/custom_apps/previewgenerator/lib/Command/PreGenerate.php line 163
    OCA\PreviewGenerator\Command\PreGenerate->processRow({id: 1518942,uid ... 2})
41. /var/www/html/custom_apps/previewgenerator/lib/Command/PreGenerate.php line 131
    OCA\PreviewGenerator\Command\PreGenerate->startProcessing()
42. /var/www/html/3rdparty/symfony/console/Command/Command.php line 255
    OCA\PreviewGenerator\Command\PreGenerate->execute(Symfony\Componen ... {}, Symfony\Componen ... {})
43. /var/www/html/3rdparty/symfony/console/Application.php line 1009
    Symfony\Component\Console\Command\Command->run(Symfony\Componen ... {}, Symfony\Componen ... {})
44. /var/www/html/3rdparty/symfony/console/Application.php line 273
    Symfony\Component\Console\Application->doRunCommand(OCA\PreviewGener ... {}, Symfony\Componen ... {}, Symfony\Componen ... {})
45. /var/www/html/3rdparty/symfony/console/Application.php line 149
    Symfony\Component\Console\Application->doRun(Symfony\Componen ... {}, Symfony\Componen ... {})
46. /var/www/html/lib/private/Console/Application.php line 215
    Symfony\Component\Console\Application->run(Symfony\Componen ... {}, Symfony\Componen ... {})
47. /var/www/html/console.php line 100
    OC\Console\Application->run()
48. /var/www/html/occ line 11
    require_once("/var/www/html/console.php")

at 2021-04-27T15:18:21+00:00

Eliminated error sources

  • this works when using a minio instance as a proxy (which uses the same credentials to access the Google bucket)
  • changing regions, path-style or domain style, makes no difference
@despens despens added 0. Needs triage Pending check for reproducibility or if it fits our roadmap bug labels Apr 27, 2021
@solracsf
Copy link
Member

See also https://help.nextcloud.com/t/support-to-google-cloud-storage/13704/14

@solracsf
Copy link
Member

Duplicate of #14074

@solracsf solracsf marked this as a duplicate of #14074 Apr 30, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
0. Needs triage Pending check for reproducibility or if it fits our roadmap bug
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@despens @solracsf