Object Storage with S3 causes `429 Too Many Request` on Startup #12537

fdoumet · 2018-11-19T18:43:03Z

When launching a fresh instance of Nextcloud with S3 Object Storage enabled, the following error occurs 90% of the time:

"user": "admin",
"app": "objectstore",
"method": "GET",
"url": "\/index.php\/apps\/files\/",
"message": {
  "Exception": "Aws\\S3\\Exception\\S3Exception",
  "Message": "Error executing \"PutObject\" on \"https:\/\/storage.googleapis.com\/mlvx-primary-storage\/urn%3Aoid%3A225782\"; AWS HTTP error: Client error: `PUT https:\/\/storage.googleapis.com\/mlvx-primary-storage\/urn%3Aoid%3A225782` resulted in a `429 Too Many Requests` response:\n<?xml version='1.0' encoding='UTF-8'?><Error><Code>SlowDown<\/Code><Message>Please reduce your request rate.<\/Message><De (truncated...)\n SlowDown (client): Please reduce your request rate. - <?xml version='1.0' encoding='UTF-8'?><Error><Code>SlowDown<\/Code><Message>Please reduce your request rate.<\/Message><Details>Changing object mlvx-primary-storage\/urn:oid:225782 too quickly, please slow down<\/Details><\/Error>",
  "Code": 0,
  "Trace": [
    {
      "file": "\/var\/www\/html\/3rdparty\/aws\/aws-sdk-php\/src\/WrappedHttpHandler.php",
      "line": 100,
      "function": "parseError",
      "class": "Aws\\WrappedHttpHandler",
      "type": "->",
      "args": [
        "*** sensitive parameter replaced ***",
        {
          "__class__": "GuzzleHttp\\Psr7\\Request"
        },
        {
          "__class__": "Aws\\Command"
        },
        [
          
        ]
      ]
    },
    {
      "file": "\/var\/www\/html\/apps\/files_external_dropbox\/vendor\/guzzlehttp\/promises\/src\/Promise.php",
      "line": 203,
      "function": "Aws\\{closure}",
      "class": "Aws\\WrappedHttpHandler",
      "type": "->",
      "args": [
        "*** sensitive parameters replaced ***"
      ]
    },
    {
      "file": "\/var\/www\/html\/apps\/files_external_dropbox\/vendor\/guzzlehttp\/promises\/src\/Promise.php",
      "line": 174,
      "function": "callHandler",
      "class": "GuzzleHttp\\Promise\\Promise",
      "type": "::",
      "args": [
        2,
        "*** sensitive parameter replaced ***",
        [
          {
            "__class__": "GuzzleHttp\\Promise\\Promise"
          },
          {
            "__class__": "Closure"
          },
          {
            "__class__": "Closure"
          }
        ]
      ]
    },
    {
      "file": "\/var\/www\/html\/apps\/files_external_dropbox\/vendor\/guzzlehttp\/promises\/src\/RejectedPromise.php",
      "line": 40,
      "function": "GuzzleHttp\\Promise\\{closure}",
      "class": "GuzzleHttp\\Promise\\Promise",
      "type": "::",
      "args": [
        "*** sensitive parameters replaced ***"
      ]
    },
    {
      "file": "\/var\/www\/html\/apps\/files_external_dropbox\/vendor\/guzzlehttp\/promises\/src\/TaskQueue.php",
      "line": 47,
      "function": "GuzzleHttp\\Promise\\{closure}",
      "class": "GuzzleHttp\\Promise\\RejectedPromise",
      "type": "::",
      "args": [
        "*** sensitive parameters replaced ***"
      ]
    },
    {
      "file": "\/var\/www\/html\/3rdparty\/guzzlehttp\/guzzle\/src\/Handler\/CurlMultiHandler.php",
      "line": 96,
      "function": "run",
      "class": "GuzzleHttp\\Promise\\TaskQueue",
      "type": "->",
      "args": [
        
      ]
    },
    {
      "file": "\/var\/www\/html\/3rdparty\/guzzlehttp\/guzzle\/src\/Handler\/CurlMultiHandler.php",
      "line": 123,
      "function": "tick",
      "class": "GuzzleHttp\\Handler\\CurlMultiHandler",
      "type": "->",
      "args": [
        
      ]
    },
    {
      "file": "\/var\/www\/html\/apps\/files_external_dropbox\/vendor\/guzzlehttp\/promises\/src\/Promise.php",
      "line": 246,
      "function": "execute",
      "class": "GuzzleHttp\\Handler\\CurlMultiHandler",
      "type": "->",
      "args": [
        true
      ]
    },
    {
      "file": "\/var\/www\/html\/apps\/files_external_dropbox\/vendor\/guzzlehttp\/promises\/src\/Promise.php",
      "line": 223,
      "function": "invokeWaitFn",
      "class": "GuzzleHttp\\Promise\\Promise",
      "type": "->",
      "args": [
        
      ]
    },
    {
      "file": "\/var\/www\/html\/apps\/files_external_dropbox\/vendor\/guzzlehttp\/promises\/src\/Promise.php",
      "line": 267,
      "function": "waitIfPending",
      "class": "GuzzleHttp\\Promise\\Promise",
      "type": "->",
      "args": [
        
      ]
    },
    {
      "file": "\/var\/www\/html\/apps\/files_external_dropbox\/vendor\/guzzlehttp\/promises\/src\/Promise.php",
      "line": 225,
      "function": "invokeWaitList",
      "class": "GuzzleHttp\\Promise\\Promise",
      "type": "->",
      "args": [
        
      ]
    },
    {
      "file": "\/var\/www\/html\/apps\/files_external_dropbox\/vendor\/guzzlehttp\/promises\/src\/Promise.php",
      "line": 267,
      "function": "waitIfPending",
      "class": "GuzzleHttp\\Promise\\Promise",
      "type": "->",
      "args": [
        
      ]
    },
    {
      "file": "\/var\/www\/html\/apps\/files_external_dropbox\/vendor\/guzzlehttp\/promises\/src\/Promise.php",
      "line": 225,
      "function": "invokeWaitList",
      "class": "GuzzleHttp\\Promise\\Promise",
      "type": "->",
      "args": [
        
      ]
    },
    {
      "file": "\/var\/www\/html\/apps\/files_external_dropbox\/vendor\/guzzlehttp\/promises\/src\/Promise.php",
      "line": 62,
      "function": "waitIfPending",
      "class": "GuzzleHttp\\Promise\\Promise",
      "type": "->",
      "args": [
        
      ]
    },
    {
      "file": "\/var\/www\/html\/3rdparty\/aws\/aws-sdk-php\/src\/S3\/S3ClientTrait.php",
      "line": 33,
      "function": "wait",
      "class": "GuzzleHttp\\Promise\\Promise",
      "type": "->",
      "args": [
        
      ]
    },
    {
      "file": "\/var\/www\/html\/lib\/private\/Files\/ObjectStore\/S3ObjectTrait.php",
      "line": 77,
      "function": "upload",
      "class": "Aws\\S3\\S3Client",
      "type": "->",
      "args": [
        "mlvx-primary-storage",
        "urn:oid:225782",
        null,
        "private",
        {
          "mup_threshold": 524288000,
          "part_size": 524288000
        }
      ]
    }
...

Possibly related to #12189

Steps to reproduce

Configure Nextcloud to use S3 Object Storage
Startup up a fresh instance
Error occurs after trying to log in

Expected behaviour

Nextcloud should load 'Files' view after logging in

Actual behaviour

Error message displays instead

The text was updated successfully, but these errors were encountered:

nextcloud-bot · 2018-11-19T18:55:54Z

GitMate.io thinks possibly related issues are #4730 (Storage), #12280 (Problem with S3 as primary storage), #12189 (Swift Object storage returns 429 Too many requests), #11826 (Encryption not working with S3 object storage as primary storage), and #3248 (Using object storage is slow because of « 429 too many request » on objectstorage auth API).

MorrisJobke · 2019-01-23T17:14:01Z

Could you try out the tip from #12189 (comment)?

I will close the ticket for now. If this didn't worked out then we can reopen it :)

fdoumet · 2019-07-29T20:54:23Z

Actually using S3 Object Storage, and still seeing the problem. Please reopen @MorrisJobke!

blieb · 2021-02-22T15:28:41Z

I've had contact my ISP about this issue. I've created a fresh install of next cloud, added a new object storage to it with s3. and tried to upload a folder with some images.

The number of requests that nextcloud does is over 380 requests per second, showing in the isp logs. mostly listing the files.

MorrisJobke · 2021-03-18T07:42:06Z

I've had contact my ISP about this issue. I've created a fresh install of next cloud, added a new object storage to it with s3. and tried to upload a folder with some images.

The number of requests that nextcloud does is over 380 requests per second, showing in the isp logs. mostly listing the files.

Which version of Nextcloud do you use? Because we heavily reduced that number recently (IIRC one of the latest 20.0.x versions but if in doubt 21 should have it).

blieb · 2021-03-18T10:50:14Z

I've had contact my ISP about this issue. I've created a fresh install of next cloud, added a new object storage to it with s3. and tried to upload a folder with some images.
The number of requests that nextcloud does is over 380 requests per second, showing in the isp logs. mostly listing the files.

Which version of Nextcloud do you use? Because we heavily reduced that number recently (IIRC one of the latest 20.0.x versions but if in doubt 21 should have it).

Yes I was testing with nextcloud 21 (clean install). I'm using the DigitalOcean Spaces. when I upload a folder with about 10 files it make that amount requests. I connect it via the s3 external storage.

szaimen · 2021-06-25T14:18:57Z

Is this Issue still valid in NC21.0.2? If not, please close this issue. Thanks! :)

blieb · 2021-06-25T15:33:51Z

Is this Issue still valid in NC21.0.2? If not, please close this issue. Thanks! :)

Yes I tested it with a clean install on 21.0.2 but still had the same issue. (But we moved away from nextcloud now since we had a big shared storage and was a lot more expensive not using s3 storage for that)

szaimen · 2023-01-09T12:09:12Z

Hi, please update to 24.0.8 or better 25.0.2 and report back if it fixes the issue. Thank you!

nextcloud-bot added bug feature: external storage labels Nov 19, 2018

MorrisJobke closed this as completed Jan 23, 2019

MorrisJobke reopened this Aug 5, 2019

MorrisJobke added the 0. Needs triage Pending check for reproducibility or if it fits our roadmap label Aug 5, 2019

szaimen added the needs info label Jun 25, 2021

szaimen added 1. to develop Accepted and waiting to be taken care of feature: object storage and removed 0. Needs triage Pending check for reproducibility or if it fits our roadmap needs info labels Jun 25, 2021

szaimen added needs info 0. Needs triage Pending check for reproducibility or if it fits our roadmap and removed 1. to develop Accepted and waiting to be taken care of labels Jan 9, 2023

szaimen closed this as completed Mar 6, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Object Storage with S3 causes `429 Too Many Request` on Startup #12537

Object Storage with S3 causes `429 Too Many Request` on Startup #12537

fdoumet commented Nov 19, 2018 •

edited

Loading

nextcloud-bot commented Nov 19, 2018

MorrisJobke commented Jan 23, 2019

fdoumet commented Jul 29, 2019

blieb commented Feb 22, 2021

MorrisJobke commented Mar 18, 2021

blieb commented Mar 18, 2021

szaimen commented Jun 25, 2021

blieb commented Jun 25, 2021

szaimen commented Jan 9, 2023

Object Storage with S3 causes 429 Too Many Request on Startup #12537

Object Storage with S3 causes 429 Too Many Request on Startup #12537

Comments

fdoumet commented Nov 19, 2018 • edited Loading

Steps to reproduce

Expected behaviour

Actual behaviour

nextcloud-bot commented Nov 19, 2018

MorrisJobke commented Jan 23, 2019

fdoumet commented Jul 29, 2019

blieb commented Feb 22, 2021

MorrisJobke commented Mar 18, 2021

blieb commented Mar 18, 2021

szaimen commented Jun 25, 2021

blieb commented Jun 25, 2021

szaimen commented Jan 9, 2023

Object Storage with S3 causes `429 Too Many Request` on Startup #12537

Object Storage with S3 causes `429 Too Many Request` on Startup #12537

fdoumet commented Nov 19, 2018 •

edited

Loading