DELETE, MOVE (maybe others except GET) requests return 503 if file is still in post-processing #9432

amrita-shrestha · 2024-06-21T08:39:32Z

Describe the bug

https://drone.owncloud.com/owncloud/ocis/36110/47/5

  Scenario Outline: move a file to existing file name                                   # /drone/src/tests/acceptance/features/coreApiWebdavMove2/moveFile.feature:129
    Given using <dav-path-version> DAV path                                             # FeatureContext::usingOldOrNewDavPath()
    And user "Alice" has uploaded file with content "test file" to "testfile.txt"       # FeatureContext::userHasUploadedAFileWithContentTo()
    And user "Alice" has uploaded file with content "some content" to "lorem.txt"       # FeatureContext::userHasUploadedAFileWithContentTo()
    When user "Alice" moves file "testfile.txt" to "lorem.txt" using the WebDAV API     # FeatureContext::userMovesFileOrFolderUsingTheWebDavAPI()
    Then the HTTP status code should be "204"                                           # FeatureContext::thenTheHTTPStatusCodeShouldBe()
    And as "Alice" file "lorem.txt" should exist                                        # FeatureContext::asFileOrFolderShouldExist()
    And the content of file "lorem.txt" for user "Alice" should be "test file"          # FeatureContext::contentOfFileForUserShouldBe()
    But as "Alice" file "testfile.txt" should not exist                                 # FeatureContext::asFileOrFolderShouldNotExist()
    And as "Alice" the file with original path "lorem.txt" should exist in the trashbin # TrashbinContext::elementIsInTrashCheckingOriginalPath()

    Examples:
      | dav-path-version |
      | old              |
        Failed step: Then the HTTP status code should be "204"
        HTTP status code 503 is not the expected value 204
        Failed asserting that 503 matches expected '204'.

API log

## Scenario Outline: move a file to existing file name (coreApiWebdavMove2/moveFile.feature:141)
	### Given using OCS API version "1"
	### And user "Alice" has been created with default attributes and without skeleton files
		_______________________________________________________________________

		==> REQUEST
		POST /graph/v1.0/users
		X-Request-ID: coreApiWebdavMove2/moveFile.feature:141-8
		==> REQ BODY
		{"onPremisesSamAccountName":"Alice","passwordProfile":{"password":"123456"},"displayName":"Alice Hansen","mail":"alice@example.org","accountEnabled":true}

		<== RESPONSE
		201 Created
		X-Xss-Protection: 1; mode=block
		<== RES BODY
		{"accountEnabled":true,"displayName":"Alice Hansen","id":"3618d746-8c34-40dd-a1eb-3287b1da74c5","mail":"alice@example.org","onPremisesSamAccountName":"Alice","surname":"Alice","userType":"Member"}

		_______________________________________________________________________

		==> REQUEST
		GET /graph/v1.0/users/Alice
		X-Request-ID: coreApiWebdavMove2/moveFile.feature:141-8

		<== RESPONSE
		200 OK
		X-Xss-Protection: 1; mode=block
		<== RES BODY
		{"accountEnabled":true,"displayName":"Alice Hansen","id":"3618d746-8c34-40dd-a1eb-3287b1da74c5","onPremisesSamAccountName":"Alice","surname":"Alice","userType":"Member"}

		_______________________________________________________________________

		==> REQUEST
		GET /graph/v1.0/users/Alice
		X-Request-ID: coreApiWebdavMove2/moveFile.feature:141-8

		<== RESPONSE
		200 OK
		X-Xss-Protection: 1; mode=block
		<== RES BODY
		{"accountEnabled":true,"displayName":"Alice Hansen","id":"3618d746-8c34-40dd-a1eb-3287b1da74c5","onPremisesSamAccountName":"Alice","surname":"Alice","userType":"Member"}

	### Given using old DAV path
	### And user "Alice" has uploaded file with content "test file" to "testfile.txt"
		_______________________________________________________________________

		==> REQUEST
		PUT /remote.php/webdav/testfile.txt
		X-Request-ID: coreApiWebdavMove2/moveFile.feature:141-131
		==> REQ BODY
		test file

		<== RESPONSE
		201 Created
		X-Xss-Protection: 1; mode=block

	### And user "Alice" has uploaded file with content "some content" to "lorem.txt"
		_______________________________________________________________________

		==> REQUEST
		PUT /remote.php/webdav/lorem.txt
		X-Request-ID: coreApiWebdavMove2/moveFile.feature:141-132
		==> REQ BODY
		some content

		<== RESPONSE
		201 Created
		X-Xss-Protection: 1; mode=block

	### When user "Alice" moves file "testfile.txt" to "lorem.txt" using the WebDAV API
		_______________________________________________________________________

		==> REQUEST
		MOVE /remote.php/webdav/testfile.txt
		X-Request-ID: coreApiWebdavMove2/moveFile.feature:141-133

		<== RESPONSE
		503 Service Unavailable
		X-Xss-Protection: 1; mode=block

	### Then the HTTP status code should be "204"


## Scenario Outline: move a file to existing file name (coreApiWebdavMove2/moveFile.feature:142)
	### Given using OCS API version "1"
	### And user "Alice" has been created with default attributes and without skeleton files
		_______________________________________________________________________

		==> REQUEST
		POST /graph/v1.0/users
		X-Request-ID: coreApiWebdavMove2/moveFile.feature:142-8
		==> REQ BODY
		{"onPremisesSamAccountName":"Alice","passwordProfile":{"password":"123456"},"displayName":"Alice Hansen","mail":"alice@example.org","accountEnabled":true}

		<== RESPONSE
		201 Created
		X-Xss-Protection: 1; mode=block
		<== RES BODY
		{"accountEnabled":true,"displayName":"Alice Hansen","id":"eb170148-78f7-4f9c-a1c6-2264aa1486ad","mail":"alice@example.org","onPremisesSamAccountName":"Alice","surname":"Alice","userType":"Member"}

		_______________________________________________________________________

		==> REQUEST
		GET /graph/v1.0/users/Alice
		X-Request-ID: coreApiWebdavMove2/moveFile.feature:142-8

		<== RESPONSE
		200 OK
		X-Xss-Protection: 1; mode=block
		<== RES BODY
		{"accountEnabled":true,"displayName":"Alice Hansen","id":"eb170148-78f7-4f9c-a1c6-2264aa1486ad","onPremisesSamAccountName":"Alice","surname":"Alice","userType":"Member"}

		_______________________________________________________________________

		==> REQUEST
		GET /graph/v1.0/users/Alice
		X-Request-ID: coreApiWebdavMove2/moveFile.feature:142-8

		<== RESPONSE
		200 OK
		X-Xss-Protection: 1; mode=block
		<== RES BODY
		{"accountEnabled":true,"displayName":"Alice Hansen","id":"eb170148-78f7-4f9c-a1c6-2264aa1486ad","onPremisesSamAccountName":"Alice","surname":"Alice","userType":"Member"}

	### Given using new DAV path
	### And user "Alice" has uploaded file with content "test file" to "testfile.txt"
		_______________________________________________________________________

		==> REQUEST
		PUT /remote.php/dav/files/Alice/testfile.txt
		X-Request-ID: coreApiWebdavMove2/moveFile.feature:142-131
		==> REQ BODY
		test file

		<== RESPONSE
		201 Created
		X-Xss-Protection: 1; mode=block

	### And user "Alice" has uploaded file with content "some content" to "lorem.txt"
		_______________________________________________________________________

		==> REQUEST
		PUT /remote.php/dav/files/Alice/lorem.txt
		X-Request-ID: coreApiWebdavMove2/moveFile.feature:142-132
		==> REQ BODY
		some content

		<== RESPONSE
		201 Created
		X-Xss-Protection: 1; mode=block

	### When user "Alice" moves file "testfile.txt" to "lorem.txt" using the WebDAV API
		_______________________________________________________________________

		==> REQUEST
		MOVE /remote.php/dav/files/Alice/testfile.txt
		X-Request-ID: coreApiWebdavMove2/moveFile.feature:142-133

		<== RESPONSE
		503 Service Unavailable
		X-Xss-Protection: 1; mode=block

	### Then the HTTP status code should be "204"

While trying to immediately DELETE the just uploaded file, the server responds 503 Service Unavailable with response body

<d:error xmlns:d="DAV" xmlns:s="http://sabredav.org/ns">
    <s:exception>Sabre\DAV\Exception\ServiceUnavailable</s:exception>
    <s:message>internal error: file is processing</s:message>
</d:error>

file is processing: Is it postprocessing? If the resource is still in postprocessing, we get 425 Too Early for GET requests. But for DELETE, it's 503.

Same for:

MOVE requests
maybe others as well

The text was updated successfully, but these errors were encountered:

ScharfViktor · 2024-06-24T10:24:44Z

I couldn't reproduce localy. Hard to understand what is problem

- coreApiWebdavMove2/moveFile.feature:129

Failed step: Then the HTTP status code should be "204"
        HTTP status code 503 is not the expected value 204

==> REQUEST
        MOVE /remote.php/webdav/testfile.txt
        X-Request-ID: coreApiWebdavMove2/moveFile.feature:141-133

        <== RESPONSE
        503 Service Unavailable
        X-Xss-Protection: 1; mode=block

coreApiShareManagementToShares/moveReceivedShare.feature:172

Then the HTTP status code of responses on each endpoint should be "201, 204" respectively # FeatureContext::theHTTPStatusCodeOfResponsesOnEachEndpointShouldBe()
      Expected HTTP status codes: "201, 204". Found HTTP status codes: "404,204"

ScharfViktor · 2024-06-24T11:22:17Z

features/coreApiWebdavMove2/moveFile.feature:129

logs localy (maybe related CI issue):

2024-06-24T13:20:24+02:00 ERR failed to remove item from index error="entity not found" Id={"opaque_id":"03d7d943-a09a-4632-b3b9-8a97eae72fb4","space_id":"3c8a1672-6e6a-4021-b4ab-3af50b115842","storage_id":"529b3e8f-9ca7-47ce-885b-35a3c0978752"} service=search
2024/06/24 13:20:25 http: TLS handshake error from [::1]:49505: remote error: tls: unknown certificate
2024/06/24 13:20:25 http: proxy error: context canceled
2024-06-24T13:20:25+02:00 ERR error walking the tree error="error: not found: stat: error: not found: " service=search
2024-06-24T13:20:25+02:00 ERR error while indexing a space error="error: not found: stat: error: not found: " service=search spaceID={"opaque_id":"529b3e8f-9ca7-47ce-885b-35a3c0978752$3c8a1672-6e6a-4021-b4ab-3af50b115842"} userID={"idp":"https://localhost:9200","opaque_id":"3c8a1672-6e6a-4021-b4ab-3af50b115842","type":1}

but I haven't any server crash or 503 error like in the CI

ScharfViktor · 2024-06-24T11:24:50Z

coreApiShareManagementToShares/moveReceivedShare.feature:172

here don't know what is trigger of the failed test, maybe it helps #9453

saw-jan · 2024-06-24T11:28:41Z

coreApiShareManagementToShares/moveReceivedShare.feature:172

here don't know what is trigger of the failed test, maybe it helps #9453

for 172 (which is very much flaky), we have separate ticket: #9428
I have reported my findings there

saw-jan · 2024-06-24T15:37:28Z

Failed builds:

Scenario Outline: file is deleted and restored to a new destination - coreApiTrashbinRestore/trashbinRestore.feature:72

https://drone.owncloud.com/owncloud/ocis/37169/49/6

Failed step: And user "Alice" has deleted file "/PARENT/parent.txt"
      HTTP status code was not 204 while trying to delete resource '/PARENT/parent.txt' for user 'Alice'
      Failed asserting that an array contains 503.

Scenario Outline: files with strange names can be restored - coreApiTrashbinRestore/trashbinRestore.feature:179:

https://drone.owncloud.com/owncloud/ocis/36237/46/6

https://drone.owncloud.com/owncloud/ocis/36274/46/5

Failed step: And user "Alice" has deleted file "😛 😜 🐱 🐭 ⌚️ ♀️ 🚴‍♂️"
HTTP status code was not 204 while trying to delete resource '😛 😜 🐱 🐭 ⌚️ ♀️ 🚴‍♂️' for user 'Alice'
Failed asserting that an array contains 503.

Failed step: And user "Alice" has deleted file "sample,1.txt"
HTTP status code was not 204 while trying to delete resource 'sample,1.txt' for user 'Alice'
Failed asserting that an array contains 503.

API log

    ### And user "Alice" has deleted file "😛 😜 🐱 🐭 ⌚️ ♀️ 🚴‍♂️"
        _______________________________________________________________________

        ==> REQUEST
        DELETE /remote.php/webdav/%F0%9F%98%9B%20%F0%9F%98%9C%20%F0%9F%90%B1%20%F0%9F%90%AD%20%E2%8C%9A%EF%B8%8F%20%E2%99%80%EF%B8%8F%20%F0%9F%9A%B4%E2%80%8D%E2%99%82%EF%B8%8F
        X-Request-ID: coreApiTrashbinRestore/trashbinRestore.feature:190-182

        <== RESPONSE
        503 Service Unavailable
        X-Xss-Protection: 1; mode=block
        <== RES BODY
        <?xml version="1.0" encoding="UTF-8"?>
        <d:error xmlns:d="DAV" xmlns:s="http://sabredav.org/ns"><s:exception>Sabre\DAV\Exception\ServiceUnavailable</s:exception><s:message>internal error: file is processing</s:message></d:error>

    ### And user "Alice" has deleted file "sample,1.txt"
    _______________________________________________________________________

    ==> REQUEST
    DELETE /remote.php/dav/files/Alice/sample,1.txt
    X-Request-ID: coreApiTrashbinRestore/trashbinRestore.feature:195-182

    <== RESPONSE
    503 Service Unavailable
    X-Xss-Protection: 1; mode=block
    <== RES BODY
    <?xml version="1.0" encoding="UTF-8"?>
    <d:error xmlns:d="DAV" xmlns:s="http://sabredav.org/ns"><s:exception>Sabre\DAV\Exception\ServiceUnavailable</s:exception><s:message>internal error: file is processing</s:message></d:error>

Scenario Outline: file is deleted and restored to a new destination - coreApiTrashbinRestore/trashbinRestore.feature:91

https://drone.owncloud.com/owncloud/ocis/36455/46/6

| new              | /PARENT/CHILD/child.txt | child.txt            |
    Failed step: And user "Alice" has deleted file "/PARENT/CHILD/child.txt"
    HTTP status code was not 204 while trying to delete resource '/PARENT/CHILD/child.txt' for user 'Alice'
    Failed asserting that an array contains 503.

	### And user "Alice" has deleted file "/PARENT/CHILD/child.txt"
	_______________________________________________________________________

	==> REQUEST
	DELETE /remote.php/dav/files/Alice/PARENT/CHILD/child.txt
	X-Request-ID: coreApiTrashbinRestore/trashbinRestore.feature:91-77

	<== RESPONSE
	503 Service Unavailable
	X-Xss-Protection: 1; mode=block
	<== RES BODY
	<?xml version="1.0" encoding="UTF-8"?>
	<d:error xmlns:d="DAV" xmlns:s="http://sabredav.org/ns"><s:exception>Sabre\DAV\Exception\ServiceUnavailable</s:exception><s:message>internal error: file is processing</s:message></d:error>

Scenario Outline: delete file from folder with dots in the path - coreApiWebdavDelete/deleteFile.feature:39
- https://drone.owncloud.com/owncloud/ocis/37177/49/5

saw-jan · 2024-07-04T05:37:30Z

While trying to immediately DELETE the just uploaded file, the server responds 503 Service Unavailable with response body

<d:error xmlns:d="DAV" xmlns:s="http://sabredav.org/ns">
    <s:exception>Sabre\DAV\Exception\ServiceUnavailable</s:exception>
    <s:message>internal error: file is processing</s:message>
</d:error>

file is processing: Is it postprocessing? If the resource is still in postprocessing, we get 425 Too Early for GET requests. But for DELETE, it's 503.

Same for:

MOVE requests

CC @micbar @2403905 @kobergj

phil-davis · 2024-07-04T05:49:37Z

Just a thought:
For upload, the server already "caches" the uploaded bytes somewhere, responds to the API request with "success" and then does some post-processing in the background.

For download, if post-processing is still happening, the server has to respond with 425 "too early" because it can't "find" the bytes to send back in an API response.

But:
For delete, the server could also "queue" the delete, and respond to the user with "success". When the existing upload post-processing is completed in the back-end, the post-processing code could then notice that there is a delete pending, and could delete the file. Or maybe there is a reasonable code-path that could interrupt the upload post-processing when a "delete" request arrives, and cleanup the file, wherever it exists - implementing synchronous delete.

For move and copy, the server could also "queue" the action. When the existing upload post-processing is completed in the back-end, the post-processing code could then notice that there is another action queued, and do the back-end move or copy.

If the server always accepts valid delete, move and copy requests, then clients do not have to bother handling the possibility of "425 too early" for those cases.

PrajwolAmatya · 2024-07-08T05:53:44Z

Failed at todays build (Jul 8) https://drone.owncloud.com/owncloud/ocis/36759/40/5

kobergj · 2024-07-16T07:49:32Z

I would go for expecting 425 always. In a perfect world delete should be possible even if the file is processing. But that brings some problems so I would just go with a consistent 425 everywhere.

Btw 503 is extremely wrong. The server should never return 503

saw-jan · 2024-07-23T03:43:31Z

Recent failures:

https://drone.owncloud.com/owncloud/ocis/37363/46/5

2403905 · 2024-07-23T20:17:08Z

I would go for expecting 425 always. In a perfect world delete should be possible even if the file is processing. But that brings some problems so I would just go with a consistent 425 everywhere.

Btw 503 is extremely wrong. The server should never return 503

Good point. Unfortunately we don't have the appropriate response code in a cs3/rpc/code

Can we add a new one?

micbar · 2024-07-24T05:41:20Z

yes, we should add one.

2403905 · 2024-07-24T08:53:05Z

@butonic objected to the use of the 425. He noticed that we completely misuse 425 Too Early: https://httpwg.org/specs/rfc8470.html#introduction

What options do we have? Could we use the 423 instead? Let's say the resource is locked by post processing.

kobergj · 2024-07-24T08:57:35Z

@butonic objected to the use of the 425. He noticed that we completely misuse 425 Too Early:

Please let's not rediscuss this again. We had a lengthy discussion about which status code to use back in the day. We decided to go for 425 to indentify this special case. If we want to change this again we need a proper discussion and planning before we do anything.

2403905 · 2024-07-24T09:03:07Z

cs3org/cs3apis#230

amrita-shrestha added the QA:team label Jun 21, 2024

micbar mentioned this issue Jun 21, 2024

feat: explicit provider for WOPI apps to handle fileinfo #9253

Merged

9 tasks

ScharfViktor self-assigned this Jun 24, 2024

ScharfViktor added the Priority:p2-high Escalation, on top of current planning, release blocker label Jun 24, 2024

saw-jan added QA:p1 and removed Priority:p2-high Escalation, on top of current planning, release blocker labels Jun 24, 2024

saw-jan self-assigned this Jun 24, 2024

saw-jan changed the title ~~Tests fail Intermittently with 503 Status Code~~ DELETE, MOVE (maybe others except GET) requests return 503 if file is still in post-processing Jul 18, 2024

saw-jan added Type:Bug and removed QA:p1 QA:team labels Jul 18, 2024

saw-jan unassigned saw-jan and ScharfViktor Jul 18, 2024

2403905 self-assigned this Jul 24, 2024

2403905 mentioned this issue Jul 25, 2024

fixed the response code when DELETE and MOVE requests to the file that is still in post-processing #9689

Merged

9 tasks

2403905 closed this as completed in #9689 Jul 25, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

DELETE, MOVE (maybe others except GET) requests return 503 if file is still in post-processing #9432

DELETE, MOVE (maybe others except GET) requests return 503 if file is still in post-processing #9432

amrita-shrestha commented Jun 21, 2024 •

edited by saw-jan

Loading

ScharfViktor commented Jun 24, 2024 •

edited

Loading

ScharfViktor commented Jun 24, 2024

ScharfViktor commented Jun 24, 2024

saw-jan commented Jun 24, 2024

saw-jan commented Jun 24, 2024 •

edited

Loading

saw-jan commented Jul 4, 2024 •

edited

Loading

phil-davis commented Jul 4, 2024

PrajwolAmatya commented Jul 8, 2024

kobergj commented Jul 16, 2024

saw-jan commented Jul 23, 2024

2403905 commented Jul 23, 2024 •

edited

Loading

micbar commented Jul 24, 2024

2403905 commented Jul 24, 2024

kobergj commented Jul 24, 2024

2403905 commented Jul 24, 2024

DELETE, MOVE (maybe others except GET) requests return 503 if file is still in post-processing #9432

DELETE, MOVE (maybe others except GET) requests return 503 if file is still in post-processing #9432

Comments

amrita-shrestha commented Jun 21, 2024 • edited by saw-jan Loading

Describe the bug

ScharfViktor commented Jun 24, 2024 • edited Loading

ScharfViktor commented Jun 24, 2024

ScharfViktor commented Jun 24, 2024

saw-jan commented Jun 24, 2024

saw-jan commented Jun 24, 2024 • edited Loading

saw-jan commented Jul 4, 2024 • edited Loading

phil-davis commented Jul 4, 2024

PrajwolAmatya commented Jul 8, 2024

kobergj commented Jul 16, 2024

saw-jan commented Jul 23, 2024

2403905 commented Jul 23, 2024 • edited Loading

micbar commented Jul 24, 2024

2403905 commented Jul 24, 2024

kobergj commented Jul 24, 2024

2403905 commented Jul 24, 2024

amrita-shrestha commented Jun 21, 2024 •

edited by saw-jan

Loading

ScharfViktor commented Jun 24, 2024 •

edited

Loading

saw-jan commented Jun 24, 2024 •

edited

Loading

saw-jan commented Jul 4, 2024 •

edited

Loading

2403905 commented Jul 23, 2024 •

edited

Loading