Collaboration utf7 ops

[jvillafanez]

Description

Implement the "PutRelativeFile" and "DeleteFile" and "RenameFile" WOPI operations. Some handling of utf7 is included because it's needed for the "PutRelativeFile" and "RenameFile" operation.

In addition, due to problems with the "PutRelativeFile.IncludeHostUrls" test in the WOPI validator, we'll be using a customized test case file. For now, only that test is commented out. The rest of the tests will be the same

Related Issue

no issue opened

Motivation and Context

These are one of the WOPI operations that optional

How Has This Been Tested?

Some parts manually tested using collabora as WOPI app, the rest using the WOPI validator

Screenshots (if appropriate):

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to change)
• Technical debt
• Tests only (no source changes)

Checklist:

• Code changes
• Unit tests added
• Acceptance tests added
• Documentation ticket raised:

[update-docs]

Thanks for opening this pull request! The maintainers of this repository would appreciate it if you would create a changelog item based on your changes.

[jvillafanez]

Failing sequence of operation is (according to the WOPI validator):

Fail: PutRelativeAndRenameFile.RenamingADeletedFileShouldReturnA404
    PutRelativeFile, response code: 200 OK
    CheckFileInfo, response code: 200 OK
    DeleteFile, response code: 500 InternalServerError
      Incorrect StatusCode. Expected: 200, Actual: 500
    DeleteFile, response code: 200 OK

related logs:

{"level":"debug","service":"collaboration","request-id":"","proto":"HTTP/1.1","method":"POST","path":"/wopi/files/1dbf2209e961da03908d11776af3f7b54c1e395f52885d925b295b1dc371a6f4","WopiOverride":"PUT_RELATIVE","WopiProof":"","WopiProofOld":"","WopiStamp":"","FileReference":"resource_id:{storage_id:\"0018fb10-6fdd-4a69-b144-218030b4097a\"  opaque_id:\"9f10d5b0-7fe5-4463-ba0b-06538eed247d\"  space_id:\"7044cba2-be2b-444e-9218-4ff6612ac7c4\"}  path:\".\"","ViewMode":"VIEW_MODE_READ_WRITE","Requester":"idp:\"https://ocis-server:9200\"  opaque_id:\"7044cba2-be2b-444e-9218-4ff6612ac7c4\"  type:USER_TYPE_PRIMARY","PutTarget":"ValidatorTestFile.wopitest","NewFileReference":"resource_id:{storage_id:\"0018fb10-6fdd-4a69-b144-218030b4097a\"  opaque_id:\"7044cba2-be2b-444e-9218-4ff6612ac7c4\"  space_id:\"7044cba2-be2b-444e-9218-4ff6612ac7c4\"}  path:\"./ValidatorTestFile.wopitest\"","time":"2024-07-25T13:19:15Z","line":"/drone/src/services/collaboration/pkg/connector/fileconnector.go:785","message":"PutRelativeFileRelative: success"}
{"level":"debug","service":"collaboration","request-id":"","proto":"HTTP/1.1","method":"GET","path":"/wopi/files/29759c18eba690c09e4351395d5ef79639ffcf9e5e6a4dd1a358f9206e727e0e","WopiOverride":"","WopiProof":"","WopiProofOld":"","WopiStamp":"","FileReference":"resource_id:{storage_id:\"0018fb10-6fdd-4a69-b144-218030b4097a\"  opaque_id:\"807c6cf0-6924-4c62-beb5-836bea0f706f\"  space_id:\"7044cba2-be2b-444e-9218-4ff6612ac7c4\"}","ViewMode":"VIEW_MODE_READ_WRITE","Requester":"idp:\"https://ocis-server:9200\"  opaque_id:\"7044cba2-be2b-444e-9218-4ff6612ac7c4\"  type:USER_TYPE_PRIMARY","FileInfo":{"BaseFileName":"ValidatorTestFile.wopitest","OwnerId":"37303434636261322d626532622d343434652d393231382d3466663636313261633763344068747470733a2f2f6f6369732d7365727665723a39323030","Size":74,"UserId":"37303434636261322d626532622d343434652d393231382d3466663636313261633763344068747470733a2f2f6f6369732d7365727665723a39323030","Version":"1721913555.63566608","SupportsCobalt":false,"SupportsContainers":false,"SupportsDeleteFile":true,"SupportsEcosystem":false,"SupportsExtendedLockLength":true,"SupportsFolders":false,"SupportsGetLock":true,"SupportsLocks":true,"SupportsRename":true,"SupportsUpdate":true,"SupportsUserInfo":false,"UserFriendlyName":"Admin","ReadOnly":false,"RestrictedWebViewOnly":false,"UserCanAttend":false,"UserCanNotWriteRelative":false,"UserCanPresent":false,"UserCanRename":true,"UserCanWrite":true,"HostEditUrl":"https://fakeoffice.owncloud.test/not/relevant?WOPISrc=http%3A%2F%2Fwopiserver%3A9300%2Fwopi%2Ffiles%2F1dbf2209e961da03908d11776af3f7b54c1e395f52885d925b295b1dc371a6f4","HostViewUrl":"https://fakeoffice.owncloud.test/not/relevant?WOPISrc=http%3A%2F%2Fwopiserver%3A9300%2Fwopi%2Ffiles%2F1dbf2209e961da03908d11776af3f7b54c1e395f52885d925b295b1dc371a6f4","AllowAdditionalMicrosoftServices":false,"AllowExternalMarketplace":false,"DisablePrint":false,"DisableTranslation":false,"BreadcrumbDocName":"ValidatorTestFile.wopitest"},"time":"2024-07-25T13:19:15Z","line":"/drone/src/services/collaboration/pkg/connector/fileconnector.go:1097","message":"CheckFileInfo: success"}
{"level":"error","service":"collaboration","request-id":"","proto":"HTTP/1.1","method":"POST","path":"/wopi/files/29759c18eba690c09e4351395d5ef79639ffcf9e5e6a4dd1a358f9206e727e0e","WopiOverride":"DELETE","WopiProof":"","WopiProofOld":"","WopiStamp":"","FileReference":"resource_id:{storage_id:\"0018fb10-6fdd-4a69-b144-218030b4097a\"  opaque_id:\"807c6cf0-6924-4c62-beb5-836bea0f706f\"  space_id:\"7044cba2-be2b-444e-9218-4ff6612ac7c4\"}","ViewMode":"VIEW_MODE_READ_WRITE","Requester":"idp:\"https://ocis-server:9200\"  opaque_id:\"7044cba2-be2b-444e-9218-4ff6612ac7c4\"  type:USER_TYPE_PRIMARY","RequestedLockID":"","StatusCode":"CODE_TOO_EARLY","StatusMsg":"file is processing","time":"2024-07-25T13:19:15Z","line":"/drone/src/services/collaboration/pkg/connector/fileconnector.go:825","message":"DeleteFile: delete failed with unexpected status"}
{"level":"error","service":"collaboration","request-id":"","proto":"HTTP/1.1","method":"POST","path":"/wopi/files/29759c18eba690c09e4351395d5ef79639ffcf9e5e6a4dd1a358f9206e727e0e","WopiOverride":"DELETE","WopiProof":"","WopiProofOld":"","WopiStamp":"","FileReference":"resource_id:{storage_id:\"0018fb10-6fdd-4a69-b144-218030b4097a\"  opaque_id:\"807c6cf0-6924-4c62-beb5-836bea0f706f\"  space_id:\"7044cba2-be2b-444e-9218-4ff6612ac7c4\"}","ViewMode":"VIEW_MODE_READ_WRITE","Requester":"idp:\"https://ocis-server:9200\"  opaque_id:\"7044cba2-be2b-444e-9218-4ff6612ac7c4\"  type:USER_TYPE_PRIMARY","RequestedLockID":"","StatusCode":"CODE_NOT_FOUND","StatusMsg":"get lock: error: not found: no lock found","time":"2024-07-25T13:19:15Z","line":"/drone/src/services/collaboration/pkg/connector/fileconnector.go:848","message":"DeleteFile: GetLock failed with unexpected status"}
{"level":"debug","service":"collaboration","request-id":"","proto":"HTTP/1.1","method":"POST","path":"/wopi/files/29759c18eba690c09e4351395d5ef79639ffcf9e5e6a4dd1a358f9206e727e0e","WopiOverride":"DELETE","WopiProof":"","WopiProofOld":"","WopiStamp":"","FileReference":"resource_id:{storage_id:\"0018fb10-6fdd-4a69-b144-218030b4097a\"  opaque_id:\"807c6cf0-6924-4c62-beb5-836bea0f706f\"  space_id:\"7044cba2-be2b-444e-9218-4ff6612ac7c4\"}","ViewMode":"VIEW_MODE_READ_WRITE","Requester":"idp:\"https://ocis-server:9200\"  opaque_id:\"7044cba2-be2b-444e-9218-4ff6612ac7c4\"  type:USER_TYPE_PRIMARY","RequestedLockID":"","time":"2024-07-25T13:19:15Z","line":"/drone/src/services/collaboration/pkg/connector/fileconnector.go:863","message":"DeleteFile: success"}

Note that the first request gets a "CODE_TOO_EARLY" status, which isn't properly handled (we assume the error comes from a locked file, so that's why we try to get the lock.)

@micbar

[micbar]

Note that the first request gets a "CODE_TOO_EARLY" status, which isn't properly handled (we assume the error comes from a locked file, so that's why we try to get the lock.)

The CODE_TOO_EARLY is the response code when the file is in processing. That means the backend does some virusscanning and the file is not unlocked. As the file has "just been created" that seems to be logical.

In this case, we need a retry mechanism with a backoff time period.

[jvillafanez]

It seems the error wasn't triggered this time, so not sure if the last commit fixes the issue.

[jvillafanez]

fix to retry the delete operation works.

{"level":"info","service":"collaboration","proto":"HTTP/1.1","request-id":"9d079347bd34/TsNqhbeXBO-000252","traceid":"00000000000000000000000000000000","remote-addr":"192.168.10.5:54824","method":"POST","wopi-action":"PUT_RELATIVE","status":200,"path":"/wopi/files/564f1e992a94898751cf419d39c88d8672395b3344b82463eb1d75e98ed72f8e","duration":27.84256,"bytes":2637,"time":"2024-07-31T10:56:44Z","line":"/drone/src/services/collaboration/pkg/middleware/accesslog.go:35","message":"access-log"}
{"level":"warn","service":"collaboration","request-id":"","proto":"HTTP/1.1","method":"POST","path":"/wopi/files/168a7b83e86e792622df783559acbdcb524988377aa54c92eba0621462f303e4","WopiOverride":"DELETE","WopiProof":"","WopiProofOld":"","WopiStamp":"","FileReference":"resource_id:{storage_id:\"6597dfc9-e311-431f-a9a4-bb2738d5176a\" opaque_id:\"65c81773-c0bd-472a-8ad5-23b5657ba1ce\" space_id:\"831dfd9c-9fca-4427-8ee6-ba7e77c3c051\"}","ViewMode":"VIEW_MODE_READ_WRITE","Requester":"idp:\"https://ocis-server:9200\" opaque_id:\"831dfd9c-9fca-4427-8ee6-ba7e77c3c051\" type:USER_TYPE_PRIMARY","RequestedLockID":"","StatusCode":"CODE_TOO_EARLY","StatusMsg":"file is processing","WaitingTime":20,"Retries":0,"time":"2024-07-31T10:56:44Z","line":"/drone/src/services/collaboration/pkg/connector/fileconnector.go:839","message":"DeleteFile: file isn't ready yet. Retrying"}
{"level":"debug","service":"collaboration","request-id":"","proto":"HTTP/1.1","method":"POST","path":"/wopi/files/168a7b83e86e792622df783559acbdcb524988377aa54c92eba0621462f303e4","WopiOverride":"DELETE","WopiProof":"","WopiProofOld":"","WopiStamp":"","FileReference":"resource_id:{storage_id:\"6597dfc9-e311-431f-a9a4-bb2738d5176a\" opaque_id:\"65c81773-c0bd-472a-8ad5-23b5657ba1ce\" space_id:\"831dfd9c-9fca-4427-8ee6-ba7e77c3c051\"}","ViewMode":"VIEW_MODE_READ_WRITE","Requester":"idp:\"https://ocis-server:9200\" opaque_id:\"831dfd9c-9fca-4427-8ee6-ba7e77c3c051\" type:USER_TYPE_PRIMARY","RequestedLockID":"","time":"2024-07-31T10:56:44Z","line":"/drone/src/services/collaboration/pkg/connector/fileconnector.go:887","message":"DeleteFile: success"}

Maybe we should check the retry periods. Right now, we wait 20ms, doubling the time for each retry until a maximum of 5 seconds. Not sure what is the expected waiting time for the first retry.

[micbar]

See comments.

[sonarcloud]

Quality Gate passed

Issues
7 New issues
0 Accepted issues

Measures
0 Security Hotspots
73.2% Coverage on New Code
1.0% Duplication on New Code

See analysis details on SonarCloud