fixed moving of master key encrypted and versioned files between shared folders

[yahesh]

This fixes #16419 for master key encrypted files.

[yahesh]

This fixes the problem for master key encrypted files and improves the situation a bit for user key encrypted files.

[yahesh]

Copied over from #16419 as suggested by @kesselb:

It seems that moving user key encrypted folders leads into some weird locking issues in the $this->file->getAccessList() call of lib/private/Files/Stream/Encryption.php:stream_open() (which we can reproduce reliably but haven't fully understood yet). At least, this locking issue doesn't break files. Copying user key encrypted folders works.

This is a write-down of what the general problem was:
Let's shortly re-iterate on what happens when a file is moved from one shared folder to another shared folder:

• a copy of the file is put into the target folder
• a copy of the file is put into the trashbin of the user moving the file
• the original copy of the file is put into the trashbin of the file owner

The general logic can be found in lib/private/Files/Storage/Wrapper/Encryption.php: moveFromStorage() which first copies the file/folder with the help of lib/private/Files/Storage/Wrapper/Encryption.php:copyBetweenStorage() and then deletes the file/folder.

One task that has to be done is to update the oc_filecache table. Most of this is done by underlying classes but the encrypted value in the database is handled here in lib/private/Files/Storage/Wrapper/Encryption.php:updateEncryptedVersion() and this is also where the problem is buried.

When copyBetweenStorage is first called in moveFromStorage it provides true as the $isRename value. This later prompts updateEncryptedVersion to overwrite the encrypted value of the source file with 1. So back to the process:

• When the source file is copied to the target file, the encrypted value of the source file is changed into a wrong value.
• When the source file is copied to the trashbin of the user moving the file, the encrypted value of the source file doesn't match and the file copy fails because the signature checks fails while reading the content.

What we did for now to remedy this problem:

• At the start of copyBetweenStorage we backup the original encrypted value.
• We let the rest of the program logic untouched because some underlying code seems to rely on this broken behaviour.
• At the end of copyBetweenStorage we restore the original encrypted value so that it is the correct value for the next copy task.

[jknockaert]

@yahesh Thanks for the PR. Ideally there would be one or more tests added to the codebase in order to avoid future regressions (and they also help to some extent in documenting the PR).
@nextcloud/encryption Reviews needed. Any other groups to invite?

[yahesh]

@jknockaert As you may have read in the corresponding issue, this problem occurs some 32 levels down into the call stack. I really have no idea how to properly mock this misbehaviour so that the requested test isn't much more than a dummy placeholder.

[jknockaert]

@yahesh Ja this is pretty low-level stuff.
@schiessle Do you have any advise as to how to design a proper test for this fix?

[papimla]

This looks like a lot of good work (diagnostic + solution) to solve a severe problem. It would be great to see some progress. I'm stuck with files that are most likely in a bad state because of this.

[hostingnuggets]

@papimla same here! It seems like @yahesh's work was a waste of time for him...

[kesselb]

@yahesh mind to have a look at the conflicting files? I don't know how soon one has time to review it but would be nice to have at least a recent ci run.

[papimla]

@papimla same here! It seems like @yahesh's work was a waste of time for him...

It certainly is frustrating when good work is not put to use and is abandoned for some time. But the excellent diagnostic is still valid and will be a great help to get this fixed. I would not go as far as to say it was wasted 😄

[papimla]

@yahesh thanks a lot for updating the pull request !

[hostingnuggets]

@papimla exactly, now we only need to hope that the Nextclouders will take their precious time to check this PR.

[juliusknorr]

Thanks a lot @yahesh for looking into this and providing such a detailed issue description together with the fix. The changes make sense and the fix works like a charm.

[MorrisJobke]

Code looks good 👍

[MorrisJobke]

Rebased to trigger CI again. Then this can be merged.

[MorrisJobke]

Unfortunately the tests fail :/

[MorrisJobke]

Moved to #22902 as it is unrelated to this PR.

[MorrisJobke]

I extracted the unrelated stuff out in #22898, #22902 and #22903. After rebasing this PR and testing it again it doesn't solve the issue for me. But also on master there were already some fixes. @juliushaertl do you mind to check it out again as well? Maybe I have overseen something?

[kesselb]

@yahesh mind to do another rebase to have recent CI run? Heads up: Your branch was changed in the meantime.

[MorrisJobke]

Let's close this one for now. If it still solves an issue we can reopen it.