externally renaming a file/folder in a files_external storage breaks the contained shares

[butonic]

Steps to reproduce

1. mount a webdav endpoint with files external to wd.
2. in oc create three folders f1, f2, f3
3. note down the file ids of these folders
4. outside of oc delete f2 and rename f3 -> f2
5. refresh the wd folder in oc
6. compare the fileids of f2

Expected behaviour

f2 should have the file id of f3 in step 3

Actual behavior

f2 still has the same fileid as f2 in step 3. The whole directory tree of f3 (now f2) is rescanned and will receive new file ids. That in itself is not yet a problem, but the root cause for any shares that have been created between step 3 & 4 inside f3 to break, because they now point to a no longer existing fileid.

The issue is a result of the rescan process which currently does not have any means to detect a rename. @icewind1991 The only hack I see for default WebDAV endpoints is to use a heuristic that compares the folder contents in the cache with those in the remote location and tries to guess the right rename ... no Idea how csync detects renames. @danimo my guess is it takes into account file ids and maybe inodes?

The problem of course applies to all external storages that are not used exclusively. The question is how do we solve it. For ownCloud DAV mounts we cold expose the file id in the webdav result ... however that is probably a bad idea security wise: http://stackoverflow.com/questions/12378220/api-design-and-security-why-hide-internal-ids cc @LukasReschke

But I don't see any other way to make renames in inter cloud shares more robust.

Thougts?

[vasyugan]

Are there any known workarounds for the time being? Is there any way to force a full rescan of the external storage? I have mounted a Dropbox folder, which is frequently modified externally. It seems they are increasingly getting out of sync.

[butonic]

Remember, this issue is about loosing shares when a rescan occurs. That being said you can rescan the files on the command line with occ:

> php occ files:scan <username>

or in the browser console with

scanFiles(true,'/')

[schiessle]

Remember, this issue is about loosing shares when a rescan occurs.

Didn't test it. But if I understand your description correct it is not only about losing shares but also about accidentally sharing stuff which shouldn't be shared. If f2 was shared to someone before you delete it and rename f3 to f2, f3 will be shared because it will have the file id from the old f2.

This could be a huge privacy risk because it can expose internal data to user you don't want to or even to the public in case of link shares.

Not sure how we can solve this problem reliable. False-positives can be even more dangerous than losing a share, because this would mean that you share something which shouldn't be shared.

Btw, as far as I know the sync client detects renames based on the file-id which will not change in this case.

[PVince81]

Linking with stable fileid ticket: #13310

[PVince81]

If a rescan finds "new" files, files that aren't in the index even when renamed remotely, these will get a new file id. So no privacy issue on this side.

It is still a big issue to be able to detect remote renames reliably, and OC cannot rely solely on file names.

At some point @icewind1991 tried saving file ids in extended attributes #14546 but this is unlikely to work with most external storages.

Maybe what's needed is a for the external storage to explicitly notify ownCloud of changes happening there. But most storages don't have such features.

Needs further research.

[icewind1991]

It's not possible to create a generic solution for this problem, there is simply no way to detect a rename without having additional meta data (#14546)

[RobinMcCorkell]

I wonder if some external storage backends expose an 'inode' or similar unique identifier for files/folders? We could have an (optional) method on the Common class, to retrieve that information, but it has the option of returning null if no unique identifier could be found. That way, the storages that do support unique file identification will get proper rename support, while those that don't still break in this way. Better than all storages breaking.

For example, Google Drive already stores items by ID. SFTP likely has some sort of inode support, given its close integration with UNIX. On the other hand, some like Dropbox do not expose a reliable identifier at all.

[rullzer]

@PVince81 @cmonteroluque I don't see this happening for 9.0

[rullzer]

I have been thinking a bit more about this. And I think this is impossible to do right with our current setup. We have no way of knowing two files are identical on external storage with the current set of information. We would need to start checksumming files. And on rescan compare, size + checksums + ?? to make sure two files are actually identical.

Without this the best we can do is a somewhat educated guess.

I think it might even be a good idea to trigger a removal of all shares on external storage if we deteced stuff has been moved. Since else you might share stuff you did not intend. CC: @LukasReschke

[PVince81]

For storages that support it, we could set extended attributes on the remote file, like "OC-fileid".
Checksumming could be expensive as it means that for every newly found file we need to download it just to do the checksum. But it could work, yeah...

[rullzer]

@PVince81 yes that is a solution for storages that have extended atrributes... but if not there is pratically now real way (besides doing checksums) to make sure. And in that case it is probabaly better to be save.

[PVince81]

So far apart from proper update detection from the storage, no good solution was found.
Moving to continue research for 9.2

[PVince81]

Moving to backlog.

So far the only good solution is having remote update detection in a way that detects renames and can preserve the fileid.

[ownclouders]

Hey, this issue has been closed because the label status/STALE is set and there were no updates for 7 days. Feel free to reopen this issue if you deem it appropriate.

(This is an automated comment from GitMate.io.)

[ownclouders]

Hey, this issue has been closed because the label status/STALE is set and there were no updates for 7 days. Feel free to reopen this issue if you deem it appropriate.

(This is an automated comment from GitMate.io.)