File ids must stay stable

[PVince81]

Here are a few features that rely on file ids:

• favorites feature
• shares
• sync client to detect file move operations. Failing to do so will cause redownloads. (CC @dragotin)

Currently there are several known situations in which the file id changes:

• File is moved to trashbin (not that bad, but causes share info + favorite info to be lost) => fixed on master / 8.1
• File is moved between storages (from/to external storage): PR here Preserve cache data when doing a cross storage move #15360
• File is moved into a shared folder, as recipient ([Favorite] Favorite file moved into a shared folder as recipient is no longer favorite #13274 Move shared file into shared folder as recipient, moved share info lost #13307 Orphaned share when moving a shared folder into another #13356) PR here: Preserve cache data when doing a cross storage move #15360

We need to make the necessary changes in the code to strive and keep the file ids stable.

@DeepDiver1975 @icewind1991 @schiesbn

[DeepDiver1975]

Thanks for the summary - let's attack this in OC8.1

[DeepDiver1975]

OC8.1

which is about to explode already ......

[PVince81]

I wrote some unit tests (in the spirit of "test first") here: #13312

[dragotin]

Especially case 2) File is moved between storages (from/to external storage) is bad as the client and also the user does not care about the storage type or knows about it. That can be considered a plain bug.

[PVince81]

@schiesbn is there any situation you can think of when the file id might change if encryption is enabled ?

[PVince81]

Partial fix here: #13385
But still need to take care of folders.

[PVince81]

There is no good short term approach for 8.0.

Hopefully we can try finding an approach similar to cross storage file renames: #13360 from @icewind1991

I also thought about using rename hooks for the shares and tags, but then it means gathering the list of all files to be renamed in the "pre-rename" hook, then in "post-rename" find out whether the file ids changed, and if yes, update the share and tags tables. But this solution would not solve the case for the sync client anyway and would be have to be thrown away once we implement a proper file-id preserving solution.

So, leaving this scheduled to 8.1.

@icewind1991 let's discuss the best solution in the coming weeks.

[PVince81]

Yet another situation that can break file ids: whenever a file was moved directly on an external storage. The file scanner might not be able to detect that the file was moved and believe that it is a new file, so it gets an new file id.
Same happens if "files:scan" is run whenever the filecache doesn't match the real storage, moved files will get a new file id. The files that stayed should keep the file id though.

[dragotin]

Reason for not detected moves in the clients.

[guruz]

Why do you guys not use the OS/storage specific identifiers for both FileID and ETags?
For example for the FileID you use the inode+some_volume_fs_specific_unique_identifier (only as initial value)
For ETag you use FileID+mtime+size (on each change)

Just because FileID and ETag are opaque strings does not mean you can not "seed" them with relevant values.

CC @ogoffart

[PVince81]

Maybe because not all external storages would support it (ex: SMB)
Also I'm not sure how it works when using "object store" as main storage.

@icewind1991 tried an alternative approach which stores the file id in extended attributes: #14546

[guruz]

Maybe because not all external storages would support it (ex: SMB)

(which could fall back to the codepath that creates a random/incremental one)

Also I'm not sure how it works when using "object store" as main storage.

I'd guess every object storage has some way of representing this by IDs or revision numbers or whatever

[PVince81]

So far we seem to have shied away from any solution that would be "local storage specific" or bad/slow/impossible for external storages. Maybe it's time to change ?

[icewind1991]

file id's are not determined by the storage backends since they need to be global (and because they are numeric prefixing them isn't trival)

[MorrisJobke]

More broken shares are there: #14084

[PVince81]

Good news! @icewind1991 has brought a PR that keeps file ids between storages (that includes shared folders): #15360

I also checked master and deleting a file to trashbin then restoring will also keep the file id. 😄

[PVince81]

Thanks to #15360 most of the fileid issues are resolved.
I'll close this ticket but we need to follow up on the remaining one, which is related to remote file changes on external storages: #11533

[PVince81]

Please note that when moving files between external storages (or when deleting from external storage), the fileid is also kept properly, which is nice. (I just tried this and it works 😄)

The outstanding issue is only when files are changed remotely (outside of ownCloud).

[moscicki]

"Stable file id" is broken by design for syncing as mentioned in owncloud/client#2725 (comment)