Cache remote files locally from external storage

[PVince81]

For the use case where specific files from external storage are accessed often, one idea is to download and cache that file locally in the ownCloud instance (for example in the "cache" folder). If the file isn't re-read for a given amount of time, it is deleted from the local OC cache folder.

The admin could specify how much cache is allocated per storage and/or user.

Whenever a cached file is modified it is first modified only in the cached version, and then "synced up" after a specific time. (note: conflicts might occur if files are modified outside OC, so not sure whether this is a good idea)

Then the question: how often would a single version of a file be accessed ? Whenever someone uses the sync client, that file is only downloaded once. But if the same file is downloaded by multiple, then there is an extra overhead of having to proxy it every time from the remote storage through OC back to the user.

When accessing files over the UI, the file would be opened/downloaded every time. In this case the local file caching would help as well.

This idea could, in general, improve remote file access speed.

What do you think ?

@icewind1991 @schiesbn @DeepDiver1975 @MTRichards @karlitschek @craigpg

[karlitschek]

Open a can of worms of "interesting" sync and conflict problems :-)

[PVince81]

Yes, this needs to be thought through first, but is interesting indeed.

If external storage is more the focus in the future, we might want to look into that.

[PVince81]

(this enhancement proposal is brought to you by the productive 🍕 we just had from the usual italian place we used to go to)

[LukasReschke]

Whenever a cached file is modified it is first modified only in the cached version, and then "synced up" after a specific time. (note: conflicts might occur if files are modified outside OC, so not sure whether this is a good idea)

Maybe some remote storage have support for things similiar to ETags? - Then we would only need to make a single request instead of getting the whole file.

[PVince81]

We actually already have change detection based on mtime (and etag if it's an "ownCloud" mount). That change detection could be wired in with the proposed cache system to invalidate the entry.

[butonic]

I have branch with the stat via filecache and cache files locally part working. Still need to implement a background job that frees up cache ... I started it to make the oc instance on my v server a huge cache for the oc instance at home to create a tiered storage. The current assumption is that I never upload files directly to the owncloud instance at home to avoid rescanning issues. I postponed work until files external is cleaned up: #11797, #11830, #11261

[PVince81]

Somewhat related: using csync to sync with external storages: #5225

[mrubioroy]

Would this allow to extend local storage space like macOS Sierrra does with iCloud?
https://support.apple.com/en-us/HT206996#storeinicloud
If yes I'd like to encourage you to bring this soon. It'd be great to have an ownCloud solution for ultraportables with limited local storage.
Kudos for your great work anyway!

[PVince81]

Sync conflicts are a problem, yes.

The way I see it is that we need to create conflict files for these and have these visible somewhere.
Considering that users usually don't see conflict files in their ownCloud storage since we don't upload them, I think we should first change the behavior to actually upload conflict files: owncloud/client#4557. Then it would be a normal thing for users to see conflict files there, regardless whether they come from a desktop client or an external storage sync.