Feature: Disable filecache for certain external storages

[RobinMcCorkell]

There is a common use-case where an external storage is used both by ownCloud and directly, and as many bug reports have stated (see #11797) this makes ownCloud not see changes correctly. There's not much we can do in the situation where this storage is needed for synchronisation, but there is also another common use-case where such an external storage is only wanted as a view into a shared file storage, not as a synchronisation area. Such storages might even want synching disabled: #7909 and #7234.

I had a possible solution for this - we could entirely disable scanning and filecache updating via an advanced storage option, which would both prevent synchronisation (no update propagation at all) and would make any changes immediately visible in the web UI in ownCloud. It could also reduce performance overheads for storages that often change or are very large and rarely accessed - ownCloud would never have to perform a full scan, but the storage would still be available in the web UI.

Example usecase: there is a shared storage containing many videos or other media files, which is many GBs if not TBs total size and could contain thousands of files. The administrator wants every user to have access to this storage to pull out or push in data, but for obvious reasons doesn't want any users synchronising the full thing, nor wants ownCloud to scan the storage. The administrator could set this special mount option, disabling the filecache for the storage, so each access through the web UI will directly access the storage every time without putting that data into the filecache.

Thoughts? @icewind1991 @PVince81 @DeepDiver1975

[mmattel]

Would such a functionality impact the use of the gallery app?

[RobinMcCorkell]

@mmattel It would stop ownCloud caching any information about such a storage, so no indexing, sharing etc would be able to occur, which would probably prevent the Gallery app from working.

[mmattel]

@oparoz FYI

[mmattel]

@Xenopathic If that get´s implemented, the storage should get some "tagging" like a different color to identify that this feature has been set.

[RobinMcCorkell]

@mmattel Sure, that would be a good UX feature

[mmattel]

and we have to propagate "no-scan" for federated sharing.
btw, better than a color would be a different icon, so we stay open to have colors for current inaccessability...
(I know, very early thinking for this, but you never know how fast it will come 😄 )

[oparoz]

Example usecase: there is a shared storage containing many videos or other media files

For which Gallery and video_player are the perfect viewers

which would probably prevent the Gallery app from working.

There seems to be a contradiction then... but that statement may not be true.
Gallery relies on local thumbnails, there is no reason those shouldn't be accessible and if some are missing, they will be generated on image access. If that doesn't work, then media files won't get a preview which would make no sense for such storage.

Is there a way to simulate this?

Sharing can be disabled on demand in Gallery, that's fine.

The administrator wants every user to have access to this storage to pull out or push in data, but for obvious reasons doesn't want any users synchronising the full thing, nor wants ownCloud to scan the storage

So, every time users change folders, even if they were there a minute ago, they have to wait for the scan to complete. Works for local storage, but for federated shares? It would make it painful to browse through those files.

Wouldn't it be possible to cache the data for the session only? Gallery does it in JS, but that could be using the filecache in order to make it accessible to all apps.

[RobinMcCorkell]

@oparoz As long as the Gallery or Video Player don't require an index of all available media files, and can instead work directly on the files within the storage, then it would work fine. The display feature of these apps in particular should be unaffected. But a global 'show me all my pictures' view would probably skip such storages.

Perhaps caching directory contents per-session is a good idea, but for the initial implementation we should just directly access the storage on each request, then later we can evaluate whether such performance optimizations are necessary. (It's very likely it will be, thinking about it).

[oparoz]

As long as the Gallery or Video Player don't require an index of all available media files, and can instead work directly on the files within the storage, then it would work fine.

@Xenopathic - Cool, Gallery uses the node API to get a list of files belonging to the node linked to a given path, so I'm hoping that the API can access files directly when not in the presence of a file cache.

But a global 'show me all my pictures' view would probably skip such storages.

That was removed with the introduction of Gallery. No more searchByMime().
But what about tagged files and comments? That wouldn't be available either for that type of storage, correct?

[RobinMcCorkell]

@oparoz RE tagged files and comments - I'm not sure on the implementation of these features, but they probably use the filecache and so would not work. This feature would only be used for storages the administrator wants to use as a raw file dump, but still wants access via the ownCloud web UI rather than [insert proprietary file access solution here].

[PVince81]

Yes, everything that requires a fileid would not work without file cache. That includes "shares", "personal tags", "system tags", "favorite", "comments", ...

One alternative would be to implement your own cache implementation using the new Storage API, but make that one read/store the metadata/fileid directly on-disk. Or maybe it could store the fileid as extended attribute if the filesystem supports it.

Have a look at https://github.com/owncloud/example-storage and https://github.com/icewind1991/files_nullcache

[oparoz]

Yes, everything that requires a fileid would not work without file cache. That includes "shares", "personal tags", "system tags", "favorite", "comments", ...

So how does that work? You get a list of paths and can get the content that way? Without a fileId, there are no previews available and so Gallery wouldn't work either then...

One alternative would be to implement your own cache implementation using the new Storage API, but make that one read/store the metadata/fileid directly on-disk.

Could work with a SSD, but I don't think each app should write their own cache, seems like a waste. Definitely looks like a good solution for a temp cache as discussed above.

[DeepDiver1975]

file cache is a central architectural concept of owncloud - there is no way to live without it.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.