From e8e5530defe5e3da7c46393783708c0a58d28f95 Mon Sep 17 00:00:00 2001 From: Louis Chemineau Date: Fri, 29 Apr 2022 13:26:45 +0200 Subject: [PATCH] Add high level filesystem documention From https://github.com/nextcloud/server/pull/26982 Adds some documentation for the filesystem layer, both a high level overview of how the various pieces interact and a high level guide for apps interacting with the filesystem. Hopefully this will be useful to anyone trying to either use the filesystem or work on the filesystem. Signed-off-by: Louis Chemineau --- .../basics/storage/filesystem.rst | 109 ++++++++++--- developer_manual/core/architecture/files.rst | 145 ++++++++++++++++++ 2 files changed, 235 insertions(+), 19 deletions(-) create mode 100644 developer_manual/core/architecture/files.rst diff --git a/developer_manual/basics/storage/filesystem.rst b/developer_manual/basics/storage/filesystem.rst index 7e43311964d..ab0c295a381 100644 --- a/developer_manual/basics/storage/filesystem.rst +++ b/developer_manual/basics/storage/filesystem.rst @@ -1,37 +1,78 @@ ========== -Filesystem +Nextcloud filesystem API ========== .. sectionauthor:: Bernhard Posselt -Because users can choose their storage backend, the filesystem should be accessed by using the appropriate filesystem classes. +High level guide to using the Nextcloud filesystem API. + +Because users can choose their storage backend, the filesystem should be accessed by using the appropriate filesystem classes. For a simplified filesystem for app specific data see `IAppData `_ + +Node API +^^^^^^^^ + +The "Node API" is the primary api for apps to access the Nextcloud filesystem, each item in the filesystem is +represented as either a File or Folder node with each node providing access to the relevant filesystem information +and actions for the node. + + +Getting access +-------------- + +Access to the filesystem is provided by the ``IRootFolder`` which can be injected into your class. +From the root folder you can either access a user's home folder or access a file or folder by its absolute path. + +.. code-block:: php + + use OCP\Files\IRootFolder; + use OCP\IUserSession; + + class FileSystemAccessExample { + private IUserSession $userSession; + private IRootFolder $rootFolder; + + public function __constructor(IUserSession $userSession, IRootFolder $rootFolder) { + $this->userSession = $userSession; + $this->rootFolder = $rootFolder; + } + + /** + * Create a new file with specified content in the home folder of the current user + * returning the size of the resulting file. + */ + public function getCurrentUserFolder(string $path, string $content): int { + $user = $this->userSession->getUser(); + + if ($user === null) { + return null; + } + + // the "user folder" corresponds to the root of the user visible files + return $this->rootFolder->getUserFolder($user->getUID()); + } + } + +For more details on the specific methods provided by file and folder nodes see the method documentation from the ``OCP\Files\File`` and ``OCP\Files\Folder`` interfaces. -Filesystem classes can be injected automatically with dependency injection. This is the user filesystem. -For a simplified filestystem for app specific data see `IAppData `_ Writing to a file ----------------- - All methods return a Folder object on which files and folders can be accessed, or filesystem operations can be performed relatively to their root. For instance for writing to file:`nextcloud/data/myfile.txt` you should get the root folder and use: .. code-block:: php - storage = $storage; } - public function writeTxt($content) { + public function writeContentToFile($content) { $userFolder = $this->storage->getUserFolder('myUser'); @@ -54,6 +95,7 @@ All methods return a Folder object on which files and folders can be accessed, o } } + Reading from a file ------------------- @@ -61,15 +103,11 @@ Files and folders can also be accessed by id, by calling the **getById** method .. code-block:: php - storage = $storage; @@ -82,7 +120,7 @@ Files and folders can also be accessed by id, by calling the **getById** method // check if file exists and read from it if possible try { $file = $userFolder->getById($id); - if($file instanceof \OCP\Files\File) { + if ($file instanceof \OCP\Files\File) { return $file->getContent(); } else { throw new StorageException('Can not read from folder'); @@ -92,3 +130,36 @@ Files and folders can also be accessed by id, by calling the **getById** method } } } + + +Direct storage access +--------------------- + +While it should be generally avoided in favor of the higher level apis, +sometimes an app needs to talk directly to the storage implementation of it's metadata cache. + +You can get access to the underlying storage of a file or folder by calling ``getStorage`` on the node or first getting +the mountpoint by calling ``getMountPoint`` and getting the storage from there. + +Once you have the storage instance you can use the storage api from ``OCP\Files\Storage\IStorage``, note however that +all paths used in the storage api are internal to the storage, the ``IMountPoint`` returned from ``getMountPoint`` provides +methods for translating between absolute filesystem paths and internal storage paths. + +If you need to query the cached metadata directory you can get the ``OCP\Files\Cache\ICache`` from the storage by calling ``getCache`. + +Implementing a storage +---------------------- + +The recommended way for implementing a storage backend is by sub-classing ``OC\Files\Storage\Common`` which provides +fallback implementations for various methods, reducing the amount of work required to implement the full storage api. +Note however that various of these fallback implementations are likely to be significantly less efficient than an +implementation of the method optimized for the abilities of the storage backend. + +Adding mounts to the filesystem +------------------------------- + +The recommended way of adding your own mounts to the filesystem from an app is implementing ``OCP\Files\Config\IMountProvider`` +and registering the provider using ``OCP\Files\Config\IMountProviderCollection::registerProvider``. + +Once registered, your provider will be called every time the filesystem is being setup for a user and your mount provider +can return a list of mounts to add for that user. \ No newline at end of file diff --git a/developer_manual/core/architecture/files.rst b/developer_manual/core/architecture/files.rst new file mode 100644 index 00000000000..969db6d8b5f --- /dev/null +++ b/developer_manual/core/architecture/files.rst @@ -0,0 +1,145 @@ +============ +Nextcloud filesystem API +============ + +High level overview +------------------- + +The Nextcloud filesystem is roughly based on the unix filesystem, consisting of multiple storages +mounted at various locations. + +.. code-block:: txt + + ┌──────────────────────────────────┐ + │Code wanting to use the filesystem│ + └─────────┬─────────────────────┬──┘ + │ │ + │ │ + ┌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┐ + ╎Filesystem │ │ ╎ + ╎layer │new │legacy ╎ + ╎ │ │ ╎ + ╎ ▼ ▼ ╎ + ╎ ┌────────┐ Partly build on ┌─┴──────┐ ╎ + ╎ │Node API├─────────────────►│View API│ ╎ + ╎ └───────┬┘ └─┬──────┘ ╎ + ╎ │ │ ╎ + └╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┘ + │ │ + ┌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┐ + ╎Storage layer │ │ ╎ + ╎ ├─────────────────────┤ ╎ + ╎ │ │ ╎ + ╎ ▼ ▼ ╎ + ╎ ┌───────┐ ┌───────┐ ┌──────┐ ╎ + ╎ │Storage│═══>│Scanner│═══>│Cache │ ╎ + ╎ └───────┘ └───────┘ └──────┘ ╎ + ╎ ╎ + ╎ ╎ + └╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┘ + +Filesystem layer +^^^^^^^^^^^^^^^^ + +Any code that wants to use the filesystem has two API options to use, the new ``Node`` api and the old ``View`` api. +New code should preferably use the ``Node`` api as it allows building systems with less overhead than the old api. + +Besides the filesystem apis, this layer also manages the available mounts, containing the logic to allow apps +to setup their mounts and translating filesystem paths into a mountpoint + "internal" path. + +### Storage layer + +The storage implementation handles the details of communicating with the filesystem or remote storage api +and provide a uniform api for Nextcloud to use the storage. + +For each storage a metadata cache/index is maintained to allow reading metadata of the storage without having +to talk to the (potentially) slow storage backend. The scanner is responsible for updating the cache with +information from the storage backend. + +## Storage/Cache wrappers + +To allow apps to customize the behavior of a storage without requiring the app to implement this for every +possible storage backend, a ``Wrapper`` system is used. + +A ``Wrapper`` encapsulates an inner storage and allows overwriting any method to customize its behavior, with +all other methods being passed through to the inner storage. + +Generally search storage wrapper has an equivalent cache wrapper encapsulating the cache of the inner storage +to provide the same behavior modifications when reading metadata from the cache. + +Wrappers can be layered to stack the behavior of the wrappers, for example the ``groupfolders`` app works by +stacking a wrapper to provide access to a single folder on the root storage with a wrapper to limit the permissions +of the storage. + +.. code-block:: txt + + ┌───────────────┐ ┌────────────────────┐ + │PermissionsMask├─────►│CachePermissionsMask│ PermissionsMask applies a mask to the permissions of a storage + └───────┬───────┘ └─────────┬──────────┘ to provide less-privilaged access to a storage + │ │ + ▼ ▼ + ┌───────────────┐ ┌────────────────────┐ + │Jail ├─────►│CacheJail │ Jail restricts access to a file or folder of a storage providing + └───────┬───────┘ └─────────┬──────────┘ a limited view into the storage (think unix chroot or bind mount) + │ │ + ▼ ▼ + ┌───────────────┐ ┌────────────────────┐ + │Base Storage ├─────►│Base Cache │ + └───────────────┘ └────────────────────┘ +Code Map +-------- + +Approximate overview of the significant filesystem code + +AppData +~~~~~~~ + +High level api for accessing "appdata" folders, based on the ``Node`` API + +Cache +~~~~~ + +- ``Cache`` implementation +- Cache wrappers +- Scanner and cache update logic +- Search infrastructure + +Mount +~~~~~ + +Mountpoint management and setup + +Node +~~~~ + +``Node`` filesystem api implementation + +ObjectStorage +~~~~~~~~~~~~~ + +Implementation of the various supported object store storage backends + +SimpleFS +~~~~~~~~ + +Simplified version of the Node api, for providing a more limited api for some filesystem bits + +Storage +~~~~~~~ + +Implementation of various storage backends and wrappers + +Streams +~~~~~~~ + +Various low-level php stream wrapper used in storage implementations + +Type +~~~~ + +Mimetype management and detection + +View.php +~~~~~~~~ + +Legacy View api \ No newline at end of file