From b2ba70beac013c2555870919de7c6d34fe9f005b Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Klemens=20B=C3=B6swirth?= Date: Sun, 25 Sep 2022 21:22:48 +0200 Subject: [PATCH 1/7] doc: update mountpoints.md --- doc/dev/mountpoints.md | 101 +++++++++++++++++++++-------------------- 1 file changed, 52 insertions(+), 49 deletions(-) diff --git a/doc/dev/mountpoints.md b/doc/dev/mountpoints.md index bd15ffb0748..4f02ad8726e 100644 --- a/doc/dev/mountpoints.md +++ b/doc/dev/mountpoints.md @@ -13,15 +13,15 @@ Every mountpoint consists of exactly three parts: The set of plugins is defined below `system:/elektra/mountpoints//plugins`. The keys below `system:/elektra/mountpoints//plugins/` define a single instance of a plugin. The part `` will be used as name by which this plugin instance will be referenced later. -The name may be arbitrary, but it must not be `backend`. The name `backend` is reserved (see below). +The name may be arbitrary, but it must not be `backend`. The name `backend` is reserved for the backend plugin (see below). -To define a plugin instance, `system:/elektra/mountpoints//plugins//name` must be set to the name of the plugin. +To define a plugin instance, `system:/elektra/mountpoints//plugins//name` must be set to the name of the plugin, i.e., it must be possible to open the plugin with `elektraPluginOpen` via this value. Optionally, the configuration keyset can be defined by the keys below `system:/elektra/mountpoints//plugins//config`. The single _backend plugin_ is the only one that the `libelektra-kdb` library communicates with. It is responsible for calling other plugins when needed. A _backend contract_ exists between this plugin and `libelektra-kdb`. -The backend plugin alone is responsible for upholding this contract. +The backend plugin alone is responsible for upholding this contract and enforcing it upon other plugins it calls. The backend plugin is defined (like the other plugins) below `system:/elektra/mountpoints//plugins`. Specifically, it uses `backend` as `` and is therefore defined by the keys below `system:/elektra/mountpoints//plugins/backend`. @@ -32,7 +32,13 @@ The _mountpoint config_ is defined by the keys below `system:/elektra/mountpoint It is merged into the configuration keyset of every plugin of this mountpoint (including the backend plugin). Specifically, the mountpoint config is put into the `system:/` namespace, while the config from `system:/elektra/mountpoints//plugins//config` is put into the `user:/` namespace of the keyset passed to a plugin. - + + +Additionally, there is the `config/needs` part of a plugins contract. +This part of the contract can be used by a plugin to provide configuration for another plugin it depends on. +In particular, these keys are put into the `default:/` namespace, meaning this config can be overriden by both the _mountpoint config_ in `system:/elektra/mountpoints//config` and the plugin's config in `system:/elektra/mountpoints//plugins//config`. + + > **Note**: This _mountpoint definition_ is separate from the normal configuration keyset passed to a plugin. > The backend plugin may still have such a keyset below its `system:/elektra/mountpoints//plugins//config` key. @@ -90,13 +96,13 @@ Explanation: - This configures a mountpoint `/hosts` that loads the plugins `resolver_fm_hpu_b`, `glob`, `hosts`, `sync`, `error`, `network` and `backend`. The plugin `glob` also has some configuration keys defined. -- It then selects the `backend` plugin as the backend plugin. +- The `backend` plugin is set as the backend plugin. - The rest is the mountpoint definition specific to the `backend` plugin. -As you can see from this example, the classic "positions" are part of the plugin-specific mountpoint definition. +As you can see from this example, the "positions" are part of the plugin-specific mountpoint definition. This allows different backend plugins to use different "positions". -For example, imagine a `version` backend plugin that always just provides a few keys with version information for Elektra. +For example, there is the `version` backend plugin that always just provides a few keys with version information for Elektra. It doesn't need to call any other plugins and requires no configuration at all. A mountpoint with this plugin could look like this: @@ -112,7 +118,7 @@ system:/elektra/mountpoints/\/version/plugins/backend/name (="version") See [KDB Operations Documentation](./kdb-operations.md) for a description of operations and phases. In each of the phases of a `get` or `set` operation, the corresponding function of the backend plugin is called. -For a description of how this work exactly read the [Backend Plugins Documentation](./backend-plugins.md). +For a description of how this works exactly read the [Backend Plugins Documentation](./backend-plugins.md). In the first example above, the phases were mapped one-to-one to what the plugin `backend` called "positions". The different terms are very much intentional, since this not a requirement. @@ -124,55 +130,53 @@ Here we will look at a few more. ``` # List of plugins with their config -system:/elektra/mountpoints/\/hosts/plugins/#0/name (="resolver_fm_hpu_b") -system:/elektra/mountpoints/\/hosts/plugins/#1/name (="glob") -system:/elektra/mountpoints/\/hosts/plugins/#1/config/set/#0 -system:/elektra/mountpoints/\/hosts/plugins/#1/config/set/#1 -system:/elektra/mountpoints/\/hosts/plugins/#1/config/set/#2 -system:/elektra/mountpoints/\/hosts/plugins/#1/config/set/#3 -system:/elektra/mountpoints/\/hosts/plugins/#1/config/set/#4/flags -system:/elektra/mountpoints/\/hosts/plugins/#2/name (="hosts") -system:/elektra/mountpoints/\/hosts/plugins/#3/name (="sync") -system:/elektra/mountpoints/\/hosts/plugins/#4/name (="network") -system:/elektra/mountpoints/\/hosts/plugins/#5/name (="other_backend") +system:/elektra/mountpoints/\/hosts/plugins/resolver/name (="resolver_fm_hpu_b") +system:/elektra/mountpoints/\/hosts/plugins/glob/name (="glob") +system:/elektra/mountpoints/\/hosts/plugins/glob/config/set/#0 +system:/elektra/mountpoints/\/hosts/plugins/glob/config/set/#1 +system:/elektra/mountpoints/\/hosts/plugins/glob/config/set/#2 +system:/elektra/mountpoints/\/hosts/plugins/glob/config/set/#3 +system:/elektra/mountpoints/\/hosts/plugins/glob/config/set/#4/flags +system:/elektra/mountpoints/\/hosts/plugins/hosts/name (="hosts") +system:/elektra/mountpoints/\/hosts/plugins/sync/name (="sync") +system:/elektra/mountpoints/\/hosts/plugins/network/name (="network") # Define backend plugin -system:/elektra/mountpoints/\/hosts/backend (="#5") +system:/elektra/mountpoints/\/hosts/backend/name (="other_backend") # Configuration for backend plugin -system:/elektra/mountpoints/\/hosts/path (="myhosts") - -system:/elektra/mountpoints/\/hosts/positions/get/resolver = (="#0") -system:/elektra/mountpoints/\/hosts/positions/get/storage = (="#2") -system:/elektra/mountpoints/\/hosts/positions/get/validation/#0 (="#1") - -system:/elektra/mountpoints/\/hosts/positions/set/resolver = (="#0") -system:/elektra/mountpoints/\/hosts/positions/set/validation/#0 = (="#1") -system:/elektra/mountpoints/\/hosts/positions/set/validation/#1 = (="#4") -system:/elektra/mountpoints/\/hosts/positions/set/storage = (="#2") -system:/elektra/mountpoints/\/hosts/positions/set/precommit/#0 = (="#3") -system:/elektra/mountpoints/\/hosts/positions/set/commit = (="#0") -system:/elektra/mountpoints/\/hosts/positions/set/rollback (="#0") +system:/elektra/mountpoints/\/hosts/definition/path (="myhosts") + +system:/elektra/mountpoints/\/hosts/definition/positions/get/resolver = (="resolver") +system:/elektra/mountpoints/\/hosts/definition/positions/get/storage = (="hosts") +system:/elektra/mountpoints/\/hosts/definition/positions/get/validation/#0 (="glob") + +system:/elektra/mountpoints/\/hosts/definition/positions/set/resolver = (="resolver") +system:/elektra/mountpoints/\/hosts/definition/positions/set/validation/#0 = (="glob") +system:/elektra/mountpoints/\/hosts/definition/positions/set/validation/#1 = (="network") +system:/elektra/mountpoints/\/hosts/definition/positions/set/storage = (="hosts") +system:/elektra/mountpoints/\/hosts/definition/positions/set/precommit/#0 = (="sync") +system:/elektra/mountpoints/\/hosts/definition/positions/set/commit = (="resolver") +system:/elektra/mountpoints/\/hosts/definition/positions/set/rollback (="resolver") ``` This example is very similar to the first one, but the plugin `other_backend` doesn't use the `postorage` and `prestorage` "positions". Instead, there is a `validation` position that is (presumably) called in the appropriate phase. The plugin `other_backend` may also impose its own restrictions on plugins configured for the `validation` position. -For example, it may define that such plugins must not generate, remove or modify keys and provide a different position for such plugins. +For example, it may define that such plugins must not generate, remove or modify keys and provide a different position for plugins that do so. ``` -system:/elektra/mountpoints/\/hosts/plugins/#0/name (="db_backend") -system:/elektra/mountpoints/\/hosts/plugins/#1/name (="network") +system:/elektra/mountpoints/\/hosts/plugins/network/name (="network") -system:/elektra/mountpoints/\/hosts/backend (="#0") +system:/elektra/mountpoints/\/hosts/plugins/backend/name (="db_backend") -system:/elektra/mountpoints/\/hosts/db/driver (="postgres") -system:/elektra/mountpoints/\/hosts/db/server (="127.0.0.1") -system:/elektra/mountpoints/\/hosts/db/port (="5432") -system:/elektra/mountpoints/\/hosts/db/user (="admin") -system:/elektra/mountpoints/\/hosts/db/password (="supersecret") +system:/elektra/mountpoints/\/hosts/definition/db/driver (="postgres") +system:/elektra/mountpoints/\/hosts/definition/db/server (="127.0.0.1") +system:/elektra/mountpoints/\/hosts/definition/db/port (="5432") +system:/elektra/mountpoints/\/hosts/definition/db/user (="admin") +system:/elektra/mountpoints/\/hosts/definition/db/password (="supersecret") -system:/elektra/mountpoints/\/hosts/phases/set/prestorage/#0 (="#1") +system:/elektra/mountpoints/\/hosts/definition/phases/set/prestorage/#0 (="network") ``` This example shows an entirely different type of backend. @@ -180,15 +184,14 @@ The hypothetical `db_backend` is backed by a database. In this case it is configured for a PostgreSQL database running on `127.0.0.1:5432` to which we connect as user `admin`. We also configured the `network` plugin to run in the `prestorage` phase of the `set` operation. -Which phases can be used and how the must be configured of course depends on `db_backend`. +Which phases can be used and how they must be configured of course depends on `db_backend`. -One might also imagine there could be a (probably quite complicated) part of the mountpoint definition that defines how the relational tables of the database are mapped into the KDB. +There might also be a (potentially quite complicated) part of the mountpoint definition that defines how the relational tables of the database are mapped into the KDB. ``` -system:/elektra/mountpoints/\/hosts/plugins/#0/name (="http_backend") -system:/elektra/mountpoints/\/hosts/plugins/#1/name (="yajl") +system:/elektra/mountpoints/\/hosts/plugins/yajl/name (="yajl") -system:/elektra/mountpoints/\/hosts/backend (="#0") +system:/elektra/mountpoints/\/hosts/plugins/backend/name (="http_backend") system:/elektra/mountpoints/\/hosts/url (="https://api.ipify.org/?format=JSON") @@ -197,5 +200,5 @@ system:/elektra/mountpoints/\/hosts/decoder (="yajl") The hypothetical `http_backend` plugin is a read-only backend plugin. In the example above, it is configured load the URL `https://api.ipify.org/?format=JSON` and use the `yajl` plugin to parse the result into a keyset. -Both the HTTP request and the decoding would happen in the `storage` phase of the `get` operation. +Both the HTTP request and the decoding would likely happen in the `storage` phase of the `get` operation. The `resolver` phase could perform an HTTP cache check, for example. From c5617f0e42f2e873383a9880eb2a6d961253007d Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Klemens=20B=C3=B6swirth?= Date: Sun, 25 Sep 2022 22:08:59 +0200 Subject: [PATCH 2/7] doc: update kdb-operations --- doc/dev/kdb-operations.md | 94 ++++++++++++++++++--------------------- src/libs/elektra/kdb.c | 12 ++--- 2 files changed, 48 insertions(+), 58 deletions(-) diff --git a/doc/dev/kdb-operations.md b/doc/dev/kdb-operations.md index bfc1c8d2213..1774ce72d97 100644 --- a/doc/dev/kdb-operations.md +++ b/doc/dev/kdb-operations.md @@ -1,9 +1,11 @@ # Proposal: KDB Operations There are four main _operations_ in `libelektra-kdb`: `open`, `get`, `set` and `close`. -For each of these there is a `kdb*` function the user calls to trigger the operation and plugins implement a `elektra*` function for each of the operations they support (at least `get`). +For each of these there is a `kdb*` function the user calls to trigger the operation and plugins export a function for each of the operations they support (at least `get`). -Additionally, plugins may implement `elektraCommit` and `elektraError`. + + +Additionally, plugins may implement `commit` and `error`. These are part of the `set` operation and there is no corresponding `kdbCommit` or `kdbError` function available in `libelektra-kdb`. The operations `get` and `set` also have different _phases_: @@ -22,33 +24,32 @@ The basic flow of this operation is: 1. Create empty `KDB` instance 2. Configure `KDB` instance for bootstrap -3. Run bootstrap `get` operation -4. Setup default global plugins -5. Process contract -6. Parse mountpoints -7. Configure `KDB` instance with real mountpoints -8. Add hard coded mountpoints to `KDB` instance - -> **Note:** Despite sharing a name, the `open` operation **does not** call the `elektraOpen` function of any plugin (except within the bootstrap `get` operation). +3. Run bootstrap `get` operation: + This loads the contents of `system:/elektra/mountpoints` so that the mountpoints can be configured. +4. Process contract and set up plugins for hooks (see [Hooks](hooks.md)) +5. Parse mountpoints: + This transforms the contents of `system:/elektra/mountpoints` into the internal state stored in a `KDB` instance. +6. Reconfigure `KDB` with real mountpoints: + This switches the `KDB` instance from bootstrap mode to use the real mountpoint state created above. +7. Add hardcoded mountpoints to `KDB` instance: + There are a few hardcoded mountpoints (root mountpoints, `system:/elektra/modules`, `system:/elektra/version`, etc.) that are always present. + They are added in this step. Namespaces in mountpoint configs: -- cascading mountpoints act as an alias for identical `proc:/`, `dir:/`, `user:/`, `system:/` and `default:/` mountpoints. -- `meta:/` mountpoints are illegal -- explicit `default:/` mountpoints are illegal - They can only exist via cascading mountpoints. - This is so that default values can be checked and validated by plugins. -- `spec:/` mountpoints may exist, but they are treated differently during `get` and `set` -- `proc:/` mountpoints may exist, but they are read-only (see below) -- `dir:/`, `user:/` and `system:/` mountpoints can be created without restrictions, except for the one below +- `dir:/`, `user:/` and `system:/` mountpoints can be created without restrictions, except for the reserved sections listed below. +- `spec:/` mountpoints can be created with the same restrictions, but they are also treated specially during `get` and `set`. +- `proc:/` mountpoints are always read-only and receive special treatment during `get` +- `default:/` mountpoints are read-only and receive special treatment during `get`, specifically they only go through the `poststorage` phase +- mountpoints in all other namespaces are entirely illegal -Other restrictions: +Reserved sections: - Creating a mountpoint for `/elektra` or below in _any namespace_ is forbidden. This section of the KDB is reserved for Elektra's own config. -- `system:/elektra/mountpoints`, `user:/elektra/mountpoints` and `dir:/elektra/mountpoints` are all required for the bootstrap process and use a hard coded backend. +- `system:/elektra/mountpoints`, `user:/elektra/mountpoints` and `dir:/elektra/mountpoints` are all required for the bootstrap process and use a hardcoded backend. The backends are implemented by a standard file-based backend plugin that is defined at compile-time of `libelektra-kdb`. -- `system:/elektra/version` and `system:/elektra/modules` will always use hard coded read-only backends containing information about this Elektra installation. +- `system:/elektra/version` and `system:/elektra/modules` will always use hardcoded read-only backends containing information about this Elektra installation. The backends are implemented by special purpose backend plugins. ## `get` Operation @@ -61,13 +62,14 @@ Properties of `kdbGet()`: - After calling `kdbGet (kdb, ks, parentKey)`, the KeySet `ks` will contain _all keys_ (including their values) that are stored in _any backend_ with a mountpoint that is _below `parentKey`_. - After calling `kdbGet (kdb, ks, parentKey)`, below `parentKey` the KeySet `ks` will _only_ contain keys that are stored in a backend. + - The KeySet `ks` _may_ contain other keys not below `parentKey`. These keys fall into one of three categories: 1. Keys that are not below `parentKey`, but are stored in a backend that contains other keys which are below `parentKey`. These keys are returned, because backends are treated as one atomic unit. Either all keys within a backend are read, or none of them are. 2. Keys that are stored in a backend that is not below `parentKey`. - `kdbGet()` may decide that it is more efficient (e.g. because of a cache) to return more keys than requested. + `kdbGet()` may decide that it is more efficient (e.g., because of a cache) to return more keys than requested. 3. Keys that were already present in `ks` when `kdbGet()` was called and do not conflict with the goal of representing the current state of the KDB. > **Note:** While it is possible that keys not below `parentKey` exist within `ks`, there are no guarantees. `kdbGet()` only makes guarantees about the keys _below `parentKey`_. - After calling `kdbGet (kdb, ks, parentKey)`, the Key `parentKey` will only have the `meta:/error/*` or `meta:/warnings/#/*` metakeys, if the errors/warnings originate from this `kdbGet()` call. @@ -81,14 +83,12 @@ Which keys are removed from `ks` depends on the backends that are read. It is only guaranteed that below `parentKey` the KeySet `ks` correctly represents the state of the KDB. For the rest of `ks` there are no such guarantees. -TODO: cache correct? notifications? (just call at the end?) - > **Note:** In the list below "phase" always refers to a phase of the `get` operation as described in [the backend plugin contract](backend-plugins.md). -The basic flow of this operation is: +The flow of this operation is: 1. Determine the backends needed to read all keys below `parentKey` -2. Run the `open` operation for all required backend that haven't been opened +2. Run the `open` operation for all required backends that haven't been opened 3. Run the `init` phase on all the backends that haven't been initialized 4. Run the `resolver` phase on all backends 5. From now on ignore all backends, which indicated that there is no update. @@ -101,29 +101,29 @@ The basic flow of this operation is: Ask the global cache plugin for the cached data and **return**. 10. Run the `prestorage` and `storage` phase on all backends. 11. Run the `poststorage` phase of all `spec:/` backends. - 12. Merge the data from all backends -13. If enabled, run the `gopts` plugin. -14. Run the `spec` plugin (to copy metakeys). +13. If enabled, run the `gopts/get` hook. +14. Run the `spec/copy` hook. 15. Split data back into individual backends. 16. Run the `poststorage` phase for all non-`spec:/` backends. 17. Remove all keys which are below the parent key of any backend that has been read from `ks`. 18. Merge the data from all backends into `ks`. 19. If a global cache plugin is enabled, update cache. +20. Run the `notification/send` hook. Then **return**. > **Note:** In case of error, we abort immediately, restore `ks` to its original state and return. Influence of namespaces: -- cascading and `meta:/` keys are always illegal in `ks` (should be enforced via different KeySet types) -- `spec:/` backends only go through `init`, `resolver`, `cache`, `presetstorage` and `storage` phases as normal, but their `poststorage` phase is called earlier. +- `spec:/` backends go through `init`, `resolver`, `cache`, `presetstorage` and `storage` phases as normal, but their `poststorage` phase is called earlier. This is required, because any validation and post-processing of `spec:/` keys needs to happen, before they are used as the specification for other keys in the actual `poststorage` phase. - `dir:/`, `user:/` and `system:/` go through all phases as described above. - `proc:/` mountpoints go through all the phases as described above, but they are not stored in the cache. - `default:/` backends only go through the `poststorage` phase. This is because `default:/` keys are generated from the specification (stored as `spec:/` keys). - Therefore, no `default:/` keys can exist before the specification is processed by the `spec` plugin. + Therefore, no `default:/` keys can exist before the specification is processed by the `spec/copy` hook. +- keys with other namespaces are always illegal in `ks` (should be enforced via different `KeySet` types) ## `set` Operation @@ -133,22 +133,18 @@ The purpose of the `set` operation is to write data from a `KDB` instance into b Properties of `kdbSet()`: + + - When calling `kdbSet (kdb, ks, parentKey)` the contents (key names, values and metadata) of `ks` _will not be modified_. - _All keys_ in `ks` that are below `parentKey` will be persisted in the KDB, when a `kdbSet (kdb, ks, parentKey)` call returns successfully. Additionally, any key in `ks` that shares a backend with another key which is below `parentKey` will also be persisted. -- Calling `kdbSet` results in an error, if `kdbGet` wasn't called on this `KDB` instance at least once. - -TODO: notifications? (just call at the end?) - -The basic flow of this operation is: +- Calling `kdbSet` may result in an error, if `kdbGet` wasn't called on this `KDB` instance with the same `parentKey` at least once. - +The flow of this operation is: 1. Determine the backends needed to write all keys below `parentKey`. -2. Check that all backends are initialized (i.e. `kdbGet()` was called). -3. Run the `spec` plugin on `ks` (to add metakeys for new keys). - - +2. Check that all backends are opened and initialized (i.e. `kdbGet()` was called). +3. Run the `spec/copy` hook on `ks` (to add metakeys to newly created keys). 4. Deep-Copy `ks` (below `parentKey`) into a new KeySet `set_ks` 5. Split `set_ks` into individual backends 6. Determine which backends contain changed data. @@ -156,30 +152,28 @@ The basic flow of this operation is: From now on ignore all backends that have not changed. From now on also ignore all backends that were initialized as read-only. Issue a warning, if a change was detected (via `KEY_FLAG_SYNC`) in a read-only backend. + > **Note**: Steps 4-6 might be combined into a single procedure that deep-copies only keys from changed backends into separate KeySets per backend 7. Run the `resolver` and `prestorage` on all backends (abort immediately on error and go to e). - + 8. Merge the results into a new version of `set_ks`. -9. Run the `spec` plugin on `set_ks` (to remove copied metakeys). +9. Run the `spec/remove` hook on `set_ks` (to remove copied metakeys). 10. Split `set_ks` into individual backends again. 11. Run the `storage` and `poststorage` phases on all backends (abort immediately on error and go to e). 12. If everything was successful: - Run the `precommit` and `commit` phases on all backends (abort immediately on error and go to e), then run the `postcommit` phase on all backends and **return**. + Run the `precommit` and `commit` phases on all backends (abort immediately on error and go to e), then run the `postcommit` phase on all backends (record all errors as warnings and ignore them) and **return**.
  1. If there was an error: - Run the coderollback, rollback and postrollback phases on all backends and return. + Run the prerollback, rollback and postrollback phases on all backends and return.
Influence of namespaces: -- cascading and `meta:/` keys are always illegal in `ks` (should be enforced via different KeySet types) - `default:/` and `proc:/` keys are completely ignored by `kdbSet()` - `spec:/`, `dir:/`, `user:/` and `system:/` go through all phases as described above. +- keys with other namespaces are always illegal in `ks` (should be enforced via different KeySet types) ## `close` Operation diff --git a/src/libs/elektra/kdb.c b/src/libs/elektra/kdb.c index fe813cd46d9..c78fe7cf075 100644 --- a/src/libs/elektra/kdb.c +++ b/src/libs/elektra/kdb.c @@ -989,7 +989,7 @@ KDB * kdbOpen (const KeySet * contract, Key * errorKey) goto error; } - // Step 4: setup default global plugins + // Step 4: process contract and set up hooks // TODO (kodebach): remove/replace step in global plugins rewrite if (mountGlobals (handle, ksDup (elektraKs), handle->modules, errorKey) == -1) { @@ -1008,14 +1008,13 @@ KDB * kdbOpen (const KeySet * contract, Key * errorKey) goto error; } - // Step 5: process contract if (contract != NULL && !ensureContract (handle, contract, errorKey)) { ksDel (elektraKs); goto error; } - // Step 6: parse mountpoints + // Step 5: parse mountpoints KeySet * backends = elektraMountpointsParse (elektraKs, handle->modules, handle->global, errorKey); if (backends == NULL) { @@ -1023,7 +1022,7 @@ KDB * kdbOpen (const KeySet * contract, Key * errorKey) goto error; } - // Step 7: switch from boostrap to real config + // Step 6: switch from boostrap to real config ksDel (elektraKs); keyCopy (errorKey, initialParent, KEY_CP_NAME | KEY_CP_VALUE); @@ -1034,7 +1033,7 @@ KDB * kdbOpen (const KeySet * contract, Key * errorKey) handle->backends = backends; - // Step 8: add hardcoded mountpoints + // Step 7: add hardcoded mountpoints if (!addHardcodedMountpoints (handle, errorKey)) { goto error; @@ -1848,18 +1847,15 @@ int kdbGet (KDB * handle, KeySet * ks, Key * parentKey) } keyDel (specRoot); - // TODO (kodebach) [Q]: can we avoid this merge and the split in step 15? // Step 12: merge data from all backends KeySet * dataKs = ksNew (ksGetSize (ks), KS_END); backendsMerge (backends, dataKs); - // Step 13: run procgetstorage global plugins if (elektraGlobalGet (handle, dataKs, parentKey, PROCGETSTORAGE, MAXONCE) == ELEKTRA_PLUGIN_STATUS_ERROR) { goto error; } - // Step 14: run postgetstorage global plugins if (elektraGlobalGet (handle, dataKs, parentKey, POSTGETSTORAGE, MAXONCE) == ELEKTRA_PLUGIN_STATUS_ERROR) { goto error; From dfd7e5ed1e5ecf0b89303a83c3b5bd0ed634b5e2 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Klemens=20B=C3=B6swirth?= Date: Sun, 25 Sep 2022 22:18:19 +0200 Subject: [PATCH 3/7] doc: update backend-plugins.md --- doc/dev/backend-plugins.md | 22 +++++++++++----------- 1 file changed, 11 insertions(+), 11 deletions(-) diff --git a/doc/dev/backend-plugins.md b/doc/dev/backend-plugins.md index 884b05ce41c..55dbf4d2609 100644 --- a/doc/dev/backend-plugins.md +++ b/doc/dev/backend-plugins.md @@ -5,7 +5,7 @@ There exists a _backend contract_ between `libelektra-kdb` and any plugin acting as a backend plugin. This contract sets, the order of the phases described above and defines the interaction between a backend plugin and `libelektra-kdb`. -TODO: update get diagram; add cachecheck phase + -The current phase is communicated to the backend plugin via the global keyset. -The value of the key `system:/elektra/kdb/backend/phase` is always set to the current phase. +The current phase is communicated to the backend plugin (and any other plugin) via the global keyset. +The current phase can be retrieved via the `elektraPluginGetPhase` function. ### Operation `get` @@ -32,8 +33,8 @@ During the `init` phase the backend plugin is called with: - A keyset `definition` containing the mountpoint definition. To make things easier for the plugin, keys in `definition` are translated into cascading keys relative to `parentKey`. For example, if the key `system:/elektra/mountpoints/system:\/hosts/path` is set in the KDB, then `definition` will contain a key `/path`. - -TODO: how does the backend plugin get access to the other plugins? Maybe, just add a plugins/# array into `ks` containing keys with `Plugin *` values? +- Additionally, the plugins for the current mountpoint are opened by `libelektra-kdb` and provided to the backend plugin via the global keyset. + They can be accessed via the `elektraPluginFromMountpoint` function. The backend plugin then: @@ -49,7 +50,7 @@ This phase exists purely for the backend plugin to initialize and configure itse > **Note**: This phase is only executed _once per instance of `KDB`_. > Only the first `kdbGet()` call will result in `libelektra-kdb` executing this phase, all future calls to `kdbGet()` (and `kdbSet()`) start with the `resolver` phase. -> The backend plugin must store the information contained in the mountpoint definition internally to accommodate this. +> The backend plugin must store the necessary information contained in the mountpoint definition internally to accommodate this. #### Resolver Phase @@ -81,8 +82,7 @@ During the `cachecheck` phase the backend plugin is called with: - The exact `parentKey` that was returned by the `resolver` phase of this `get` operation. The key name and value of this key are read-only. - Additionally, the metakey `internal/kdb/cachetime` is set to a value indicating the update time of the cache entry. - TODO: exact format of `internal/kdb/cachetime` TBD + Additionally, the metakey `internal/kdb/cachehandle` is set to a value indicating the cache handle (usually modification time) of the cache entry. - An empty keyset `ks`. The backend plugin then: @@ -130,7 +130,7 @@ It is where validation, generation of implicit values and similar tasks happen. Finally, `libelektra-kdb` merges the keyset returned by the `poststorage` phase with the ones returned by other backend plugins for different mountpoints and then returns it to the user. -TODO: Are modifications to `parentKey` visible to the user? + ### Operation `set` @@ -237,7 +237,7 @@ This makes the `postcommit` phase mostly useful for logging. Finally, `libelektra-kdb` merges the keyset returned by the `postcommit` phase (which is still the same one that was returned by the `prestorage` phase) with the ones returned by other backend plugins for different mountpoints and then returns it to the user. -TODO: Are modifications to `parentKey` visible to the user? + #### Rollback Phases (`set` only) @@ -268,4 +268,4 @@ In the `rollback` phase the backend plugin: Finally, `libelektra-kdb` will restore `ks` to the state in which the user provided it and return. -TODO: Are modifications to `parentKey` visible to the user? + From 05b515e19f87adad0daa17df5a53d22da29321c5 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Klemens=20B=C3=B6swirth?= Date: Sun, 25 Sep 2022 22:21:31 +0200 Subject: [PATCH 4/7] doc: remove outdated stuff from data-structures.md --- doc/dev/data-structures.md | 112 +------------------------------------ 1 file changed, 1 insertion(+), 111 deletions(-) diff --git a/doc/dev/data-structures.md b/doc/dev/data-structures.md index 19f961e464c..b3986a1b2e3 100644 --- a/doc/dev/data-structures.md +++ b/doc/dev/data-structures.md @@ -37,7 +37,7 @@ It performs well for lookup, but needs more memory allocations. Currently the `KeySet` is implemented as a sorted array. It is fast on appending and iterating, and has nearly no size-overhead. -To improve the lookup-time, an additional **hash** will be used. +To improve the lookup-time, an additional **hash** is used, see [OPMPHM](#order-preserving-minimal-perfect-hash-map-aka-opmphm) below. ### ABI compatibility @@ -194,40 +194,6 @@ indicating where the new value should be inserted when the key is not found. Elektra now also uses this trick internally. -### Internal Cursor - -`KeySet` supports an -**external iterator** -with the two functions -`ksRewind()` to go to the beginning and `ksNext()` to -advance the _internal cursor_ to the next key. -This side effect is used to indicate a position for -operations on a `KeySet` without any additional parameter. -This technique is comfortable to see which -key has caused an error after an unsuccessful key database operation. - -Elektra only has some functions to change the cursor of a key set. -But these allow the user to compose powerful functions. -Plugins do that extensively as we will see later -in `ksLookupRE()`. -The user can additionally write more such functions for -his or her own purposes. -To change the internal cursor, it is -sufficient to iterate -over the `KeySet` and stop at the wanted key. -With this technique, we can, for example, realize -lookup by value, by specific metadata and by -parts of the name. -Without an additional index, it is not possible that -such operations perform more efficiently -than by a linear iteration key by key. -For that reason, Elektra’s core does not provide -such functions. -The function `ksLookupByName()`, however, -uses the more efficient binary search -because the array inside the `KeySet` -is ordered by name. - ### External Cursor External cursor is an alternative to the approach explained above. @@ -282,82 +248,6 @@ For example, for every key in a `KeySet` without having null pointer or out of range problems. -## Trie vs. Split - -Up to now, -we have discussed external data structures visible to the user of the -library. -The application and plugin programmer needs them -to access configuration. -Last, but not least, -we will show two internal data structures. -The user will not see them. -To understand the algorithm, however, -the user needs to understand them as well. - -### Trie - -A _Trie_ or prefix tree is an ordered tree -data structure. -In Elektra, -it provides the information -to decide -in which backend a key resides. -The algorithm, presented in [algorithm](algorithm.md), -also needs a list of all backends. -The initial approach was to iterate over the `Trie` -to get a list of all backends. -But the transformation of a `Trie` to a list of backends, contained -many bugs caused by corner cases in connection with the default backend -and cascading mount points. - -### Split - -So, instead of transforming the trie to a list of backends, -we introduced a new data structure called `Split`. -The name `Split` comes from the fact that -an initial key set is split into many key sets. -These key sets are stored in the `Split` object. -`Split` advanced to the central data structure for the algorithm: - -```c -typedef struct _Split Split; - -struct _Split { - size_t size; - size_t alloc; - KeySet **keysets; - Backend **handles; - Key **parents; - int *syncbits; -}; -``` - -The data structure `Split` contains the following fields: - -- **size**: contains the number of key sets currently in `Split`. - -- **alloc**: allows us to allocate more items than currently in use. - -- **keysets** represents a list of key sets. - The keys in one of the key sets are known to belong to a specific - backend. - -- **handles**: contains a list of handles to backends. - -- **parents**: represents a list of keys. - Each `parentKey` contains the - root key of a backend. No key of the respective key set is above the - `parentKey`. - The key name of `parentKey` contains the mount point of a backend. - The resolver writes the file name into the value of the `parentKey`. - -- **syncbits**: are some bits that can be set for every backend. - The algorithm uses the `syncbits` to decide if the key set needs to be - synchronized. - -Continue reading [with the error handling](error-handling.md). - ## Order Preserving Minimal Perfect Hash Map (aka OPMPHM) The OPMPHM is a non-dynamic randomized hash map of the Las Vegas type, that creates an index over the elements, From ebb55d8733840d7a61850e8d46b376b609fc8a62 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Klemens=20B=C3=B6swirth?= Date: Fri, 30 Sep 2022 21:48:15 +0200 Subject: [PATCH 5/7] update outdated docs --- doc/AUTHORS.md | 4 +- doc/dev/algorithm.md | 10 +- doc/dev/architecture.md | 7 +- doc/dev/backend-plugins.md | 2 +- doc/dev/kdb-operations.md | 2 +- doc/dev/mountpoints.md | 2 +- doc/dev/plugins-framework_old.md | 163 ------------------------------- doc/dev/plugins-ordering.md | 7 ++ doc/tutorials/mount.md | 2 + 9 files changed, 26 insertions(+), 173 deletions(-) delete mode 100644 doc/dev/plugins-framework_old.md diff --git a/doc/AUTHORS.md b/doc/AUTHORS.md index e630cbcf722..2a7a28ed54d 100644 --- a/doc/AUTHORS.md +++ b/doc/AUTHORS.md @@ -19,11 +19,11 @@ maintainer, development of the cache and mmapstorage plugins ## Klemens Böswirth -development of the highlevel API and code-generation; various other things +a bit of everything - email: k.boeswirth+git@gmail.com - github user: [kodebach](https://github.com/kodebach) -- devel/test on: Fedora +- devel/test on: Manjaro ## Robert Sowula diff --git a/doc/dev/algorithm.md b/doc/dev/algorithm.md index c4d52c783c7..373fa606bd8 100644 --- a/doc/dev/algorithm.md +++ b/doc/dev/algorithm.md @@ -1,9 +1,11 @@ # Algorithm -You might want to read -[about architecture](architecture.md) -and -[data structures](data-structures.md) first. +You might want to read [about architecture](architecture.md) and [data structures](data-structures.md) first. + +## Outdated + +> **Warning** Many of the things described below (especially about `KDB` and the `kdb*` functions) are outdated. +> See [`kdb-operations.md`](kdb-operations.md) and [`kdb-contracts.md`](kdb-contracts.md) for more up-to-date information. ## Introduction diff --git a/doc/dev/architecture.md b/doc/dev/architecture.md index 52298b9c8b5..07c1059a502 100644 --- a/doc/dev/architecture.md +++ b/doc/dev/architecture.md @@ -14,7 +14,7 @@ To help readers to understand the algorithm that glues together the plugins, we first describe some details of the [data structures](data-structures.md). Full knowledge of the [algorithm](algorithm.md) is not presumed to be able to develop -most plugins (with the exception of [the resolver](/src/plugins/resolver/)). +most plugins. Further important concepts are explained in: @@ -22,6 +22,11 @@ Further important concepts are explained in: - [granularity](/doc/help/elektra-granularity.md) - [sync-flag](/doc/help/elektra-sync-flag.md) +## Outdated + +> **Warning** Many of the things described below (especially in relation to backends and mountpoints) are outdated. +> See [`kdb-operations.md`](kdb-operations.md), [`backend-plugins.md`](backend-plugins.md) and [`mountpoints.md`](mountpoints.md) for more up-to-date information. + ## API The aim of the Elektra Initiative is to design and implement a powerful diff --git a/doc/dev/backend-plugins.md b/doc/dev/backend-plugins.md index 55dbf4d2609..95318ee6feb 100644 --- a/doc/dev/backend-plugins.md +++ b/doc/dev/backend-plugins.md @@ -1,4 +1,4 @@ -# Proposal: Backend Plugins +# Backend Plugins ## Backend Contract diff --git a/doc/dev/kdb-operations.md b/doc/dev/kdb-operations.md index 1774ce72d97..032ee8985e8 100644 --- a/doc/dev/kdb-operations.md +++ b/doc/dev/kdb-operations.md @@ -1,4 +1,4 @@ -# Proposal: KDB Operations +# KDB Operations There are four main _operations_ in `libelektra-kdb`: `open`, `get`, `set` and `close`. For each of these there is a `kdb*` function the user calls to trigger the operation and plugins export a function for each of the operations they support (at least `get`). diff --git a/doc/dev/mountpoints.md b/doc/dev/mountpoints.md index 4f02ad8726e..19786b6583c 100644 --- a/doc/dev/mountpoints.md +++ b/doc/dev/mountpoints.md @@ -1,4 +1,4 @@ -# Proposal: New mountpoint config structure +# New mountpoint config structure A mountpoint is defined by adding keys below `system:/elekta/mountpoints/`, where `` is the key name for the parent key of the mountpoint with slashes properly escaped. For example for the mountpoint `user:/mymountpoint` you have to add keys below `system:/elekta/mountpoints/user:\/mymountpoint`. diff --git a/doc/dev/plugins-framework_old.md b/doc/dev/plugins-framework_old.md deleted file mode 100644 index 62ce7bc52a0..00000000000 --- a/doc/dev/plugins-framework_old.md +++ /dev/null @@ -1,163 +0,0 @@ -# Plugins Framework - -Many component systems pass information between the various components -by calling methods of each other. This is not the way Elektra’s plugin -system works. Instead, the core passes a `KeySet` object in one direction -from plugin to plugin. So they form a so called pipes-and-filter. Each of -the plugins can modify the configuration or add any other information -using metakeys. While this approach is in general less flexible, this -information flow still allows powerful chaining. Because plugins do -not have to bother to call other plugins, the plugin development is -also easier. The ordering of plugins in backends is controlled using -contracts. - -## Contracts - -Every plugin should provide a full contract to give information how it -will work with other plugins. Most parts of the contract are obligatory. -Plugins cannot be loaded without this information. For example, plugins -must provide the clause `infos/version`. It is vital so that the plugin -loader knows which version of Elektra the plugin was built for. - -## Conditions - -It is, however, up to the plugin not to have every clause of the contract. -For example, the plugin might not tell what it provides or needs. It can -also leave any description out. In this situation it is unclear what -the plugin will do. Such a plugin can add or remove keys, and changes -values and metadata regardless what other plugins expect. If only such -plugins existed there would be chaos. It would be impossible to determine -the behavior of a backend which uses a multiple of such plugins. - -To avoid this situation, every plugin exports a contract describing how -the plugin modifies the `KeySet` `returned`. Most often it is enough -to state that it is a _storage_ plugin or that it will _filter_ keys. - -The data structures, however, are already responsible for most of the -pre- and postconditions. Every condition the data structure guarantees, -takes away a concern for the plugins. All the parts that are already -guaranteed by data structures do not need to be stated in the contract. - -Plugins should not be burdened to check too many postconditions. Instead, -plugins focus on their task. The plugin does not need to check the sync -flag of keys or if the keys are below the mount point. The core already -guarantees correct behavior as described -in [algorithm](/doc/dev/algorithm.md). - -To sum up, contracts give the information how a plugin interacts with -others. It describes if, and how, the `KeySet` `returned` is changed. -Using contracts, we get a predictable behavior, but still support every -kind of plugin. - -## Exporting Contracts - - - -As already stated, some parts of the contracts are obligatory. -`kdb mount` needs to know which symbols the plugin exports. Only the -`elektraPluginGet()` symbol is mandatory - it is used to yield this -information. Elektra’s core also uses the functions `elektraPluginSet()`, -`elektraPluginError()`, `elektraPluginOpen()` and `elektraPluginClose()` -if available. Other functions like `serialize`, `unserialize` or -`lookup` which implement special features can be supported, but are -ignored by the core. For the user of the library these functions can -be very useful. These functions shall either belong to the concern of -the plugin or be implemented within the plugin because of the dependences. - -As described in [infos/provides](/doc/CONTRACT.ini), the plugin can -also provide descriptive information, for example about the author and -the licence. Advanced plugins can also export plugin configuration for -other plugins so that the overall backend works properly. Last, but not -least, as enumerated in [infos/placement](/doc/CONTRACT.ini) dependency -and placement information makes the system reliable and robust. With that -information, plugins can be placed into a backend in an automatic and -secure way. - -`system:/elektra/modules` provides for every module the information -described above. The entry exists once a plugin of that module is loaded. -For each module a special _module backend_ is generated and mounted at -`system:/elektra/modules/`. The `elektraPluginGet()` function -generates this described contract on requests. - -For example, the ccode plugin, implements: - -```c -int elektraCcodeGet(Plugin *handle, KeySet *returned, Key *parentKey) -{ - if (!strcmp (keyName(parentKey), "system:/elektra/modules/ccode")) - { - KeySet *contract = ksNew (30, - keyNew ("system:/elektra/modules/ccode", - KEY_END), - keyNew ("system:/elektra/modules/ccode/exports", - KEY_END), - //... - KS_END); - ksAppend (returned, contract); - ksDel (contract); - return 1; - } - // implementation of elektraCcodeGet -} -``` - -We see in the listing above that the plugin generates and returns -the contract if, and only if, the name of the `parentKey` is -`system:/elektra/modules/ccode`. The user and the contract checker can -access the contract of ccode below the key `system:/elektra/modules/ccode` -in the same way other configuration is accessed. Note that we also -have to `return 1` at the end of the contract to not execute the regular -functionality of the plugin. - -## Exports - -Plugins can also export arbitrary additional functions. -To export it, simply add another `exports` symbol to -the contract: - -```c -keyNew ("system:/elektra/modules/dump/exports/checkconf", KEY_FUNC, - elektraCcodeCheckConf, KEY_END); -``` - -`checkconf` is the most important function that is not -available as built-in function. It is used to validate -or even correct/finish the plugin's configuration. - -If you want to use an exported function from a symbol, -please look at [Plugin::parse](/src/libs/tools/src/plugin.cpp). - -## Changing Plugins - -This configuration is static and contains the contract information. -In theory, the contract can be changed without any problems in ways that -it provides more and obligates less. But the problem is that it will -not be checked if this is the case because a recheck of the contracts -of a backend is very expensive. The contract checker doing this, only -runs once during mount-time. Changing contracts in an incompatible way -forces the user to remove all mount points where the plugin is and mount -it again. Such actions are only sustainable in a development phase and -not in a productive environment. - -But the plugin's implementation is allowed to change without being -remounted if it is a subtype of the earlier version. Only in this -situation it can be a drop-in replacement. With a good testing framework -the behavior can be checked to some extent. - -We also see in the listing above that the code responsible for generating -the contract and the code for the implementation are next to each other. -Plugins need to satisfy those self-imposed obligations that are described -in contracts. They ensure that plugins interact in predictable ways. -So the process of writing individual plugins and composing them together -can be described as Component-Based Software Engineering. - -Plugins can also be viewed as framework extensions. A component abstracts -plugins. But this term is misleading in our case, because components -usually can choose which interfaces they implement. Elektra’s plugins, -however, are restricted to implement one specific interface. Without -contracts, plugins could not interact as described in this chapter. - -## SEE ALSO - -- [plugins-ordering](plugins-ordering.md) -- [elektra-contracts(7)](/doc/help/elektra-contracts.md) diff --git a/doc/dev/plugins-ordering.md b/doc/dev/plugins-ordering.md index 0d98c002e6b..e23fec93b74 100644 --- a/doc/dev/plugins-ordering.md +++ b/doc/dev/plugins-ordering.md @@ -1,5 +1,12 @@ # Plugins Ordering +## Outdated + +> **Warning** Many of the things described below are outdated. +> Parts of this document are still valid for the `backend` plugin. + +## Introduction + You should first read [elektra-plugins](/src/plugins/) to get an idea about plugins. diff --git a/doc/tutorials/mount.md b/doc/tutorials/mount.md index 631489cfff1..d06a665db02 100644 --- a/doc/tutorials/mount.md +++ b/doc/tutorials/mount.md @@ -1,5 +1,6 @@ # Mounting + From 7a75cc33efe9b0ad45fc52ea184c8bf0bf913bc1 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Klemens=20B=C3=B6swirth?= Date: Thu, 6 Oct 2022 22:40:00 +0200 Subject: [PATCH 6/7] incorporate review --- doc/dev/algorithm.md | 2 ++ doc/dev/architecture.md | 2 ++ doc/dev/backend-plugins.md | 19 +++++++++++-------- doc/dev/kdb-operations.md | 19 ++++++++++++------- doc/dev/mountpoints.md | 15 ++++++--------- doc/dev/plugins-ordering.md | 4 ++++ doc/tutorials/mount.md | 2 +- 7 files changed, 38 insertions(+), 25 deletions(-) diff --git a/doc/dev/algorithm.md b/doc/dev/algorithm.md index 373fa606bd8..58d003ccf56 100644 --- a/doc/dev/algorithm.md +++ b/doc/dev/algorithm.md @@ -4,6 +4,8 @@ You might want to read [about architecture](architecture.md) and [data structure ## Outdated + + > **Warning** Many of the things described below (especially about `KDB` and the `kdb*` functions) are outdated. > See [`kdb-operations.md`](kdb-operations.md) and [`kdb-contracts.md`](kdb-contracts.md) for more up-to-date information. diff --git a/doc/dev/architecture.md b/doc/dev/architecture.md index 07c1059a502..340eacc2a89 100644 --- a/doc/dev/architecture.md +++ b/doc/dev/architecture.md @@ -24,6 +24,8 @@ Further important concepts are explained in: ## Outdated + + > **Warning** Many of the things described below (especially in relation to backends and mountpoints) are outdated. > See [`kdb-operations.md`](kdb-operations.md), [`backend-plugins.md`](backend-plugins.md) and [`mountpoints.md`](mountpoints.md) for more up-to-date information. diff --git a/doc/dev/backend-plugins.md b/doc/dev/backend-plugins.md index 95318ee6feb..4914d621b97 100644 --- a/doc/dev/backend-plugins.md +++ b/doc/dev/backend-plugins.md @@ -19,6 +19,15 @@ Similarly, for the phases of a `set` operation `elektraSet` is called. The current phase is communicated to the backend plugin (and any other plugin) via the global keyset. The current phase can be retrieved via the `elektraPluginGetPhase` function. +### `parentKey` + +The key `parentKey` that is given to the backend plugin as an input at various points, must be treated carefully. +Currently, _all_ modifications to this key will be propagated to the `parentKey` that was used to call `kdbGet`. + +The name of the `parentKey` is marked read-only and therefore cannot be changed. +The value and metadata can, and in some cases must be, changed. +In future this may be restricted further to ensure a more structured communication. + ### Operation `get` The `get` operation is mandatory and all backend plugins must implement it. @@ -31,8 +40,8 @@ During the `init` phase the backend plugin is called with: The key name and value of this key are read-only. The name of `parentKey` is chosen to make it easier for the plugin to produce good error messages. - A keyset `definition` containing the mountpoint definition. - To make things easier for the plugin, keys in `definition` are translated into cascading keys relative to `parentKey`. - For example, if the key `system:/elektra/mountpoints/system:\/hosts/path` is set in the KDB, then `definition` will contain a key `/path`. + To make things easier for the plugin, keys in `definition` are renamed to be below `system:/`. + For example, if the key `system:/elektra/mountpoints/system:\/hosts/path` is set in the KDB, then `definition` will contain a key `system:/path`. - Additionally, the plugins for the current mountpoint are opened by `libelektra-kdb` and provided to the backend plugin via the global keyset. They can be accessed via the `elektraPluginFromMountpoint` function. @@ -130,8 +139,6 @@ It is where validation, generation of implicit values and similar tasks happen. Finally, `libelektra-kdb` merges the keyset returned by the `poststorage` phase with the ones returned by other backend plugins for different mountpoints and then returns it to the user. - - ### Operation `set` The `set` operation is optional. @@ -237,8 +244,6 @@ This makes the `postcommit` phase mostly useful for logging. Finally, `libelektra-kdb` merges the keyset returned by the `postcommit` phase (which is still the same one that was returned by the `prestorage` phase) with the ones returned by other backend plugins for different mountpoints and then returns it to the user. - - #### Rollback Phases (`set` only) If any of the phases `prestorage`, `storage`, `poststorage`, `precommit` or `commit` fail, `libelektra-kdb` will continue with the rollback phases. @@ -267,5 +272,3 @@ In the `rollback` phase the backend plugin: - **MAY** act differently depending on which phase failed. Finally, `libelektra-kdb` will restore `ks` to the state in which the user provided it and return. - - diff --git a/doc/dev/kdb-operations.md b/doc/dev/kdb-operations.md index 032ee8985e8..75094c95f26 100644 --- a/doc/dev/kdb-operations.md +++ b/doc/dev/kdb-operations.md @@ -3,7 +3,12 @@ There are four main _operations_ in `libelektra-kdb`: `open`, `get`, `set` and `close`. For each of these there is a `kdb*` function the user calls to trigger the operation and plugins export a function for each of the operations they support (at least `get`). - + Additionally, plugins may implement `commit` and `error`. These are part of the `set` operation and there is no corresponding `kdbCommit` or `kdbError` function available in `libelektra-kdb`. @@ -61,8 +66,10 @@ The purpose of the `get` operation is to read data stored in backends into a `KD Properties of `kdbGet()`: - After calling `kdbGet (kdb, ks, parentKey)`, the KeySet `ks` will contain _all keys_ (including their values) that are stored in _any backend_ with a mountpoint that is _below `parentKey`_. -- After calling `kdbGet (kdb, ks, parentKey)`, below `parentKey` the KeySet `ks` will _only_ contain keys that are stored in a backend. - +- After calling `kdbGet (kdb, ks, parentKey)`, below `parentKey` the KeySet `ks` will _mostly_ contain keys that are stored in a backend. + The exception here are `proc:/` and `spec:/` keys. + For other namespaces, all keys below `parentKey` will be removed from `ks`. + For `proc:/` and `spec:/` only keys that overlap with a backend that was loaded will be removed from `ks`. - The KeySet `ks` _may_ contain other keys not below `parentKey`. These keys fall into one of three categories: 1. Keys that are not below `parentKey`, but are stored in a backend that contains other keys which are below `parentKey`. @@ -133,9 +140,8 @@ The purpose of the `set` operation is to write data from a `KDB` instance into b Properties of `kdbSet()`: - - -- When calling `kdbSet (kdb, ks, parentKey)` the contents (key names, values and metadata) of `ks` _will not be modified_. +- When calling `kdbSet (kdb, ks, parentKey)` the contents (key names, values and metadata) of `ks` will _mostly_ not be modified. + The only modifications that are made to `ks` are those that originate from the `spec/copy` hook. - _All keys_ in `ks` that are below `parentKey` will be persisted in the KDB, when a `kdbSet (kdb, ks, parentKey)` call returns successfully. Additionally, any key in `ks` that shares a backend with another key which is below `parentKey` will also be persisted. - Calling `kdbSet` may result in an error, if `kdbGet` wasn't called on this `KDB` instance with the same `parentKey` at least once. @@ -154,7 +160,6 @@ The flow of this operation is: Issue a warning, if a change was detected (via `KEY_FLAG_SYNC`) in a read-only backend. > **Note**: Steps 4-6 might be combined into a single procedure that deep-copies only keys from changed backends into separate KeySets per backend 7. Run the `resolver` and `prestorage` on all backends (abort immediately on error and go to e). - 8. Merge the results into a new version of `set_ks`. 9. Run the `spec/remove` hook on `set_ks` (to remove copied metakeys). 10. Split `set_ks` into individual backends again. diff --git a/doc/dev/mountpoints.md b/doc/dev/mountpoints.md index 19786b6583c..58361710d53 100644 --- a/doc/dev/mountpoints.md +++ b/doc/dev/mountpoints.md @@ -28,17 +28,16 @@ Specifically, it uses `backend` as `` and is therefore defined by the keys The _mountpoint definition_ for the backend plugin is defined by the keys below `system:/elektra/mountpoints//definition`. + + The _mountpoint config_ is defined by the keys below `system:/elektra/mountpoints//config`. It is merged into the configuration keyset of every plugin of this mountpoint (including the backend plugin). -Specifically, the mountpoint config is put into the `system:/` namespace, while the config from `system:/elektra/mountpoints//plugins//config` is put into the `user:/` namespace of the keyset passed to a plugin. - - +Specifically, the mountpoint config is put into the `user:/` namespace, while the config from `system:/elektra/mountpoints//plugins//config` is put into the `dir:/` namespace of the keyset passed to a plugin. -Additionally, there is the `config/needs` part of a plugins contract. -This part of the contract can be used by a plugin to provide configuration for another plugin it depends on. -In particular, these keys are put into the `default:/` namespace, meaning this config can be overriden by both the _mountpoint config_ in `system:/elektra/mountpoints//config` and the plugin's config in `system:/elektra/mountpoints//plugins//config`. +Additionally, the `config/needs` config from plugins' contract is moved to `system:/`. +This config is not stored in `system:/elektra/mountpoints` and instead loaded at runtime by `libelektra-kdb` during the `open` operation. - +This leaves the `default:/` namespace for plugins to add their own defaults before calling `ksLookup` to access the config keyset. > **Note**: This _mountpoint definition_ is separate from the normal configuration keyset passed to a plugin. > The backend plugin may still have such a keyset below its `system:/elektra/mountpoints//plugins//config` key. @@ -186,8 +185,6 @@ In this case it is configured for a PostgreSQL database running on `127.0.0.1:54 We also configured the `network` plugin to run in the `prestorage` phase of the `set` operation. Which phases can be used and how they must be configured of course depends on `db_backend`. -There might also be a (potentially quite complicated) part of the mountpoint definition that defines how the relational tables of the database are mapped into the KDB. - ``` system:/elektra/mountpoints/\/hosts/plugins/yajl/name (="yajl") diff --git a/doc/dev/plugins-ordering.md b/doc/dev/plugins-ordering.md index e23fec93b74..aa3d7d703ca 100644 --- a/doc/dev/plugins-ordering.md +++ b/doc/dev/plugins-ordering.md @@ -2,6 +2,10 @@ ## Outdated + + > **Warning** Many of the things described below are outdated. > Parts of this document are still valid for the `backend` plugin. diff --git a/doc/tutorials/mount.md b/doc/tutorials/mount.md index d06a665db02..bd268b9f502 100644 --- a/doc/tutorials/mount.md +++ b/doc/tutorials/mount.md @@ -1,6 +1,6 @@ # Mounting ->-storage:   + storage-->>-backend:   + backend-->>-kdb:   + kdb->>+backend: poststorage + backend->>+validation:   + validation-->>-backend:   + backend-->>-kdb:   + kdb->>kdb: merge backends + kdb-->>-user:   +``` + +```mermaid +sequenceDiagram + actor user + participant kdb as libelektra-kdb + participant backend as backend + participant other as other-backend + participant storage + participant validation + participant sa as storage-unit-a (e.g. file) + + user->>+kdb: kdbSet + kdb->>kdb: check backends initialized + kdb->>+backend: resolver + backend-->>-kdb: "storage-unit-a" + critical try to store + kdb->>+backend: prestorage + backend->>+validation:   + validation-->>-backend:   + backend-->>-kdb:   + kdb->>+backend: storage + backend->>+storage:   + storage->>+sa: write to temp + sa-->>-storage:   + storage-->>-backend:   + backend-->>-kdb:   + kdb->>backend: poststorage + kdb->>backend: precommit + backend->>sa: make changes permanent + kdb->>+backend: commit + backend-->-kdb:   + kdb->>backend: postcommit + option on failure + kdb->>backend: prerollback + kdb->>+backend: rollback + backend->>sa: revert changes + backend-->>-kdb:   + kdb->>backend: postrollback + end + kdb-->>-user:   +``` + +The diagrams above show possible sequences of phases during a `get` and a `set` operation. For each of the phases of a `get` operation `libelektra-kdb` calls the backend plugin's `elektraGet` function once. Similarly, for the phases of a `set` operation `elektraSet` is called. ---> +The backend plugin can also (optionally) delegate to other plugins. The current phase is communicated to the backend plugin (and any other plugin) via the global keyset. -The current phase can be retrieved via the `elektraPluginGetPhase` function. +It can be retrieved via the `elektraPluginGetPhase` function. ### `parentKey` diff --git a/doc/dev/kdbGet.svg b/doc/dev/kdbGet.svg deleted file mode 100644 index dd77bfbb523..00000000000 --- a/doc/dev/kdbGet.svg +++ /dev/null @@ -1 +0,0 @@ -Sequence of phase during get operationbackendsOther pluginsstorageuserlibelektra-kdbbackend-pluginother-backendstoragevalidationstorage-unit-astorage-unit-bkdbGetinitinitresolver"storage-unit-a", update neededresolver"storage-unit-b", up-to-dateprestoragestoragepoststoragemerge backends \ No newline at end of file diff --git a/doc/dev/kdbSet.svg b/doc/dev/kdbSet.svg deleted file mode 100644 index af031fc1b34..00000000000 --- a/doc/dev/kdbSet.svg +++ /dev/null @@ -1 +0,0 @@ -Sequence of phases during set operationbackendsOther pluginsstorageuserlibelektra-kdbbackend-pluginother-backendstoragevalidationstorage-unit-astorage-unit-bkdbSetresolver"storage-unit-a"resolver"storage-unit-b"prestoragestoragewrite to temppoststorageprestoragestoragewrite to temppoststorageprecommitcommitmake changes permanentpostcommitprecommitcommitmake changes permanentpostcommitmerge backendsstoragewrite to tempERRORprerollbackrollbackrevert changespostrollbackprerollbackrollbackrevert changespostrollbackrevert keysetalt[Success][Error] \ No newline at end of file