Update new-backend docs

doc/decisions/0_drafts/commit_function.md

@@ -0,0 +1,63 @@
+# Commit Function
+
+## Problem
+
+When `kdbSet()` is called, plugins implementing the commit role, need to internally track their state to distinguish between carrying out that role and carrying out potential other roles (commit and setresolver for the resolver plugin, for example).
+This limits the possibilities of plugin reuse and the ways plugins can be combined.
+
+More generally, this problem applies to all plugins with one function that is called multiple times for different reasons.
+
+Between `libelektra-kdb` and a backend plugins there is a [contract](../../dev/backend-plugins.md) that uses "phases" for this.
+
+With the introduction of the new backend, a new `kdbCommit` function was introduced to `struct _Plugin` (matching `kdbError`).
+We also added a `kdbInit`.
+Therefore, some phases of `kdbGet`/`kdbSet` now use separate functions, but not all of them do.
+This partial use of separate functions can be weird and unintuitive.
+
+## Constraints
+
+- Plugins should not have internal state that tracks how often a function has been called to determine what the function should do next.
+
+## Assumptions
+
+## Considered Alternatives
+
+1. Keep the current "partial separation" approach.
+
+   So far, there has been no technical reason why some phases use separate functions and others do not.
+   At least there has been no explanation, why separate `kdbCommit`/`kdbError` would be better than having some other way of knowing the phase.
+
+2. Fully separate the functions with one function for each of the phases `libelektra-kdb` knows.
+
+   This would introduce a lot of symbols, which would bloat `struct _Plugin`.
+   The bloat could be avoided by choosing the approach in the ["Plugin Contract Function"](plugin_contract_function.md) decision that removes the function pointers from `struct _Plugin`.
+
+3. Only use `kdbOpen`, `kdbGet`, `kdbSet` and `kdbClose` functions matching the ones in `libelektra-kdb`.
+   Communicate the current phase via the `elektraPluginGetPhase` function.
+
+   Matching the functions from `libelektra-kdb` makes it clear, which plugin function gets used for which `kdb*` call.
+   The API does not make it obvious that phases exist.
+   Plugins that handle multiple phases must call `elektraPluginGetPhase` to know what phase is being executed.
+   Also using a single `elektraPluginGetPhase` function means all phases must share a single type, even if not all phases are shared between `kdbGet` and `kdbSet`.
+
+4. Like 3, but instead of using a `elektraPluginGetPhase` function, directly pass a `ElektraGetPhase phase`/`ElektraSetPhase phase` parameter.
+
+   This API makes it very clear that phases exist and that the plugin might need to handle them.
+   It also allows using separate types for the phases of `kdbGet` and `kdbSet`.
+   However, a big downside of this approach is that it adds an extra parameter to the API that will be unused by many plugins.
+
+   This API also allows plugins to call other plugins to execute a different phase.
+   This could be a good thing or a bad thing.
+
+## Decision
+
+## Rationale
+
+## Implications
+
+## Related Decisions
+
+- [Plugin Contract Function](plugin_contract_function.md)
+- [Plugin Struct](plugin_struct.md)
+
+## Notes

doc/decisions/0_drafts/plugin_contract_function.md

@@ -0,0 +1,98 @@
+# Plugin Contract Function
+
+## Problem
+
+Contract for a plugin is currently retrieved by calling the plugin's `get` function with a special `parentKey`.
+This makes the `get` function awkward to write.
+You must always first check for the special `parentKey`.
+It also makes the `get` function mandatory, even if the plugin doesn't implement any real `get` functionality.
+
+Additionally, the actual public API exported from shared library to make it a plugin is the `ELEKTRA_PLUGIN_EXPORT` function.
+This function always calls `elektraPluginExport`, which allocates and half initializes a `struct _Plugin`.
+Specifically, only the `name` field and the `kdb*` function pointers are set.
+This same information is also provided via the contract.
+The function pointers are all listed in `system:/elektra/modules/<plugin>/exports/*` and the `<plugin>` part of the name is the `name` of the plugin.
+
+## Constraints
+
+## Assumptions
+
+## Considered Alternatives
+
+1. Create a separate `kdbContract` that returns the contract of a plugin.
+
+   ```c
+   KeySet * ELEKTRA_PLUGIN_FUNCTION(contract) ();
+
+   // OR
+
+   void ELEKTRA_PLUGIN_FUNCTION(contract) (KeySet * ks);
+   ```
+
+   This solves the problem of the awkward `get` function, but it does not address the issue of the duplicate information in `ELEKTRA_PLUGIN_EXPORT`.
+
+2. Change the `ELEKTRA_PLUGIN_EXPORT` to return the contract of the plugin instead of creating half a `struct _Plugin`.
+
+   ```c
+   KeySet * ELEKTRA_PLUGIN_EXPORT ();
+
+   // OR
+
+   void ELEKTRA_PLUGIN_EXPORT (KeySet * ks);
+   ```
+
+   This also removes the duplicate information inside the plugin.
+
+3. (combined with 2) Change what the `KeySet * modules` stores.
+
+   Currently, `modules` stores data dependent on how modules are loaded.
+   When dynamic linking is used, the keys store the handle from `dlopen` and a pointer to the `ELEKTRA_PLUGIN_EXPORT` function.
+   With static linking, the keys store a custom struct with function pointers.
+
+   Instead, we would now store the contracts directly in `KeySet * modules`.
+
+   The implementations of `elektraModulesLoad` would simply find the correct function and call it.
+
+   ```c
+   typedef void (*elektraPluginExportFn) (KeySet * ks);
+
+   void elektraModulesLoad (KeySet * modules, const char * name, Key * errorKey)
+   {
+        Key * moduleKey = keyNew ("system:/elektra/modules", KEY_END);
+   keyAddBaseName (moduleKey, name);
+   if (ksLookup (modules, moduleKey, 0) != NULL)
+   {
+                // already loaded
+   	return;
+   }
+
+        elektraPluginExportFn exportFn;
+        // [...] find exportFn for plugin
+
+        exportFn (modules); // inserts the contrat with all the exported functions
+
+        if (ksLookup (modules, moduleKey, 0) == NULL)
+   {
+                ksAppendKey (modules, moduleKey);
+   }
+        else
+        {
+                keyDel (moduleKey);
+        }
+   }
+   ```
+
+   > **Note** to avoid allocated and deleting a temporary `KeySet *` the `void ELEKTRA_PLUGIN_EXPORT (KeySet * ks)` API works best here.
+
+## Decision
+
+## Rationale
+
+## Implications
+
+## Related Decisions
+
+- [Commit Function](commit_function.md)
+- [Plugin Struct](plugin_struct.md)
+
+## Notes

doc/decisions/0_drafts/plugin_struct.md

@@ -0,0 +1,86 @@
+# Plugin Struct
+
+## Problem
+
+`struct _Plugin` contains some unnecessary data
+
+## Constraints
+
+## Assumptions
+
+## Considered Alternatives
+
+> **Note**: Not all of these are alternatives, most of them could be combined.
+
+1. Replace `KeySet * global` and `KeySet * modules` with `KDB * kdb`.
+
+   This would require changing the `elektraPluginOpen` API to take a `KDB * kdb`, or a `Plugin * existing` from which we extract the `KDB *` (the first plugin could use a stack allocated `Plugin` with just `KDB *` set).
+
+2. Replace all the `kdb*` function pointers and `char * name` with a `KeySet * contract`.
+
+   The contract also contains the function pointers, so this avoids duplication and should reduce memory usage.
+   This could potentially affect runtime performance, because instead of having a pointer directly, we need to first do a lookup in a keyset.
+
+3. Remove `size_t refcounter`.
+
+   Plugins are never shared between mountpoints.
+   There is also no way of increasing `refcounter` via the API.
+
+4. (Based on all above) Change how `struct _Plugin` is used.
+
+   This is the most comprehensive change, it combines all the above to remove as much duplication as possible.
+   It consists of these parts:
+
+   - Create a `struct _PluginData` that will be used in `struct _BackendData` instead of `struct _Plugin`.
+
+     ```c
+     struct _PluginData {
+        KeySet * config; // config of plugin
+        void * data; // private data of plugin
+        char * name; // name of the plugin
+     };
+     ```
+
+     The `name` is needed to get the correct functions from `kdb->contracts` (see below).
+
+   - Change `struct _Plugin` to the following and use stack allocated instances created when calling a plugin function.
+
+     ```c
+     struct _Plugin {
+       KeySet * config; // copied from struct _PluginData
+       void * data; // copied from struct _PluginData
+       KDB * kdb; // pointer to KDB instance calling this plugin right now, may be copied from other plugin calling this plugin
+     };
+     ```
+
+   - Add a `KeySet * contracts` to `struct _KDB`.
+     This contains all the contracts for all the plugins currently loaded.
+
+     To call a plugin function you need an existing `Plugin *` from which the `KDB * kdb` will be taken.
+     Then the function will be looked up in `kdb->contracts`.
+     Inside `libelektra-kdb` the existing `Plugin *` is faked with a stack allocated `struct _Plugin` containing only a `KDB *`.
+
+     > **Note** If the changes to `KeySet * modules` from the ["Plugin Contract Function"](plugin_contract_function.md) decision are implemented, then `kdb->contracts` becomes `kdb->modules`.
+
+5. Introduce a new `KeySet * shared` for sharing data among plugins of a single backend.
+
+   This is similar to `KeySet * global`, but not shared between different mountpoints.
+   The `KeySet * shared` would be a copy of the one in `struct _BackendData` (also new field), which owns the `KeySet`.
+
+   This solves the issue that `libelektra-kdb` must currently use `KeySet * global` to communicate extra data like the plugins belonging to a backend (for access in backend plugins).
+
+   When combined with the stack allocated `struct _Plugin` from option 4, this would not affect memory usage as much as otherwise.
+   In that case, it would only be one `struct _KeySet` per backend, when otherwise we would also add one pointer for every loaded plugin instance.
+
+## Decision
+
+## Rationale
+
+## Implications
+
+## Related Decisions
+
+- [Plugin Contract Function](plugin_contract_function.md)
+- [Commit Function](commit_function.md)
+
+## Notes

doc/decisions/4_partially_implemented/hooks.md

@@ -53,7 +53,7 @@ These hooks are not shared, so no `list` plugin is needed.
 
 Installed plugins will be used.
 
-In the beginning, we'll hard code the names of the plugins. For changing those plugins symlinks will have to be used.
+We'll hard code the names of the plugins. For changing those plugins symlinks will have to be used.
 
 ## Rationale
 

doc/decisions/5_implemented/backend_plugin.md

@@ -2,35 +2,33 @@
 
 ## Problem
 
-- Backends store plugins in arrays which have a fixed number of slots for each plugin role. The number of plugins which can be assigned is limited,
-  making it easy to reach the limit if many plugins are in use.
-- As structs, backends are separate from the plugin interface and integrated into the core of Elektra. This makes it difficult to perform operations
-  such as nesting plugins, or to develop other implementations for backends.
+- Old backends (before Elektra 0.9.12) store plugins in arrays which have a fixed number of slots for each plugin role.
+  The number of plugins which can be assigned is limited, making it easy to reach the limit if many plugins are in use.
+- As structs, old backends are separate from the plugin interface and integrated into the core of Elektra.
+  This makes it difficult to perform operations such as nesting plugins, or to develop other implementations for backends.
 
 ## Constraints
 
-- It should be possible for all existing plugins to run normally
+- All existing plugins, except [hooks](../4_partially_implemented/hooks.md), should continue working as before.
 
 ## Assumptions
 
 ## Considered Alternatives
 
 - Multiple storage plugins within a single backend
-- Plugin containing more plugin slots
+- Improve backend to contain more plugin slots
+- Making number of plugins per slot unlimited
 
 ## Decision
 
-- The current backend implementation will be redeveloped into a backend plugin. That way, the core of Elektra will only access backends through
-  the standard plugin interface.
-- The new backend plugin will support an unlimited number in any position where more than one plugin is sensible.
-  e.g., unlimited plugins in `poststorage`, but only a single one in `storage`
+- The current backend implementation was redeveloped into a backend plugin.
+  The core of Elektra accesses backends through the standard plugin interface.
+- The new backend plugin supports an unlimited number of plugins in any position where more than one plugin is sensible, e.g., unlimited plugins in `poststorage`, but only a single one in `storage` and `resolver`
 
 ## Rationale
 
-- Making backends plugins themselves detaches their implementation from the core of Elektra, making it possible to develop new kinds of backends
-  without major changes to the core itself.
-- As plugins, backends can contain further backends, making it possible to nest plugins and enabling new kinds of plugin combinations such as
-  fallback storage options.
+- Making backends also plugins detaches their implementation from the core of Elektra, making it possible to develop new kinds of backends without major changes to the core itself.
+- As plugins, backends can contain further backends, making it possible to nest plugins and enabling new kinds of plugin combinations such as fallback storage options.
 
 ## Implications
 

doc/dev/algorithm.md

@@ -7,7 +7,7 @@ You might want to read [about architecture](architecture.md) and [data structure
 <!-- TODO [new_backend]: Update the text below using the docs listed in the warning. -->
 
 > **Warning** Many of the things described below (especially about `KDB` and the `kdb*` functions) are outdated.
-> See [`kdb-operations.md`](kdb-operations.md) and [`kdb-contracts.md`](kdb-contracts.md) for more up-to-date information.
+> See [`kdb-operations.md`](kdb-operations.md) and [`kdb-contracts.md`](kdb-contracts.md) for up-to-date information.
 
 ## Introduction