Skip to content

Latest commit

 

History

History
667 lines (510 loc) · 25.5 KB

2016-02-16_0.8.15.md

File metadata and controls

667 lines (510 loc) · 25.5 KB
Raw

0.8.15 Release

  • guid: 1ab4a560-c286-46d2-a058-1a8e7e208fe8
  • author: Markus Raab
  • pubDate: Tue, 16 Feb 2016 17:47:00 +0100
  • shortDesc: adds improved mounting, split of library & configuration validation

In case you do not yet know about it, here is an abstract about Elektra:

Elektra serves as a universal and secure framework to access configuration parameters in a global, hierarchical key database. Elektra provides a mature, consistent and easily comprehensible API. Its modularity effectively avoids code duplication across applications and tools regarding configuration tasks. Elektra abstracts from cross-platform-related issues and allows applications to be aware of other applications' configurations, leveraging easy application integration.

See https://libelektra.org

Overview

This is one of the largest release up to now. It includes many user-visible improvements. Some highlights:

  • Mounting is vastly improved: think about Debian's "dpkg" to "apt"-like functionality
  • In previous releases you could already specify the structure of the configuration. Now you can also automatically enforce this property.
  • We split the shared library libelektra into smaller parts. Now users can link against the parts of the library they need.
  • As always, the ABI and API is fully forward-compatible.
  • The release contains improvements in the bootstrapping process.
  • We improved the ini, rename and crypto plugins.
  • The tool kdb now supports bookmarks and profiles.
  • The new tool kdb editor allows you to edit KDB configuration in your favorite text editor.
  • We are glad of the new packages for Debian, Arch Linux and OpenWRT.

The same text as follows is also available here as html and here on github

Global Mount

Sometimes you simply want some functionality for the whole key database. For example, you want to enable logging or notification of configuration changes. In previous versions, you had to change every mountpoint individually. Even more problematic, every mountpoint created its individual logs and notifications, without any way for someone to know if an application has issued its last log/notification.

These problems are now solved by global plugins. The same plugins are reused for this purpose. Also the mounting works nearly in the same way, you only have to omit the configuration file and the mountpoint:

kdb global-mount syslog journald dbus

Voilà, from now on every configuration change gets logged to syslog and journald. Additionally, every application gets notified via dbus.

If you want it more verbose for debugging, you can easily do so by:

kdb global-mount logchange counter

Which gives you detailed information to standard output which keys were changed/edited/deleted. Additionally, Elektra counts how often the counter plugin is invoked.

The underlying work for the global plugins, i.e. hooks in the core libraries and the list plugin that allows us to use many plugins without bloating the core was done by Thomas Waser.

It was already possible in earlier versions of Elektra to specify the configuration of your program. Until now, this specification could be mainly used to to generate code as described here. This release adds two more interesting options:

  1. the spec plugin, and
  2. the spec mount.

Spec Plugin

The most important global plugin that is now newly introduced with this release (thanks to Thomas Waser) is the spec plugin. By default it will be added for every global-mount. So for a new installation make sure you executed at least once:

kdb global-mount

The spec plugin is a global plugin that copies metadata from the spec-namespace to other namespaces. That means, it reads the specification, and makes sure that the configuration conforms to it. The actual validation is done by the many validation plugins already present.

Lets start by saying a key is a long and must have at least the value 10:

kdb setmeta spec/example/longkey check/type long

Then we can create a key in a different namespace and see if the spec plugin applies the metadata correctly:

kdb set /example/longkey 25
kdb lsmeta /example/longkey

Should now at least print check/type. By itself, this is useful for documentation of keys. For example, the application developer says:

kdb setmeta spec/example/longkey description "Do not change"
kdb setmeta spec/example/longkey example 30

The user can retrieve this documentation by:

kdb getmeta /example/longkey description

But we want check/type to be not only a documentation, but also enforced.

Spec Mount

Using kdb setmeta extensively and always looking out that all plugins are mounted correctly is error-prone. So instead, one can directly mount the plugins as specified. For the example above one simply needs:

kdb setmeta spec/example mountpoint example.ecf
kdb spec-mount /example

Now, when we try to modify /example/longkey it will be validated:

kdb set /example/longkey a
#> Error (#52) [...] long [not] matched [...] a

Based on the present metadata, the correct plugins (in this case type because of the metadata check/type) will be loaded.

We can also create a whole specification file, first mount the specification and then the mountpoint according the specification, e.g when we have battery.ini in the folder $(dirname $(kdb file spec)) with following content:

[]
mountpoint = battery.ini
infos/plugins = ini

[level]
check/enum = 'critical', 'low', 'high', 'full'

Then we can use:

# mount the file itself:
kdb mount battery.ini spec/example/battery ni
# make sure all plugins are present (e.g. enum for check/enum):
kdb spec-mount /example/battery

Now lets verify if it worked:

kdb lsmeta /example/battery/level
# we see it has a check/enum
kdb getmeta /example/battery/level check/enum
# now we know allowed values
kdb set /example/battery/level low
# success, low is ok!
kdb set /example/battery/level wrong
# fails, not one of the allowed values!

The main idea of the spec-mount is: search a plugin for every specification (metadata) found in the spec-namespace.

General Mount Improvements

In earlier versions kdb mount failed when plugin dependencies were not satisfied. Now dependencies will automatically be fulfilled, e.g.

kdb mount /etc/modules system/modules line

In earlier versions you would have got an error because of the missing null plugin. Now it simply adds the needed plugins.

The plugins given in the command-line used to be real plugins. Now also so called providers are accepted.

To make providers useful we need to actually know which plugin is the best candidate. To get the information we added a infos/status clause in the contract. In this clause the plugin developer adds many details how well the plugin is tested, reviewed, documented, maintained and so on. Based on this information, the best suited plugin will be chosen. For example, you now can use:

kdb mount /etc/security/limits.conf system/limits augeas \
	lens=Limits.lns logging

And the best suitable logger will automatically be chosen.

The configuration variable /sw/kdb/current/plugins now allows us to pass plugin configuration with the same syntax as the plugin specification passed on the commandline. A subtle difference is that thus the shell-splitting of arguments is missing, it is not possible to include whitespaces in the plugin configuration that way.

Now it is also possible to include the same plugin multiple times and also give them individual names. This feature is essential for script-based plugins, e.g. you now might add:

kdb mount file.x /example/mountpoint \
	lua#resolver script=resolver.lua \
	lua#storage script=storage.lua

Furthermore, kdb mount now supports recommendations, which can be enabled with --with-recommends. E.g. supplied to the mount command using augeas above, comments will automatically transformed to metadata to avoid cluttering of the real configuration.

Library Split

Up to now, Elektra consisted only of a single shared library, libelektra.so. Not all symbols in it were relevant to end users, for example, some were only needed by plugins. Others were only proposed and not yet part of the stable API. And finally, other symbols were not needed in some situations, e.g. the plugins do not need the kdb interface.

Thus, we split libelektra.so, so that only coherent parts are in the same library:

  • libelektra-core.so only contains the KeySet data structure and module loading. Every binary using Elektra should link against it.
  • libelektra-kdb.so contains the missing KDB symbols. Together with the core they contain everything declared in kdb.h. Michael Zehender plans to have multiple variants of libelektra-kdb.so that use different kinds of concurrency. Headerfile: <kdb.h>
  • libelektra-ease.so adds functionality missing in core to make the life for C programmers easier. New headerfile: kdbease.h
  • libelektra-proposal.so adds functionality proposed for core. It directly uses internal structures of core, thus they always need to have exactly the same version. Headerfile: kdbproposal.h

The source code is now better organized by the introduction of a libs folder. The explanation of every library can be found in /src/libs/.

The rationale of the library split is documented here. Shortly put, it was needed for further evolution and allowing us to grow and enhance without getting a fat core.

Thanks to Manuel Mausz for fixing many bugs related to the library split and also adapting all the bindings for it.

Benchmark

To show that the split does not make Elektra slower, Mihael Pranjić made a small benchmark. The real time of benchmarks/large and revision 634ad924109d3cf5d9f83c33aacfdd341b8de17a

  1. kdb-static: Median :0.8800
  2. kdb-full: Median :0.8700
  3. kdb: Median :0.8700

So it seems that the split does not influence the time of running elektrified processes.

Times were measured with time(1) and the median is calculated for 21 runs of benchmarks/large. This was done using scripts/benchmark_libsplit.sh

Compatibility

As always, the ABI and API is fully forward-compatible, i.e. programs compiled against an older 0.8 version of Elektra will continue to work (ABI) and you will be able to recompile every program without errors (API).

We added keyGetNamespace which allows us to handle all namespaces in a switch and to iterate all namespaces. This is essential, when a new namespace is added, thus then the compiler can warn you about every part where the new namespace is not yet considered. We currently do not plan to add a new namespace, but better be safe than sorry.

The internal function keyCompare now also detects any metadata change.

libtools was nearly rewritten. Even though it is mostly API compatible (you should not use the low-level Backend anymore but instead use the BackendBuilder), it is certainly not ABI compatible. If you have an undefined symbol: _ZN3kdb5tools7Backend9addPluginESsNS_6KeySetE you need to recompile your tool. Even the merging part has ABI incompatibility (different size of _ZTVN3kdb5tools7merging14NewKeyStrategyE). Unfortunately, we still cannot guarantee compatibility in libtools, further changes are planned (e.g. implementing mounting of lazy plugins).

The python(2) and lua interfaces changed, an additional argument (the plugin configuration) is passed to open.

The INI plugin was rewritten, so many options changed in incompatible ways.

The default format plugin (e.g. for import/export) is no longer hard coded to be dump. Instead KDB_DEFAULT_STORAGE will be used. To get KDB_DEFAULT_STORAGE you can use the constants plugin:

kdb mount-info
kdb get system/info/elektra/constants/cmake/KDB_DEFAULT_STORAGE

Thanks to Manuel Mausz plugins do no longer export any method other than elektraPluginSymbol. It now will fail if you directly linked against plugins and did not correctly use their public interface. Please use the module loading and access functions via the contract.

The CMake and Pkgconfig Files now only link against elektra-core and elektra-kdb. If you used some symbols not present in kdb.h, your application might not work anymore.

libelektra.so is still present for compatibility reasons. It should not be used for new applications. Some unimportant parts, however, moved to the "sugar" libraries. E.g. the symbol elektraKeyCutNamePart is no longer part of libelektra.so.

Bootstrapping

When you use kdbOpen in Elektra as first step it reads mountpoint configuration from /elektra. This step is the only hard coded one, and all other places of the KDB's tree can be customized as desired via mounting.

The bootstrapping now changed, so that /elektra is not part of the default configuration default.ecf (or as configured with KDB_DB_FILE), but instead we use elektra.ecf (or as configured with KDB_DB_INIT). This behaviour solves the problem that default.ecf was read twice (and even differently, once for /elektra and once for /).

To be fully compatible with previous mountpoints, Elektra will still read mountpoints from default.ecf as long as elektra.ecf is not present.

To migrate the mountpoints to the new method, simply use:

kdb upgrade-bootstrap

which basically exports system/elektra/mountpoints, then does kdb rm -r system/elektra/mountpoints so that default.ecf will be without an mountpoint and thus the compatibility mode does not apply anymore. As last step it will import again what it exported before.

Details are here

Plugins

We already highlighted the new spec plugin, but also other plugins were improved at many places. Small other changes are:

  • Conditionals now also support assign/condition syntax, thanks to Thomas Waser
  • Lua and Python are not tagged experimental anymore. They now correctly add their configuration to the open-call.
  • The plugin yajl (the json parser and generator) now also accepts the type string and yields better warnings on wrong types.
  • Improved error message in the type plugin.

Larger changes were done in the following plugins:

INI

The INI plugin was rewritten and a huge effort was taken so that it fully-roundtrips and additionally preserves all comments and ordering. Currently, it is brand new. It is planned that it will replace dump in the future.

Currently is has some minor problems when used as KDB_DEFAULT_STORAGE. You can avoid most problems by mounting a different file into root:

kdb mount root.ini /

Read here about the details.

A huge thanks to Thomas Waser.

Rename

Thanks to Thomas Waser rename is now fixed for cascading keys. Additionally, it does not change the sync status of the keys so that notification plugins work properly afterwards.

Crypto

The crypto plugin is a facility for securing sensitive Keys by symmetric encryption of the value. It acts as a filter plugin and it will only operate on Keys, which have the meta-key „crypto/encrypt“ set.

The key derivation is still work-in-progress, so the plugin does not work with kdb yet. A planned method for key derivation is to utilize the gpg-agent.

For now the plugin requires the following Keys within the plugin configuration in order to work properly:

  1. /crypto/key - the cryptographic key (binary 256 bit long)
  2. /crypto/iv - the initialization vector (binary 128 bit long)

Please note that this method of key input is for testing purposes only and should never be used in a productive environment!

Thanks to Peter Nirschl, especially the patience for also supporting different platforms where dependencies need to be handled differently.

KDB

A technical preview of a new tool was added: kdb editor allows you to edit any part of Elektra’s configuration with any editor and any syntax. It uses 3-way merging and other stable technology, but it currently does not provides a way to abort editing. So you only should use it with care.

The tool kdb list now searches in the rpath for libraries and thus will also find plugins not present at compile time (using glob). Additionally, it sorts the plugins by infos/status score, which can also be printed with -v. The last plugins printed are the ones ranked highest.

When running as root, kdb will now use the system namespace when writing configuration to cascading key names.

Long paths are cumbersome to enter in the CLI. Thus one can define bookmarks now. Bookmarks are key names that start with +. They are only recognized by the kdb tool or tools that explicitly have support for it. Applications should not depend on the presence of a bookmark. For example, if you set the bookmark kdb:

kdb set user/sw/elektra/kdb/#0/current/bookmarks
kdb set user/sw/elektra/kdb/#0/current/bookmarks/kdb user/sw/elektra/kdb/#0/current

You are able to use:

kdb ls +kdb/bookmarks
kdb set +kdb/format ini

The kdb tool got much more robust when the initial configuration is broken, no man page viewer is present or Elektra was installed wrongly.

The --help usage is unified and improved.

The new key naming conventions are now used for configuration of the kdb-tool itself: /sw/elektra/kdb/#0/%/ and /sw/elektra/kdb/#0/current/ are additionally read. The option -p/--profile is now supported for every command, it allows to fetch configuration from /sw/elektra/kdb/#0/<profile>/ instead of current. KDB is more robust when it cannot fetch its own configuration due to KDB errors.

Coding Guidelines

Thanks to Kurt Micheli the code guidelines are now properly documented. Thanks to René Schwaiger we also provide a style file for clang-format.

Furthermore we unified and fixed the source:

  • only use @ for doxygen commands
  • always use elektra* functions for allocation
  • added a file header for every file

C++11 migration

Since we now only use C++11, we applied clang-modernize which simplified many loops and replaced many 0 to nullptr. Additionally, we added override and default at many places.

We removed all places where we had ifdefs to use auto_ptr if no modern smart pointer is available.

Because of these changes there is no easy way to compile Elektra without C++11 support, except by avoiding the C++ parts all together.

The C++ KeySet now also supports a get to retrieve whole containers at once. By specializing KeySetTypeWrapper you can support your own containers. Currently only map<string, T> is supported (T is any type supported by Key::get).

If you haven't removed it from your flags already, there is no use anymore to set ENABLE_CXX11.

Documentation

The documentation was improved vastly. Most thanks to Kurt Micheli who did a lot of editing and fixed many places throughout the documentation Also thanks to Michael Zehender who added two paragraphs in the main README.md.

Key names of applications should be called /sw/org/app/#0/current, where current is the default profile (non given). org and app is supposed to not contain / and be completely lowercase. Key names are documented here. See also here. The main reason for long paths is the provided flexibility in the future (e.g. to use profiles and have a compatible path for new major versions of configuration). By using bookmarks, users should not be confronted by it too often.

  • many man pages were improved
  • many typos were fixed, thanks to Pino Toscano!
  • Fix documentation for kdb list, thanks to Christian Berrer
  • Compilation variants are explained better, thanks to Peter Nirschl for pointing out what was missing
  • document ronn as dependency, thanks to Michael Zehender
  • fix broken links, thanks to Daniel Bugl

Thanks to Kurt Micheli, developers are now warned during compilation on broken links in Markdown. The output format is the same as for gcc. Additionally, the markdown documentation of plugins now get a proper title in the pdf and html output of doxygen.

Qt-Gui 0.0.10

Raffael Pancheri again updated qt-gui with many nice improvements:

  • update existing nodes in case the config was changed outside the gui
  • safely update tree
  • add update signal to metadata
  • fix crash when closing the GUI
  • move Actions in separate file to make main.qml less clustered
  • plugins do not output messages at startup
  • BackendBuilder is now used, which automatically takes cares of the correct ordering of plugins
  • Qt 5.4 compatibility is still ensured
  • C++11 is now used in Qt-Gui, too

Packaging and Build System

Elektra 0.8.14 now in Debian with qt-gui, man pages, thanks to Pino Toscano read more here

Thanks to Gustavo Alvarez for updating and splitting the packages on Arch Linux!

Thanks to Harald Geyer, Elektra is now packaged for OpenWRT. We fixed a number of cross-compilation issues and now officially support building against musl libc, with one minor limitation: RPATH works differently on musl so you need to install all plugins directly in /usr/lib/ or set LD_LIBRARY_PATH. Report any bugs in Harald's OpenWRT packaging issue tracker.

  • export errors/symbols are now called elektra-export-symbols and elektra-export-symbols and can be installed using INSTALL_BUILD_TOOLS (by default off). This is needed for cross-compilation. Thanks to Harald Geyer for reporting.
  • some header files are renamed because they endlessly included themselves if the header files were found in wrong order. Thanks to Harald Geyer for reporting.
  • fixed compilation when built on more than 20 cores with >= -j15, thanks to Gustavo Alvarez for reporting and Manuel Mausz for analyzing
  • lua 5.1 now works too (except for iterators), thanks to Harald Geyer for reporting. thanks to Manuel Mausz for adding a new FindLua.cmake
  • pdf builds do not fail due to half written files, reported by René Schwaiger and fixed by Kurt Micheli

Read about other packages here.

Fixes and Improvements

  • 3 way merge now properly deals with binary data, thanks to Felix Berlakovich
  • getenv: fix wrapping on powerpc, thanks to Pino Toscano
  • markdownlinkconverter: fix char/int mismatch, thanks to Pino Toscano
  • wresolver: use KDB_MAX_PATH_LENGTH instead of PATH_MAX, thanks to Pino Toscano
  • Cleaning up #ifdefs that break statements, thanks to Romero Malaquias
  • Daniel Bugl tested the INI plugin
  • cmake list_filter was broken because of different behaviour in cmake_parse_arguments, thanks to Christian Berrer for reporting
  • g++5.3 is now supported
  • gtest does not link against pthread if not needed
  • test cases that are built with BUILD_SHARED also successfully work
  • kdb list works when libs are in same path as plugins, thanks to Harald Geyer for reporting
  • fix macOS issues, thanks to Peter Nirschl, René Schwaiger and Mihael Pranjic
  • fix resolver-baseflag docu, thanks to Harald Geyer for reporting
  • do not create wrong directories called ( and ) in source, thanks to René Schwaiger
  • fix cmake for systems where iconv is not part of libc, thanks to Michael Zehender and Peter Kümmel (for FindIconv.cmake)
  • fix segfault in libgetenv if root keys are present
  • lua: fix Key:tostring(), thanks to Manuel Mausz
  • add list of supported bindings, thanks to Manuel Mausz

Get It!

You can download the release from here and now also here on github

  • name: elektra-0.8.15.tar.gz
  • size: 2338297
  • md5sum: 33ec1e5982fb7fbd8893bf7b579b80f0
  • sha1: 6b1fdd5aa5aaad6ba377b4bb5ef437e0c85319ff
  • sha256: 6a406986cecb8d4a44485ced118ee803bc039b0824b72298e123b4dd47eb0b22
  • sha512: 86a408dd546b33e3b437f92f415de7aee6a235189f9eab0762b3f44ab4c453ee369a53de10a9f5b0df1b446460b12c57c6b8b77c282648ec2a49f2328d9af13d

This release tarball now is also available signed by me using gpg

already built API-Docu can be found here

Stay tuned!

Subscribe to the RSS feed to always get the release notifications.

For any questions and comments, please contact the Mailing List the issue tracker on github or by mail elektra@markus-raab.org.

Permalink to this NEWS entry

For more information, see https://libelektra.org

Best regards, Markus