Skip to content

Latest commit

 

History

History
441 lines (317 loc) · 21.8 KB

2018-11-18_0.8.25.md

File metadata and controls

441 lines (317 loc) · 21.8 KB
Raw

0.8.25 Release

We are proud to present Elektra 0.8.25.

What is Elektra?

Elektra serves as a universal and secure framework to access configuration settings in a global, hierarchical key database. For more information, visit https://libelektra.org.

For a small demo see here:

asciicast

You can also read the news on our website

Metadata

  • guid: 472392e0-cc4f-4826-a0a9-2764d90c5f89
  • author: Markus Raab
  • pubDate: Sun, 18 Nov 2018 21:24:34 +0100
  • shortDesc: faster storage and lookup

Highlight

This release is dedicated to drastically improve the speed of Elektra. Two non-trivial features facilitate most of the improvement:

  • mmap storage for very fast retrieval of configuration
  • Hybrid Search Algorithm for ksLookup (...) for very fast access of configuration

mmap storage

We added a new, binary and fast storage plugin called mmapstorage. It leverages the mmap() syscall and supports full Elektra semantics. We provide two compile variants: mmapstorage and mmapstorage_crc. The mmapstorage_crc variant enables CRC32 checksums for critical data, while the mmapstorage variant omits the checksum for maximum performance.

We ran a synthetic benchmark with 257 iterations using 40k keys in a keyset, and compared the performance to the dump storage plugin.

Median write time in microseconds:

Plugin Time
dump 71079
mmapstorage 2964
mmapstorage_crc 7644

Median read time in microseconds:

Plugin Time
dump 82737
mmapstorage 1145
mmapstorage_crc 5744

In our benchmark, the mmapstorage plugin writes more than 23x faster, and reads more than 72x faster than the dump storage plugin.

For this release the plugin is marked as experimental, even though it is already used as default storage plugin in a build job on our build server.

Thanks to Mihael Pranjić for this improvement.

Hybrid Search Algorithm for ksLookup (...)

The hybrid search algorithm is now implemented, this concludes the extension of the ksLookup (...) search with the order preserving minimal perfect hash map (OPMPHM). The hybrid search combines the best properties of the binary search and the OPMPHM. The hybrid search decides dynamically which search algorithm to use.

Because of the automatic decision, usually nothing needs to be done by API users to take advantage of this improvement. Advanced API user, however, can overrule the hybrid search by passing KDB_O_OPMPHM or KDB_O_BINSEARCH to ksLookup (...). The constants are defined in kdbproposal.h. For low-memory systems the building of the hash map can be disabled altogether at build-time by disabling the CMake variable ENABLE_OPTIMIZATIONS (by default enabled now).

The implemented randomized OPMPHM algorithm is in 99.5% of the measured random cases optimal. However the randomization property of the algorithm leaves an uncertainty.

The results made with random cases had shown that the hybrid search is, except for small keyset sizes, almost always faster compared to the standalone binary search. The performance increase strongly depended on the measured hardware. In the random cases where the hybrid search is faster, on average ~8.53% to ~20.92% of time was saved. The implemented hybrid search works only above a keyset size of 599 to exclude the small keyset sizes.

Thanks to Kurt Micheli for this improvement.

Plugins

The following section lists news about the plugins we updated in this release.

Directory Value

We improved the performance of the directoryvalue plugin. (René Schwaiger) This plugin is used for configuration file formats that do not support that directories contain values, like it is the case in JSON. A program manipulating a 13 MB JSON file which first did not succeed within 10 hours is now finished in 44 seconds.

Process

There is a new, experimental plugin called process. This plugin utilizes the pluginprocess library in order to execute arbitrary other plugins in an own process, acting as a proxy itself. Therefore it is not required to explicitly change a plugin's implementation if it shall be executed in an own process. This plugin is not completely finished yet, as currently there is no way for it to mimic the proxied plugin's contract in Elektra. It can be used with simple plugins like dump however, check the limitations in the readme for more details. (Armin Wurzinger)

FSTab

The detection of the mntent functions now also works correctly, if you use the compiler switch -Werror. (René Schwaiger)

passwd

We fixed an issue with the passwd plugin not properly setting compile flags. This resolves a problem with undefined functions when building with musl. (Lukas Winkler)

gpgme

The experimental gpgme plugin was brought into existence to provide cryptographic functions using GnuGP via the libgpgme library. (Peter Nirschl)

network

The network plugin now also allows for non-numerical hosts (i.e. "localhost") to be set and tries to resolve it via DNS. (Michael Zronek)

YAMBi

This new plugin parses a subset of YAML using a parser generated by Bison. (René Schwaiger)

YAML CPP

The build system now disables the plugin automatically, if you use a GCC compiler (6.x or earlier) and enable the option ENABLE_ASAN. We updated the behavior, since otherwise the plugin will report memory leaks at runtime. (René Schwaiger)

Yan LR

  • The plugin does not modify the (original) parent key. As a consequence, setting values at the root of a mountpoint:

    sudo kdb mount config.yaml user/tests/yambi yambi
kdb set user/tests/yanlr 'Mount Point Value'
kdb get user/tests/yanlr
#> Mount Point Value

    now works correctly. (René Schwaiger)

  • We now use C++ code to test the plugin. (René Schwaiger)

  • The CMake code of the plugin now also recognizes antlr as ANTLR executable, if antlr4 is not available. (René Schwaiger)

  • The build system now disables the unit test for the plugin, if you use GCC (6.x or earlier) to translate Elektra. We introduced this behavior, since the code generated by ANTLR (YAML.h) seems to contain a double free that causes a segmentation fault on systems that use the GNU C library. (René Schwaiger)

  • The build system now disables the plugin automatically, if you use a GCC compiler (6.x or earlier) and enable the option ENABLE_ASAN. (René Schwaiger)

YAwn

This new plugin parses a subset of YAML using the Earley Parser library YAEP. (René Schwaiger)

Reference

This new plugin can be used to validate that the value of a key is a reference to another key. (Klemens Böswirth)

Libraries

The text below summarizes updates to the C (and C++)-based libraries of Elektra.

Compatibility

As always, the ABI and API of kdb.h is fully compatible, i.e. programs compiled against an older 0.8 version of Elektra will continue to work (ABI) and you will be able to recompile programs without errors (API).

This is the last release for which we have built Jessie packages:

deb     [trusted=yes] https://debian-stable.libelektra.org/elektra-stable/ jessie main
deb-src [trusted=yes] https://debian-stable.libelektra.org/elektra-stable/ jessie main

Obviously, we will continue to update the stretch package:

deb     [trusted=yes] https://debian-stretch-repo.libelektra.org/ stretch main
deb-src [trusted=yes] https://debian-stretch-repo.libelektra.org/ stretch main

Infos for Package Maintainers

Following plugins got added:

  • libelektra-gpgme.so
  • libelektra-mmapstorage_crc.so
  • libelektra-mmapstorage.so
  • libelektra-process.so
  • libelektra-reference.so
  • libelektra-yambi.so

A new library got added (should be packaged privately for now):

  • libelektra-globbing.so

Core

Optimize elektraKsFilter to not duplicate keys (Markus Raab)

Globbing

A new library which can be used to match keys against globbing patterns was introduced. (Klemens Böswirth)

The API is still experimental, so it should not be used externally for now.

Ease

libease provides the function elektraArrayValidateBaseNameString, which can be used to validate that a given string is an Elektra array name. (Klemens Böswirth)

Bindings

Bindings allow you to utilize Elektra using various programming languages. This section keeps you up to date with the multi-language support provided by Elektra.

Ruby

Do not use private Elektra headers for Ruby bindings as preparation for a Ruby libelektra gem. (Bernhard Denner)

Tools

  • Added benchmarks for storage plugins. The currently benchmarked plugins are dump and mmapstorage. (Mihael Pranjić)
  • Avoid raw pointers in KDB tools. (Markus Raab)
  • Improved error text of KDB tool cp. (Markus Raab)
  • Document hidden feature of KDB tool mount. (Markus Raab)
  • Add more tags to the KDB tools to be used with kdb find-tools. (Markus Raab)

Scripts

  • We now require clang-format 6.0 for formatting C and C++ code. (René Schwaiger)
  • The command reformat-source now displays information about the installed version of clang-format, if it is unable to locate a supported version of the tool. (René Schwaiger)
  • We now also check the POSIX compatibility of our scripts with shfmt. (René Schwaiger)
  • The new command [reformat-shfmt][https://master.libelektra.org/scripts/reformat-shfmt] reformats Shell scripts using the tool shfmt. (René Schwaiger)

Documentation

  • We fixed some minor spelling mistakes in the documentation. (René Schwaiger)
  • Improved the plugins documentation. (Michael Zronek)
  • The ReadMe now includes two badges that show the latest released version of Elektra and the status of the Travis build. (René Schwaiger)
  • Fixed documenation error on Ruby plugin Readme. (Bernhard Denner)
  • Go into more detail in BUILDSERVER.md. (Lukas Winkler)

Tests

  • Fix potential parallel execution of maven tests, which write to KDB. (Markus Raab)
  • The unit test for the dbus plugin does not leak memory anymore, if it fails on macOS. (Thomas Wahringer)
  • The tests testkdb_allplugins and testscr_check_kdb_internal_check do not test a plugin on an ASAN enabled build anymore, if you specify the status tag memleak in the plugin contract. (René Schwaiger)
  • The CFramework macro compare_key now also checks if the meta values of keys are equal. (René Schwaiger)
  • The test testscr_check_bashisms does not print warnings about skipped files anymore. (René Schwaiger)
  • We added a Markdown Shell Recorder test for the shell plugin . (René Schwaiger)
  • Added many storage plugin tests. Most tests use the keyset, key name and value APIs. Currently, the tests are only active for dump and mmapstorage. (Mihael Pranjić)
  • The test testcpp_contextual_basic now compiles without warnings, if we use Clang 7 as compiler. (René Schwaiger)
  • crypto, fcrypt and gpgme properly shut down the gpg-agent after the unit test is done. See #1973 . (Peter Nirschl)
  • minor refactoring of the unit tests for crypto, fcrypt, gpgme: moved shared code to separate module in order to avoid code duplication. (Peter Nirschl)
  • The CMake targets for plugin tests (testmod_[plugin]) now depend on the respective CMake targets for the plugins themselves (elektra-[plugin]). (Klemens Böswirth)
  • Fixed bug in CMake plugin tests, if only BUILD_FULL but not BUILD_SHARED is used. (Klemens Böswirth)
  • The test testscr_check_formatting now also checks the formatting of Shell code. (René Schwaiger)
  • We pumped version numbers in XML-test files. (Markus Raab)
  • We fixed a crash in the unit test of the JNA binding. (René Schwaiger)
  • The command kdb run_all now only prints the output of tests that failed. To print the full output of all test, please use the option -v. (René Schwaiger)
  • The Shell Recorder does not use the non-POSIX grep option --text any more. (René Schwaiger)
  • The test suite now uses Google Test 1.8.1. (René Schwaiger)

Build

CMake

  • We improved the detection of Python 2 and Python 3 in the CMake code of the Python bindings/plugins. (René Schwaiger)
  • We restructured the code of the CMake module we use to detect Haskell tools . (René Schwaiger)
  • Building the Haskell binding should now work again. (René Schwaiger)
  • The CMake configuration step now displays less debug messages about found libraries. (René Schwaiger)
  • Provide a wrapper around check_symbol_exists that handles issues with -Werror -Wpedantic. (Lukas Winkler)
  • The argument INCLUDE_SYSTEM_DIRECTORIES of the function add_plugin now supports multiple include directories. (René Schwaiger)
  • We reformatted all CMake source files with cmake-format 0.4.3. (René Schwaiger)
  • Generating coverage data (ENABLE_COVERAGE=ON) should now also work on macOS. (René Schwaiger)
  • You can use the new target run_checkshell to run all shell checks (testscr_check.*). (René Schwaiger)
  • The new target run_nocheckshell runs all tests except for shell checks. (René Schwaiger)
  • The target run_all now runs tests that do not modify the key database in parallel. (René Schwaiger)
  • Fix CMake inclusion logic for GLib/Gi (Markus Raab)

Docker

  • The Docker image for Debian stretch now contains all (optional) dependencies for Elektra. (René Schwaiger)
  • The docker images used by our build system are now available to download from our systems without authentication. Try it out and list available images via docker run --rm anoxis/registry-cli -r https://hub-public.libelektra.org. You can search for images using --images-like, for example: docker run --rm anoxis/registry-cli -r https://hub-public.libelektra.org --images-like alpine. Afterwards pull your desired image as you would do from any public registry, for example: docker pull hub-public.libelektra.org/build-elektra-alpine:201811-37597a34fed4988639cdaf4d6a2c54754d09918586f53389e4fde5fd3b3a7180. (Lukas Winkler)

Vagrant

  • Added Vagrantfile for Ubuntu artful 32-bit. (Mihael Pranjić)

Infrastructure

Jenkins

  • We enabled tests that write to the hard disk on the build job alpine. (René Schwaiger)
  • The build jobs now print less non-relevant output. (René Schwaiger)
  • Enable -Werror in debian-stable-full. (Lukas Winkler)
  • We added the compiler switch -Werror to the build jobs:
    • alpine,
    • debian-stable-full-i386,
    • debian-stable-full-mmap-asan,
    • debian-stable-full-mmap,
    • debian-stable-full-optimizations-off,
    • debian-stable-full-xdg,
    • debian-stable-minimal,
    • debian-stable-multiconf,
    • debian-unstable-clang-asan,
    • debian-unstable-full-clang,
    • debian-unstable-full,
    • ubuntu-xenial, and
    • debian-stable-asan. (René Schwaiger)
  • Build Artifacts for all PR's to detect issues before merging (Lukas Winkler)
  • Stricter removal of temporary docker images on docker nodes (Lukas Winkler)
  • Added jenkins build jobs debian-stable-full-mmap and debian-stable-full-mmap-asan with mmapstorage as the default storage. (Mihael Pranjić)
  • We added basic support for coverage analysis via Coveralls. (René Schwaiger)

Travis

  • Travis now also checks the code for memory leaks in the build job 🍏 Clang ASAN. (René Schwaiger)
  • The Travis build jobs 🍏 Clang ASAN and 🐧 GCC ASAN now only translates a minimal set of plugins, since we had various timeout problems with these jobs before. We explicitly excluded plugins, to make sure that the build jobs still test newly added plugins. (René Schwaiger)
  • Added travis build job 🍏 mmap on macOS with mmapstorage as the default storage. (Mihael Pranjić)
  • Travis now prints the CMake configuration for each build job. (René Schwaiger)
  • We now test Elektra using the latest version of Xcode (10.0). (René Schwaiger)
  • We added the build job 🍏 Check Shell, which only runs shell checks such as testscr_check_oclint. This update allows us to remove the shell checks from the jobs 🍏 MMap and 🍏 Clang, which sometimes hit the timeout limit for public repositories before. (René Schwaiger)
  • All Travis build jobs now use the compiler switch -Werror. (René Schwaiger)
  • The new job 🍏 FULL and the build job 🐧 FULL build Elektra using the CMake options BUILD_FULL=ON and BUILD_SHARED=OFF. (René Schwaiger)
  • The script stage of the build jobs print less non-relevant output. Usually the commands in this stage should now only print verbose output if a test fails. (René Schwaiger)

Website

The website is generated from the repository, so all information about plugins, bindings and tools are always up to date.

Outlook

We are currently working on following topics:

  • Global mmap cache: This feature will enable Elektra to return configuration without parsing configuration files or executing other plugins as long as the configuration files are not changed. (Mihael Pranjić)
  • Finish high-level API. (Klemens Böswirth)
  • Validation improvements. (Michael Zronek)
  • Improve YAML plugins. (René Schwaiger)

Statistics

Following authors made this release possible:

Commits Author
1 Thomas Wahringer thomas.wahringer@libelektra.org
2 Bernhard Denner bernhard.denner@gmail.com
7 Kurt Micheli e1026558@student.tuwien.ac.at
12 Michael Zronek michael.zronek@gmail.com
33 Lukas Winkler derwinlu+git@gmail.com
28 Klemens Böswirth k.boeswirth+git@gmail.com
30 Armin Wurzinger e1528532@student.tuwien.ac.at
38 Peter Nirschl peter.nirschl@gmail.com
100 Markus Raab markus@libelektra.org
180 Mihael Pranjic mpranj@limun.org
418 René Schwaiger sanssecours@me.com

849 commits, 581 files changed, 18503 insertions(+), 3192 deletions(-)

We welcome new contributors!

Finished Thesis

Get It!

You can download the release from here or GitHub

The hashsums are:

  • name: elektra-0.8.25.tar.gz
  • size: 6233918
  • md5sum: d5614b2049fb8431a80842a4909b140e
  • sha1: c7dfb5fa87284d8f5ba4d4753e0e47a0e362c733
  • sha256: 37829256e102e967fe3d58613a036d9fb9b8f9658e20c23fa787eac0bfbb8a79

The release tarball is also available signed by Markus Raab using GnuPG from here or GitHub

Already built API-Docu can be found here or GitHub.

Stay tuned!

Subscribe to the RSS feed to always get the release notifications.

If you also want to participate, or for any questions and comments please contact us via the issue tracker on GitHub.

Permalink to this NEWS entry

For more information, see https://libelektra.org

Best regards, Elektra Initiative