| author | brief | tags | review |
|---|---|---|---|
Markus Raab <elektra@markus-raab.org> |
Describes validation capabilities of Elektra. |
validation, spec, metadata |
pending |
Configuration in FLOSS unfortunately is often stored
completely without validation. Notable exceptions are sudo
(visudo), or user accounts (adduser) but in most cases
you only get feedback of non-validating configuration when
the application fails to start.
Elektra provides a generic way to validate any configuration before it is written to disc.
Any of Elektra's user interfaces will work with the technique described in this tutorial, e.g.:
-
kdb qt-gui: graphical user interface -
kdb editor: starts up your favorite text editor and allows you to edit configuration in any syntax. (generalization ofvisudo) -
kdb set: manipulate or add individual configuration entries. (generalization ofadduser) -
Any other tool using Elektra to store configuration (e.g. if the application itself has capabilities to modify its configuration)
The most direct way to validate keys is
sudo kdb mount validation.dump user/tests/together dump validation
kdb vset user/tests/together/test 123 "[1-9][0-9]*" "Not a number"
kdb set user/tests/together/test abc
# STDERR: The command kdb.* set failed while accessing the key database .*
# ERROR: 42
# RET:5For all other plugins (except validation) the convenience tool kdb vset
is missing. Let us see what kdb vset actually did:
kdb lsmeta user/tests/together/test
#> check/validation
#> check/validation/match
#> check/validation/messageSo it only appended some metadata (data describing the data) next to the key, which we also could do by:
# Following lines are (except for error conditions) identical to
# kdb vset user/tests/together/test 123 "[1-9][0-9]*" "Not a number"
kdb setmeta user/tests/together/test check/validation "[1-9][0-9]*"
kdb setmeta user/tests/together/test check/validation/match LINE
kdb setmeta user/tests/together/test check/validation/message "Not a number"
kdb set user/tests/together/test 123
#> Set string to "123"
# Undo modifications
kdb rm -r user/tests/together
sudo kdb umount user/tests/togetherThe approach is not limited to validation via regular expressions, but any values-validation plugin can be used, e.g. enum. For a full list refer to the section "Value Validation" in the list of all plugins.
Note that it also easy to write your own (value validation) plugin.
The drawbacks of this approach are:
- The administrator needs to take care that the validation plugins are available where needed.
- Some metadata (in this case
check/validation) needs to be stored next to the key which won't work with most configuration files. This is the reason why we explicitly useddumpas storage inkdb mount. - After the key is removed, the validation information is gone, too.
- It only works for the namespace where
vsetwas used. In the example above we could override the cascading key/tutorial/together/testwith the unvalidated keydir/tutorial/together/test. - You cannot validate structure of which keys must be present or absent.
These issues are resolved straightforward by separating the configuration from its configuration specification (often called schemata in XML or JSON). The purpose of the spec namespace is to hold the configuration specification, i.e., the description of how to validate the keys of all other namespaces.
To make this work, we need a plugin that applies all metadata found in the spec-namespace
to all other namespaces. This plugin is called spec and needs to be mounted
globally (will be added by default and also with any kdb global-mount call).
Before we start, let us make a backup of the current data in the spec and user namespace:
kdb set system/tests/spec $(mktemp)
kdb set system/tests/user $(mktemp)
kdb export spec dump > $(kdb get system/tests/spec)
kdb export user dump > $(kdb get system/tests/user)We write metadata to the namespace spec and the plugin spec applies it to every cascading key:
kdb setmeta spec/tests/spec/test hello world
kdb set /tests/spec/test value
#> Using name user/tests/spec/test
#> Create a new key user/tests/spec/test with string "value"
kdb lsmeta spec/tests/spec/test | grep -v '^internal/ini'
#> hello
kdb lsmeta /tests/spec/test | grep -v '^internal/ini'
#> hello
kdb getmeta /tests/spec/test hello
#> world
kdb getmeta user/tests/spec/test hello
#> worldBut it also supports globbing (_ for any key, ? for any char, [] for character classes):
kdb setmeta "spec/tests/spec/_" new metaval
kdb set /tests/spec/test value
#> Using name user/tests/spec/test
#> Set string to "value"
kdb lsmeta /tests/spec/test | grep -v '^internal/ini'
#> hello
#> new
# Remove keys and metadata from the commands above
kdb rm -r spec/tests/spec
kdb rm -r user/tests/specSo let us combine this functionality with validation plugins. So we would specify:
kdb setmeta spec/tests/spec/test check/validation "[1-9][0-9]*"
kdb setmeta spec/tests/spec/test check/validation/match LINE
kdb setmeta spec/tests/spec/test check/validation/message "Not a number"If we now set a new key with
kdb set /tests/spec/test "not a number"
#> Using name user/tests/spec/test
#> Create a new key user/tests/spec/test with string "not a number"this key has adopted all metadata from the spec namespace:
kdb lsmeta /tests/spec/test | grep -v '^internal/ini'
#> check/validation
#> check/validation/match
#> check/validation/messageNote that this key should not have passed the validation that we defined in the spec namespace. Nonetheless we were able to set this key, because the validation plugin was not active for this key. On that behalf we have to make sure that the validation plugin is loaded for this key with:
kdb mount tutorial.dump user/tutorial dump validation
This mounts the backend tutorial.dump to the mountpoint
user/tutorial and activates the validation plugin for the keys below the mountpoint.
The validation plugin now uses the metadata of the keys below user/tutorial
to validate values before storing them in tutorial.dump.
Although this is better than defining metadata in the same place as the data itself, we can still do better. The reason for that is that one of the aims of Elektra is to remove the trouble of validation and finding the files that hold your configuration from the users. At the moment a user still has to know which files should hold the configuration and which plugins must be loaded when he mounts configuration files.
This problem can be addressed by recognizing that the location of the configuration files and the plugins that must be loaded is part of the schema of our configuration and therefore should be stored in the spec namespace.
# Undo modifications
kdb rm -r spec/tests/spec/test
kdb rm -r user/tests/spec/testWe call the files, that contain a complete schema for configuration below a specific path in form of metadata, Specfiles.
Particularly a Specfile contains metadata that defines
- the mountpoints of paths,
- the plugins to load and
- the behavior of these plugins.
Let us create an example Specfile in the dump format, which supports metadata
(altough the specfile is stored in the dump format, we can still create it using
the human readable ni format by using kdb import):
sudo kdb mount tutorial.dump spec/tests/tutorial dump
cat << HERE | kdb import spec/tests/tutorial ni \
[] \
mountpoint = tutorial.dump \
infos/plugins = dump validation \
\
[/links/_] \
check/validation = https?://.*\..* \
check/validation/match = LINE \
check/validation/message = not a valid URL \
description = A link to some website \
HERE
kdb lsmeta spec/tests/tutorial
#> infos/plugins
#> mountpointWe now have all the metadata that we need to mount and validate the data below
/tutorial in one file.
Now we apply this Specfile to the key database to all keys below tests/tutorial.
kdb spec-mount /tests/tutorialThis command automatically mounts /tests/tutorial to the backend tutorial.dump and
loads the validation plugin.
kdb set /tests/tutorial/links/url "invalid url"
#> Using name user/tests/tutorial/links/url
# STDERR: .*key value failed to validate.*not a valid URL.*
# ERROR: 42
# RET: 5Note that the backend tutorial.dump is mounted for all namespaces:
kdb file user/tests/tutorial
# STDOUT-REGEX: /.*/tutorial\.dump
kdb file system/tests/tutorial
# STDOUT-REGEX: /.*/tutorial\.dump
kdb file dir/tests/tutorial
# STDOUT-REGEX: /.*/tutorial\.dumpIf you want to set a key for another namespace and do not want to go without validation,
consider that the spec plugin works only when you use cascading keys.
You can work around that by setting the keys with the -N option:
kdb set -N system /tests/tutorial/links/elektra https://www.libelektra.org
#> Using name system/tests/tutorial/links/elektra
#> Create a new key system/tests/tutorial/links/elektra with string "https://www.libelektra.org"Up to now we only discussed how to reject keys that have unwanted values. Sometimes, however, applications require the presence or absence of keys. There are many ways to do so directly supported by the spec plugin. Another way is to trigger errors with the error plugin:
kdb setmeta /tests/tutorial/spec/should_not_be_here trigger/error 10
#> Using keyname spec/tests/tutorial/spec/should_not_be_here
kdb spec-mount /tests/tutorial
kdb set /tests/tutorial/spec/should_not_be_here abc
#> Using name user/tests/tutorial/spec/should_not_be_here
# RET: 5
# STDERR: .*error.*10.*occurred.*
kdb get /tests/tutorial/spec/should_not_be_here
# RET: 11
# STDERR: Did not find key '/tests/tutorial/spec/should_not_be_here'If we want to reject every optional key (and only want to allow required keys)
we can use the plugin required as further discussed below.
Before we look further let us undo the modifications to the key database.
kdb rm -r spec/tests/tutorial
kdb rm -r system/tests/tutorial
kdb umount spec/tests/tutorial
kdb umount /tests/tutorial
kdb rm -rf spec
kdb rm -rf user
kdb import spec dump < $(kdb get system/tests/spec)
kdb import user dump < $(kdb get system/tests/user)
rm $(kdb get system/tests/spec)
rm $(kdb get system/tests/user)
kdb rm system/tests/spec
kdb rm system/tests/userSometimes we already have configuration specifications given in some other format which is more compact and more directed to the needs of an individual application. We can write a plugin that parses that format and transform the content to key-value and metadata (describing how to validate).
For example, let us assume we have enum validations in the file schema.txt:
cat > "$PWD/schema.txt" << HERE \
%: notation TBD ? graph text semi \
%: tool-support* TBD ? none compiler ide \
%: applied-to TBD ? none small real-world \
mountpoint file.txt \
plugins required \
HERE
And by convention for keys ending with *, multiple values are allowed.
So we want to transform above syntax to:
%:notation TBD ? graph text semi
%:tool-support* TBD ? none compiler ide
%:applied-to TBD ? none small real-world
Lucky, we already have a plugin which allows us to so:
kdb mount "$PWD/schema.txt" spec/tutorial/schema simplespeclang keyword/enum=%:,keyword/assign=TBD
kdb spec-mount /tutorial/schema
We configure the plugin simplespeclang so that it conforms to our "weird" syntax.
Because in schema.txt we have the line mountpoint file.txt we can also mount the
schema using spec-mount.
Now we have enforced that the 3 configuration options notation tool-support* applied-to
need to be present (and no other). For example we can import:
kdb import -s validate -c "format=% : %" /tutorial/schema simpleini << HERE \
notation : graph \
tool-support : ? none \
applied-to : small \
HERE \
Or (afterwards) setting individual values:
kdb set /tutorial/schema/applied-to smal # fails, not a valid enum
Or (in visudo fashion):
kdb editor -s validate /tutorial/schema simpleini