cache: kdb export system

[kodebach]

In #3115 I noticed that kdb export system (kdb export system:/ in the PR), also exports everything in system/elektra/modules. Is this intended?

For the specific circumstance see #3115 (comment)

[markus2330]

Of course, why not? It also exports version and other stuff not useful for import. You can add --without-elektra to avoid exporting these parts.

For the shell-recorder I would also expect that only the given part is backed up and restored.

[kodebach]

Of course, why not?

Does kdbSet remove them somehow? It seems wrong to serialize these keys, when they are hard coded via a special part of plugin get functions...

[markus2330]

For the version information kdbSet checks if they are identical and ignores kdbSet if they are. Otherwise an error gets generated.

For modules (iirc) there is no kdbSet, so the keys to set are simply ignored. Iirc the "constants" plugin has the same behavior It is usually mounted system/info/elektra so not in system/elektra. Maybe we should change that to make --without-elektra more useful?

But anyway: Both behaviors (ignoring and checking for identical keys) should not harm an import.

[markus2330]

So for the current situation of the quite unstable Elektra it makes perfect sense: imports of system/elektra are rejected unless the version is identical. To avoid this error, --without-elektra can be passed.

After 1.0 we can actually be more relaxed and accept imports from Elektra versions 1.0 or greater. (@mpranj what do you think?)

The text needs to be adapted then, currently it is:

kdb import system < sys
Sorry, module kdb issued the error C01320:
Interface: Read only plugin, 'kdbSet' not supported but the key system/elektra/version/constants/KDB_VERSION_MICRO (expected system/elektra/version/constants/KDB_VERSION_MICRO) was tried to be modified to '1' (expected '2')

Anyway, your question gives insight about how we can improve the documentation of --without-elektra. The idea of --without-elektra in context of import/export is that either:

1. all the internas of Elektra are exported but then we also have the version check to not mess up Elektra.
2. you do not import/export internas of Elektra, so version of Elektra does not matter (only the version of the storage plugin does)

[mpranj]

After 1.0 we can actually be more relaxed

I agree with the suggestions.

As @kodebach mentioned, it is somehow weird that this appeared just now in #3115 and did not appear before. While adding --without-elektra probably should have been there in the first place for backup-and-restore, it still seems that the PR changed some behavior.

@kodebach did you activate cache again for those tests? I'd like to wait for the rest of the PR to be done so I can fix mmapstorage and cache. I have seen such/similar errors (Postcondition of backend was violated: drop key [...] not belonging to [...]) while the cache was not implemented correctly.

[kodebach]

did you activate cache again for those tests?

Yes, cache is fully enabled.

so I can fix mmapstorage and cache

mmapstorage works already. Turns out I had a key instead of ukey somewhere.

The cache also seems to work, apart from the problem with the global unmount test. Last I checked, the error didn't happen when cache isn't compiled.

I did change parts of kdbGet, elektraCacheGet and related functions. It's likely I broke something, but since the things I changed probably have to be changed again after #2969, I don't think you need to take a look right now.

[mpranj]

Ok. I suggest if the cache+mmapstorage causes the problems you simply keep it deactivated until the very end and we activate adapt it then.

The PR is a quite fundamental change so it will be much easier if you don't get stuck because of some broken cache behavior. When it's basically ready simply activate cache again and ping me so I can take a look and fix the remainders.

[markus2330]

@mpranj: I assigned you as this is cache related.

[mpranj]

I can not reproduce anything similar on current master. I can't do much there unless someone knows how to reproduce it there, but I suspect it is not present on master.

On the branch from #3115 I get the errors as described. @kodebach should I go in there and try to fix the cache related stuff on top of your changes or is it too soon to work on top of that? How much of a pain is it to rebase the branch from master currently (pre-backend plugin)?

[kodebach]

I think it is not worth working on #3115 until #2969 and the related PRs are merged. After that I will rebase and try to finish the PR. I'd like to rebase as little as possible, because you basically have to check all commits for anything that relies on the syntax of keynames to ensure a proper rebase.

If you want to, you can of course look for the bug in the current version of the PR. However, as far as I remember, I tried this on master before creating the issue, so maybe it has been fixed in the meantime.

[mpranj]

I think it is not worth working on #3115 until #2969 and the related PRs are merged.

Ok, thank you! Then let's wait and check this issue after the related PRs are merged!

[markus2330]

Is it possible to tag issues somehow that they are waiting for some PRs? I changed the title for now.

[kodebach]

I actually managed to reproduce this on master today. Specifically, I built scripts/docker/fedora/32/Dockerfile (technically from #3447 not from master, but the file is the same and completely standalone). I then started a container with

docker run -it -w /home/jenkins elektra-fedora-32:latest

and inside the container ran the following lines:

git clone https://github.com/ElektraInitiative/libelektra
cd libelektra && mkdir build && cd build
cmake .. -DCMAKE_BUILD_TYPE=Debug -DKDB_DB_SPEC="$PWD/config/spec" -DKDB_DB_SYSTEM="$PWD/config/system" -DKDB_DB_USER="cdbg-1/.config" -DCMAKE_INSTALL_PREFIX="$PWD/install" -DINSTALL_SYSTEM_FILES=OFF -DENABLE_DEBUG=ON -DENABLE_LOGGER=ON -DCMAKE_EXPORT_COMPILE_COMMANDS=TRUE -DPLUGINS="ALL;-lua;-ccode;-tcl" -DBINDINGS="ALL" -DCOMMON_FLAGS="-Werror"
make -j 24
bin/kdb export system toml

The output from the last command was:

elektra.modules.cache = "cache plugin waits for your orders"
elektra.modules.cache.exports = ""
elektra.modules.cache.exports.close = "(binary)"
elektra.modules.cache.exports.get = "(binary)"
elektra.modules.cache.exports.open = "(binary)"
elektra.modules.cache.exports.set = "(binary)"
elektra.modules.cache.infos = "Information about the cache plugin is in keys below"
elektra.modules.cache.infos.author = "Mihael Pranjic <mpranj@limun.org>"
elektra.modules.cache.infos.licence = "BSD"
elektra.modules.cache.infos.metadata = ""
elektra.modules.cache.infos.needs = ""
elektra.modules.cache.infos.placements = "pregetcache postgetcache"
elektra.modules.cache.infos.provides = ""
elektra.modules.cache.infos.recommends = ""
elektra.modules.cache.infos.status = "maintained unittest shelltest specific global"
elektra.modules.cache.infos.version = 1
elektra.modules.dump = "dump plugin waits for your orders"
elektra.modules.dump.config.needs.fcrypt.textmode = 0
elektra.modules.dump.exports = ""
elektra.modules.dump.exports.get = "(binary)"
elektra.modules.dump.exports.serialise = "(binary)"
elektra.modules.dump.exports.set = "(binary)"
elektra.modules.dump.exports.unserialise = "(binary)"
elektra.modules.dump.infos = "Information about the dump plugin is in keys below"
elektra.modules.dump.infos.author = "Markus Raab <elektra@libelektra.org>"
elektra.modules.dump.infos.licence = "BSD"
elektra.modules.dump.infos.metadata = ""
elektra.modules.dump.infos.needs = ""
elektra.modules.dump.infos.placements = "getstorage setstorage"
elektra.modules.dump.infos.provides = "storage/dump storage dump"
elektra.modules.dump.infos.recommends = ""
elektra.modules.dump.infos.status = "productive maintained conformant unittest tested nodep -1000 default"
elektra.modules.dump.infos.version = 1
elektra.modules.list = "list plugin waits for your orders"
elektra.modules.list.exports = ""
elektra.modules.list.exports.addPlugin = "(binary)"
elektra.modules.list.exports.close = "(binary)"
elektra.modules.list.exports.deferredCall = "(binary)"
elektra.modules.list.exports.editPlugin = "(binary)"
elektra.modules.list.exports.error = "(binary)"
elektra.modules.list.exports.findplugin = "(binary)"
elektra.modules.list.exports.get = "(binary)"
elektra.modules.list.exports.mountplugin = "(binary)"
elektra.modules.list.exports.open = "(binary)"
elektra.modules.list.exports.set = "(binary)"
elektra.modules.list.exports.unmountplugin = "(binary)"
elektra.modules.list.infos = "Information about the list plugin is in keys below"
elektra.modules.list.infos.author = "Thomas Waser <thomas.waser@libelektra.org>"
elektra.modules.list.infos.licence = "BSD"
elektra.modules.list.infos.needs = ""
elektra.modules.list.infos.placements = "pregetstorage procgetstorage postgetstorage postgetcleanup presetstorage presetcleanup precommit postcommit prerollback postrollback"
elektra.modules.list.infos.provides = ""
elektra.modules.list.infos.status = "unittest nodep libc configurable global"
elektra.modules.list.infos.version = 1
elektra.modules.resolver_fm_hpu_b = "resolver_fm_hpu_b plugin waits for your orders"
elektra.modules.resolver_fm_hpu_b.constants = ""
elektra.modules.resolver_fm_hpu_b.constants.ELEKTRA_VARIANT_BASE = "fm"
elektra.modules.resolver_fm_hpu_b.constants.ELEKTRA_VARIANT_SYSTEM = "b"
elektra.modules.resolver_fm_hpu_b.constants.ELEKTRA_VARIANT_USER = "hpu"
elektra.modules.resolver_fm_hpu_b.constants.KDB_DB_DIR = ".dir"
elektra.modules.resolver_fm_hpu_b.constants.KDB_DB_HOME = "/home"
elektra.modules.resolver_fm_hpu_b.constants.KDB_DB_SPEC = "/home/jenkins/libelektra/build/config/spec"
elektra.modules.resolver_fm_hpu_b.constants.KDB_DB_SYSTEM = "/home/jenkins/libelektra/build/config/system"
elektra.modules.resolver_fm_hpu_b.constants.KDB_DB_USER = "cdbg-1/.config"
elektra.modules.resolver_fm_hpu_b.exports = ""
elektra.modules.resolver_fm_hpu_b.exports.checkfile = "(binary)"
elektra.modules.resolver_fm_hpu_b.exports.close = "(binary)"
elektra.modules.resolver_fm_hpu_b.exports.commit = "(binary)"
elektra.modules.resolver_fm_hpu_b.exports.error = "(binary)"
elektra.modules.resolver_fm_hpu_b.exports.filename = "(binary)"
elektra.modules.resolver_fm_hpu_b.exports.freeHandle = "(binary)"
elektra.modules.resolver_fm_hpu_b.exports.get = "(binary)"
elektra.modules.resolver_fm_hpu_b.exports.open = "(binary)"
elektra.modules.resolver_fm_hpu_b.exports.set = "(binary)"
elektra.modules.resolver_fm_hpu_b.infos = "All information you want to know is in keys below"
elektra.modules.resolver_fm_hpu_b.infos.author = "Markus Raab <elektra@markus-raab.org>"
elektra.modules.resolver_fm_hpu_b.infos.licence = "BSD"
elektra.modules.resolver_fm_hpu_b.infos.needs = ""
elektra.modules.resolver_fm_hpu_b.infos.placements = "rollback getresolver setresolver commit"
elektra.modules.resolver_fm_hpu_b.infos.provides = "resolver"
elektra.modules.resolver_fm_hpu_b.infos.status = "productive maintained specific unittest tested libc nodep configurable default"
elektra.modules.resolver_fm_hpu_b.infos.version = 1
elektra.version = "Below are version information of the Elektra Library you are currently using"
elektra.version.constants = ""
elektra.version.constants.KDB_VERSION = "0.9.2"
elektra.version.constants.KDB_VERSION_MAJOR = 0
elektra.version.constants.KDB_VERSION_MINOR = 9
elektra.version.constants.KDB_VERSION_PATCH = 2
elektra.version.constants.SO_VERSION = 4
elektra.version.infos = "All information you want to know"
elektra.version.infos.author = "Markus Raab <elektra@markus-raab.org>"
elektra.version.infos.description = "Information of your Elektra Installation"
elektra.version.infos.licence = "BSD"
elektra.version.infos.version = 1

(NOTE: I removed the .description keys from the output, because they contain the string ``` and therefore break Github's Markdown formatting)

IMO the output should actuall be empty, since all the keys are derived from the specific Elektra installation and should never be imported into any KDB. Possibly exporting the version.constants keys could make sense, if kdb import properly ignores them (the other version keys also shouldn't be exported).

The Postcondition of backend was violated: drop key system:/elektra/modules/cache/infos/licence not belonging to "system:/elektra/modules/cache" with name "modules" but instead to "(null)" with name "(null)" because it is hidden by other mountpoint issues only occur in #3447 and only for the cache plugin. So there is still a bug in #3447. However, fixing this kdb export issue would also remove the problem.

[kodebach]

CORRECTION: The postcondition issues do actually occur in the docker container as well.

In the container from above, I ran (directly after the other stuff):

ctest -R kdb_global_umount --output-on-failure

After that any kdb command that tries to access the KDB reports lots of postcondition warnings, e.g. kdb ls system.

[markus2330]

Are there now remaining bugs described in this issue?

[kodebach]

AFAIK this one and the next comment (i.e. the original issue) are still unresolved: #3299 (comment)

[mpranj]

Thank you so much for reporting this!

The cache definitely stores these (any) Keys if they are present in the returned KeySet from kdbGet. There are no exceptions for the cache. However, as you said, it makes no sense to allow these keys on kdb import. I'll see if I can reproduce this from your description and fix it.

If the problem occurs only when using kdb import, I would simply drop these special keys on import.

[markus2330]

I agree with @mpranj that the cache should fully return everything.

Its output should never contain any keys below system/elektra/modules or system/elektra/version since these are specific to the Elektra installation and should not be imported into another KDB.

Exporting is not only for importing but also to display multiple key/values for a user. There is nothing wrong with kdb export system/elektra/version/constants simpleini to see all version information.

I didn't investigate where those keys come from, so I'm not sure the cache plugin is actually involved here. The best solution might be to just always ksCut these parts in kdb export. That way the problem can't reappear if something changes.

Such a ksCut happens when you say --without-elektra. What we could do is to change the default: exclude elektra by default, except when explicitly importing/exporting /elektra or saying --with-elektra.

Postcondition of backend was violated

This seems to be like an unrelated bug. (Maybe shellrecorder is not using --without-elektra). #3299 (comment), however, looks like it should be. (I cannot reproduce, I get Error response from daemon: pull access denied for elektra-fedora-32, repository does not exist or may require 'docker login': denied: requested access to the resource is denied.)

[mpranj]

I cannot reproduce

We have not published this image to docker hub.

The precondition here is that he build the docker image from scripts/docker/fedora/32/Dockerfile and tagged the build with -t elektra-fedora-32:latest when running docker build.

Something like:

cd scripts/docker/fedora/32/
docker build -t elektra-fedora-32:latest -f ./Dockerfile .

[markus2330]

Thank you, yes I can reproduce the Postcondition of backend was violated problems now on master (as described by @kodebach in the docker image and then using global mount commands):

 Sorry, 17 warnings were issued ;(
        Sorry, module kdb issued the warning C01320:
        Interface: Postcondition of backend was violated: drop key system/elektra/modules/cache not belonging to "system/elektra/modules/cache" with name "modules" but instead to "" with name "default" because it is hidden by other mountpoint
        Sorry, module kdb issued the warning C01320:
        Interface: Postcondition of backend was violated: drop key system/elektra/modules/cache/exports not belonging to "system/elektra/modules/cache" with name "modules" but instead to "" with name "default" because it is hidden by other mountpoint
        Sorry, module kdb issued the warning C01320:
        Interface: Postcondition of backend was violated: drop key system/elektra/modules/cache/exports/close not belonging to "system/elektra/modules/cache" with name "modules" but instead to "" with name "default" because it is hidden by other mountpoint
        Sorry, module kdb issued the warning C01320:
        Interface: Postcondition of backend was violated: drop key system/elektra/modules/cache/exports/get not belonging to "system/elektra/modules/cache" with name "modules" but instead to "" with name "default" because it is hidden by other mountpoint
        Sorry, module kdb issued the warning C01320:
        Interface: Postcondition of backend was violated: drop key system/elektra/modules/cache/exports/open not belonging to "system/elektra/modules/cache" with name "modules" but instead to "" with name "default" because it is hidden by other mountpoint
        Sorry, module kdb issued the warning C01320:
        Interface: Postcondition of backend was violated: drop key system/elektra/modules/cache/exports/set not belonging to "system/elektra/modules/cache" with name "modules" but instead to "" with name "default" because it is hidden by other mountpoint
        Sorry, module kdb issued the warning C01320:
        Interface: Postcondition of backend was violated: drop key system/elektra/modules/cache/infos not belonging to "system/elektra/modules/cache" with name "modules" but instead to "" with name "default" because it is hidden by other mountpoint
        Sorry, module kdb issued the warning C01320:
        Interface: Postcondition of backend was violated: drop key system/elektra/modules/cache/infos/author not belonging to "system/elektra/modules/cache" with name "modules" but instead to "" with name "default" because it is hidden by other mountpoint
        Sorry, module kdb issued the warning C01320:
        Interface: Postcondition of backend was violated: drop key system/elektra/modules/cache/infos/description not belonging to "system/elektra/modules/cache" with name "modules" but instead to "" with name "default" because it is hidden by other mountpoint
        Sorry, module kdb issued the warning C01320:
        Interface: Postcondition of backend was violated: drop key system/elektra/modules/cache/infos/licence not belonging to "system/elektra/modules/cache" with name "modules" but instead to "" with name "default" because it is hidden by other mountpoint
        Sorry, module kdb issued the warning C01320:
        Interface: Postcondition of backend was violated: drop key system/elektra/modules/cache/infos/metadata not belonging to "system/elektra/modules/cache" with name "modules" but instead to "" with name "default" because it is hidden by other mountpoint
        Sorry, module kdb issued the warning C01320:
        Interface: Postcondition of backend was violated: drop key system/elektra/modules/cache/infos/needs not belonging to "system/elektra/modules/cache" with name "modules" but instead to "" with name "default" because it is hidden by other mountpoint
        Sorry, module kdb issued the warning C01320:
        Interface: Postcondition of backend was violated: drop key system/elektra/modules/cache/infos/placements not belonging to "system/elektra/modules/cache" with name "modules" but instead to "" with name "default" because it is hidden by other mountpoint
        Sorry, module kdb issued the warning C01320:
        Interface: Postcondition of backend was violated: drop key system/elektra/modules/cache/infos/provides not belonging to "system/elektra/modules/cache" with name "modules" but instead to "" with name "default" because it is hidden by other mountpoint
        Sorry, module kdb issued the warning C01320:
        Interface: Postcondition of backend was violated: drop key system/elektra/modules/cache/infos/recommends not belonging to "system/elektra/modules/cache" with name "modules" but instead to "" with name "default" because it is hidden by other mountpoint
        Sorry, module kdb issued the warning C01320:
        Interface: Postcondition of backend was violated: drop key system/elektra/modules/cache/infos/status not belonging to "system/elektra/modules/cache" with name "modules" but instead to "" with name "default" because it is hidden by other mountpoint
        Sorry, module kdb issued the warning C01320:
        Interface: Postcondition of backend was violated: drop key system/elektra/modules/cache/infos/version not belonging to "system/elektra/modules/cache" with name "modules" but instead to "" with name "default" because it is hidden by other mountpoint

The problem is clearly unrelated to kdb export as it also occurs, e.g. with kdb ls or kdb cache clear.

Interestingly kdb cache clear does not help but it still could be a problem of the cache (as kdb tools first use KDB to get their config).

@mpranj do you have an idea?

[kodebach]

What we could do is to change the default: exclude elektra by default, except when explicitly importing/exporting /elektra or saying --with-elektra.

Yes, that would definitely be a better solution. Ideally we would also differentiate between system/elektra/modules/system/elektra/version and the rest of system/elektra. The mountpoint config and global plugin config are a lot more useful than the modules and version keys (which are generated at runtime).

The problem is clearly unrelated to kdb export as it also occurs, e.g. with kdb ls or kdb cache clear.

Like I stated the "Postcondition" stuff is likely a bug in kdb import.

Interestingly kdb cache clear does not help

Why would it? Like I said, the issue is that system/elektra/modules keys are stored in elektra.ecf. If you followed my steps above (specifically used the same cmake options), you can see that by running

cat config/system/elektra.ecf

within the build directory.

[markus2330]

Like I stated the "Postcondition" stuff is likely a bug in kdb import.

As kdb import does not do much more than creating a KeySet and calling kdbSet I am afraid the problem is somewhere in kdbSet. The correct behavior of storing anything in system/elektra/modules must be an error or discarding, and definitely not persisting. But obviously this is not the behavior:

kdb set system/elektra/modules/NOTALLOWED
kdb ls system/elektra/modules
#> system/elektra/modules/NOTALLOWED
#> system/elektra/modules/cache
...

So when a NOTALLOWED plugin gets mounted, we have the problem as described here. So we need kdbSet to fail on any attempts to write to system/elektra/modules.

Actually we need to define the semantics of the whole /elektra hierarchy, which is probably a bigger task...

the issue is that system/elektra/modules keys are stored in elektra.ecf

Sorry, I did not see it when I was jumping around between the comments trying to reproduce it.

[stale]

I mark this stale as it did not have any activity for one year. I'll close it in two weeks if no further activity occurs. If you want it to be alive again, ping by writing a message here or create a new issue/PR with the remainder of this issue/PR.
Thank you for your contributions 💖

[stale]

I closed this now because it has been inactive for more than one year. If I closed it by mistake, please do not hesitate to reopen it or create a new issue/PR with the remainder of this issue/PR.
Thank you for your contributions 💖

[github-actions]

I mark this stale as it did not have any activity for one year. I'll close it in two weeks if no further activity occurs. If you want it to be alive again, ping by writing a message here or create a new issue with the remainder of this issue.
Thank you for your contributions 💖

[github-actions]

I closed this now because it has been inactive for more than one year. If I closed it by mistake, please do not hesitate to reopen it or create a new issue with the remainder of this issue.
Thank you for your contributions 💖