kola run-ext: Add support for running an externally defined test

[cgwalters]

I'd like to propose something like:

$ kola run-ext-script /path/to/binary

Takes a single binary at /path/to/binary, injects it and a generated basic systemd unit kola-test-ext-$(basename binary).service that runs it.
kola connects via ssh and runs systemctl --wait is-active (does that work? if not find another command).

When the service exits successfully the test is considered successful. When it exits with an EXITCODE error, the test has failed. If it exits from SIGTERM, then we assume the host is rebooting and continue waiting.

Then we support a much bigger generalization of this,

kola run-ext /path/to/dir

which injects a full directory for dependencies, and which would have at least a kola-test.service unit, and maybe we support a container image?

This would allow us to more easily adapt e.g. the ostree test suite to run inside kola. So in ostree's upstream CI we could use that. And importantly the reverse works - run the ostree tests downstream too, i.e. reverse dependency testing. e.g. in FCOS pipeline, when the kernel/systemd change, run ostree's tests.

[arithx]

The proposal sounds like it has a lot of overlap to what we're already doing with kolet so it should be relatively straightforward to implement.

One idea is to also allow some specification of the binaries and whether or not the tests are entirely self-contained to allow for re-use of the same machine which could be valuable when running on non-qemu platforms.

[cgwalters]

Another idea here is:

kola run-ext-container quay.io/coreos/ostree-tests:latest

We could define a simple schema for "test containers" - perhaps two cases:

• run unprivileged with the host filesystem mounted readonly at /host, can run multiple in parallel/sequence
• run privileged, assumed to be destructive and only one run per VM

[cgwalters]

This exists now.