Support execution contexts

[spbnick]

At the moment we support expressing only one test hierarchy: a test being a test suite or a part of one, encoded in the test path field. However, we have no way of specifying whether a particular subtest executed as part of this or that execution of its test suite.
This is particularly important for troubleshooting correlation and interference between test executions, in some cases.

We need a solution which would be optional, as not all CI systems would have enough capacity to implement it, and likely not from the start. We need to be able to group recursively, as test suites often have multiple levels of nesting and there's also interest in grouping e.g. multi-host tests, and test runs spread across multiple VMs with some running on their hosts. We also need to be able to encode ordering of test plans, before the tests are executed.

Here's the current proposal:

• Add a new field called context or execution_context (maybe with _id or _path on the end) to tests.
• This field is expected to contain a dot-separated path, similarly to the path, but prepended with <origin>..
• If a test has a particular context, and another test has that same context with a dot and an arbitrary path appended, then the latter is part of the execution of the former.
• Particular positions in the path and the path as a whole have no assigned meaning (although we might work on standardizing that later), and only serve to group test executions.
• Not all nodes in a given path need to have corresponding tests reported, and may not necessarily correspond to a test. E.g. suite12.host3.test1 and suite12.host2.test2 could be specified to two different test entries, and there doesn't have to be a test specifying suite12.host3, or suite12.host2, or suite12, but such paths could serve to combine tests into hosts they executed on, and e.g. a particular parallel test. Same goes for interim "nodes". Still, a dashboard would be unable to assign meaning to these "non-test" nodes and may skip grouping by them. We can work later on assigning more meaning to particular nodes, e.g. by creating other objects in the database referencing them.
• In absence of start_time values, the values of context should be used to represent test ordering within each node by sorting them lexicographically. E.g. for each execution environment (we need to come up with IDs for them, including labs) for a particular build, you can simply sort all tests by context to get their execution order.
• If start_time is specified, it provides information on actual execution order, but context could be used to refine ordering for tests with the same start_time. In practice using ORDER BY start_time ASC NULLS LAST, context ASC NULLS LAST would implement this and the above requirements, and also support reports without context.
• Context (paths) are global, and it's up to the submitting CI system to make sure they're unique, and maintain correct ordering (e.g. by padding them where necessary).

This is still a work in progress, feedback is very much welcome.

[spbnick]

I think just context (or context_path) would be fine as the name of the field, because our test objects are already test "executions".

[spbnick]

Note that we won't be able to use the PostgreSQL ltree extension for this one, as the set of "labels" would be too large for indexing, but we also won't need the kind of queries it helps with for these. A basic prefix search would probably be enough.

[helen-fornazier]

Just to make sure I understand, it could be like:
parent node: maestro.lava_id_deadbeef
children nodes: maestro.lava_id_deadbeef.unittest1, maestro.lava_id_deadbeef.unittest2

?

[spbnick]

The maestro. part is right, but the rest is up to you. Assuming that lava_id_deadbeef corresponds to an execution of a test suite, such as LTP, and unittest1 corresponds to a test inside it, it makes sense. If this was e.g. an LTP execution, then you could create a test entry with path set to ltp and context set to maestro.lava_id_deadbeef (but you don't have to), and then the ltp.unittest1 would have maestro.lava_id_deadbeef.unittest1.

Now I'm thinking that perhaps we better use the : as the origin separator, to match everything else, as we won't be able to use ltree on this anyway. I.e. maestro:lava_id_deadbeef.unittest1.

[spbnick]

We need to also think about how this would be represented in a dashboard, perhaps ask ProFUSION folks too.