Add TMT plans for all our notable use cases

[comps]

I split these according to what seemed reasonable. Milan mentioned we should probably use auto-ptp via Beaker directly, so I didn't include a plan for it.

Feel free to add suggestions what to change - I'm not sure if having automated errata runs is even useful if we test everything before that. And we got rid of manual tests since (replaced them with JIRA).

I left the original tags (NoProductization, daily, etc.) in place, due to TCMS, but I expect we remove them after we switch away - #151 .

[comps]

We should also consider splitting productization into ie. daily and weekly.

The reason is our ever-growing test coverage - we already have barely enough resources to run tests daily. Once we add UEFI testing next to with-gui, it will grow even more. Not to mention the fact that we already don't test non-x86_64 outside of stabilization (presumably).

What will happen when we start auto-scheduling up to 2 reruns of failed/errored tests?

Maybe we should run

• /plans/daily - set of tests that can 100% finish overnight (from cca 00:00 to 07:00)
  • all of /hardening/{oscap,anaconda,image-builder,ansible}, but without any extra variants (eg. no gui/uefi)
  • one gui test per hardening type (probably STIG)
  • one uefi test per hardening type (maybe OSPP?)
  • x86_64 only, VMs only, no host-os
  • anything reasonably cheap from /scanning and /static-checks
  • on supported RHEL majors (latest minors we plan to target with next SSG release)
• /plans/weekly - to be run over weekend (saturday 00:00 until monday 07:00; 24+24+7 hours)
  • everything except any manual tasks (which include /static-checks/diff and any other tool-like tests)
  • incl. all GUI/UEFI tests, incl. host-os on all architectures
  • on supported RHEL majors (latest minors we plan to target with next SSG release)
• /plans/stabilization
  • like weekly, but incl. manual tests (or tests needing manual review)
  • on multiple ZStreams too, not just one minor per major

(alternatively, we could make weekly run x86_64 only, but on all ZStreams)

[mildas]

/plans/daily - set of tests that can 100% finish overnight (from cca 00:00 to 07:00)

Well, if we talk runcontest context, where we save a lot of time thanks to test grouping, I believe finishing overnight shouldn't be problem even if we include GUI variants + host-os.
To other points: one uefi - yes, x86_64 only - yes, everything reasonably cheap - yes but also reliable (e.g. skip HTML links), on latest RHEL majors - yes.

/plans/weekly - to be run over weekend (saturday 00:00 until monday 07:00; 24+24+7 hours)

Agree with all points. I'm open to discuss if weekly should be on latest majors or also Z-stream.

/plans/stabilization

Sounds good

[comps]

As agreed with @mildas , I have split productization into daily/weekly test sets, for now.

With that, this PR is ready for review.
If merged, I plan to use /plans/daily + /plans/weekly for runcontest.

To recap - this is about

• using TMT plans for test set selection
• using TMT test tags as "traits", which can then be used by plans
  • (rather than using tags to pre-select tests into given use cases like productization)

Maybe something like

args=()
case "$PROD_TYPE" in
    daily)
       args+=(
            --plan /plans/daily
            --arch x86_64
            --rhel 8 --rhel 9 --compose-hint '9:RHEL-9.4.0-%'
       ) ;;
    weekly_rhels)
        args+=(
            --plan /plans/weekly
            --arch x86_64
            --rhel 8.6 --rhel 8.8 --rhel 8.10
            --rhel 9.0 --rhel 9.2 --rhel 9.4
        ) ;;
    weekly_arches)
        args+=(
            --plan /plans/weekly
            # all architectures by default
            --rhel 8 --rhel 9 --compose-hint '9:RHEL-9.4.0-%'
        ) ;;
esac

as we likely won't have the resources to run all minors + all arches + all tests during one weekend, so we could alter odd/even weekends.

[mildas]

broken tag is misleading. You have to read the tag description to understand it's about broken functionality it tests, not broken test. If we could agree on something clearer, it would be great. (I couldn't come with something better during review so I can live with broken tag though)

[comps]

@mildas Maybe suspended instead of broken?

Also, agreed on running with-gui for stig_gui at least, but possibly for all tests that use snapshots (oscap, ansible), as they should be comparatively cheap.

Maybe we should add a new snapshottable tag, so the tests can be selected without explicitly listing stig_gui in the plans?

[comps]

Rebased, amended.

[mildas]

@comps suspended doesn't sound much better than broken, to that I'd rather stay with broken.

snapshotable (or just snapshot) tag sounds ok

[comps]

Ok, changed back to broken + left GUI only for stig_gui. No need for snapshottable, so I didn't add it.