Skip flaky tests

[perdasilva]

Signed-off-by: perdasilva perdasilva@redhat.com

Description of the change:
This PR updates the e2e test mechanism to skip tests marked with a [FLAKY] tag.

Note

With Ginkgo v2.0 we can actually label tests, which might make this system less brittle. I just don't know what the upfront work required would be to make the upgrade. So, let's use this as a stop gap solution until we migrate and it should be easy enough to then migrate the flaky tests to use labels instead.

Motivation for the change:
Merging PRs is a PITA atm because of flakes. This gives us a way to mark flakes and remove them from the critical path while they get fixed

Reviewer Checklist

• Implementation matches the proposed design, or proposal is updated to match implementation
• Sufficient unit test coverage
• Sufficient end-to-end test coverage
• Docs updated or added to /doc
• Commit messages sensible and descriptive

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: perdasilva

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [perdasilva]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[perdasilva]

/hold

[timflannagan]

Any value in adding another workflow that's responsible for running the "[FLAKY]" regex so we don't skip the coverage of these tests? The overall CI signal is fairly watered down right now, so offloading them to their own workflow (instead of directly skipping them all together) wouldn't add a ton of value, but at least we wouldn't have to re-enable them, or manually promote a flaky test to a stable one, etc. over time.

That can be done as a follow-up as well.

[perdasilva]

Any value in adding another workflow that's responsible for running the "[FLAKY]" regex so we don't skip the coverage of these tests? The overall CI signal is fairly watered down right now, so offloading them to their own workflow (instead of directly skipping them all together) wouldn't add a ton of value, but at least we wouldn't have to re-enable them, or manually promote a flaky test to a stable one, etc. over time.

That can be done as a follow-up as well.

Yup - that's something I've had in mind to do as well. I want to find a way to execute these in isolation one at a time. To further reduce the failure noise while we fix them. I'm just re-running the e2e jobs over and over a few times until I feel the PR is "stable enough" and we have accounted for all (as many as possible) flaky tests and have those with a corresponding ticket.

[perdasilva]

Any value in adding another workflow that's responsible for running the "[FLAKY]" regex so we don't skip the coverage of these tests? The overall CI signal is fairly watered down right now, so offloading them to their own workflow (instead of directly skipping them all together) wouldn't add a ton of value, but at least we wouldn't have to re-enable them, or manually promote a flaky test to a stable one, etc. over time.

That can be done as a follow-up as well.

My main worry though is, what is the value of tests that just add confusion? i.e. you don't know if they passed or failed or failed because of change, etc. I'd say it's hard to get away from the disable/fix/enable pattern. If a test is flaky, it will need to be fixed. When it's fixed, just remove [FLAKY] from the name. If it's flaky, add [FLAKY] to the name and remove it from the critical path and create a tracking ticket. Meaning, the toil can be piggyback off the natural PRs.

[timflannagan]

Yeah agreed - my main worry with simply skipping these tests as there's no tracking vehicle for fixing them right now, outside of creating individual issues and burning down those over time. Having a dedicated workflow that still gives CI signal, while not blocking a PR from merging as that flaky workflow isn't a required check, might be decent short term balance. I don't have a strong opinion on this approach as long as we commit to tracking these flakes $somewhere so they don't inevitably get swept under the rug.

[perdasilva]

Closing and reopening to see if it removes the required flag from the flaky-e2e tests

[perdasilva]

re-created as #2625