Initial flaky test mitigation

[perdasilva]

Description of the change:
This PR makes the following changes and resolved aspects of #2589:

• Adds a method to mark e2e tests as flaky
• Separates flaky test execution into its own, non-required, gh action
• Adds an issue template for flaky tests
• Identifies some flaky tests (maybe most, but I fear not all)
• Adds contributor documentation around filing flaky tests
• Updates the PR template with additional reviewer criteria
• Adds an utility function to the e2e tests to make strings k8s friendly

Note
With Ginkgo v2.0 we can actually label tests, which might make this system less brittle. I just don't know what the upfront work required would be to make the upgrade. So, let's use this as a stop gap solution until we migrate and it should be easy enough to then migrate the flaky tests to use labels instead.

Motivation for the change:
Merging PRs is a PITA atm because of flakes. This gives us a way to mark flakes and remove them from the critical path while they get fixed
Reviewer Checklist

• Implementation matches the proposed design, or proposal is updated to match implementation
• Sufficient unit test coverage
• Sufficient end-to-end test coverage
• Docs updated or added to /doc
• Commit messages sensible and descriptive

Targets aspects of #2589

[perdasilva]

/hold

[njhale]

Reference: Using label decorators in Ginkgo 2.0

[njhale]

/lgtm

[njhale]

/lgtm

[tylerslaton]

/lgtm

[njhale]

@perdasilva Looks like the tests are generating invalid label values ATM:

unable to parse requirement: values[0][label]: Invalid value: "[FLAKY]cleanupcsvswithbadowneroperatorgroups": a valid label must be an empty string or consist of alphanumeric characters, '-', '_' or '.', and must start and end with an alphanumeric character (e.g. 'MyValue',  or 'my_value',  or '12345', regex used for validation is '(([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9])?')

see https://github.com/operator-framework/operator-lifecycle-manager/runs/5202410751?check_suite_focus=true

[tylerslaton]

/lgtm

[timflannagan]

/lgtm

[openshift-ci]

New changes are detected. LGTM label has been removed.

[timflannagan]

Looks like the filtering isn't working as intended?

Summarizing 2 Failures:

[Fail] Operator API [It] [FLAKE] should surface components in its status 
/home/runner/work/operator-lifecycle-manager/operator-lifecycle-manager/test/e2e/util.go:894

/hold

[openshift-ci]

New changes are detected. LGTM label has been removed.

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: perdasilva

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [perdasilva]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[openshift-ci]

New changes are detected. LGTM label has been removed.