Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

End-to-end tests don't clean up scans when run individually #231

Closed
rhmdnd opened this issue Feb 23, 2023 · 4 comments
Closed

End-to-end tests don't clean up scans when run individually #231

rhmdnd opened this issue Feb 23, 2023 · 4 comments

Comments

@rhmdnd
Copy link

rhmdnd commented Feb 23, 2023

If you run a single e2e test that produces a scan using the -testRegex flag (introduced in PR #227), you'll notice that the scan created by the test isn't cleaned up.

This causes issues when you try and run the same test twice.

Here's what I did to reproduce:

$ E2E_SKIP_CLEANUP_ON_ERROR=false E2E_GO_TEST_FLAGS="-v -testRegex TestScanProducesRemediations" make e2e

Which leaves the following scan after the test completes:

NAMESPACE                                       NAME                                                                     PHASE   RESULT
osdk-e2e-3ae56ab8-d508-4146-846d-bf28fc751e91   compliancescan.compliance.openshift.io/test-scan-produces-remediations   DONE    NON-COMPLIANT

Running the test test again fails because the test times out (the scan setting binding doesn't process).
@rhmdnd
Copy link
Author

rhmdnd commented Feb 23, 2023

Looking at the namespace, it appears the scan finalizer tripped:

- lastTransitionTime: "2023-02-23T19:53:57Z"
    message: 'Some content in the namespace has finalizers remaining: scan.finalizers.compliance.openshift.io
      in 1 resource instances, suite.finalizers.compliance.openshift.io in 1 resource
      instances'
    reason: SomeFinalizersRemain
    status: "True"
    type: NamespaceFinalizersRemaining
  phase: Terminating

@rhmdnd
Copy link
Author

rhmdnd commented Mar 2, 2023 •
edited
Loading

Also noticed that if a test fails, the cluster role bindings are not cleaned up.

Is this intentional?

    helpers.go:251: 2023-03-02T14:44:54-06:00: Running e2e test on Namespace osdk-e2e-dd3cb879-31cb-4e59-a2eb-43f8e3587d4a
    client.go:58: resource type ServiceAccount with namespace/name (osdk-e2e-dd3cb879-31cb-4e59-a2eb-43f8e3587d4a/api-resource-collector) created
    client.go:58: resource type ServiceAccount with namespace/name (osdk-e2e-dd3cb879-31cb-4e59-a2eb-43f8e3587d4a/compliance-operator) created
    client.go:58: resource type ServiceAccount with namespace/name (osdk-e2e-dd3cb879-31cb-4e59-a2eb-43f8e3587d4a/profileparser) created
    client.go:58: resource type ServiceAccount with namespace/name (osdk-e2e-dd3cb879-31cb-4e59-a2eb-43f8e3587d4a/remediation-aggregator) created
    client.go:58: resource type ServiceAccount with namespace/name (osdk-e2e-dd3cb879-31cb-4e59-a2eb-43f8e3587d4a/rerunner) created
    client.go:58: resource type ServiceAccount with namespace/name (osdk-e2e-dd3cb879-31cb-4e59-a2eb-43f8e3587d4a/resultscollector) created
    client.go:58: resource type ServiceAccount with namespace/name (osdk-e2e-dd3cb879-31cb-4e59-a2eb-43f8e3587d4a/resultserver) created
    client.go:58: resource type Role with namespace/name (osdk-e2e-dd3cb879-31cb-4e59-a2eb-43f8e3587d4a/api-resource-collector) created
    client.go:58: resource type Role with namespace/name (osdk-e2e-dd3cb879-31cb-4e59-a2eb-43f8e3587d4a/compliance-operator) created
    client.go:58: resource type Role with namespace/name (osdk-e2e-dd3cb879-31cb-4e59-a2eb-43f8e3587d4a/leader-election-role) created
    client.go:58: resource type Role with namespace/name (osdk-e2e-dd3cb879-31cb-4e59-a2eb-43f8e3587d4a/profileparser) created
    client.go:58: resource type Role with namespace/name (osdk-e2e-dd3cb879-31cb-4e59-a2eb-43f8e3587d4a/remediation-aggregator) created
    client.go:58: resource type Role with namespace/name (osdk-e2e-dd3cb879-31cb-4e59-a2eb-43f8e3587d4a/rerunner) created
    client.go:58: resource type Role with namespace/name (osdk-e2e-dd3cb879-31cb-4e59-a2eb-43f8e3587d4a/resultscollector) created
    client.go:58: resource type Role with namespace/name (osdk-e2e-dd3cb879-31cb-4e59-a2eb-43f8e3587d4a/resultserver) created
    helpers.go:400: failed to initialize cluster resources: clusterroles.rbac.authorization.k8s.io "api-resource-collector" already exists
    client.go:79: resource type Role with namespace/name (osdk-e2e-dd3cb879-31cb-4e59-a2eb-43f8e3587d4a/resultserver) successfully deleted
    client.go:79: resource type Role with namespace/name (osdk-e2e-dd3cb879-31cb-4e59-a2eb-43f8e3587d4a/resultscollector) successfully deleted
    client.go:79: resource type Role with namespace/name (osdk-e2e-dd3cb879-31cb-4e59-a2eb-43f8e3587d4a/rerunner) successfully deleted
    client.go:79: resource type Role with namespace/name (osdk-e2e-dd3cb879-31cb-4e59-a2eb-43f8e3587d4a/remediation-aggregator) successfully deleted
    client.go:79: resource type Role with namespace/name (osdk-e2e-dd3cb879-31cb-4e59-a2eb-43f8e3587d4a/profileparser) successfully deleted
    client.go:79: resource type Role with namespace/name (osdk-e2e-dd3cb879-31cb-4e59-a2eb-43f8e3587d4a/leader-election-role) successfully deleted
    client.go:79: resource type Role with namespace/name (osdk-e2e-dd3cb879-31cb-4e59-a2eb-43f8e3587d4a/compliance-operator) successfully deleted
    client.go:79: resource type Role with namespace/name (osdk-e2e-dd3cb879-31cb-4e59-a2eb-43f8e3587d4a/api-resource-collector) successfully deleted
    client.go:79: resource type ServiceAccount with namespace/name (osdk-e2e-dd3cb879-31cb-4e59-a2eb-43f8e3587d4a/resultserver) successfully deleted
    client.go:79: resource type ServiceAccount with namespace/name (osdk-e2e-dd3cb879-31cb-4e59-a2eb-43f8e3587d4a/resultscollector) successfully deleted
    client.go:79: resource type ServiceAccount with namespace/name (osdk-e2e-dd3cb879-31cb-4e59-a2eb-43f8e3587d4a/rerunner) successfully deleted
    client.go:79: resource type ServiceAccount with namespace/name (osdk-e2e-dd3cb879-31cb-4e59-a2eb-43f8e3587d4a/remediation-aggregator) successfully deleted
    client.go:79: resource type ServiceAccount with namespace/name (osdk-e2e-dd3cb879-31cb-4e59-a2eb-43f8e3587d4a/profileparser) successfully deleted
    client.go:79: resource type ServiceAccount with namespace/name (osdk-e2e-dd3cb879-31cb-4e59-a2eb-43f8e3587d4a/compliance-operator) successfully deleted
    client.go:79: resource type ServiceAccount with namespace/name (osdk-e2e-dd3cb879-31cb-4e59-a2eb-43f8e3587d4a/api-resource-collector) successfully deleted
--- FAIL: TestE2E (99.30s)

@rhmdnd
Copy link
Author

rhmdnd commented Mar 2, 2023

Something else I noticed is that scans aren't cleaned up in time.

=== CONT  TestE2E
    client.go:79: resource type  with namespace/name (osdk-e2e-da534140-399d-4b6a-8472-cafa94a42ec3/hide-rules-test) successfully deleted
    client.go:79: resource type  with namespace/name (osdk-e2e-da534140-399d-4b6a-8472-cafa94a42ec3/hide-rules-test) successfully deleted
    client.go:79: resource type  with namespace/name (osdk-e2e-da534140-399d-4b6a-8472-cafa94a42ec3/test-hide-rule) successfully deleted
    client.go:79: resource type Deployment with namespace/name (osdk-e2e-da534140-399d-4b6a-8472-cafa94a42ec3/compliance-operator) successfully deleted
    client.go:79: resource type ClusterRoleBinding with namespace/name (/remediation-aggregator) successfully deleted
    client.go:79: resource type ClusterRoleBinding with namespace/name (/compliance-operator) successfully deleted
    client.go:79: resource type ClusterRoleBinding with namespace/name (/api-resource-collector) successfully deleted
    client.go:79: resource type RoleBinding with namespace/name (osdk-e2e-da534140-399d-4b6a-8472-cafa94a42ec3/resultserver) successfully deleted
    client.go:79: resource type RoleBinding with namespace/name (osdk-e2e-da534140-399d-4b6a-8472-cafa94a42ec3/resultscollector) successfully deleted
    client.go:79: resource type RoleBinding with namespace/name (osdk-e2e-da534140-399d-4b6a-8472-cafa94a42ec3/rerunner) successfully deleted
    client.go:79: resource type RoleBinding with namespace/name (osdk-e2e-da534140-399d-4b6a-8472-cafa94a42ec3/remediation-aggregator) successfully deleted
    client.go:79: resource type RoleBinding with namespace/name (osdk-e2e-da534140-399d-4b6a-8472-cafa94a42ec3/profileparser) successfully deleted
    client.go:79: resource type RoleBinding with namespace/name (osdk-e2e-da534140-399d-4b6a-8472-cafa94a42ec3/leader-election-rolebinding) successfully deleted
    client.go:79: resource type RoleBinding with namespace/name (osdk-e2e-da534140-399d-4b6a-8472-cafa94a42ec3/compliance-operator-metrics) successfully deleted
    client.go:79: resource type RoleBinding with namespace/name (osdk-e2e-da534140-399d-4b6a-8472-cafa94a42ec3/compliance-operator) successfully deleted
    client.go:79: resource type RoleBinding with namespace/name (osdk-e2e-da534140-399d-4b6a-8472-cafa94a42ec3/api-resource-collector) successfully deleted
    client.go:79: resource type ClusterRole with namespace/name (/tailoredprofile-viewer-role) successfully deleted
    client.go:79: resource type ClusterRole with namespace/name (/tailoredprofile-editor-role) successfully deleted
    client.go:79: resource type ClusterRole with namespace/name (/scansettingbinding-viewer-role) successfully deleted
    client.go:79: resource type ClusterRole with namespace/name (/scansettingbinding-editor-role) successfully deleted
    client.go:79: resource type ClusterRole with namespace/name (/remediation-aggregator) successfully deleted
    client.go:79: resource type ClusterRole with namespace/name (/profilebundle-viewer-role) successfully deleted
    client.go:79: resource type ClusterRole with namespace/name (/profilebundle-editor-role) successfully deleted
    client.go:79: resource type ClusterRole with namespace/name (/compliancesuite-viewer-role) successfully deleted
    client.go:79: resource type ClusterRole with namespace/name (/compliancesuite-editor-role) successfully deleted
    client.go:79: resource type ClusterRole with namespace/name (/compliancescan-viewer-role) successfully deleted
    client.go:79: resource type ClusterRole with namespace/name (/compliancescan-editor-role) successfully deleted
    client.go:79: resource type ClusterRole with namespace/name (/complianceremediation-viewer-role) successfully deleted
    client.go:79: resource type ClusterRole with namespace/name (/complianceremediation-editor-role) successfully deleted
    client.go:79: resource type ClusterRole with namespace/name (/compliance-operator-metrics) successfully deleted
    client.go:79: resource type ClusterRole with namespace/name (/compliance-operator) successfully deleted
    client.go:79: resource type ClusterRole with namespace/name (/api-resource-collector) successfully deleted
    client.go:79: resource type Role with namespace/name (osdk-e2e-da534140-399d-4b6a-8472-cafa94a42ec3/resultserver) successfully deleted
    client.go:79: resource type Role with namespace/name (osdk-e2e-da534140-399d-4b6a-8472-cafa94a42ec3/resultscollector) successfully deleted
    client.go:79: resource type Role with namespace/name (osdk-e2e-da534140-399d-4b6a-8472-cafa94a42ec3/rerunner) successfully deleted
    client.go:79: resource type Role with namespace/name (osdk-e2e-da534140-399d-4b6a-8472-cafa94a42ec3/remediation-aggregator) successfully deleted
    client.go:79: resource type Role with namespace/name (osdk-e2e-da534140-399d-4b6a-8472-cafa94a42ec3/profileparser) successfully deleted
    client.go:79: resource type Role with namespace/name (osdk-e2e-da534140-399d-4b6a-8472-cafa94a42ec3/leader-election-role) successfully deleted
    client.go:79: resource type Role with namespace/name (osdk-e2e-da534140-399d-4b6a-8472-cafa94a42ec3/compliance-operator) successfully deleted
    client.go:79: resource type Role with namespace/name (osdk-e2e-da534140-399d-4b6a-8472-cafa94a42ec3/api-resource-collector) successfully deleted
    client.go:79: resource type ServiceAccount with namespace/name (osdk-e2e-da534140-399d-4b6a-8472-cafa94a42ec3/resultserver) successfully deleted
    client.go:79: resource type ServiceAccount with namespace/name (osdk-e2e-da534140-399d-4b6a-8472-cafa94a42ec3/resultscollector) successfully deleted
    client.go:79: resource type ServiceAccount with namespace/name (osdk-e2e-da534140-399d-4b6a-8472-cafa94a42ec3/rerunner) successfully deleted
    client.go:79: resource type ServiceAccount with namespace/name (osdk-e2e-da534140-399d-4b6a-8472-cafa94a42ec3/remediation-aggregator) successfully deleted
    client.go:79: resource type ServiceAccount with namespace/name (osdk-e2e-da534140-399d-4b6a-8472-cafa94a42ec3/profileparser) successfully deleted
    client.go:79: resource type ServiceAccount with namespace/name (osdk-e2e-da534140-399d-4b6a-8472-cafa94a42ec3/compliance-operator) successfully deleted
    client.go:79: resource type ServiceAccount with namespace/name (osdk-e2e-da534140-399d-4b6a-8472-cafa94a42ec3/api-resource-collector) successfully deleted
--- PASS: TestE2E (318.87s)
    --- PASS: TestE2E/Parallel_tests (0.00s)
        --- PASS: TestE2E/Parallel_tests/TestHideRule (115.55s)
PASS
ok      github.com/ComplianceAsCode/compliance-operator/tests/e2e       321.139s
╭─lbragstad@p1 ~/go/src/github.com/rhmdnd/compliance-operator ‹OCPBUGS-3009*›
╰─➤  $ oc get scans,suite,ssb -A
NAMESPACE                                       NAME                                                     PHASE   RESULT
osdk-e2e-da534140-399d-4b6a-8472-cafa94a42ec3   compliancescan.compliance.openshift.io/hide-rules-test   DONE    NOT-APPLICABLE

NAMESPACE                                       NAME                                                      PHASE   RESULT
osdk-e2e-da534140-399d-4b6a-8472-cafa94a42ec3   compliancesuite.compliance.openshift.io/hide-rules-test   DONE    NOT-APPLICABLE
Error from server (NotFound): Unable to list "compliance.openshift.io/v1alpha1, Resource=scansettingbindings": the server could not find the requested resource (get scansettingbindings.compliance.openshift.io)

I was able to recreate this using $ E2E_SKIP_CLEANUP_ON_ERROR=false E2E_GO_TEST_FLAGS="-v -run TestE2E/Parallel_tests/TestHideRule" make e2e from latest master.

I know the tests do clean up the SSB, but I wonder if the tests are cleaning up the rest of the environment/CRDs before the scan and suite get deleted.

@rhmdnd
Copy link
Author

rhmdnd commented Mar 3, 2023

Added some debugging to the e2e suite that might help us track down what's going on.

#246

@rtheis rtheis mentioned this issue Jun 29, 2023
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@rhmdnd