Add a minimum cluster stability timeout for OpenShift e2e remediation #12184
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
The OpenShift content has a manual remeidation for setting up an identity provider, which includes creating a secret, updating the authentication configuration, and bouncing the authentication operator. The test suite needs to make sure the authentication operator is up and ready before it continues, and we recently updated the logic to make sure it was ready by using the `oc adm wait-for-stable-cluster` command. This command is ideal because it checks all cluster operators are running and stable, not just the authentication operator. One side-effect of using this command though is that it polls on successful cluster conditions. We were using it from the perspective that the command would exit gracefully when the cluster was stable, which isn't the case. However, we can pass and argument to the command to force an exit after the cluster is stable for a certain period of time. This commit updates the command to do that so that the command doesn't hang and cause timeouts in our testing.
|
Start a new ephemeral environment with changes proposed in this pull request: Fedora Environment Oracle Linux 8 Environment |
|
/test |
|
@rhmdnd: The
Use
In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
|
/test 4.13-e2e-aws-ocp4-cis |
|
🤖 A k8s content image for this PR is available at: Click here to see how to deploy itIf you alread have Compliance Operator deployed: Otherwise deploy the content and operator together by checking out ComplianceAsCode/compliance-operator and: |
|
Code Climate has analyzed commit 7e3abc1 and detected 0 issues on this pull request. The test coverage on the diff in this pull request is 100.0% (50% is the threshold). This pull request will bring the total coverage in the repository to 59.4% (0.0% change). View more on Code Climate. |
yuumasato
approved these changes
Jul 19, 2024
This was referenced Jul 19, 2024
rhmdnd
added a commit
to rhmdnd/content
that referenced
this pull request
Jul 26, 2024
Lately, we've been experiencing issues with manual remediations timing
out during functional testing. This manifests in the following error:
=== RUN TestE2e/Apply_manual_remediations
<snip>
helpers.go:1225: Running manual remediation '/tmp/content-3345141771/applications/openshift/networking/default_ingress_ca_replaced/tests/ocp4/e2e-remediation.sh'
helpers.go:1225: Running manual remediation '/tmp/content-3345141771/applications/openshift/general/file_integrity_notification_enabled/tests/ocp4/e2e-remediation.sh'
helpers.go:1231: Command '/tmp/content-3345141771/applications/openshift/authentication/idp_is_configured/tests/ocp4/e2e-remediation.sh' timed out
In this particular case, it looks like the remediation to add an
Identity Provider to the cluster failed, but this is actually an
unintended side-effect of another change that updated the
idp_is_configured remediation to use a more robust technique for
determining if the cluster applied the remediation successfully:
ComplianceAsCode#12120
ComplianceAsCode#12184
Because we updated the remediation to use `oc adm
wait-for-stable-cluster`, we're effectively checking all cluster
operators to ensure they're healthy.
This started causing timeouts because a separate, unrelated remediation
was also getting applied in our testing that updated the default CA, but
didn't include a ConfigMap that contained the CA bundle. As a result,
one of the operators didn't come up because it was looking for a
ConfigMap that didn't exist. The `oc adm wait-for-stable-cluster`
command was hanging on a legitimate issue in a separate remediation.
This commit attempts to fix that issue by updating the trusted CA
remediation by generating a certificate for testing purposes, then
creates a ConfigMap called `trusted-ca-bundle`, before updating the
trusted CA.
rhmdnd
added a commit
to rhmdnd/content
that referenced
this pull request
Jul 26, 2024
Lately, we've been experiencing issues with manual remediations timing
out during functional testing. This manifests in the following error:
=== RUN TestE2e/Apply_manual_remediations
<snip>
helpers.go:1225: Running manual remediation '/tmp/content-3345141771/applications/openshift/networking/default_ingress_ca_replaced/tests/ocp4/e2e-remediation.sh'
helpers.go:1225: Running manual remediation '/tmp/content-3345141771/applications/openshift/general/file_integrity_notification_enabled/tests/ocp4/e2e-remediation.sh'
helpers.go:1231: Command '/tmp/content-3345141771/applications/openshift/authentication/idp_is_configured/tests/ocp4/e2e-remediation.sh' timed out
In this particular case, it looks like the remediation to add an
Identity Provider to the cluster failed, but this is actually an
unintended side-effect of another change that updated the
idp_is_configured remediation to use a more robust technique for
determining if the cluster applied the remediation successfully:
ComplianceAsCode#12120
ComplianceAsCode#12184
Because we updated the remediation to use `oc adm
wait-for-stable-cluster`, we're effectively checking all cluster
operators to ensure they're healthy.
This started causing timeouts because a separate, unrelated remediation
was also getting applied in our testing that updated the default CA, but
didn't include a ConfigMap that contained the CA bundle. As a result,
one of the operators didn't come up because it was looking for a
ConfigMap that didn't exist. The `oc adm wait-for-stable-cluster`
command was hanging on a legitimate issue in a separate remediation.
This commit attempts to fix that issue by updating the trusted CA
remediation by generating a certificate for testing purposes.
rhmdnd
added a commit
to rhmdnd/content
that referenced
this pull request
Jul 30, 2024
Lately, we've been experiencing issues with manual remediations timing
out during functional testing. This manifests in the following error:
=== RUN TestE2e/Apply_manual_remediations
<snip>
helpers.go:1225: Running manual remediation '/tmp/content-3345141771/applications/openshift/networking/default_ingress_ca_replaced/tests/ocp4/e2e-remediation.sh'
helpers.go:1225: Running manual remediation '/tmp/content-3345141771/applications/openshift/general/file_integrity_notification_enabled/tests/ocp4/e2e-remediation.sh'
helpers.go:1231: Command '/tmp/content-3345141771/applications/openshift/authentication/idp_is_configured/tests/ocp4/e2e-remediation.sh' timed out
In this particular case, it looks like the remediation to add an
Identity Provider to the cluster failed, but this is actually an
unintended side-effect of another change that updated the
idp_is_configured remediation to use a more robust technique for
determining if the cluster applied the remediation successfully:
ComplianceAsCode#12120
ComplianceAsCode#12184
Because we updated the remediation to use `oc adm
wait-for-stable-cluster`, we're effectively checking all cluster
operators to ensure they're healthy.
This started causing timeouts because a separate, unrelated remediation
was also getting applied in our testing that updated the default CA, but
didn't include a ConfigMap that contained the CA bundle. As a result,
one of the operators didn't come up because it was looking for a
ConfigMap that didn't exist. The `oc adm wait-for-stable-cluster`
command was hanging on a legitimate issue in a separate remediation.
This commit attempts to fix that issue by updating the trusted CA
remediation by creating a configmap for the expected certificate bundle.
rhmdnd
added a commit
to rhmdnd/content
that referenced
this pull request
Jul 31, 2024
Lately, we've been experiencing issues with manual remediations timing
out during functional testing. This manifests in the following error:
=== RUN TestE2e/Apply_manual_remediations
<snip>
helpers.go:1225: Running manual remediation '/tmp/content-3345141771/applications/openshift/networking/default_ingress_ca_replaced/tests/ocp4/e2e-remediation.sh'
helpers.go:1225: Running manual remediation '/tmp/content-3345141771/applications/openshift/general/file_integrity_notification_enabled/tests/ocp4/e2e-remediation.sh'
helpers.go:1231: Command '/tmp/content-3345141771/applications/openshift/authentication/idp_is_configured/tests/ocp4/e2e-remediation.sh' timed out
In this particular case, it looks like the remediation to add an
Identity Provider to the cluster failed, but this is actually an
unintended side-effect of another change that updated the
idp_is_configured remediation to use a more robust technique for
determining if the cluster applied the remediation successfully:
ComplianceAsCode#12120
ComplianceAsCode#12184
Because we updated the remediation to use `oc adm
wait-for-stable-cluster`, we're effectively checking all cluster
operators to ensure they're healthy.
This started causing timeouts because a separate, unrelated remediation
was also getting applied in our testing that updated the default CA, but
didn't include a ConfigMap that contained the CA bundle. As a result,
one of the operators didn't come up because it was looking for a
ConfigMap that didn't exist. The `oc adm wait-for-stable-cluster`
command was hanging on a legitimate issue in a separate remediation.
This commit attempts to fix that issue by updating the trusted CA
remediation by creating a configmap for the expected certificate bundle.
vojtapolasek
pushed a commit
to vojtapolasek/content
that referenced
this pull request
Aug 1, 2024
Lately, we've been experiencing issues with manual remediations timing
out during functional testing. This manifests in the following error:
=== RUN TestE2e/Apply_manual_remediations
<snip>
helpers.go:1225: Running manual remediation '/tmp/content-3345141771/applications/openshift/networking/default_ingress_ca_replaced/tests/ocp4/e2e-remediation.sh'
helpers.go:1225: Running manual remediation '/tmp/content-3345141771/applications/openshift/general/file_integrity_notification_enabled/tests/ocp4/e2e-remediation.sh'
helpers.go:1231: Command '/tmp/content-3345141771/applications/openshift/authentication/idp_is_configured/tests/ocp4/e2e-remediation.sh' timed out
In this particular case, it looks like the remediation to add an
Identity Provider to the cluster failed, but this is actually an
unintended side-effect of another change that updated the
idp_is_configured remediation to use a more robust technique for
determining if the cluster applied the remediation successfully:
ComplianceAsCode#12120
ComplianceAsCode#12184
Because we updated the remediation to use `oc adm
wait-for-stable-cluster`, we're effectively checking all cluster
operators to ensure they're healthy.
This started causing timeouts because a separate, unrelated remediation
was also getting applied in our testing that updated the default CA, but
didn't include a ConfigMap that contained the CA bundle. As a result,
one of the operators didn't come up because it was looking for a
ConfigMap that didn't exist. The `oc adm wait-for-stable-cluster`
command was hanging on a legitimate issue in a separate remediation.
This commit attempts to fix that issue by updating the trusted CA
remediation by creating a configmap for the expected certificate bundle.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The OpenShift content has a manual remeidation for setting up an
identity provider, which includes creating a secret, updating the
authentication configuration, and bouncing the authentication operator.
The test suite needs to make sure the authentication operator is up and
ready before it continues, and we recently updated the logic to make
sure it was ready by using the
oc adm wait-for-stable-clustercommand.This command is ideal because it checks all cluster operators are
running and stable, not just the authentication operator.
One side-effect of using this command though is that it polls on
successful cluster conditions. We were using it from the perspective
that the command would exit gracefully when the cluster was stable,
which isn't the case.
However, we can pass and argument to the command to force an exit after
the cluster is stable for a certain period of time. This commit updates
the command to do that so that the command doesn't hang and cause
timeouts in our testing.