Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[WIP] OCPBUGS-42045: Fix IBMCloud DNS Propagation Issues #1164

Draft
wants to merge 1 commit into
base: master
Choose a base branch
from
Draft

[WIP] OCPBUGS-42045: Fix IBMCloud DNS Propagation Issues #1164

wants to merge 1 commit into from

Conversation

gcs278
Copy link
Contributor

@gcs278 gcs278 commented Oct 29, 2024

Resolves IBMCloud DNS names inside the cluster because they don't propagate to the test runner clusters in a reasonable time.

@openshift-ci openshift-ci bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Oct 29, 2024
@openshift-ci OpenShift CI
Copy link
Contributor

openshift-ci bot commented Oct 29, 2024

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

@openshift-ci OpenShift CI
Copy link
Contributor

openshift-ci bot commented Oct 29, 2024

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please ask for approval from gcs278. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@gcs278
Copy link
Contributor Author

gcs278 commented Oct 29, 2024

/test e2e-ibmcloud-operator

@gcs278 gcs278 force-pushed the ibmcloud-e2e-dns-fix branch from 66cbc36 to c029765 Compare October 29, 2024 04:05
@gcs278
Copy link
Contributor Author

gcs278 commented Oct 29, 2024

/test e2e-ibmcloud-operator

@gcs278 gcs278 force-pushed the ibmcloud-e2e-dns-fix branch from c029765 to 133382a Compare October 29, 2024 17:03
@gcs278
Copy link
Contributor Author

gcs278 commented Oct 29, 2024

/test e2e-ibmcloud-operator

@gcs278 gcs278 force-pushed the ibmcloud-e2e-dns-fix branch 2 times, most recently from afee45c to 34c2e67 Compare October 29, 2024 19:48
@gcs278
Copy link
Contributor Author

gcs278 commented Oct 29, 2024

/test e2e-ibmcloud-operator

@gcs278
Copy link
Contributor Author

gcs278 commented Oct 29, 2024

To make sure I didn't break anything:
/test e2e-aws-operator

@gcs278
Copy link
Contributor Author

gcs278 commented Oct 30, 2024

Success!
/test e2e-ibmcloud-operator

@gcs278 gcs278 mentioned this pull request Oct 30, 2024
Open
@gcs278
Copy link
Contributor Author

gcs278 commented Nov 11, 2024

I've temporarily added an availability check for IngressController in TestScopeChange to confirm that it would catch the issue fixed in #1133. I'll remove and place in #1133 once confirmed it fails here.
/test e2e-ibmcloud-operator

@gcs278 gcs278 force-pushed the ibmcloud-e2e-dns-fix branch from 4f8b44e to 34c2e67 Compare November 12, 2024 14:28
@gcs278
Copy link
Contributor Author

gcs278 commented Nov 12, 2024

Yup, it failed, that's a good thing 👍 Added the TestScopeChange updates to #1133 and removed from here.

@gcs278 gcs278 force-pushed the ibmcloud-e2e-dns-fix branch from 34c2e67 to 9f6e396 Compare November 12, 2024 15:05
@gcs278
Copy link
Contributor Author

gcs278 commented Nov 12, 2024

/test e2e-ibmcloud-operator

@gcs278
Copy link
Contributor Author

gcs278 commented Nov 13, 2024

success, but failed on deprovisioning:
/test e2e-ibmcloud-operator

@gcs278
Copy link
Contributor Author

gcs278 commented Nov 13, 2024

Failure with:

=== NAME  TestAll/parallel/TestMTLSWithCRLs/multiple-intermediate-ca
    client_tls_test.go:1079: failed to get hostname of route "echo-pod-8ccjt"

Looks like a flake in which the router didn't admit the router within 1 minute. That's odd, but I doubt specific to IBMCloud.
/test e2e-ibmcloud-operator

@gcs278 gcs278 changed the title [WIP] Fix IBMCloud DNS Propagation Issues OCPBUGS-42045: Fix IBMCloud DNS Propagation Issues Nov 13, 2024
@openshift-ci-robot openshift-ci-robot added jira/severity-important Referenced Jira bug's severity is important for the branch this PR is targeting. jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. labels Nov 13, 2024
@openshift-ci-robot
Copy link
Contributor

@gcs278: This pull request references Jira Issue OCPBUGS-42045, which is valid.

3 validation(s) were run on this bug
  • bug is open, matching expected state (open)
  • bug target version (4.18.0) matches configured target version for branch (4.18.0)
  • bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, POST)

Requesting review from QA contact:
/cc @lihongan

The bug has been updated to refer to the pull request using the external bug tracker.

In response to this:

Resolves IBMCloud DNS names inside the cluster because they don't propagate to the test runner clusters in a reasonable time.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-ci-robot openshift-ci-robot added the jira/valid-bug Indicates that a referenced Jira bug is valid for the branch this PR is targeting. label Nov 13, 2024
@openshift-ci openshift-ci bot requested a review from lihongan November 13, 2024 16:10
@gcs278 gcs278 mentioned this pull request Nov 13, 2024
Closed
@gcs278 gcs278 changed the title OCPBUGS-42045: Fix IBMCloud DNS Propagation Issues [WIP] OCPBUGS-42045: Fix IBMCloud DNS Propagation Issues Nov 13, 2024
@gcs278
OCPBUGS-42045: Fix IBMCloud DNS Propagation Issues
941bff1 
Resolves IBMCloud DNS names within the cluster, as they do not propagate
to the test runner clusters in a reasonable time. Additionally, it adds
a warmup period since IBMCloud clusters have an unconfigurable 30-minute
negative cache. The warmup period delays DNS queries until the name is
resolvable to prevent it from being negatively cached.
@gcs278 gcs278 force-pushed the ibmcloud-e2e-dns-fix branch from 9f6e396 to 941bff1 Compare November 14, 2024 01:28
@gcs278
Copy link
Contributor Author

gcs278 commented Nov 14, 2024

okay DNS failed again. Increased warmup period to 5 minute which is what I found in #1132 (comment)
/test e2e-ibmcloud-operator

@gcs278
Copy link
Contributor Author

gcs278 commented Nov 15, 2024

At 6 minute warmup time, it failed 1/6 of the time. So 7 minutes is the minimum.

@openshift-ci OpenShift CI
Copy link
Contributor

openshift-ci bot commented Jan 9, 2025

@gcs278: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
ci/prow/e2e-ibmcloud-operator 941bff1 link false /test e2e-ibmcloud-operator
ci/prow/hypershift-e2e-aks 941bff1 link true /test hypershift-e2e-aks

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@lihongan lihongan Awaiting requested review from lihongan

Assignees
No one assigned
Labels
do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. jira/severity-important Referenced Jira bug's severity is important for the branch this PR is targeting. jira/valid-bug Indicates that a referenced Jira bug is valid for the branch this PR is targeting. jira/valid-reference Indicates that this PR references a valid Jira ticket of any type.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@gcs278 @openshift-ci-robot