Skip to content

[Flaking Test] [sig-api-machinery] k8s.io/kubernetes/test/integration/apiserver/coordinatedleaderelection.coordinatedleaderelection #129802

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
Rajalakshmi-Girish opened this issue Jan 24, 2025 · 10 comments · May be fixed by #129812
Closed

[Flaking Test] [sig-api-machinery] k8s.io/kubernetes/test/integration/apiserver/coordinatedleaderelection.coordinatedleaderelection #129802

Rajalakshmi-Girish opened this issue Jan 24, 2025 · 10 comments · May be fixed by #129812
Assignees
@Jefftree
Labels
kind/flake Categorizes issue or PR as related to a flaky test. needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. sig/api-machinery Categorizes an issue or PR as relevant to SIG API Machinery.

Comments

@Rajalakshmi-Girish
Copy link
Contributor

Which jobs are flaking?

master-blocking

  • integration-master

Which tests are flaking?

k8s.io/kubernetes/test/integration/apiserver/coordinatedleaderelection.coordinatedleaderelection
Prow
Triage

Since when has it been flaking?

1/14/2025, 12:41:26 AM
1/18/2025, 1:29:23 AM
1/20/2025, 1:20:29 PM
1/20/2025, 1:37:27 PM
1/22/2025, 1:04:24 AM
1/23/2025, 1:45:57 PM

Testgrid link

https://testgrid.k8s.io/sig-release-master-blocking#integration-master

Reason for failure (if possible)

{Failed  === RUN   TestCoordinatedLeaderElectionLeaseTransfer
    testserver.go:582: Resolved testserver package path to: "/home/prow/go/src/k8s.io/kubernetes/cmd/kube-apiserver/app/testing"
    testserver.go:402: runtime-config=map[api/all:true]
    testserver.go:403: Starting kube-apiserver on port 43809...
    testserver.go:438: Waiting for /healthz to be ok...
[-]poststarthook/start-apiextensions-controllers failed: not finished
[-]poststarthook/crd-informer-synced failed: not finished
[-]poststarthook/start-service-ip-repair-controllers failed: not finished
[-]poststarthook/scheduling/bootstrap-system-priority-classes failed: not finished
[-]poststarthook/priority-and-fairness-config-producer failed: not finished
[-]poststarthook/bootstrap-controller failed: not finished
[-]poststarthook/apiservice-registration-controller failed: not finished
[-]poststarthook/apiservice-discovery-controller failed: not finished
[-]autoregister-completion failed: missing APIService: [v1. v1.admissionregistration.k8s.io v1.apiextensions.k8s.io v1.apps v1.authentication.k8s.io v1.authorization.k8s.io v1.autoscaling v1.batch v1.certificates.k8s.io v1.coordination.k8s.io v1.discovery.k8s.io v1.events.k8s.io v1.flowcontrol.apiserver.k8s.io v1.networking.k8s.io v1.node.k8s.io v1.policy v1.rbac.authorization.k8s.io v1.scheduling.k8s.io v1.storage.k8s.io v1alpha1.admissionregistration.k8s.io v1alpha1.internal.apiserver.k8s.io v1alpha1.storage.k8s.io v1alpha2.coordination.k8s.io v1alpha3.resource.k8s.io v1beta1.admissionregistration.k8s.io v1beta1.networking.k8s.io v1beta1.resource.k8s.io v1beta1.storage.k8s.io v2.autoscaling]
[-]poststarthook/start-apiextensions-controllers failed: not finished
[-]poststarthook/crd-informer-synced failed: not finished
[-]poststarthook/scheduling/bootstrap-system-priority-classes failed: not finished
[-]poststarthook/priority-and-fairness-config-producer failed: not finished
[-]poststarthook/bootstrap-controller failed: not finished
[-]poststarthook/apiservice-registration-controller failed: not finished
[-]poststarthook/apiservice-discovery-controller failed: not finished
[-]autoregister-completion failed: missing APIService: [v1. v1.admissionregistration.k8s.io v1.apiextensions.k8s.io v1.apps v1.authentication.k8s.io v1.authorization.k8s.io v1.autoscaling v1.batch v1.certificates.k8s.io v1.coordination.k8s.io v1.discovery.k8s.io v1.events.k8s.io v1.flowcontrol.apiserver.k8s.io v1.networking.k8s.io v1.node.k8s.io v1.policy v1.rbac.authorization.k8s.io v1.scheduling.k8s.io v1.storage.k8s.io v1alpha1.admissionregistration.k8s.io v1alpha1.internal.apiserver.k8s.io v1alpha1.storage.k8s.io v1alpha2.coordination.k8s.io v1alpha3.resource.k8s.io v1beta1.admissionregistration.k8s.io v1beta1.networking.k8s.io v1beta1.resource.k8s.io v1beta1.storage.k8s.io v2.autoscaling]
[-]poststarthook/start-apiextensions-controllers failed: not finished
[-]poststarthook/crd-informer-synced failed: not finished
[-]poststarthook/scheduling/bootstrap-system-priority-classes failed: not finished
[-]poststarthook/priority-and-fairness-config-producer failed: not finished
[-]autoregister-completion failed: missing APIService: [v1. v1.admissionregistration.k8s.io v1.apiextensions.k8s.io v1.apps v1.authentication.k8s.io v1.authorization.k8s.io v1.autoscaling v1.batch v1.certificates.k8s.io v1.coordination.k8s.io v1.discovery.k8s.io v1.events.k8s.io v1.flowcontrol.apiserver.k8s.io v1.networking.k8s.io v1.node.k8s.io v1.policy v1.rbac.authorization.k8s.io v1.scheduling.k8s.io v1.storage.k8s.io v1alpha1.admissionregistration.k8s.io v1alpha1.internal.apiserver.k8s.io v1alpha1.storage.k8s.io v1alpha2.coordination.k8s.io v1alpha3.resource.k8s.io v1beta1.admissionregistration.k8s.io v1beta1.networking.k8s.io v1beta1.resource.k8s.io v1beta1.storage.k8s.io v2.autoscaling]
[-]poststarthook/scheduling/bootstrap-system-priority-classes failed: not finished
[-]poststarthook/priority-and-fairness-config-producer failed: not finished
[-]autoregister-completion failed: missing APIService: [v1. v1.admissionregistration.k8s.io v1.apiextensions.k8s.io v1.apps v1.authentication.k8s.io v1.authorization.k8s.io v1.autoscaling v1.batch v1.certificates.k8s.io v1.coordination.k8s.io v1.discovery.k8s.io v1.events.k8s.io v1.flowcontrol.apiserver.k8s.io v1.networking.k8s.io v1.node.k8s.io v1.policy v1.rbac.authorization.k8s.io v1.scheduling.k8s.io v1.storage.k8s.io v1alpha1.admissionregistration.k8s.io v1alpha1.internal.apiserver.k8s.io v1alpha1.storage.k8s.io v1alpha2.coordination.k8s.io v1alpha3.resource.k8s.io v1beta1.admissionregistration.k8s.io v1beta1.networking.k8s.io v1beta1.resource.k8s.io v1beta1.storage.k8s.io v2.autoscaling]
[-]poststarthook/scheduling/bootstrap-system-priority-classes failed: not finished
[-]poststarthook/priority-and-fairness-config-producer failed: not finished
[-]autoregister-completion failed: missing APIService: [v1. v1.admissionregistration.k8s.io v1.apiextensions.k8s.io v1.apps v1.authentication.k8s.io v1.authorization.k8s.io v1.autoscaling v1.batch v1.certificates.k8s.io v1.coordination.k8s.io v1.discovery.k8s.io v1.events.k8s.io v1.flowcontrol.apiserver.k8s.io v1.networking.k8s.io v1.node.k8s.io v1.policy v1.rbac.authorization.k8s.io v1.scheduling.k8s.io v1.storage.k8s.io v1alpha1.admissionregistration.k8s.io v1alpha1.internal.apiserver.k8s.io v1alpha1.storage.k8s.io v1alpha2.coordination.k8s.io v1alpha3.resource.k8s.io v1beta1.admissionregistration.k8s.io v1beta1.networking.k8s.io v1beta1.resource.k8s.io v1beta1.storage.k8s.io v2.autoscaling]
  	Type:               "Dangling",
- 	Status:             "",
+ 	Status:             "False",
- 	LastTransitionTime: v1.Time{},
+ 	LastTransitionTime: v1.Time{Time: s"2025-01-21 19:59:28.22697537 +0000 UTC m=+2.405278625"},
- 	Reason:             "",
+ 	Reason:             "Found",
- 	Message:            "",
+ 	Message:            `This FlowSchema references the PriorityLevelConfiguration object named "system" and it exists`,
  }
  	Type:               "Dangling",
- 	Status:             "",
+ 	Status:             "False",
- 	LastTransitionTime: v1.Time{},
+ 	LastTransitionTime: v1.Time{Time: s"2025-01-21 19:59:28.232527805 +0000 UTC m=+2.410831090"},
- 	Reason:             "",
+ 	Reason:             "Found",
- 	Message:            "",
+ 	Message:            `This FlowSchema references the PriorityLevelConfiguration object named "node-high" and it exists`,
  }
  	Type:               "Dangling",
- 	Status:             "",
+ 	Status:             "True",
- 	LastTransitionTime: v1.Time{},
+ 	LastTransitionTime: v1.Time{Time: s"2025-01-21 19:59:28.236808066 +0000 UTC m=+2.415111321"},
- 	Reason:             "",
+ 	Reason:             "NotFound",
- 	Message:            "",
+ 	Message:            `This FlowSchema references the PriorityLevelConfiguration object named "exempt" but there is no such object`,
  }
  	Type:               "Dangling",
- 	Status:             "",
+ 	Status:             "False",
- 	LastTransitionTime: v1.Time{},
+ 	LastTransitionTime: v1.Time{Time: s"2025-01-21 19:59:28.236810496 +0000 UTC m=+2.415113761"},
- 	Reason:             "",
+ 	Reason:             "Found",
- 	Message:            "",
+ 	Message:            `This FlowSchema references the PriorityLevelConfiguration object named "leader-election" and it exists`,
  }
  	Type:               "Dangling",
- 	Status:             "",
+ 	Status:             "False",
- 	LastTransitionTime: v1.Time{},
+ 	LastTransitionTime: v1.Time{Time: s"2025-01-21 19:59:28.244349304 +0000 UTC m=+2.422652559"},
- 	Reason:             "",
+ 	Reason:             "Found",
- 	Message:            "",
+ 	Message:            `This FlowSchema references the PriorityLevelConfiguration object named "leader-election" and it exists`,
  }
  	Type:               "Dangling",
- 	Status:             "",
+ 	Status:             "False",
- 	LastTransitionTime: v1.Time{},
+ 	LastTransitionTime: v1.Time{Time: s"2025-01-21 19:59:28.244351044 +0000 UTC m=+2.422654299"},
- 	Reason:             "",
+ 	Reason:             "Found",
- 	Message:            "",
+ 	Message:            `This FlowSchema references the PriorityLevelConfiguration object named "workload-high" and it exists`,
  }
[-]poststarthook/scheduling/bootstrap-system-priority-classes failed: not finished
[-]poststarthook/priority-and-fairness-config-producer failed: not finished
  	Type:               "Dangling",
- 	Status:             "",
+ 	Status:             "False",
- 	LastTransitionTime: v1.Time{},
+ 	LastTransitionTime: v1.Time{Time: s"2025-01-21 19:59:28.253019695 +0000 UTC m=+2.431322950"},
- 	Reason:             "",
+ 	Reason:             "Found",
- 	Message:            "",
+ 	Message:            `This FlowSchema references the PriorityLevelConfiguration object named "workload-high" and it exists`,
  }
  	Type:               "Dangling",
- 	Status:             "",
+ 	Status:             "False",
- 	LastTransitionTime: v1.Time{},
+ 	LastTransitionTime: v1.Time{Time: s"2025-01-21 19:59:28.253021215 +0000 UTC m=+2.431324470"},
- 	Reason:             "",
+ 	Reason:             "Found",
- 	Message:            "",
+ 	Message:            `This FlowSchema references the PriorityLevelConfiguration object named "workload-high" and it exists`,
  }
  	Type:               "Dangling",
- 	Status:             "",
+ 	Status:             "False",
- 	LastTransitionTime: v1.Time{},
+ 	LastTransitionTime: v1.Time{Time: s"2025-01-21 19:59:28.253025715 +0000 UTC m=+2.431328970"},
- 	Reason:             "",
+ 	Reason:             "Found",
- 	Message:            "",
+ 	Message:            `This FlowSchema references the PriorityLevelConfiguration object named "workload-high" and it exists`,
  }
  	Type:               "Dangling",
- 	Status:             "",
+ 	Status:             "False",
- 	LastTransitionTime: v1.Time{},
+ 	LastTransitionTime: v1.Time{Time: s"2025-01-21 19:59:28.266023747 +0000 UTC m=+2.444327022"},
- 	Reason:             "",
+ 	Reason:             "Found",
- 	Message:            "",
+ 	Message:            `This FlowSchema references the PriorityLevelConfiguration object named "exempt" and it exists`,
  }
  	Type:               "Dangling",
- 	Status:             "",
+ 	Status:             "False",
- 	LastTransitionTime: v1.Time{},
+ 	LastTransitionTime: v1.Time{Time: s"2025-01-21 19:59:28.266032617 +0000 UTC m=+2.444335882"},
- 	Reason:             "",
+ 	Reason:             "Found",
- 	Message:            "",
+ 	Message:            `This FlowSchema references the PriorityLevelConfiguration object named "global-default" and it exists`,
  }
  	Type:               "Dangling",
- 	Status:             "",
+ 	Status:             "False",
- 	LastTransitionTime: v1.Time{},
+ 	LastTransitionTime: v1.Time{Time: s"2025-01-21 19:59:28.266040247 +0000 UTC m=+2.444343502"},
- 	Reason:             "",
+ 	Reason:             "Found",
- 	Message:            "",
+ 	Message:            `This FlowSchema references the PriorityLevelConfiguration object named "workload-low" and it exists`,
  }
  	Type:               "Dangling",
- 	Status:             "True",
+ 	Status:             "False",
- 	LastTransitionTime: v1.Time{Time: s"2025-01-21 19:59:28 +0000 UTC"},
+ 	LastTransitionTime: v1.Time{Time: s"2025-01-21 19:59:28.266050447 +0000 UTC m=+2.444353702"},
- 	Reason:             "NotFound",
+ 	Reason:             "Found",
  	Message: strings.Join({
  		"This FlowSchema references the PriorityLevelConfiguration object",
  		` named "exempt" `,
- 		"but there is no such object",
+ 		"and it exists",
  	}, ""),
  }
  	Type:               "Dangling",
- 	Status:             "",
+ 	Status:             "False",
- 	LastTransitionTime: v1.Time{},
+ 	LastTransitionTime: v1.Time{Time: s"2025-01-21 19:59:28.284227301 +0000 UTC m=+2.462530576"},
- 	Reason:             "",
+ 	Reason:             "Found",
- 	Message:            "",
+ 	Message:            `This FlowSchema references the PriorityLevelConfiguration object named "catch-all" and it exists`,
  }
[-]poststarthook/scheduling/bootstrap-system-priority-classes failed: not finished
[-]poststarthook/scheduling/bootstrap-system-priority-classes failed: not finished
[-]poststarthook/scheduling/bootstrap-system-priority-classes failed: not finished
[-]poststarthook/scheduling/bootstrap-system-priority-classes failed: not finished
[-]poststarthook/scheduling/bootstrap-system-priority-classes failed: not finished
[-]poststarthook/scheduling/bootstrap-system-priority-classes failed: not finished
[-]poststarthook/scheduling/bootstrap-system-priority-classes failed: not finished
[-]poststarthook/scheduling/bootstrap-system-priority-classes failed: not finished
[-]poststarthook/scheduling/bootstrap-system-priority-classes failed: not finished
[-]poststarthook/scheduling/bootstrap-system-priority-classes failed: not finished
[-]poststarthook/scheduling/bootstrap-system-priority-classes failed: not finished
[-]poststarthook/scheduling/bootstrap-system-priority-classes failed: not finished
[-]poststarthook/scheduling/bootstrap-system-priority-classes failed: not finished
[-]poststarthook/scheduling/bootstrap-system-priority-classes failed: not finished
[-]poststarthook/scheduling/bootstrap-system-priority-classes failed: not finished
leases.coordination.k8s.io "leader-election-controller" not found
leases.coordination.k8s.io "leader-election-controller" not found
    testserver.go:582: Resolved testserver package path to: "/home/prow/go/src/k8s.io/kubernetes/cmd/kube-apiserver/app/testing"
    testserver.go:402: runtime-config=map[api/all:true]
    testserver.go:403: Starting kube-apiserver on port 42539...
    testserver.go:438: Waiting for /healthz to be ok...
[-]poststarthook/start-apiextensions-controllers failed: not finished
[-]poststarthook/crd-informer-synced failed: not finished
[-]poststarthook/start-service-ip-repair-controllers failed: not finished
[-]poststarthook/scheduling/bootstrap-system-priority-classes failed: not finished
[-]poststarthook/priority-and-fairness-config-producer failed: not finished
[-]poststarthook/bootstrap-controller failed: not finished
[-]poststarthook/apiservice-registration-controller failed: not finished
[-]poststarthook/apiservice-discovery-controller failed: not finished
[-]autoregister-completion failed: missing APIService: [v1. v1.admissionregistration.k8s.io v1.apiextensions.k8s.io v1.apps v1.authentication.k8s.io v1.authorization.k8s.io v1.autoscaling v1.batch v1.certificates.k8s.io v1.coordination.k8s.io v1.discovery.k8s.io v1.events.k8s.io v1.flowcontrol.apiserver.k8s.io v1.networking.k8s.io v1.node.k8s.io v1.policy v1.rbac.authorization.k8s.io v1.scheduling.k8s.io v1.storage.k8s.io v1alpha1.admissionregistration.k8s.io v1alpha1.internal.apiserver.k8s.io v1alpha1.storage.k8s.io v1alpha2.coordination.k8s.io v1alpha3.resource.k8s.io v1beta1.admissionregistration.k8s.io v1beta1.networking.k8s.io v1beta1.resource.k8s.io v1beta1.storage.k8s.io v2.autoscaling]
[-]poststarthook/scheduling/bootstrap-system-priority-classes failed: not finished
[-]poststarthook/priority-and-fairness-config-producer failed: not finished
[-]poststarthook/bootstrap-controller failed: not finished
[-]poststarthook/apiservice-registration-controller failed: not finished
[-]poststarthook/apiservice-discovery-controller failed: not finished
[-]poststarthook/scheduling/bootstrap-system-priority-classes failed: not finished
[-]poststarthook/priority-and-fairness-config-producer failed: not finished
[-]poststarthook/apiservice-registration-controller failed: not finished
[-]poststarthook/apiservice-discovery-controller failed: not finished
[-]poststarthook/scheduling/bootstrap-system-priority-classes failed: not finished
[-]poststarthook/scheduling/bootstrap-system-priority-classes failed: not finished
[-]poststarthook/scheduling/bootstrap-system-priority-classes failed: not finished
[-]poststarthook/scheduling/bootstrap-system-priority-classes failed: not finished
[-]poststarthook/scheduling/bootstrap-system-priority-classes failed: not finished
[-]poststarthook/scheduling/bootstrap-system-priority-classes failed: not finished
[-]poststarthook/scheduling/bootstrap-system-priority-classes failed: not finished
[-]poststarthook/scheduling/bootstrap-system-priority-classes failed: not finished
[-]poststarthook/scheduling/bootstrap-system-priority-classes failed: not finished
[-]poststarthook/scheduling/bootstrap-system-priority-classes failed: not finished
[-]poststarthook/scheduling/bootstrap-system-priority-classes failed: not finished
[-]poststarthook/scheduling/bootstrap-system-priority-classes failed: not finished
[-]poststarthook/scheduling/bootstrap-system-priority-classes failed: not finished
[-]poststarthook/scheduling/bootstrap-system-priority-classes failed: not finished
[-]poststarthook/scheduling/bootstrap-system-priority-classes failed: not finished
[-]poststarthook/scheduling/bootstrap-system-priority-classes failed: not finished
[-]poststarthook/scheduling/bootstrap-system-priority-classes failed: not finished
client rate limiter Wait returned an error: context deadline exceeded
    leaderelection_test.go:162: Expected the cle lease lock to transition to the first apiserver
--- FAIL: TestCoordinatedLeaderElectionLeaseTransfer (25.47s)
}

Anything else we need to know?

N/A

Relevant SIG(s)

/sig api-machinery
@Rajalakshmi-Girish Rajalakshmi-Girish added the kind/flake Categorizes issue or PR as related to a flaky test. label Jan 24, 2025
@k8s-ci-robot k8s-ci-robot added sig/api-machinery Categorizes an issue or PR as relevant to SIG API Machinery. needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Jan 24, 2025
@k8s-ci-robot
Copy link
Contributor

This issue is currently awaiting triage.

If a SIG or subproject determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@Jefftree Jefftree mentioned this issue Jan 24, 2025
Open
@Jefftree
Copy link
Member

/assign

@Rajalakshmi-Girish
Copy link
Contributor Author

@Jefftree Can you please triage this issue and tell whether this issue will block the v1.33.0-alpha.1 cut, which is scheduled for Tuesday, 4th February UTC?

@Rajalakshmi-Girish
Copy link
Contributor Author

Rajalakshmi-Girish commented Jan 29, 2025 •
edited
Loading

@Jefftree Can you please triage this issue and tell whether this issue will block the v1.33.0-alpha.1 cut, which is scheduled for Tuesday, 4th February UTC?

Got a non release blocking reply here in k8s channel https://kubernetes.slack.com/archives/C0EG7JC6T/p1738171062259389?thread_ts=1737719943.361149&cid=C0EG7JC6T

@stmcginnis stmcginnis moved this from FLAKY to PASSING in CI Signal (SIG Release / Release Team) Feb 20, 2025
@stmcginnis
Copy link
Contributor

No failures in testgrid since Feb 11. Appears to be fixed, will wait to close for confirmation.

@wendy-ha18 wendy-ha18 moved this from PASSING to FLAKY in CI Signal (SIG Release / Release Team) Feb 25, 2025
@enj
Copy link
Member

enj commented May 1, 2025 •
edited
Loading

Seen in #131573 -> prow

@enj enj mentioned this issue May 1, 2025
Merged
@aojea
Copy link
Member

aojea commented May 2, 2025

There is still a fix without merging, @Jefftree @sttts do we need to follow up on #129812

@liggitt
Copy link
Member

liggitt commented May 2, 2025

seen again today in #130989

@liggitt liggitt mentioned this issue May 2, 2025
Merged
3 tasks
@wendy-ha18
Copy link
Member

Hi everyone, thanks a lot for your support and attention on this issue. After the merge of #130989 and the stable of testgrid since 8th May, I would like to close this issue now.
Please feel free to reopen or reference it in the future if you see the test fail again with the same issue.

/close

@k8s-ci-robot
Copy link
Contributor

@wendy-ha18: Closing this issue.

In response to this:

Hi everyone, thanks a lot for your support and attention on this issue. After the merge of #130989 and the stable of testgrid since 8th May, I would like to close this issue now.
Please feel free to reopen or reference it in the future if you see the test fail again with the same issue.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@Jefftree Jefftree

Labels
kind/flake Categorizes issue or PR as related to a flaky test. needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. sig/api-machinery Categorizes an issue or PR as relevant to SIG API Machinery.
Projects
CI Signal (SIG Release / Release Team)
Status: RESOLVED
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Forcefully expire lease in integration test to fix flake Jefftree/kubernetes
8 participants
@liggitt @stmcginnis @aojea @enj @Jefftree @k8s-ci-robot @Rajalakshmi-Girish @wendy-ha18