FakeClientset: bump watch channel size #1174

ingvagabund · 2023-06-15T09:11:06Z

Huge clusters with thousands of pods can quickly exceed the default watch channel size of the fake clientset. Causing the channel to panic with "channel full".

a7i · 2023-06-15T12:18:41Z

/approve

damemi · 2023-06-15T13:26:35Z

cmd/descheduler/app/server.go

@@ -111,6 +112,9 @@ func NewDeschedulerCommand(out io.Writer) *cobra.Command {
 }

 func Run(ctx context.Context, rs *options.DeschedulerServer) error {
+	// increase the fake watch channel so the dry-run mode can be run
+	// over a cluster with thousands of pods
+	watch.DefaultChanSize = 100000


should this be configurable? Or is there any downside to defaulting to a larger channel size

In general you create one channel for each resource watch. We don't have that many. It's also used only when the fake clients are utilized. Unfortunately, there's currently no way to target a specific watch given the underlying apimachinery code does not distinguish fake clientsets on the level of informers.

Or is there any downside to defaulting to a larger channel size

Blind guess is only few MBs up given the Event type carries a reference to an object instead of its copy: https://github.com/kubernetes-sigs/descheduler/blob/master/vendor/k8s.io/apimachinery/pkg/watch/watch.go#L57-L70. I.e. make(chan Event, DefaultChanSize) will eat (2 + some const) * size(integer) * DefaultChanSize.

kevchu3

lgtm!

damemi · 2023-06-16T16:46:53Z

/approve

k8s-ci-robot · 2023-06-16T16:47:19Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: a7i, damemi

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [a7i,damemi]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

a7i · 2023-06-16T16:54:58Z

/lgtm

a7i · 2023-06-16T17:00:40Z

/retest-required

ingvagabund · 2023-06-19T10:08:31Z

/retest-required

ingvagabund · 2023-06-19T10:31:27Z

/retest-required

ingvagabund · 2023-06-19T11:38:27Z

/retest-required

ingvagabund · 2023-06-19T12:50:51Z

golangci/golangci-lint info checking GitHub for tag 'v1.52.1'
golangci/golangci-lint info found version: 1.52.1 for v1.52.1/linux/amd64
golangci/golangci-lint info installed ./_output/bin/golangci-lint
./_output/bin/golangci-lint run
level=error msg="Timeout exceeded: try increasing it by passing --timeout option"

$ /usr/bin/golangci-lint --version
golangci-lint has version 1.52.1 built with go1.20.2 from d92b38cc on 2023-03-21T19:48:38Z
$ /usr/bin/golangci-lint run

No timeout locally.

$ time /usr/bin/golangci-lint run

real	0m0.883s
user	0m3.328s
sys	0m0.524s

$ /usr/bin/golangci-lint run --help
Run the linters
...
--timeout duration               Timeout for total work (default 1m0s)

A minute should be enough. Unless the CI container runs for longer.

It takes 15m15s to run the entire pull-descheduler-verify-master /O\

a7i · 2023-06-20T17:50:18Z

/retest-required

This PR contains the following updates: | Package | Update | Change | |---|---|---| | [descheduler](https://github.com/kubernetes-sigs/descheduler) | minor | `0.27.1` -> `0.28.0` | --- ### Release Notes <details> <summary>kubernetes-sigs/descheduler (descheduler)</summary> ### [`v0.28.0`](https://github.com/kubernetes-sigs/descheduler/releases/tag/v0.28.0): Descheduler v0.28.0 [Compare Source](kubernetes-sigs/descheduler@v0.27.1...v0.28.0) #### What's Changed - add unit tests for version compatibility check by [@a7i](https://github.com/a7i) in kubernetes-sigs/descheduler#1096 - fix plugin arg conversion when using multiple profiles with same plugin by [@a7i](https://github.com/a7i) in kubernetes-sigs/descheduler#1143 - Use dl.k8s.io instead of kubernetes-release bucket by [@ratnopamc](https://github.com/ratnopamc) in kubernetes-sigs/descheduler#1145 - update deprecated sets.String to generic sets by [@a7i](https://github.com/a7i) in kubernetes-sigs/descheduler#1146 - helm: ability to override command and args. set args inline by [@a7i](https://github.com/a7i) in kubernetes-sigs/descheduler#1151 - fix: imagepullsecrets indentation for kind: Deployment by [@a7i](https://github.com/a7i) in kubernetes-sigs/descheduler#1150 - docs: supplement missing link by [@mikutas](https://github.com/mikutas) in kubernetes-sigs/descheduler#1159 - bump chart to v0.27.1 by [@a7i](https://github.com/a7i) in kubernetes-sigs/descheduler#1160 - Custom labels for ServiceMonitor resource by [@nlamirault](https://github.com/nlamirault) in kubernetes-sigs/descheduler#1147 - pod anti-affinity check among nodes by [@10hin](https://github.com/10hin) in kubernetes-sigs/descheduler#1033 - PodLifeTime: support CrashLoopBackOff container state by [@a7i](https://github.com/a7i) in kubernetes-sigs/descheduler#1164 - update pause image from 'kubernetes/pause' to 'registry.k8s.io/pause' by [@a7i](https://github.com/a7i) in kubernetes-sigs/descheduler#1166 - TooManyRestart: state filter for CrashLoopBackOff by [@a7i](https://github.com/a7i) in kubernetes-sigs/descheduler#1165 - Replace deprecated command with environment file by [@jongwooo](https://github.com/jongwooo) in kubernetes-sigs/descheduler#1173 - use pod informers for listing pods in removepodsviolatingtopologyspreadconstraint and removepodsviolatinginterpodantiaffinity by [@a7i](https://github.com/a7i) in kubernetes-sigs/descheduler#1163 - removepodsviolatingtopologyspreadconstraint: implement explicit constraints by [@a7i](https://github.com/a7i) in kubernetes-sigs/descheduler#1148 - FakeClientset: bump watch channel size by [@ingvagabund](https://github.com/ingvagabund) in kubernetes-sigs/descheduler#1174 - deschedule/balance order (continuation) by [@ingvagabund](https://github.com/ingvagabund) in kubernetes-sigs/descheduler#1177 - bump log level for processing info by [@a7i](https://github.com/a7i) in kubernetes-sigs/descheduler#1141 - fix priority threshold by name alone by [@knelasevero](https://github.com/knelasevero) in kubernetes-sigs/descheduler#1186 - feat: Enable open telemetry tracing by [@harshanarayana](https://github.com/harshanarayana) in kubernetes-sigs/descheduler#1189 - bump to k8s 1.28 beta.0 by [@a7i](https://github.com/a7i) in kubernetes-sigs/descheduler#1201 - nodefit: aggregate errors by [@lucming](https://github.com/lucming) in kubernetes-sigs/descheduler#1203 - fix: base configmap missing plugin RemoveDuplicates by [@a7i](https://github.com/a7i) in kubernetes-sigs/descheduler#1207 - feat: Implement preferredDuringSchedulingIgnoredDuringExecution for RemovePodsViolatingNodeAffinity by [@jordipiqueselles](https://github.com/jordipiqueselles) in kubernetes-sigs/descheduler#1210 - fix: descheduler_loop_duration_seconds has wrong value by [@Abirdcfly](https://github.com/Abirdcfly) in kubernetes-sigs/descheduler#1215 - .gitattribute to not pollute PRs or stats by [@a7i](https://github.com/a7i) in kubernetes-sigs/descheduler#1202 - Bump Kubernetes dependencies to v1.28.0 by [@JaneLiuL](https://github.com/JaneLiuL) in kubernetes-sigs/descheduler#1216 - profile: fix span attribute typo by [@antoinedeschenes](https://github.com/antoinedeschenes) in kubernetes-sigs/descheduler#1221 - k8s 1.28: update docs and go-version by [@a7i](https://github.com/a7i) in kubernetes-sigs/descheduler#1224 #### New Contributors - [@ratnopamc](https://github.com/ratnopamc) made their first contribution in kubernetes-sigs/descheduler#1145 - [@mikutas](https://github.com/mikutas) made their first contribution in kubernetes-sigs/descheduler#1159 - [@nlamirault](https://github.com/nlamirault) made their first contribution in kubernetes-sigs/descheduler#1147 - [@10hin](https://github.com/10hin) made their first contribution in kubernetes-sigs/descheduler#1033 - [@jongwooo](https://github.com/jongwooo) made their first contribution in kubernetes-sigs/descheduler#1173 - [@jordipiqueselles](https://github.com/jordipiqueselles) made their first contribution in kubernetes-sigs/descheduler#1210 - [@Abirdcfly](https://github.com/Abirdcfly) made their first contribution in kubernetes-sigs/descheduler#1215 - [@antoinedeschenes](https://github.com/antoinedeschenes) made their first contribution in kubernetes-sigs/descheduler#1221 **Full Changelog**: kubernetes-sigs/descheduler@v0.27.0...v0.28.0 </details> --- ### Configuration 📅 **Schedule**: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined). 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] If you want to rebase/retry this PR, check this box --- This PR has been generated by [Renovate Bot](https://github.com/renovatebot/renovate).  Reviewed-on: https://git.devmem.ru/projects/infra/pulls/949 Co-authored-by: Renovate Bot <renovate@devmem.ru> Co-committed-by: Renovate Bot <renovate@devmem.ru>

FakeClientset: bump watch channel size

4c272e6

Huge clusters with thousands of pods can quickly exceed the default watch channel size of the fake clientset. Causing the channel to panic with "channel full".

k8s-ci-robot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. labels Jun 15, 2023

k8s-ci-robot requested review from a7i and knelasevero June 15, 2023 09:11

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jun 15, 2023

damemi reviewed Jun 15, 2023

View reviewed changes

kevchu3 reviewed Jun 16, 2023

View reviewed changes

k8s-ci-robot assigned a7i Jun 16, 2023

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jun 16, 2023

ingvagabund mentioned this pull request Jun 19, 2023

Bump golangci lint timeout #1178

Closed

k8s-ci-robot merged commit a497541 into kubernetes-sigs:master Jun 20, 2023

ingvagabund deleted the bump-fake-watch-channel-size branch June 21, 2023 09:41

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

FakeClientset: bump watch channel size #1174

FakeClientset: bump watch channel size #1174

ingvagabund commented Jun 15, 2023

a7i commented Jun 15, 2023

damemi Jun 15, 2023

ingvagabund Jun 15, 2023 •

edited

Loading

kevchu3 left a comment

damemi commented Jun 16, 2023

k8s-ci-robot commented Jun 16, 2023

a7i commented Jun 16, 2023

a7i commented Jun 16, 2023

ingvagabund commented Jun 19, 2023

ingvagabund commented Jun 19, 2023

ingvagabund commented Jun 19, 2023

ingvagabund commented Jun 19, 2023 •

edited

Loading

a7i commented Jun 20, 2023

FakeClientset: bump watch channel size #1174

FakeClientset: bump watch channel size #1174

Conversation

ingvagabund commented Jun 15, 2023

a7i commented Jun 15, 2023

damemi Jun 15, 2023

Choose a reason for hiding this comment

ingvagabund Jun 15, 2023 • edited Loading

Choose a reason for hiding this comment

kevchu3 left a comment

Choose a reason for hiding this comment

damemi commented Jun 16, 2023

k8s-ci-robot commented Jun 16, 2023

a7i commented Jun 16, 2023

a7i commented Jun 16, 2023

ingvagabund commented Jun 19, 2023

ingvagabund commented Jun 19, 2023

ingvagabund commented Jun 19, 2023

ingvagabund commented Jun 19, 2023 • edited Loading

a7i commented Jun 20, 2023

ingvagabund Jun 15, 2023 •

edited

Loading

ingvagabund commented Jun 19, 2023 •

edited

Loading