Pod Topology Spread blog post (KEP-3022 KEP-3094 KEP-3243) #39777

sanposhiho · 2023-03-04T05:15:16Z

We added some features in recent releases and those are now graduated to beta by 1.27.

We'd like to have a blog post for all of these features.

/cc @kerthcet @denkensk

sanposhiho · 2023-03-04T05:16:40Z

/sig scheduling

netlify · 2023-03-04T05:27:00Z

✅ Pull request preview available for checking

Built without sensitive environment variables

Name	Link
🔨 Latest commit	`1841857`
🔍 Latest deploy log	https://app.netlify.com/sites/kubernetes-io-main-staging/deploys/64385b2f0092510008291e64
😎 Deploy Preview	https://deploy-preview-39777--kubernetes-io-main-staging.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site settings.

kerthcet · 2023-03-04T10:06:02Z

Great! Thanks for opening this.

sftim · 2023-03-06T09:23:02Z

/area blog

bradmccoydev · 2023-03-24T06:05:56Z

Hello, Comms Shadow for the 1.27 release here. This feature blog is tracked for release, the deadline for submitting the draft is 4th of April- the sooner the better since there's still editing to be done afterward. Any doubt, the Comms team is here to help. Thanks!
/cc @harshitasao

kerthcet · 2023-03-24T08:09:17Z

FYI: I'll start to write the blog next week.

sanposhiho

@kerthcet @denkensk
I wrote the introduction and closing sections so that each person just needs to focus on own section.
Could you give me reviews or some ideas to improve them more?

sanposhiho · 2023-03-25T05:39:52Z

content/en/blog/_posts/2023-04-11-topology-spread-features.md

@@ -0,0 +1,43 @@
+---
+layout: blog
+title: "TBD" // TODO: have a cool title.


@denkensk @kerthcet any ideas? 🙂

How about "Introducing the advanced pod topology spread policy for Kubernetes"?

or More fine-gained pod topology spread policies in Kubernetes scheduling.

I like the word "fine-grained".
Also, I'd like to mention that these features are coming to beta in v1.27 like many other blog posts did.

So... like Kubernetes 1.27: More fine-grained pod topology spread policies reached beta?

Kubernetes 1.27: More fine-grained pod topology spread policies reached beta
sgtm.

content/en/blog/_posts/2023-04-11-topology-spread-features.md

harshitasao · 2023-03-31T07:19:56Z

Hello @sanposhiho, we're doing a global reminder about submitting a draft for review for all opted-in feature blogs. If it's at all possible, it is very helpful for the release team to have drafts submitted for review before the hard deadline date, to better plan the release dates and avoid missing out. Thank you!

content/en/blog/_posts/2023-04-11-topology-spread-features.md

kerthcet · 2023-04-02T17:16:31Z

My commit is 7656b49, I can't append directly, maybe I should be a co-author.

Signed-off-by: Kante Yin <kerthcet@gmail.com>

denkensk · 2023-04-05T15:01:52Z

denkensk@060f4ec

@sanposhiho plz help to cherry-pick this commit. Thanks

Signed-off-by: Alex Wang <wangqingcan1990@gmail.com>

kerthcet · 2023-04-06T06:52:56Z

ping @sanposhiho a new commit to address the comments about nodeInclusionPolicy. c01127b

Signed-off-by: Kante Yin <kerthcet@gmail.com>

alculquicondor

LGTM from SIG Scheduling (technical PoV)

alculquicondor · 2023-04-06T19:01:51Z

content/en/blog/_posts/2023-04-11-topology-spread-features.md

+**Authors:** [Alex Wang](https://github.com/denkensk)(Shopee), [Kante Yin](https://github.com/kerthcet)(DaoCloud), [Kensei Nakada](https://github.com/sanposhiho)(Mercari)
+
+In Kubernetes v1.19, [Pod Topology Spread Constraints](https://kubernetes.io/docs/concepts/scheduling-eviction/topology-spread-constraints/) went to GA.
+It is the feature to control how Pods are spread to each failure-domain (regions, zones, nodes etc).


Suggested change

It is the feature to control how Pods are spread to each failure-domain (regions, zones, nodes etc).

It is the feature to control how Pods are spread in the cluster topology or failure domains (regions, zones, nodes etc).

alculquicondor · 2023-04-06T19:04:22Z

content/en/blog/_posts/2023-04-11-topology-spread-features.md

+## KEP-3094: Take taints/tolerations into consideration when calculating PodTopologySpread skew
+
+Before this enhancement, when you deploy a pod with `podTopologySpread` configured, kube-scheduler would
+take all inclined nodes(satisfied with pod nodeAffinity and nodeSelector) into consideration


Suggested change

take all inclined nodes(satisfied with pod nodeAffinity and nodeSelector) into consideration

take the Nodes that satisfy the Pod's nodeAffinity and nodeSelector into consideration

alculquicondor · 2023-04-06T19:05:10Z

content/en/blog/_posts/2023-04-11-topology-spread-features.md

+take all inclined nodes(satisfied with pod nodeAffinity and nodeSelector) into consideration
+in filtering and scoring, but would not care about whether the node taints are tolerated by the incoming pod or not.
+This may lead to a node with untolerated taint best fit the pod in podTopologySpread plugin, and as a result,
+the pod will stuck in pending for it violates the nodeTaint plugin.


Suggested change

the pod will stuck in pending for it violates the nodeTaint plugin.

the pod will stuck in Pending if it doesn't tolerate the taint.

alculquicondor · 2023-04-06T19:05:51Z

content/en/blog/_posts/2023-04-11-topology-spread-features.md

+Before this enhancement, when you deploy a pod with `podTopologySpread` configured, kube-scheduler would
+take all inclined nodes(satisfied with pod nodeAffinity and nodeSelector) into consideration
+in filtering and scoring, but would not care about whether the node taints are tolerated by the incoming pod or not.
+This may lead to a node with untolerated taint best fit the pod in podTopologySpread plugin, and as a result,


Suggested change

This may lead to a node with untolerated taint best fit the pod in podTopologySpread plugin, and as a result,

This may lead to a node with untolerated taint as the only candidate for spreading, and as a result,

alculquicondor · 2023-04-06T19:09:14Z

content/en/blog/_posts/2023-04-11-topology-spread-features.md

+revision label to Deployment and update it manually at each rolling upgrade (both the label on the
+podTemplate and the `labelSelector` in the `topologySpreadConstraints`).
+
+To solve this problem once and for all, and to make more accurate decisions in scheduling, we added a new named


Suggested change

To solve this problem once and for all, and to make more accurate decisions in scheduling, we added a new named

To solve this problem with a simpler API, we added a new field named

alculquicondor · 2023-04-06T19:10:34Z

content/en/blog/_posts/2023-04-11-topology-spread-features.md

+existing pods over which spreading will be calculated for the incoming pod.
+
+With `matchLabelKeys`, you don't need to update the `pod.spec` between different revisions.
+The controller/operator just needs to set different values to the same label key for different revisions.


Suggested change

The controller/operator just needs to set different values to the same label key for different revisions.

The controller or operator managing rollouts just needs to set different values to the same label key for different revisions.

sanposhiho · 2023-04-06T23:55:47Z

@alculquicondor I fixed based on your suggestions.

@kerthcet @denkensk I applied what @alculquicondor suggested as it is for now. Please create a commit if you have another idea for your section.

kerthcet · 2023-04-07T06:08:51Z

Thanks @sanposhiho , LGTMed.

denkensk · 2023-04-07T10:21:20Z

@alculquicondor I fixed based on your suggestions.

@kerthcet @denkensk I applied what @alculquicondor suggested as it is for now. Please create a commit if you have another idea for your section.

OK Thanks LGTM @sanposhiho

harshitasao · 2023-04-07T19:28:00Z

Hello, Comms Lead for v1.27 here. The publication order and date for the Feature Blog series has been finalized and the tracking board is updated.

The publication date for this article is 17-04-2023 (April 17).

Thank you!

sanposhiho · 2023-04-08T05:44:32Z

/assign @onlydole

For approval from sig/docs side.

content/en/blog/_posts/2023-04-11-topology-spread-features.md

sftim

Hi. Here's some feedback based on how we usually write and style articles. I hope it helps.

I also suggested an update to use the official publication date.

sftim · 2023-04-11T12:30:27Z

content/en/blog/_posts/2023-04-11-topology-spread-features.md

+---
+layout: blog
+title: "Kubernetes 1.27: More fine-grained pod topology spread policies reached beta"
+date: 2023-04-11


Suggested change

date: 2023-04-11

date: 2023-04-17

Please update the filename to match.

sftim · 2023-04-11T12:31:14Z

content/en/blog/_posts/2023-04-11-topology-spread-features.md

+title: "Kubernetes 1.27: More fine-grained pod topology spread policies reached beta"
+date: 2023-04-11
+slug: fine-grained-pod-topology-spread-features-beta
+evergreen: true


Are you sure? This represents a commitment by the SIG to keep the content current. However, this doesn't feel like a post that would stay up to date as it is tied to a specific release.

Suggested change

evergreen: true

This represents a commitment by the SIG to keep the content current.

Ah, OK. I'll delete then.

sftim · 2023-04-11T12:31:47Z

content/en/blog/_posts/2023-04-11-topology-spread-features.md

+But, there wasn't a way to control the number of domains over which we should spread.
+Some users want to force spreading Pods over a minimum number of domains, and if there aren't enough already present, make the cluster-autoscaler provision them.
+
+Then, we introduced the `minDomains` parameter in the Pod Topology Spread.


Suggested change

Then, we introduced the `minDomains` parameter in the Pod Topology Spread.

Kubernetes v1.24 introduced the `minDomains` parameter for pod topology spread constraints,

as an alpha feature.

sftim · 2023-04-11T12:32:02Z

content/en/blog/_posts/2023-04-11-topology-spread-features.md

+Via `minDomains` parameter, you can define the minimum number of domains.
+
+For example, assume there are 3 Nodes with the enough capacity,
+and a newly created replicaset has the following `topologySpreadConstraints` in template.


Suggested change

and a newly created replicaset has the following `topologySpreadConstraints` in template.

and a newly created ReplicaSet has the following `topologySpreadConstraints` in its Pod template.

sftim · 2023-04-11T12:32:20Z

content/en/blog/_posts/2023-04-11-topology-spread-features.md

+and a newly created replicaset has the following `topologySpreadConstraints` in template.
+
+```yaml
+topologySpreadConstraints:


Suggested change

topologySpreadConstraints:

...

topologySpreadConstraints:

sftim · 2023-04-11T12:41:03Z

content/en/blog/_posts/2023-04-11-topology-spread-features.md

+
+## Getting involved
+
+These features are managed by the [SIG/Scheduling](https://github.com/kubernetes/community/tree/master/sig-scheduling).


Suggested change

These features are managed by the [SIG/Scheduling](https://github.com/kubernetes/community/tree/master/sig-scheduling).

These features are managed by Kubernetes [SIG Scheduling](https://github.com/kubernetes/community/tree/master/sig-scheduling).

sftim · 2023-04-11T12:41:24Z

content/en/blog/_posts/2023-04-11-topology-spread-features.md

+
+## How can I learn more?
+
+- [Pod Topology Spread Constraints | Kubernetes](/docs/concepts/scheduling-eviction/topology-spread-constraints/)


Suggested change

- [Pod Topology Spread Constraints | Kubernetes](/docs/concepts/scheduling-eviction/topology-spread-constraints/)

- [Pod Topology Spread Constraints](/docs/concepts/scheduling-eviction/topology-spread-constraints/) in the Kubernetes documentation

sftim · 2023-04-11T12:42:17Z

content/en/blog/_posts/2023-04-11-topology-spread-features.md

+
+**Authors:** [Alex Wang](https://github.com/denkensk)(Shopee), [Kante Yin](https://github.com/kerthcet)(DaoCloud), [Kensei Nakada](https://github.com/sanposhiho)(Mercari)
+
+In Kubernetes v1.19, [Pod Topology Spread Constraints](/docs/concepts/scheduling-eviction/topology-spread-constraints/) went to GA.


(nit)

Suggested change

In Kubernetes v1.19, [Pod Topology Spread Constraints](/docs/concepts/scheduling-eviction/topology-spread-constraints/) went to GA.

In Kubernetes v1.19, [Pod topology spread constraints](/docs/concepts/scheduling-eviction/topology-spread-constraints/)

went to general availability (GA).

sftim · 2023-04-11T12:42:37Z

content/en/blog/_posts/2023-04-11-topology-spread-features.md

+In Kubernetes v1.19, [Pod Topology Spread Constraints](/docs/concepts/scheduling-eviction/topology-spread-constraints/) went to GA.
+It is the feature to control how Pods are spread in the cluster topology or failure domains (regions, zones, nodes etc).
+
+As time passed, we received feedback from users,


Suggested change

As time passed, we received feedback from users,

As time passed, we - SIG Scheduling - received feedback from users,

sftim · 2023-04-11T12:44:26Z

content/en/blog/_posts/2023-04-11-topology-spread-features.md

+```yaml
+topologySpreadConstraints:
+- maxSkew: 1
+  minDomains: 5 # requires 5 Nodes at least.


Suggested change

minDomains: 5 # requires 5 Nodes at least.

minDomains: 5 # requires 5 Nodes at least (because each Node has a unique hostname)

kerthcet · 2023-04-12T02:41:03Z

@sanposhiho can you help to solve these comments? Or another two cherry-picks?

sanposhiho · 2023-04-12T13:51:27Z

@sftim Thanks for the reviews!

@kerthcet @denkensk
by tomorrow, I will check and probably apply the suggestions from @sftim as they are. If you want to have another change in your section, let me know the commit hash then 🙂

sanposhiho · 2023-04-13T03:23:54Z

I applied the changes to fix the points from @sftim

sftim · 2023-04-13T19:41:27Z

/hold cancel

content/en/blog/_posts/2023-04-17-topology-spread-features.md

sftim

Thanks
/lgtm
/approve

k8s-ci-robot · 2023-04-13T19:43:16Z

LGTM label has been added.

Git tree hash: 6b6be3cc3058d90117a5245fe819e4249a340e59

k8s-ci-robot · 2023-04-13T19:43:30Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: sftim

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~content/en/blog/OWNERS~~ [sftim]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

k8s-ci-robot requested review from denkensk and kerthcet March 4, 2023 05:15

k8s-ci-robot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. labels Mar 4, 2023

k8s-ci-robot added the sig/scheduling Categorizes an issue or PR as relevant to SIG Scheduling. label Mar 4, 2023

sanposhiho marked this pull request as draft March 4, 2023 05:21

k8s-ci-robot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Mar 4, 2023

k8s-ci-robot added the area/blog Issues or PRs related to the Kubernetes Blog subproject label Mar 6, 2023

k8s-ci-robot requested a review from harshitasao March 24, 2023 06:05

add general section

bd64321

sanposhiho force-pushed the topologyspread branch from 412d3c5 to bd64321 Compare March 25, 2023 05:37

sanposhiho commented Mar 25, 2023

View reviewed changes

sanposhiho changed the title ~~Place holder for the Pod Topology Spread blog post (KEP-3022 KEP-3094 KEP-3243)~~ Pod Topology Spread blog post (KEP-3022 KEP-3094 KEP-3243) Mar 26, 2023

kerthcet reviewed Mar 31, 2023

View reviewed changes

content/en/blog/_posts/2023-04-11-topology-spread-features.md Outdated Show resolved Hide resolved

add min domains

528e113

Add section about nodeInclusionPolicy

ba1d6bd

Signed-off-by: Kante Yin <kerthcet@gmail.com>

k8s-ci-robot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Apr 3, 2023

blog: update content about matchLabelKeys

81482cf

Signed-off-by: Alex Wang <wangqingcan1990@gmail.com>

Address the comment about NodeInclusionPolicy

a141976

Signed-off-by: Kante Yin <kerthcet@gmail.com>

alculquicondor reviewed Apr 6, 2023

View reviewed changes

fix based on reviews

ae626b9

k8s-ci-robot assigned onlydole Apr 8, 2023

kerthcet reviewed Apr 8, 2023

View reviewed changes

content/en/blog/_posts/2023-04-11-topology-spread-features.md Outdated Show resolved Hide resolved

fix link

2d14bb7

sftim reviewed Apr 11, 2023

View reviewed changes

fix based on the suggestion

f03ac42

k8s-ci-robot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Apr 13, 2023

sftim reviewed Apr 13, 2023

View reviewed changes

content/en/blog/_posts/2023-04-17-topology-spread-features.md Outdated Show resolved Hide resolved

Fix typography

1841857

sftim reviewed Apr 13, 2023

View reviewed changes

k8s-ci-robot assigned sftim Apr 13, 2023

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Apr 13, 2023

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Apr 13, 2023

k8s-ci-robot merged commit 543dd54 into kubernetes:main Apr 13, 2023

	It is the feature to control how Pods are spread to each failure-domain (regions, zones, nodes etc).
	It is the feature to control how Pods are spread in the cluster topology or failure domains (regions, zones, nodes etc).

	take all inclined nodes(satisfied with pod nodeAffinity and nodeSelector) into consideration
	take the Nodes that satisfy the Pod's nodeAffinity and nodeSelector into consideration

	the pod will stuck in pending for it violates the nodeTaint plugin.
	the pod will stuck in Pending if it doesn't tolerate the taint.

	This may lead to a node with untolerated taint best fit the pod in podTopologySpread plugin, and as a result,
	This may lead to a node with untolerated taint as the only candidate for spreading, and as a result,

	To solve this problem once and for all, and to make more accurate decisions in scheduling, we added a new named
	To solve this problem with a simpler API, we added a new field named

	The controller/operator just needs to set different values to the same label key for different revisions.
	The controller or operator managing rollouts just needs to set different values to the same label key for different revisions.

	Then, we introduced the `minDomains` parameter in the Pod Topology Spread.
	Kubernetes v1.24 introduced the `minDomains` parameter for pod topology spread constraints,
	as an alpha feature.

	and a newly created replicaset has the following `topologySpreadConstraints` in template.
	and a newly created ReplicaSet has the following `topologySpreadConstraints` in its Pod template.


		## Getting involved

		These features are managed by the [SIG/Scheduling](https://github.com/kubernetes/community/tree/master/sig-scheduling).

	These features are managed by the [SIG/Scheduling](https://github.com/kubernetes/community/tree/master/sig-scheduling).
	These features are managed by Kubernetes [SIG Scheduling](https://github.com/kubernetes/community/tree/master/sig-scheduling).


		## How can I learn more?

		- [Pod Topology Spread Constraints \| Kubernetes](/docs/concepts/scheduling-eviction/topology-spread-constraints/)

	- [Pod Topology Spread Constraints \| Kubernetes](/docs/concepts/scheduling-eviction/topology-spread-constraints/)
	- [Pod Topology Spread Constraints](/docs/concepts/scheduling-eviction/topology-spread-constraints/) in the Kubernetes documentation


		Authors: [Alex Wang](https://github.com/denkensk)(Shopee), [Kante Yin](https://github.com/kerthcet)(DaoCloud), [Kensei Nakada](https://github.com/sanposhiho)(Mercari)

		In Kubernetes v1.19, [Pod Topology Spread Constraints](/docs/concepts/scheduling-eviction/topology-spread-constraints/) went to GA.

	In Kubernetes v1.19, [Pod Topology Spread Constraints](/docs/concepts/scheduling-eviction/topology-spread-constraints/) went to GA.
	In Kubernetes v1.19, [Pod topology spread constraints](/docs/concepts/scheduling-eviction/topology-spread-constraints/)
	went to general availability (GA).

	As time passed, we received feedback from users,
	As time passed, we - SIG Scheduling - received feedback from users,

	minDomains: 5 # requires 5 Nodes at least.
	minDomains: 5 # requires 5 Nodes at least (because each Node has a unique hostname)

Pod Topology Spread blog post (KEP-3022 KEP-3094 KEP-3243) #39777

Pod Topology Spread blog post (KEP-3022 KEP-3094 KEP-3243) #39777

Conversation

sanposhiho commented Mar 4, 2023 • edited Loading

sanposhiho commented Mar 4, 2023

netlify bot commented Mar 4, 2023 • edited Loading

✅ Pull request preview available for checking

kerthcet commented Mar 4, 2023

sftim commented Mar 6, 2023

bradmccoydev commented Mar 24, 2023

kerthcet commented Mar 24, 2023

sanposhiho left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

denkensk Mar 27, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sanposhiho Apr 1, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

harshitasao commented Mar 31, 2023

kerthcet commented Apr 2, 2023

denkensk commented Apr 5, 2023

kerthcet commented Apr 6, 2023

alculquicondor left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sanposhiho commented Apr 6, 2023 • edited Loading

kerthcet commented Apr 7, 2023

denkensk commented Apr 7, 2023

harshitasao commented Apr 7, 2023

sanposhiho commented Apr 8, 2023

sftim left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kerthcet commented Apr 12, 2023

sanposhiho commented Apr 12, 2023 • edited Loading

sanposhiho commented Apr 13, 2023

sftim commented Apr 13, 2023

sftim left a comment

Choose a reason for hiding this comment

k8s-ci-robot commented Apr 13, 2023

k8s-ci-robot commented Apr 13, 2023

sanposhiho commented Mar 4, 2023 •

edited

Loading

netlify bot commented Mar 4, 2023 •

edited

Loading

sanposhiho left a comment •

edited

Loading

denkensk Mar 27, 2023 •

edited

Loading

sanposhiho Apr 1, 2023 •

edited

Loading

sanposhiho commented Apr 6, 2023 •

edited

Loading

sftim left a comment •

edited

Loading

sanposhiho commented Apr 12, 2023 •

edited

Loading