IP reclaim: differentiate stateless workload under deleting-timeout state on ready node and not-ready node #3002

Icarus9913 · 2023-12-25T09:53:07Z

Refer to #2967

we should add switch to choose whether GC the IP for the following situations

terminating timeout pod on not-ready node
terminating timeout pod on ready node

Signed-off-by: Icarus9913 icaruswu66@qq.com

What this PR does / why we need it:
new feature

codecov · 2023-12-25T09:56:21Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (ca8606e) 81.10% compared to head (65e406f) 81.07%.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3002      +/-   ##
==========================================
- Coverage   81.10%   81.07%   -0.04%     
==========================================
  Files          49       50       +1     
  Lines        5351     5358       +7     
==========================================
+ Hits         4340     4344       +4     
- Misses        854      856       +2     
- Partials      157      158       +1

Flag	Coverage Δ
unittests	`81.07% <100.00%> (-0.04%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files	Coverage Δ
pkg/nodemanager/utils.go	`100.00% <100.00%> (ø)`

... and 1 file with indirect coverage changes

weizhoublue · 2023-12-27T01:07:57Z

For the stateful case， it barely make sense to release ip after dectecting ip conflict . So this feature should only work for stateless workload

weizhoublue · 2024-01-08T06:43:11Z

any update on this ?

cyclinder · 2024-01-10T02:14:50Z

docs/concepts/ipam-des-zh_CN.md


 节点意外宕机后，集群中的 Pod 永久处于 `deleting` 状态，Pod 占用的 IP 地址无法被释放。

- 对处于 `Terminating` 状态的 Pod，Spiderpool 将在 Pod 的 `spec.terminationGracePeriodSecond` 后，自动释放其 IP 地址。该功能可通过环境变量 `SPIDERPOOL_GC_TERMINATING_POD_IP_ENABLED` 来控制。该能力能够用以解决 `节点意外宕机` 的故障场景。
+- 对处于 `Terminating` 状态的 Pod，Spiderpool 将在 Pod 的 `spec.terminationGracePeriodSecond` 后，自动释放其 IP 地址。该功能可通过环境变量 `SPIDERPOOL_GC_TERMINATING_NODE_NOT_READY_POD_IP_ENABLED` 来控制。该能力能够用以解决 `节点意外宕机` 的故障场景。


这句话是我理解错还是描述不对:

节点意外宕机后，集群中的 Pod 永久处于 deleting 状态

对处于 deleting 状态的 Pod?

换成了Terminating

docs/reference/spiderpool-controller.md

docs/concepts/ipam-des-zh_CN.md

weizhoublue · 2024-01-10T02:23:16Z

charts/spiderpool/templates/deployment.yaml

@@ -161,10 +161,12 @@ spec:
 value: {{ .Values.spiderpoolController.httpPort | quote }}
 - name: SPIDERPOOL_GC_IP_ENABLED
 value: {{ .Values.ipam.gc.enabled | quote }}
- - name: SPIDERPOOL_GC_TERMINATING_POD_IP_ENABLED
- value: {{ .Values.ipam.gc.GcDeletingTimeOutPod.enabled | quote }}
+ - name: SPIDERPOOL_GC_TERMINATING_NODE_READY_POD_IP_ENABLED


SPIDERPOOL_GC_STATELESS_TERMINATING_POD_ON_READY_NODE_ENABLE

weizhoublue · 2024-01-10T02:23:49Z

charts/spiderpool/templates/deployment.yaml

- value: {{ .Values.ipam.gc.GcDeletingTimeOutPod.enabled | quote }}
+ - name: SPIDERPOOL_GC_TERMINATING_NODE_READY_POD_IP_ENABLED
+ value: {{ .Values.ipam.gc.enableGcDeletingTimeOutPodWithNodeReady | quote }}
+ - name: SPIDERPOOL_GC_TERMINATING_NODE_NOT_READY_POD_IP_ENABLED


SPIDERPOOL_GC_STATELESS_TERMINATING_POD_ON_NOT_READY_NODE_ENABLE

weizhoublue · 2024-01-10T02:25:40Z

charts/spiderpool/values.yaml

- ## @param ipam.gc.GcDeletingTimeOutPod.enabled enable retrieve IP for the pod who times out of deleting graceful period
- enabled: true
+ ## @param ipam.gc.enableGcDeletingTimeOutPodWithNodeReady enable reclaim IP for the pod who times out of deleting graceful period with its node ready
+ enableGcDeletingTimeOutPodWithNodeReady: true


enableGcStatelesTerminatingPodOnReadyNode

weizhoublue · 2024-01-10T02:26:59Z

docs/concepts/coordinator-zh_CN.md

@@ -75,6 +75,8 @@ spec:
 detectIPConflict: true # Enable detectIPConflict
 ```

+> 若 IP 冲突检查发现某 IP 已被集群中其他处于 `Terminating` 状态的 Pod 所占用，请参考 [IP 回收机制](./ipam-des-zh_CN.md#ip-回收机制) 相关配置。


这个是个歧义，“IP 冲突检查发现”功能并不能发现冲突的 ip 是否处于 Terminating` 状态，只可能是可能的原因之一

docs/concepts/ipam-des-zh_CN.md

docs/concepts/coordinator-zh_CN.md

charts/spiderpool/values.yaml

docs/concepts/ipam-des-zh_CN.md

weizhoublue · 2024-01-15T14:45:21Z

现有有了两种场景，是否有不同的用例闭环验证

weizhoublue · 2024-01-22T09:58:38Z

charts/spiderpool/values.yaml

- ## @param ipam.gc.GcDeletingTimeOutPod.enabled enable retrieve IP for the pod who times out of deleting graceful period
- enabled: true
+ ## @param ipam.gc.enableGcStatelesTerminatingPodOnReadyNode enable reclaim IP for the stateless pod who is over deleting graceful period on a ready node
+ enableGcStatelesTerminatingPodOnReadyNode: true


enableGcStatelesTerminatingPodOnReadyNode
enableGcStatelesTerminatingPodOnNotReadyNode

调整为如下，是否会清爽点，将来可能还有 stateful 其它层级其中 zombie pod 定义为 deleting 超时了的 pod

StatelesPod: zombieOnReadyNode: true zombieOnNotReadyNode: true

docs/concepts/ipam-des-zh_CN.md

Signed-off-by: Icarus9913 <icaruswu66@qq.com>

Icarus9913 added pr/not-ready not ready for merging release/feature-new release note for new feature labels Dec 25, 2023

Icarus9913 requested a review from weizhoublue as a code owner December 25, 2023 09:53

Icarus9913 force-pushed the feat/wk/gc-env branch from 7a2c234 to 0158dee Compare January 9, 2024 06:25

Icarus9913 requested a review from windsonsea as a code owner January 9, 2024 06:25

Icarus9913 added pr/ready-review This pull is ready for review and removed pr/not-ready not ready for merging labels Jan 9, 2024

Icarus9913 force-pushed the feat/wk/gc-env branch from 0158dee to 5eb4e14 Compare January 9, 2024 09:04

Icarus9913 requested review from ty-dc and bzsuni as code owners January 9, 2024 09:04

cyclinder reviewed Jan 10, 2024

View reviewed changes

weizhoublue reviewed Jan 10, 2024

View reviewed changes

docs/concepts/ipam-des-zh_CN.md Outdated Show resolved Hide resolved

Icarus9913 force-pushed the feat/wk/gc-env branch 2 times, most recently from e014491 to 94d9827 Compare January 10, 2024 03:13

cyclinder reviewed Jan 10, 2024

View reviewed changes

docs/concepts/coordinator-zh_CN.md Outdated Show resolved Hide resolved

Icarus9913 force-pushed the feat/wk/gc-env branch from 94d9827 to 1795182 Compare January 10, 2024 04:50

cyclinder previously approved these changes Jan 10, 2024

View reviewed changes

weizhoublue reviewed Jan 11, 2024

View reviewed changes

charts/spiderpool/values.yaml Outdated Show resolved Hide resolved

Icarus9913 dismissed cyclinder’s stale review via 8870338 January 11, 2024 02:53

Icarus9913 force-pushed the feat/wk/gc-env branch 2 times, most recently from 8870338 to 8b4f4a0 Compare January 11, 2024 02:58

weizhoublue reviewed Jan 11, 2024

View reviewed changes

docs/concepts/ipam-des-zh_CN.md Outdated Show resolved Hide resolved

Icarus9913 force-pushed the feat/wk/gc-env branch from 8b4f4a0 to dc6ea2a Compare January 12, 2024 03:04

weizhoublue changed the title ~~supplement IP GC ENV for various scenarios~~ IP reclaim: differentiate stateless workload under deleting-timeout state on ready node and not-ready node Jan 15, 2024

weizhoublue reviewed Jan 22, 2024

View reviewed changes

docs/concepts/ipam-des-zh_CN.md Outdated Show resolved Hide resolved

Icarus9913 force-pushed the feat/wk/gc-env branch 4 times, most recently from 92d1f4e to 63213d4 Compare January 24, 2024 09:57

supplement IP GC ENV for various scenarios

65e406f

Signed-off-by: Icarus9913 <icaruswu66@qq.com>

Icarus9913 force-pushed the feat/wk/gc-env branch from 63213d4 to 65e406f Compare January 24, 2024 10:53

weizhoublue approved these changes Jan 25, 2024

View reviewed changes

ty-dc approved these changes Jan 25, 2024

View reviewed changes

Icarus9913 merged commit af4632d into spidernet-io:main Jan 25, 2024
44 checks passed

Icarus9913 deleted the feat/wk/gc-env branch January 25, 2024 02:37

ty-dc mentioned this pull request Jan 29, 2024

IP reclaim for various scenarios #2967

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

IP reclaim: differentiate stateless workload under deleting-timeout state on ready node and not-ready node #3002

IP reclaim: differentiate stateless workload under deleting-timeout state on ready node and not-ready node #3002

Icarus9913 commented Dec 25, 2023

codecov bot commented Dec 25, 2023 •

edited

Loading

weizhoublue commented Dec 27, 2023 •

edited

Loading

weizhoublue commented Jan 8, 2024

cyclinder Jan 10, 2024

Icarus9913 Jan 10, 2024

weizhoublue Jan 10, 2024

weizhoublue Jan 10, 2024

weizhoublue Jan 10, 2024

weizhoublue Jan 10, 2024 •

edited

Loading

weizhoublue commented Jan 15, 2024

weizhoublue Jan 22, 2024

IP reclaim: differentiate stateless workload under deleting-timeout state on ready node and not-ready node #3002

IP reclaim: differentiate stateless workload under deleting-timeout state on ready node and not-ready node #3002

Conversation

Icarus9913 commented Dec 25, 2023

codecov bot commented Dec 25, 2023 • edited Loading

Codecov Report

weizhoublue commented Dec 27, 2023 • edited Loading

weizhoublue commented Jan 8, 2024

cyclinder Jan 10, 2024

Choose a reason for hiding this comment

Icarus9913 Jan 10, 2024

Choose a reason for hiding this comment

weizhoublue Jan 10, 2024

Choose a reason for hiding this comment

weizhoublue Jan 10, 2024

Choose a reason for hiding this comment

weizhoublue Jan 10, 2024

Choose a reason for hiding this comment

weizhoublue Jan 10, 2024 • edited Loading

Choose a reason for hiding this comment

weizhoublue commented Jan 15, 2024

weizhoublue Jan 22, 2024

Choose a reason for hiding this comment

codecov bot commented Dec 25, 2023 •

edited

Loading

weizhoublue commented Dec 27, 2023 •

edited

Loading

weizhoublue Jan 10, 2024 •

edited

Loading