Fast-tracked rollback to stable with dynamicStableScale went under maxUnavailable #3020
Comments
|
I can think of a few workarounds:
Unfortunately, if the old stable spec is applied on a Rollout with dynamicStableScale and in the middle of an update, we may not honor maxUnavailable and shift too much traffic to the undersized stable/desired ReplicaSet. |
|
Here is how to reproduce it
apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
name: istio-canary
spec:
replicas: 10
revisionHistoryLimit: 3
selector:
matchLabels:
app: istio-canary
template:
metadata:
labels:
app: istio-canary
sidecar.istio.io/inject: "true"
spec:
containers:
- name: istio-canary
image: argoproj/rollouts-demo:red
ports:
- name: http
containerPort: 8080
protocol: TCP
resources:
requests:
memory: 32Mi
cpu: 5m
readinessProbe:
initialDelaySeconds: 30
httpGet:
path: /
port: 8080
periodSeconds: 30
strategy:
canary:
dynamicStableScale: true
canaryService: istio-canary-canary
stableService: istio-canary-stable
trafficRouting:
istio:
virtualService:
name: istio-canary
steps:
- setWeight: 90
- pause: {}
---
apiVersion: v1
kind: Service
metadata:
name: istio-canary
spec:
ports:
- port: 80
targetPort: http
protocol: TCP
name: http
selector:
app: istio-canary
---
apiVersion: v1
kind: Service
metadata:
name: istio-canary-canary
spec:
ports:
- port: 80
targetPort: http
protocol: TCP
name: http
selector:
app: istio-canary
---
apiVersion: v1
kind: Service
metadata:
name: istio-canary-stable
spec:
ports:
- port: 80
targetPort: http
protocol: TCP
name: http
selector:
app: istio-canary
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: istio-canary
spec:
gateways:
- istio-canary
hosts:
- istio-canary
- istio-canary.localhost
- jesse-rollout-scaledown.demo.akuity.io
http:
- route:
- destination:
host: istio-canary-stable
weight: 100
- destination:
host: istio-canary-canary
weight: 0
|
Describe the bug
Argo Rollouts v1.4.1
The following sequence of events occurred on a user's rollout, with:
See the following abbreviated logs (with emphasis on problematic behavior). The logs are grepped by only Patch events and Kube Events:
To Reproduce
I have not yet reproduced this.
Expected behavior
Dynamic stable scale should not have sent 100% to the new desired/stable RS because it only had 2 pods..
I think the bug is that while we may be handling this properly in the abort case, we may not be handling it in the fast-tracked rollback to stable.
Message from the maintainers:
Impacted by this bug? Give it a 👍. We prioritize the issues with the most 👍.
The text was updated successfully, but these errors were encountered: