VirtualService's Retries stanza gets deleted by Argo

[m-wynn]

Checklist:

• I've included steps to reproduce the bug.
• I've inclued the version of argo rollouts.

Describe the bug

Argo Rollouts is managing routes on a couple Istio delegate VirtualServices. These VirtualServices have a route and a retries stanza. Whenever Argo updates the VirtualService, the retries stanza is lost. I did not notice this issue before upgrading to Istio 1.15.0 and changing the application to use delegate VirtualServices

To Reproduce

1. Create a delegate VirtualService with a retries stanza:

kind: VirtualService
apiVersion: networking.istio.io/v1beta1
metadata:
  name: my-application-delegate
  namespace: prod
spec:
  http:
    - name: my-application-main
      route:
        - destination:
            host: my-application
            subset: stable
          weight: 100
        - destination:
            host: my-application
            subset: canary
      retries:
        attempts: 3
        perTryTimeout: 10s
        retryOn: 'gateway-error,connect-failure,refused-stream'

2. Route from another virtualservice

kind: VirtualService
apiVersion: networking.istio.io/v1beta1
metadata:
  name: my-application
  namespace: prod
spec:
  hosts:
    - mywebsite.com
  gateways:
    - istio-system/external
    - mesh
  http:
    - name: my-application
      delegate:
        name: my-application-delegate
        namespace: prod

3. Manage the routes from an argo rollout:

apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: my-application
  namespace: prod
spec:
  revisionHistoryLimit: 3
  template:
    spec:
      ...
  strategy:
    canary:
      canaryMetadata:
        annotations:
          role: canary
        labels:
          role: canary

      stableMetadata:
        annotations:
          role: stable
        labels:
          role: stable

      maxUnavailable: 20%
      maxSurge: 100%

      scaleDownDelaySeconds: 120

      trafficRouting:
        managedRoutes:
          - name: internal-header
        istio:
          virtualServices:
            - name: my-application-delegate
              routes:
                - my-application
          destinationRule:
            name: my-application
            canarySubsetName: canary
            stableSubsetName: stable
      steps:
        - setWeight: 0
        - setCanaryScale:
            replicas: 3
        - setHeaderRoute: 
            name: "internal-header"
            match:
              - headerName: cookie
                headerValue:
                  regex: ^(.*;?)?(internal=true)(;.*)?
        - pause: {}

4. Update the rollout with a new release

Expected behavior

The route should retain the retries stanza. Instead, as soon as it is updated, the retries stanza is deleted.

Screenshots

N/A

Version

1.3.0

Logs

Logs

time="2022-09-26T19:42:51Z" level=info msg="Start processing" resource=prod/my-application
time="2022-09-26T19:42:51Z" level=info msg="Processing completed" resource=prod/my-application
time="2022-09-26T19:42:51Z" level=info msg="Started syncing rollout" generation=36 namespace=prod resourceVersion=758342599 rollout=my-application
time="2022-09-26T19:42:51Z" level=info msg="ComputePodTemplateHash hash changed (expected: 6ccc4dbffd, actual: 5bff6b9764)" namespace=prod rollout=my-application
time="2022-09-26T19:42:51Z" level=info msg="Patched conditions: {\"status\":{\"conditions\":[{\"lastTransitionTime\":\"2022-09-26T14:24:23Z\",\"lastUpdateTime\":\"2022-09-26T14:24:23Z\",\"message\":\"Rollout has minimum availability\",\"reason\":\"AvailableReason\",\"status\":\"True\",\"type\":\"Available\"},{\"lastTransitionTime\":\"2022-09-26T19:32:39Z\",\"lastUpdateTime\":\"2022-09-26T19:32:39Z\",\"message\":\"Rollout is not healthy\",\"reason\":\"RolloutHealthy\",\"status\":\"False\",\"type\":\"Healthy\"},{\"lastTransitionTime\":\"2022-09-26T19:32:39Z\",\"lastUpdateTime\":\"2022-09-26T19:32:39Z\",\"message\":\"RolloutCompleted\",\"reason\":\"RolloutCompleted\",\"status\":\"False\",\"type\":\"Completed\"},{\"lastTransitionTime\":\"2022-09-26T19:42:18Z\",\"lastUpdateTime\":\"2022-09-26T19:42:51Z\",\"message\":\"Rollout is resumed\",\"reason\":\"RolloutResumed\",\"status\":\"Unknown\",\"type\":\"Progressing\"},{\"lastTransitionTime\":\"2022-09-26T19:42:51Z\",\"lastUpdateTime\":\"2022-09-26T19:42:51Z\",\"message\":\"Rollout is paused\",\"reason\":\"RolloutPaused\",\"status\":\"False\",\"type\":\"Paused\"}],\"message\":\"more replicas need to be updated\",\"phase\":\"Progressing\"}}" generation=36 namespace=prod resourceVersion=758342599 rollout=my-application
time="2022-09-26T19:42:51Z" level=info msg="Found 1 TrafficRouting Reconcilers" namespace=prod rollout=my-application
time="2022-09-26T19:42:51Z" level=info msg="Reconciling TrafficRouting with type 'Istio'" namespace=prod rollout=my-application
time="2022-09-26T19:42:51Z" level=info msg="Reconciling canary pause step (stepIndex: 3/6)" namespace=prod rollout=my-application
time="2022-09-26T19:42:51Z" level=info msg="Not finished reconciling Canary Pause" namespace=prod rollout=my-application
time="2022-09-26T19:42:51Z" level=info msg="Rollout has been unpaused" namespace=prod rollout=my-application
time="2022-09-26T19:42:51Z" level=info msg="Rollout step 4/6 completed (pause)" event_reason=RolloutStepCompleted namespace=prod rollout=my-application
time="2022-09-26T19:42:51Z" level=info msg="Event(v1.ObjectReference{Kind:\"Rollout\", Namespace:\"prod\", Name:\"my-application\", UID:\"03b464f0-0178-4837-8773-492ddaf83f41\", APIVersion:\"argoproj.io/v1alpha1\", ResourceVersion:\"758342599\", FieldPath:\"\"}): type: 'Normal' reason: 'RolloutStepCompleted' Rollout step 4/6 completed (pause)"
time="2022-09-26T19:42:51Z" level=info msg="Start processing" resource=prod/my-application
time="2022-09-26T19:42:51Z" level=info msg="Processing completed" resource=prod/my-application
time="2022-09-26T19:42:51Z" level=info msg="Patched: {\"status\":{\"conditions\":[{\"lastTransitionTime\":\"2022-09-26T14:24:23Z\",\"lastUpdateTime\":\"2022-09-26T14:24:23Z\",\"message\":\"Rollout has minimum availability\",\"reason\":\"AvailableReason\",\"status\":\"True\",\"type\":\"Available\"},{\"lastTransitionTime\":\"2022-09-26T19:32:39Z\",\"lastUpdateTime\":\"2022-09-26T19:32:39Z\",\"message\":\"Rollout is not healthy\",\"reason\":\"RolloutHealthy\",\"status\":\"False\",\"type\":\"Healthy\"},{\"lastTransitionTime\":\"2022-09-26T19:32:39Z\",\"lastUpdateTime\":\"2022-09-26T19:32:39Z\",\"message\":\"RolloutCompleted\",\"reason\":\"RolloutCompleted\",\"status\":\"False\",\"type\":\"Completed\"},{\"lastTransitionTime\":\"2022-09-26T19:33:10Z\",\"lastUpdateTime\":\"2022-09-26T19:33:10Z\",\"message\":\"Rollout is paused\",\"reason\":\"RolloutPaused\",\"status\":\"True\",\"type\":\"Paused\"},{\"lastTransitionTime\":\"2022-09-26T19:42:51Z\",\"lastUpdateTime\":\"2022-09-26T19:42:51Z\",\"message\":\"ReplicaSet \\\"my-application-5bff6b9764\\\" is progressing.\",\"reason\":\"ReplicaSetUpdated\",\"status\":\"True\",\"type\":\"Progressing\"}],\"controllerPause\":null,\"currentStepIndex\":4,\"message\":\"more replicas need to be updated\",\"phase\":\"Progressing\"}}" generation=36 namespace=prod resourceVersion=758342599 rollout=my-application
time="2022-09-26T19:42:51Z" level=info msg="persisted to informer" generation=36 namespace=prod resourceVersion=758342602 rollout=my-application
time="2022-09-26T19:42:51Z" level=info msg="Reconciliation completed" generation=36 namespace=prod resourceVersion=758342599 rollout=my-application time_ms=60.706250000000004
time="2022-09-26T19:42:51Z" level=info msg="Started syncing rollout" generation=36 namespace=prod resourceVersion=758342602 rollout=my-application
time="2022-09-26T19:42:51Z" level=info msg="ComputePodTemplateHash hash changed (expected: 6ccc4dbffd, actual: 5bff6b9764)" namespace=prod rollout=my-application
time="2022-09-26T19:42:51Z" level=info msg="Start processing" resource=prod/my-application
time="2022-09-26T19:42:51Z" level=info msg="Processing completed" resource=prod/my-application
time="2022-09-26T19:42:51Z" level=info msg="Patched conditions: {\"status\":{\"conditions\":[{\"lastTransitionTime\":\"2022-09-26T14:24:23Z\",\"lastUpdateTime\":\"2022-09-26T14:24:23Z\",\"message\":\"Rollout has minimum availability\",\"reason\":\"AvailableReason\",\"status\":\"True\",\"type\":\"Available\"},{\"lastTransitionTime\":\"2022-09-26T19:32:39Z\",\"lastUpdateTime\":\"2022-09-26T19:32:39Z\",\"message\":\"Rollout is not healthy\",\"reason\":\"RolloutHealthy\",\"status\":\"False\",\"type\":\"Healthy\"},{\"lastTransitionTime\":\"2022-09-26T19:32:39Z\",\"lastUpdateTime\":\"2022-09-26T19:32:39Z\",\"message\":\"RolloutCompleted\",\"reason\":\"RolloutCompleted\",\"status\":\"False\",\"type\":\"Completed\"},{\"lastTransitionTime\":\"2022-09-26T19:42:51Z\",\"lastUpdateTime\":\"2022-09-26T19:42:51Z\",\"message\":\"ReplicaSet \\\"my-application-5bff6b9764\\\" is progressing.\",\"reason\":\"ReplicaSetUpdated\",\"status\":\"True\",\"type\":\"Progressing\"},{\"lastTransitionTime\":\"2022-09-26T19:42:51Z\",\"lastUpdateTime\":\"2022-09-26T19:42:51Z\",\"message\":\"Rollout is paused\",\"reason\":\"RolloutPaused\",\"status\":\"False\",\"type\":\"Paused\"}]}}" generation=36 namespace=prod resourceVersion=758342602 rollout=my-application
time="2022-09-26T19:42:51Z" level=info msg="Found 1 TrafficRouting Reconcilers" namespace=prod rollout=my-application
time="2022-09-26T19:42:51Z" level=info msg="Reconciling TrafficRouting with type 'Istio'" namespace=prod rollout=my-application
time="2022-09-26T19:42:51Z" level=info msg="Start processing" resource=prod/my-application
time="2022-09-26T19:42:51Z" level=info msg="Processing completed" resource=prod/my-application
time="2022-09-26T19:42:51Z" level=info msg="VirtualService `my-application-delegate` set to desiredWeight '100'" event_reason="Updated VirtualService" namespace=prod rollout=my-application
time="2022-09-26T19:42:51Z" level=info msg="Previous weights: &TrafficWeights{Canary:WeightDestination{Weight:0,ServiceName:,PodTemplateHash:5bff6b9764,},Stable:WeightDestination{Weight:100,ServiceName:,PodTemplateHash:6468cd49fc,},Additional:[]WeightDestination{},Verified:nil,}" namespace=prod rollout=my-application
time="2022-09-26T19:42:51Z" level=info msg="New weights: &TrafficWeights{Canary:WeightDestination{Weight:100,ServiceName:,PodTemplateHash:5bff6b9764,},Stable:WeightDestination{Weight:0,ServiceName:,PodTemplateHash:6468cd49fc,},Additional:[]WeightDestination{},Verified:nil,}" namespace=prod rollout=my-application
time="2022-09-26T19:42:51Z" level=info msg="Event(v1.ObjectReference{Kind:\"Rollout\", Namespace:\"prod\", Name:\"my-application\", UID:\"03b464f0-0178-4837-8773-492ddaf83f41\", APIVersion:\"argoproj.io/v1alpha1\", ResourceVersion:\"758342602\", FieldPath:\"\"}): type: 'Normal' reason: 'Updated VirtualService' VirtualService `my-application-delegate` set to desiredWeight '100'"
time="2022-09-26T19:42:51Z" level=info msg="Event(v1.ObjectReference{Kind:\"Rollout\", Namespace:\"prod\", Name:\"my-application\", UID:\"03b464f0-0178-4837-8773-492ddaf83f41\", APIVersion:\"argoproj.io/v1alpha1\", ResourceVersion:\"758342602\", FieldPath:\"\"}): type: 'Normal' reason: 'TrafficWeightUpdated' Traffic weight updated from 0 to 100"
time="2022-09-26T19:42:51Z" level=info msg="Traffic weight updated from 0 to 100" event_reason=TrafficWeightUpdated namespace=prod rollout=my-application
time="2022-09-26T19:42:51Z" level=info msg="Rollout step 5/6 completed (setWeight: 100)" event_reason=RolloutStepCompleted namespace=prod rollout=my-application
time="2022-09-26T19:42:51Z" level=info msg="Event(v1.ObjectReference{Kind:\"Rollout\", Namespace:\"prod\", Name:\"my-application\", UID:\"03b464f0-0178-4837-8773-492ddaf83f41\", APIVersion:\"argoproj.io/v1alpha1\", ResourceVersion:\"758342602\", FieldPath:\"\"}): type: 'Normal' reason: 'RolloutStepCompleted' Rollout step 5/6 completed (setWeight: 100)"
time="2022-09-26T19:42:51Z" level=info msg="Patched: {\"status\":{\"canary\":{\"weights\":{\"canary\":{\"weight\":100},\"stable\":{\"weight\":0}}},\"currentStepIndex\":5}}" generation=36 namespace=prod resourceVersion=758342602 rollout=my-application
time="2022-09-26T19:42:51Z" level=info msg="persisted to informer" generation=36 namespace=prod resourceVersion=758342607 rollout=my-application
time="2022-09-26T19:42:51Z" level=info msg="Reconciliation completed" generation=36 namespace=prod resourceVersion=758342602 rollout=my-application time_ms=73.964032
time="2022-09-26T19:42:51Z" level=info msg="Started syncing rollout" generation=36 namespace=prod resourceVersion=758342607 rollout=my-application
time="2022-09-26T19:42:51Z" level=info msg="ComputePodTemplateHash hash changed (expected: 6ccc4dbffd, actual: 5bff6b9764)" namespace=prod rollout=my-application
time="2022-09-26T19:42:51Z" level=info msg="Found 1 TrafficRouting Reconcilers" namespace=prod rollout=my-application
time="2022-09-26T19:42:51Z" level=info msg="Reconciling TrafficRouting with type 'Istio'" namespace=prod rollout=my-application
time="2022-09-26T19:42:51Z" level=info msg="Start processing" resource=prod/my-application
time="2022-09-26T19:42:51Z" level=info msg="Processing completed" resource=prod/my-application
time="2022-09-26T19:42:51Z" level=info msg="Reconciling canary pause step (stepIndex: 5/6)" namespace=prod rollout=my-application
time="2022-09-26T19:42:51Z" level=info msg="Not finished reconciling Canary Pause" namespace=prod rollout=my-application
time="2022-09-26T19:42:51Z" level=info msg="Adding pause reason CanaryPauseStep with start time 2022-09-26T19:42:51Z" namespace=prod rollout=my-application
time="2022-09-26T19:42:51Z" level=info msg="Rollout is paused (CanaryPauseStep)" event_reason=RolloutPaused namespace=prod rollout=my-application
time="2022-09-26T19:42:51Z" level=info msg="Patched: {\"status\":{\"controllerPause\":true,\"message\":\"CanaryPauseStep\",\"pauseConditions\":[{\"reason\":\"CanaryPauseStep\",\"startTime\":\"2022-09-26T19:42:51Z\"}],\"phase\":\"Paused\"}}" generation=36 namespace=prod resourceVersion=758342607 rollout=my-application
time="2022-09-26T19:42:51Z" level=info msg="Event(v1.ObjectReference{Kind:\"Rollout\", Namespace:\"prod\", Name:\"my-application\", UID:\"03b464f0-0178-4837-8773-492ddaf83f41\", APIVersion:\"argoproj.io/v1alpha1\", ResourceVersion:\"758342607\", FieldPath:\"\"}): type: 'Normal' reason: 'RolloutPaused' Rollout is paused (CanaryPauseStep)"
time="2022-09-26T19:42:51Z" level=info msg="persisted to informer" generation=36 namespace=prod resourceVersion=758342609 rollout=my-application
time="2022-09-26T19:42:51Z" level=info msg="Reconciliation completed" generation=36 namespace=prod resourceVersion=758342607 rollout=my-application time_ms=34.155287
time="2022-09-26T19:42:51Z" level=info msg="Started syncing rollout" generation=36 namespace=prod resourceVersion=758342609 rollout=my-application
time="2022-09-26T19:42:51Z" level=info msg="ComputePodTemplateHash hash changed (expected: 6ccc4dbffd, actual: 5bff6b9764)" namespace=prod rollout=my-application
time="2022-09-26T19:42:51Z" level=info msg="Start processing" resource=prod/my-application
time="2022-09-26T19:42:51Z" level=info msg="Processing completed" resource=prod/my-application
time="2022-09-26T19:42:51Z" level=info msg="Patched conditions: {\"status\":{\"conditions\":[{\"lastTransitionTime\":\"2022-09-26T14:24:23Z\",\"lastUpdateTime\":\"2022-09-26T14:24:23Z\",\"message\":\"Rollout has minimum availability\",\"reason\":\"AvailableReason\",\"status\":\"True\",\"type\":\"Available\"},{\"lastTransitionTime\":\"2022-09-26T19:32:39Z\",\"lastUpdateTime\":\"2022-09-26T19:32:39Z\",\"message\":\"Rollout is not healthy\",\"reason\":\"RolloutHealthy\",\"status\":\"False\",\"type\":\"Healthy\"},{\"lastTransitionTime\":\"2022-09-26T19:32:39Z\",\"lastUpdateTime\":\"2022-09-26T19:32:39Z\",\"message\":\"RolloutCompleted\",\"reason\":\"RolloutCompleted\",\"status\":\"False\",\"type\":\"Completed\"},{\"lastTransitionTime\":\"2022-09-26T19:42:51Z\",\"lastUpdateTime\":\"2022-09-26T19:42:51Z\",\"message\":\"Rollout is paused\",\"reason\":\"RolloutPaused\",\"status\":\"Unknown\",\"type\":\"Progressing\"},{\"lastTransitionTime\":\"2022-09-26T19:42:51Z\",\"lastUpdateTime\":\"2022-09-26T19:42:51Z\",\"message\":\"Rollout is paused\",\"reason\":\"RolloutPaused\",\"status\":\"True\",\"type\":\"Paused\"}]}}" generation=36 namespace=prod resourceVersion=758342609 rollout=my-application
time="2022-09-26T19:42:51Z" level=info msg="Found 1 TrafficRouting Reconcilers" namespace=prod rollout=my-application
time="2022-09-26T19:42:51Z" level=info msg="Reconciling TrafficRouting with type 'Istio'" namespace=prod rollout=my-application
time="2022-09-26T19:42:51Z" level=info msg="Reconciling canary pause step (stepIndex: 5/6)" namespace=prod rollout=my-application
time="2022-09-26T19:42:51Z" level=info msg="Enqueueing Rollout in 4m59.559550001s seconds" namespace=prod rollout=my-application
time="2022-09-26T19:42:51Z" level=info msg="Not finished reconciling Canary Pause" namespace=prod rollout=my-application
time="2022-09-26T19:42:51Z" level=info msg="No status changes. Skipping patch" generation=36 namespace=prod resourceVersion=758342609 rollout=my-application
time="2022-09-26T19:42:51Z" level=info msg="persisted to informer" generation=36 namespace=prod resourceVersion=758342611 rollout=my-application
time="2022-09-26T19:42:51Z" level=info msg="Reconciliation completed" generation=36 namespace=prod resourceVersion=758342609 rollout=my-application time_ms=35.44591
time="2022-09-26T19:42:51Z" level=info msg="Start processing" resource=prod/my-application
time="2022-09-26T19:42:51Z" level=info msg="Processing completed" resource=prod/my-application
time="2022-09-26T19:42:51Z" level=info msg="Started syncing rollout" generation=36 namespace=prod resourceVersion=758342611 rollout=my-application
time="2022-09-26T19:42:51Z" level=info msg="ComputePodTemplateHash hash changed (expected: 6ccc4dbffd, actual: 5bff6b9764)" namespace=prod rollout=my-application
time="2022-09-26T19:42:51Z" level=info msg="Found 1 TrafficRouting Reconcilers" namespace=prod rollout=my-application
time="2022-09-26T19:42:51Z" level=info msg="Reconciling TrafficRouting with type 'Istio'" namespace=prod rollout=my-application
time="2022-09-26T19:42:51Z" level=info msg="Reconciling canary pause step (stepIndex: 5/6)" namespace=prod rollout=my-application
time="2022-09-26T19:42:51Z" level=info msg="Enqueueing Rollout in 4m59.547951417s seconds" namespace=prod rollout=my-application
time="2022-09-26T19:42:51Z" level=info msg="Not finished reconciling Canary Pause" namespace=prod rollout=my-application
time="2022-09-26T19:42:51Z" level=info msg="No status changes. Skipping patch" generation=36 namespace=prod resourceVersion=758342611 rollout=my-application
time="2022-09-26T19:42:51Z" level=info msg="Reconciliation completed" generation=36 namespace=prod resourceVersion=758342611 rollout=my-application time_ms=5.0035039999999995

Message from the maintainers:

Impacted by this bug? Give it a 👍. We prioritize the issues with the most 👍.

[zachaller]

Can you test this against Argo Rollouts 1.2.2? This might be related to some changes in 1.3 I want to confirm.

[m-wynn]

I downgraded to 1.2.2, removed the setHeaderRoute bit as it is not compatible with 1.2, and attempted a rollback. You are correct, I no longer lose the retries stanza on upgrade.

I also realized that argo was deleting the port stanza from the routes it manages as well. This is also resolved with a rollback to 1.2.2.

[zachaller]

Ok, I think I have an idea on what is going on will try to get a fix out soon.