Canary stuck after failed deployment

[giggio]

We are trying to upgrade a release from a Helm chart, and skip analysis is being used. It seems that some problem brought flagger to a state that locked any progress.

Logs show, in order, several of each:

Synced ns/release

Halt advancement release-primary.ns deployment \"release-primary\" exceeded its progress deadline

Halt advancement release-primary.ns waiting for rollout to finish: 1 old replicas are pending termination

After that, there is one log every 20 seconds with the same message:

Halt advancement release-primary.ns deployment \"release-primary\" exceeded its progress deadline

If you try to create a new release, the canary deployment is updated, but the primary is not, and the canary succeeds. I have pods running with old specs and crashed, which were promoted to primary in the past. As the primary deployment is not updated, no updated pods are updated. This is because of #362.

I can't solve this unless I manually remove the whole canary structure, which seems stuck.

[stefanprodan]

Hi @giggio what provider are you using?

[giggio]

Hey @stefanprodan, we are using Istio.

[stefanprodan]

Can you please update Flagger image to weaveworks/flagger:skip-primary-check-9af6ade you should be able to rollout the new version with skip analysis on even if the primary fails. Please let me know if it worked. Thanks!

[stefanprodan]

Also please join Flagger slack channel and give me a ping https://slack.weave.works/