Inconclusive state is not respected for background analysis

[jahvon]

Checklist:

• I've included steps to reproduce the bug.
• I've included the version of argo rollouts.

Describe the bug

When I create a Rollout that uses background analysis + a canary pause step, I am unable to get the Rollout to stay in the paused state when the canary analysis is inconclusive. Instead, I'm seeing the AnalysisRun getting into the Inconclusive state but and restarted/recreated over and over until it finally promotes the Rollout as healthy (at the end of the pause duration).

I've inspected the Rollout and AnalysisRuns during the canary analysis and noticed that the status.pauseConditions never includes Inconclusive as a reason which may be contributing to this issue. In spot checking the logs for Argo Rollouts after rerunning my test a couple of times, I do see logs that indicate that the inconclusive pause condition was added but it seems to be inconsistent, in most cases the Rollout never reflects that the background AnalysisRuns were inconclusive.

To Reproduce

Below is the exact AnalysisTemplate and Rollout that I've used to reproduce this.

apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
  name: misconfigured-analysis
spec:
  metrics:
  - name: faked-success
    successCondition: result.status < 300
    failureCondition: result.status > 400
    count: 5
    interval: 10s
    inconclusiveLimit: 0
    provider:
      web:
        url: https://httpbin.org/anything
        method: POST
        jsonBody:
          status: 302
        jsonPath: "{$.json}"
---
apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: inconclusive-analysis-test-rollout
spec:
  replicas: 2
  strategy:
    canary:
      canaryService: inconclusive-analysis-rollout-canary
      stableService: inconclusive-analysis-rollout-stable
      trafficRouting:
        nginx:
          stableIngress: inconclusive-analysis-rollout-stable
      steps:
      - setWeight: 33
      - pause: {duration: 5m}
      analysis:
        templates:
        - templateName: inconclusive-analysis-misconfigured-analysis
        startingStep: 1 
  revisionHistoryLimit: 2
  selector:
    matchLabels:
      app: inconclusive-analysis-rollout
  template:
    metadata:
      labels:
        app: inconclusive-analysis-rollout
    spec:
      containers:
        - name: rollouts-demo
          image: argoproj/rollouts-demo:green
          ports:
            - name: http
              containerPort: 8080
              protocol: TCP
          resources:
            requests:
              memory: 32Mi
              cpu: 5m

After applying those alongside the references service and ingress, you should notice that the AnalysisRuns are created and enter an Inconclusive state but after the pause duration, the Rollout promotes the canary revision to stable.

Expected behavior

The Rollout reflects that it's encountered Inconclusive analysis and pauses as described in the docs.

Screenshots

Version

v1.7.2

Logs

There's a ton of logs generated for this Rollout so I've omitted them but happy to pull them if needed.

---

Message from the maintainers:

Impacted by this bug? Give it a 👍. We prioritize the issues with the most 👍.