Prevent workers from running flow runs scheduled for in process retry

[desertaxle]

This PR adds the retry_type field to FlowRunPolicy, so the server can determine if a flow will be retried via an already running process or needs to be rescheduled. Automatic retries will have retry_type set to in_process via the existing RetryFailedFlows orchestration policy. The query to get scheduled flow runs has been updated to filter out flow runs that are in AwaitingRetry but will be retried via an existing process. When transition out of a terminal state into a SCHEDULED state (i.e. manual retries) the retry_type will be set to reschedule and will be given to workers for execution by the server.

Closes #15458

[desertaxle]

@zzstoatzz has pointed out that this change will break flow retries from the UI since it uses the AwaitingRetry state. Closing this PR and exploring other solutions.

Edit: This was for the initial implementation that used run_count. The updated implementation doesn't have this problem.

[codspeed-hq]

CodSpeed Performance Report

Merging #15482 will not alter performance

_{Comparing fix-duplicate-runs-on-awaiting-retry (8bff9da) with main (f168d96)}

Summary

✅ 3 untouched benchmarks

[zzstoatzz]

🥇 lgtm

[jashbycu]

Hi all. Is there a plan to get this into a release soon? This issue has been a major pain for us at CU Boulder and we would love to see this fix deployed. Thank you!

[github-actions]

This pull request is stale because it has been open 14 days with no activity. To keep this pull request open remove stale label or comment.