Fail connection manager workflow on non-deterministic exception

[lmossman]

What

Resolves #13973

How

Uses the temporal api to have non-deterministic exceptions cause workflows to be marked as failed, rather than retrying indefinitely.

Note: This should probably only be merged once this other issue has been completed, so that the failed workflows are automatically restarted when this occurs: #14043

[jdpgrailsdev]

[gosusnp]

I am wondering, how complex would it be to write a test for this?

[lmossman]

I'm not exactly sure what a test for this would look like, since it would require changing the order or names of activities for a running workflow, which I've only done by changing code. Might be possible though - probably worth having someone spend some time looking into that.

[lmossman]

@benmoriceau I added a note to the description of this ticket - it's up to you all, but we may want to wait to wait to merge this until the issue to find and restart failed workflows has been completed (#14043).

If we merge this first, then non-deterministic exceptions will cause workflows to be set to Failed without anything automatically fixing them. Though, I'm not sure if this is actually worse or better than the current behavior (Temporal just retrying the workflows indefinitely, running into the non-deterministic exception over and over). It feels like the current behavior is probably preferable, as fixing that just requires rolling back to the previous deployment, which should result in workflows automatically recovering. Whereas if we mark them as failed without the automatic recovery process, then we will have to manually go and restart a bunch of workflows even if we roll back.

I'll leave that decision up to the team!

[evantahler]

#14043 has been merged. @jdpgrailsdev / @benmoriceau it it time to merge this?

[lmossman]

@evantahler if we merge this before we have the process in place that automatically restarts failed workflows (I think this ticket #15218 ?), then non-deterministic exceptions will result in workflows being set to Failed with nothing automatically restarting them.

See my comment here - I think it may be slightly preferable to keep the current behavior until we have that, but its up to you all

[benmoriceau]

This is ready to be merged, I will do it once green.