Update function state change traces to improve orchestration monitoring query

[bachuv]

This PR adds a new RuntimeStatus field to track function state changes. This helps us improve our monitoring query in the portal (Functions -> OrchestratorFunction -> Monitor tab -> App Insights query).

Issues that are fixed in this PR:

• The portal shows the incorrect status for Terminated orchestrations
• We don't support showing the Suspended runtime state

New query to list the status of orchestration instances:

traces
| where customDimensions.Category == "Host.Triggers.DurableTask"
| extend functionName = tostring(customDimensions["prop__functionName"])
| extend instanceId = tostring(customDimensions["prop__instanceId"])
| extend state = tostring(customDimensions["prop__state"])
| extend isReplay = tobool(tolower(customDimensions["prop__isReplay"]))
| extend hubName = tostring(tolower(customDimensions["prop__hubName"]))
| extend runtimeStatus = tostring((customDimensions["prop__runtimeStatus"]))
| where isReplay != true
| where functionName == "<functionName>"
| where state != "Awaited"
| where runtimeStatus != ""
| summarize arg_max(timestamp, *) by instanceId
| order by timestamp asc
| project timestamp, instanceId, runtimeStatus

Pull request checklist

• My changes do not require documentation changes
  • Otherwise: Documentation PR is ready to merge and referenced in pending_docs.md
• My changes should not be added to the release notes for the next release
  • Otherwise: I've added my notes to release_notes.md
• My changes do not need to be backported to a previous version
  • Otherwise: Backport tracked by issue/PR #issue_or_pr
• I have added all required tests (Unit tests, E2E tests)
• My changes do not require any extra work to be leveraged by OutOfProc SDKs
  • Otherwise: That work is being tracked here: #issue_or_pr_in_each_sdk
• My changes do not change the version of the WebJobs.Extensions.DurableTask package
  • Otherwise: major or minor version updates are reflected in /src/Worker.Extensions.DurableTask/AssemblyInfo.cs

[davidmrdavid]

Thanks for doing this work! I added a request for a manual test, please see my comment below

[bachuv]

As an update, I removed the functionState property to make it simpler and updated the query in the description based on that change. I'm now looking into removing the additional "orchestrator state changed" log.

[bachuv]

I looked into removing the additional state trace log and updating the existing Activity.Tags with the correct runtime status to try and not change the query used in the portal that's used populate the orchestration state. I realized that doesn't work for client APIs like Terminate and Suspend though. For those APIs, I tried updating the Tags in the relevant methods in DurableClient but was seeing different values for Activity.Current which meant the runtime status value wouldn't get updated correctly. Let me know if there's another way of approaching this that I haven't considered.

I'm looking into updating the existing state change trace with a new State field now.

[bachuv]

I was able to remove the additional state trace log by adding a RuntimeStatus field to the existing state trace log instead. I also updated the runtime status query.

I haven't updated all of the E2E tests since it's a tedious process, but I will update them once I get some reviews on this new approach of adding a runtime status field.

traces
| where customDimensions.Category == "Host.Triggers.DurableTask"
| extend functionName = tostring(customDimensions["prop__functionName"])
| extend instanceId = tostring(customDimensions["prop__instanceId"])
| extend state = tostring(customDimensions["prop__state"])
| extend isReplay = tobool(tolower(customDimensions["prop__isReplay"]))
| extend hubName = tostring(tolower(customDimensions["prop__hubName"]))
| extend runtimeStatus = tostring((customDimensions["prop__runtimeStatus"]))
| where isReplay != true
| where state != "Awaited"
| where runtimeStatus != ""
| summarize arg_max(timestamp, *) by instanceId
| order by timestamp asc
| project timestamp, instanceId, runtimeStatus

[davidmrdavid]

Left 2 small questions

[davidmrdavid]

With the exception of the ResumingOrchestration log, the code changes look good to me! It is certainly confusing that we log both a State and RuntimeStatus field. We'll need to tackle that eventually. Other than that, the overall change strategy makes sense to me. I think you should be good to update the test code

[bachuv]

@davidmrdavid, I know the Netherite smoke test is expected to fail right now, but is the Python smoke test expected to fail?

[davidmrdavid]

@bachuv: it isn't, but the error suggests the problem was a failure in downloading the docker image. I'm re-running the test now to see if it's transient.

[davidmrdavid]

LGTM so long as all smoke tests (except the Netherite one) pass

[cgillum]

LGTM!