Staging - [Alerting] Servicing Builds Running in R&D Pools Alert

[dotnet-eng-status-staging]

💔 Metric state changed to alerting

Please run the following query to identify the delinquent pipelines:

TimelineBuilds
| where QueueTime > ago(1d)
| where SourceBranch has "release"
| join kind=inner (TimelineRecords
                    | where Order != 0
                    | where strlen(Path) == 11
                    | where WorkerName has "NetCore1ESPool"
                    | where not(WorkerName has "-Svc")
                    | summarize arg_min(WorkerName, *) by RecordId, BuildId ) on BuildId

• count_ 2837

Go to rule

@dotnet/dnceng, please investigate

Automation information below, do not change

Grafana-Automated-Alert-Id-a5641c4a6d8e4e499f1710aa8386d81b

[MattGal]

I'm working on this via #10578.

[dotnet-eng-status-staging]

💚 Metric state changed to ok

Please run the following query to identify the delinquent pipelines:

TimelineBuilds
| where QueueTime > ago(1d)
| where SourceBranch has "release"
| join kind=inner (TimelineRecords
                    | where Order != 0
                    | where strlen(Path) == 11
                    | where WorkerName has "NetCore1ESPool"
                    | where not(WorkerName has "-Svc")
                    | summarize arg_min(WorkerName, *) by RecordId, BuildId ) on BuildId

Go to rule

[dotnet-eng-status-staging]

💔 Metric state changed to alerting

Please run the following query to identify the delinquent pipelines:

TimelineBuilds
| where QueueTime > ago(1d)
| where SourceBranch has "release"
| join kind=inner (TimelineRecords
                    | where Order != 0
                    | where strlen(Path) == 11
                    | where WorkerName has "NetCore1ESPool"
                    | where not(WorkerName has "-Svc")
                    | summarize arg_min(WorkerName, *) by RecordId, BuildId ) on BuildId

• count_ 7

Go to rule

[dotnet-eng-status-staging]

💚 Metric state changed to ok

Please run the following query to identify the delinquent pipelines:

TimelineBuilds
| where QueueTime > ago(1d)
| where SourceBranch has "release"
| join kind=inner (TimelineRecords
                    | where Order != 0
                    | where strlen(Path) == 11
                    | where WorkerName has "NetCore1ESPool"
                    | where not(WorkerName has "-Svc")
                    | summarize arg_min(WorkerName, *) by RecordId, BuildId ) on BuildId

Go to rule

[MattGal]

Wow it's clear, as I still have #10578, closing this. (will likely reactivate)