ci-analysis: structured output, MCP integration, and deep investigation guides

[lewing]

ci-analysis: structured output, MCP integration, and deep investigation guides

Changes to Get-CIStatus.ps1 (+216/-132)

• Add [CI_ANALYSIS_SUMMARY] JSON block — structured summary emitted at end of script with all key facts (builds, failed jobs, known issues, PR correlation, recommendation hint)
• Replace 47-line if/elseif recommendation chain with a single recommendationHint field in JSON (one of: BUILD_SUCCESSFUL, KNOWN_ISSUES_DETECTED, LIKELY_PR_RELATED, POSSIBLY_TRANSIENT, REVIEW_REQUIRED, MERGE_CONFLICTS, NO_BUILDS)
• Add failedJobDetails to JSON — per-job errorCategory (test-failure, build-error, test-timeout, crash, tests-passed-reporter-failed, unclassified), errorSnippet, and helixWorkItems
• Add failedJobDetailsTruncated — boolean flag indicating when -MaxJobs cap means failedJobDetails is incomplete vs failedJobNames
• Add top-level knownIssues from Build Analysis (not per-job — Build Analysis reports at the PR level, not per-job)
• Add timeout pattern to Format-TestFailure — catches Timed Out (timeout that was previously invisible
• Show log tail in PR mode when no failure pattern matches (Helix Job mode already did this)
• Add accumulation variables for cross-build aggregation (totalFailedJobs, totalLocalFailures, lastBuildJobSummary)
• Fix early-continue scoping bug — job summary computation was at end of build loop, after 3 continue paths that skipped it
• Fix empty array falsy check — if ($listFiles) → proper count check
• Fix mergeable_state trimming — gh api --jq output trimmed to prevent whitespace comparison failures
• Remove interpretive prose — "These failures are likely PR-related" moved from script to agent reasoning
• Fix empty catch — merge state error now logged via Write-Verbose

Changes to SKILL.md (+97/-144, net reduction)

• Add Step 0: Gather Context — PR type classification table (code, flow, backport, merge, dependency update)
• Add Step 3: Verify before claiming — systematic checklist
• Add build progression analysis (Step 2, item 4) — comparing pass/fail across PR builds to narrow down which commit introduced a failure
• Add prior-build mismatch detection (Step 2, item 6) — ask user when they reference jobs not in current results
• Document failedJobDetails — per-failure error categories in Interpreting Results
• Add Build Analysis check status enforcement — red check means unaccounted failures exist, never claim "all known" when it's red
• Add timeout recovery workflow — explicit guidance for verifying timed-out builds have passing Helix results via hlx_status
• Add crash/canceled job recovery procedure — step-by-step using hlx_batch_status, hlx_files, hlx_download_url to recover results from crashed Helix work items
• Fix MCP tool references — use canonical short-form tool names consistently
• Condense anti-patterns — tighter, more targeted, near relevant steps
• Net token reduction — despite adding new content, SKILL.md shrank from ~4.6K to ~3.5K tokens

New reference files

• references/azure-cli.md — Azure CLI deep investigation guide
• references/binlog-comparison.md — binlog comparison workflow
• references/delegation-patterns.md — subagent delegation patterns (5 patterns including parallel artifact extraction and canceled job recovery)
• references/build-progression-analysis.md — commit-to-build correlation using triggerInfo.pr.sourceSha, SQL-based progression tracking, MCP-first with AzDO MCP tools as primary

Updated reference files

• references/manual-investigation.md — fix nonexistent msbuild-mcp analyze tool refs, use real mcp-binlog-tool-* tools

Design principles

• Data/reasoning boundary: Script emits structured JSON facts → agent synthesizes recommendations. No more canned prose from the script.
• MCP-first: AzDO MCP tools (get_builds, get_build_log_by_id) and Helix MCP tools (hlx_status, hlx_logs) positioned as primary, CLI/script as fallback.
• Token budget: Orchestrating SKILL.md kept within 2K-4K token budget by extracting depth to references/.
• SQL for structured investigations: Build progression tracking uses SQL tables to persist SHAs across context, enabling queries for pass→fail transitions and target branch movement.

Testing

• Multi-model subagent testing (Sonnet 4 + GPT-5 + Opus 4.5) — two review rounds with findings addressed
• Live MCP integration test confirmed hlx_status, hlx_logs, get_builds, get_build_log_by_id all work
• Real-world validation against PRs [release/8.0] Update dependencies from dotnet/arcade #123245, [release/10.0] Rework and enable Wasm.Build.Tests.Blazor.AssetCachingTests #123883, Fix WASM boot config ContentRoot to use IntermediateOutputPath #124125, [main] Source code updates from dotnet/dotnet #124232

[Copilot]

Pull request overview

This PR updates the ci-analysis Copilot skill to follow the “data vs. reasoning boundary” pattern by having the script emit a structured JSON summary block ([CI_ANALYSIS_SUMMARY]) and moving recommendation synthesis into the agent guidance in SKILL.md.

Changes:

• Added Helix ListFiles-based work item file retrieval to avoid broken artifact URIs from the Details endpoint.
• Replaced the script’s canned recommendation if/elseif chain with a structured [CI_ANALYSIS_SUMMARY] JSON block (including correlation and hint fields).
• Updated SKILL.md to instruct agents to generate recommendations from the JSON summary (decision table + nuance guidance).

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 5 comments.

File	Description
.github/skills/ci-analysis/scripts/Get-CIStatus.ps1	Adds Helix ListFiles workaround and emits [CI_ANALYSIS_SUMMARY] JSON instead of canned recommendation prose.
.github/skills/ci-analysis/SKILL.md	Documents the new workflow: parse JSON summary + agent-generated recommendations, with decision logic guidance.

[Copilot]

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

[Copilot]

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated 7 comments.

[Copilot]

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated 1 comment.

[Copilot]

Pull request overview

Copilot reviewed 8 out of 8 changed files in this pull request and generated 1 comment.

Comments suppressed due to low confidence (1)

.github/skills/ci-analysis/scripts/Get-CIStatus.ps1:2140

• In the non-Helix (build failure) path, jobDetail.errorCategory is only set to build-error when Extract-BuildErrors finds matches. If extraction returns 0 errors (pattern miss, truncated logs, etc.), the JSON will report unclassified with an empty snippet even though the job is clearly a build failure. Consider setting errorCategory to build-error as soon as you enter the non-Helix branch, and populating errorSnippet with something deterministic (e.g., failed task name and/or a short tail of the task log) when no errors are extracted.

                    # No Helix tasks - this is a build failure, extract actual errors
                    $buildTasks = $timeline.records | Where-Object {
                        $_.parentId -eq $job.id -and $_.result -eq "failed"
                    }

                    foreach ($task in $buildTasks | Select-Object -First 3) {
                        Write-Host "  Failed task: $($task.name)" -ForegroundColor Red

                        # Fetch and parse the build log for actual errors
                        if ($task.log) {
                            $logUrl = "https://dev.azure.com/$Organization/$Project/_build/results?buildId=$currentBuildId&view=logs&j=$($job.id)&t=$($task.id)"
                            Write-Host "  Log: $logUrl" -ForegroundColor Gray
                            $logContent = Get-BuildLog -Build $currentBuildId -LogId $task.log.id

                            if ($logContent) {
                                $buildErrors = Extract-BuildErrors -LogContent $logContent

                                if ($buildErrors.Count -gt 0) {
                                    # Collect for PR correlation
                                    $allFailuresForCorrelation += @{
                                        TaskName = $task.name
                                        JobName = $job.name
                                        Errors = $buildErrors
                                        HelixLogs = @()
                                        FailedTests = @()
                                    }
                                    $jobDetail.errorCategory = "build-error"
                                    if (-not $jobDetail.errorSnippet) {
                                        $snippet = ($buildErrors | Select-Object -First 2) -join "; "
                                        $jobDetail.errorSnippet = $snippet.Substring(0, [Math]::Min(200, $snippet.Length))
                                    }

                                    # Extract Helix log URLs from the full log content
                                    $helixLogUrls = Extract-HelixLogUrls -LogContent $logContent

                                    if ($helixLogUrls.Count -gt 0) {
                                        Write-Host "  Helix failures ($($helixLogUrls.Count)):" -ForegroundColor Red
                                        foreach ($helixLog in $helixLogUrls | Select-Object -First 5) {
                                            Write-Host "    - $($helixLog.WorkItem)" -ForegroundColor White
                                            Write-Host "      Log: $($helixLog.Url)" -ForegroundColor Gray
                                        }
                                        if ($helixLogUrls.Count -gt 5) {
                                            Write-Host "    ... and $($helixLogUrls.Count - 5) more" -ForegroundColor Gray
                                        }
                                    }
                                    else {
                                        Write-Host "  Build errors:" -ForegroundColor Red
                                        foreach ($err in $buildErrors | Select-Object -First 5) {
                                            Write-Host "    $err" -ForegroundColor White
                                        }
                                        if ($buildErrors.Count -gt 5) {
                                            Write-Host "    ... and $($buildErrors.Count - 5) more errors" -ForegroundColor Gray
                                        }
                                    }

                                    # Search for known issues
                                    Show-KnownIssues -ErrorMessage ($buildErrors -join "`n") -IncludeMihuBot:$SearchMihuBot
                                }
                                else {
                                    Write-Host "  (No specific errors extracted from log)" -ForegroundColor Gray
                                }
                            }

[Copilot]

Pull request overview

Copilot reviewed 8 out of 8 changed files in this pull request and generated 4 comments.

[Copilot]

Pull request overview

Copilot reviewed 9 out of 9 changed files in this pull request and generated 1 comment.

Comments suppressed due to low confidence (1)

.github/skills/ci-analysis/scripts/Get-CIStatus.ps1:2150

• $allFailedJobDetails += $jobDetail is executed only on the success path. If an exception is thrown while processing a failed job and -ContinueOnError is set, that job will be missing from failedJobDetails even though it’s still in failedJobNames, and the missing entry will also affect failedJobDetailsTruncated. Consider appending a best-effort jobDetail in a finally block (and populating errorSnippet from the exception) so the JSON remains structurally complete.

            $allFailedJobDetails += $jobDetail
            $processedJobs++
        }
        catch {
            $errorCount++

[Copilot]

Pull request overview

Copilot reviewed 9 out of 9 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (1)

.github/skills/ci-analysis/scripts/Get-CIStatus.ps1:2011

• Extract-HelixUrls is called even when $logContent is $null (it’s outside the if ($logContent) block). Extract-HelixUrls does regex matching on the content and can throw on null input, which will abort processing this job/build. Move the Helix URL extraction inside the $logContent guard (or make Extract-HelixUrls return an empty array when $LogContent is null/empty).

                            # Extract and optionally fetch Helix URLs
                            $helixUrls = Extract-HelixUrls -LogContent $logContent

[davidfowl]

This looks like a banger.