Initial version of a build duty report generator instructions for copilot#52678
Initial version of a build duty report generator instructions for copilot#52678marcpopMSFT wants to merge 1 commit intomainfrom
Conversation
…ilot. See below for an example report.
SDK Build Duty Triage ReportGenerated: January 26, 2026 Summary
🟢 Ready to Merge PRs (1)These PRs have all checks passing and are ready for merge. dotnet/dotnet VMR
🟠 New/Pending Validation PRs (2)These PRs were just created and are awaiting initial CI validation. dotnet/sdk
dotnet/templating
⏳ Waiting/On Hold PRs (1)These PRs have passing checks but have comments indicating they should wait before merging. dotnet/templating
🟡 Branch Lockdown PRs (10)These PRs are in branches with lockdown labels and require approval to merge. dotnet/sdk (8)
dotnet/templating (2)
🔴 Failing/Blocked PRs (13)PRs with pending or failing status checks. dotnet/sdk (11)
github-actions[bot] Merge PRs (2)Note: These are cross-branch merge PRs that require manual attention.
Notes
Total PR Count by Repo
Key Issues Blocking Multiple PRs
Report generated by SDK Build Duty skill |
|
@DonnaChen888 this might make your report generations easier. This is just a first try experiment though. |
There was a problem hiding this comment.
Pull request overview
This PR introduces two new documentation files to help automate and guide build duty PR triage for the .NET SDK team using GitHub Copilot. The files provide instructions for monitoring, classifying, and reporting on pull requests from automated bots and specific team members across multiple .NET repositories.
Changes:
- Added a skill definition file that describes the build duty PR triage capability for Copilot
- Added a prompt file with detailed instructions for executing build duty triage and generating reports
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 8 comments.
| File | Description |
|---|---|
| .github/copilot/skills/build-duty-pr-triage.md | Defines the build duty PR triage skill with monitored repositories, authors, PR classification categories, and example output format |
| .github/copilot/prompts/build-duty.prompt.md | Provides detailed step-by-step instructions for Copilot to perform PR triage, including query syntax, classification logic, and report generation format |
| ### ❌ Category 4: Failing / Blocked | ||
| PRs with ANY of: | ||
| - `mergeable_state` = `"blocked"` (even if pending - this means required checks failing) | ||
| - `get_status` returns `state: "failure"` or `state: "pending"` |
There was a problem hiding this comment.
Inconsistent classification logic for failing PRs. Line 78 states that get_status with "pending" state should be classified as "Failing / Blocked", but "pending" typically means checks are still running, not that they have failed. This could lead to premature classification of PRs as failing when they're simply waiting for CI to complete. Consider separating "pending" (waiting) from "failure" (actually failed) or clarifying that "pending" only belongs in this category if it's been pending for an extended period.
| - `get_status` returns `state: "failure"` or `state: "pending"` | |
| - `get_status` returns `state: "failure"` (if `state` is `"pending"`, treat as checks still running and **do not** put in this category unless the PR has been pending unusually long and appears stuck) |
|
|
||
| ``` | ||
| # All maestro PRs in SDK | ||
| is:open is:pr repo:dotnet/sdk author:app/dotnet-maestro |
There was a problem hiding this comment.
Inconsistency in query examples between files. In the skill file, line 156 shows "is:open is:pr repo:dotnet/sdk author:app/dotnet-maestro" but in the prompt file at line 21, it references the same bot as "dotnet-maestro[bot]". While both may work, having consistent naming across the instruction files would improve clarity. The query format "author:app/dotnet-maestro" is correct for GitHub searches, but it should be consistently documented.
| ### Special: dotnet/dotnet VMR Filtering | ||
|
|
||
| For the dotnet/dotnet repo, only include PRs that: | ||
| 1. Are authored by `dotnet-maestro[bot]}}` |
There was a problem hiding this comment.
There is a syntax error in the bot mention. The line contains dotnet-maestro[bot]}} with extra closing braces. This should be dotnet-maestro[bot] to match the format used elsewhere in the document.
| 1. Are authored by `dotnet-maestro[bot]}}` | |
| 1. Are authored by `dotnet-maestro[bot]` |
|
|
||
| ### 1. ✅ Ready to Merge | ||
| PRs that meet all of the following criteria: | ||
| - mergeable_state is "clean" or not "blocked" |
There was a problem hiding this comment.
The logic for "Ready to Merge" classification is contradictory. Line 49 states the criteria includes 'mergeable_state is "clean" or not "blocked"', which would include states like "unstable" or "pending". However, this conflicts with the more precise definition in the prompt file (lines 52-55) and the Category 4 definition (lines 64-68) which treats "unstable" and "blocked" states as failing. The criterion should be simplified to only accept mergeable_state as "clean" for consistency.
| - mergeable_state is "clean" or not "blocked" | |
| - mergeable_state is exactly "clean" |
|
|
||
| 1. **Fetch Open PRs**: Query for all open pull requests in the monitored repositories | ||
| 2. **Check CI Status**: For each PR, determine the status of all required checks | ||
| 3. **Classify PRs**: Sort PRs into the three categories above |
There was a problem hiding this comment.
The Workflow Instructions mention "three categories" but the document defines four categories: Ready to Merge, Branch Lockdown, Changes Requested, and Failing/Blocked. The text should be updated to accurately reflect the number of categories.
| 3. **Classify PRs**: Sort PRs into the three categories above | |
| 3. **Classify PRs**: Sort PRs into the four categories above |
| applyTo: chat | ||
| description: "Triage pull requests across .NET SDK repositories for build duty. Classifies PRs into: Ready to Merge, Passing but Branch Lockdown, and Failing Checks." | ||
| --- |
There was a problem hiding this comment.
The frontmatter at the top of the file appears to be incomplete or malformed. Line 1 contains "applyTo: chat" but line 3 has "---" which typically closes YAML frontmatter, leaving the description on line 2 in an ambiguous position. Standard YAML frontmatter should open with "---" on the first line, contain the metadata, and close with "---". Either add an opening "---" before line 1, or clarify the intended format for this file type.
| 1. **dotnet-maestro[bot]** - Automated dependency updates and codeflow PRs | ||
| - Query: `author:app/dotnet-maestro` | ||
|
|
||
| 2. **github-actions[bot]** - Only inter-branch merge PRs |
There was a problem hiding this comment.
Inconsistent terminology for the same tool. The document uses "github-actions[bot]" (lowercase with hyphens) in some places but refers to the same entity as "app/github-actions" in queries. While "app/github-actions" is correct for GitHub search queries, the bot should be consistently referred to using its actual username format. Consider standardizing the reference format or clarifying which context requires which format.
| 2. **github-actions[bot]** - Only inter-branch merge PRs | |
| 2. **github-actions[bot]** - Only inter-branch merge PRs (GitHub search represents this bot as `app/github-actions`) |
| - `mergeable_state` is `"clean"` (this is the primary check) | ||
| - `get_status` returns `state: "success"` | ||
| - Not a draft PR |
There was a problem hiding this comment.
The instruction states to check if 'mergeable_state is "clean"' as the primary check, but then immediately adds an OR condition with get_status. However, the logic should be AND, not OR. A PR should have both mergeable_state as "clean" AND get_status as "success" to be truly ready to merge. The current wording with separate bullet points makes this unclear. Consider clarifying that ALL criteria must be met simultaneously.
| @@ -0,0 +1,193 @@ | |||
| # Build Duty PR Triage Skill | |||
There was a problem hiding this comment.
issue: Skill does not conform to the Agent Skill specification.
| ## Usage | ||
|
|
||
| To use this skill, ask Copilot to: | ||
|
|
||
| ``` | ||
| @workspace Check the build duty PR status for today | ||
| ``` | ||
|
|
||
| or | ||
|
|
||
| ``` | ||
| @workspace Triage open PRs in dotnet/sdk for build duty | ||
| ``` |
There was a problem hiding this comment.
issue: Skill definition should not include instructions on how to use the skill. Instead, examples of when the skill should be invoked should be in the YAML frontmatter/description.
| 1. **Fetch Open PRs**: Query for all open pull requests in the monitored repositories | ||
| 2. **Check CI Status**: For each PR, determine the status of all required checks | ||
| 3. **Classify PRs**: Sort PRs into the three categories above | ||
| 4. **Report Summary**: Provide a formatted summary with: | ||
| - PR number and title | ||
| - Author | ||
| - Target branch | ||
| - Status category | ||
| - Link to the PR | ||
| - Age of the PR (days open) | ||
| - Any blocking issues or notes |
There was a problem hiding this comment.
suggestion: Most of this seems well defined and could be extracted into a file-based app that is included in the skill. It's faster and more efficient to have something like "For each PR, determine the status of all required checks" encoded in a script than rely on the LLM to do that.
There was a problem hiding this comment.
Yeah, I had originally envisioned a bunch of gh CLI calls to pull this info but copilot advocated for using the GH MCP instead (it seemed to think it could be more parallelized). Does it only do CLI calls in the terminal so would it end up being serial calls? I can have it try the other way and see what it does as the results were definitely a bit unpredictable and having specific script calls would be better.
There was a problem hiding this comment.
A file-based app can be parallelized however you want. Then the SKILL.md file can include instructions on how to use it.
2 participants
See below for an example report.
Drawback of this is that I had to keep reminding it not to include extra PRs like backports or non-SDK PRs into the VMR as even though that's in the skill, it would forget that. It also had some trouble always remembering to include failure details. Still an interesting first try.