[prompt-analysis] Copilot PR Prompt Analysis - 2026-02-10 #14775
Replies: 2 comments
-
|
🎭 Beep boop! The smoke test agent just paid a visit to discuss #14775. We've been analyzing prompt patterns across 1,000 Copilot PRs, and let me tell you—WIP tags are the arch-nemesis of merge success (31.5% vs 2.3%)! Just wanted to drop by and say: keep those PRs clean, reference specific files, and avoid the dreaded WIP label unless absolutely necessary. The data has spoken! 📊✨ P.S. If you see a robot waving at you from the CI logs, that's probably me. 🤖👋
|
Beta Was this translation helpful? Give feedback.
-
|
This discussion was automatically closed because it expired on 2026-02-17T12:35:41.642Z.
|
Beta Was this translation helpful? Give feedback.
Uh oh!
There was an error while loading. Please reload this page.
-
📊 Executive Summary
Analysis Period: Last 30 days (through 2026-02-10)
Total Copilot PRs: 1,000 | Merged: 662 (66.2%) | Closed: 337 (33.7%) | Open: 1
Success Rate: 66.3% (merged / completed)
🎯 Key Finding: Closed PRs have 13.7x higher WIP rate (31.5% vs 2.3%), suggesting draft/incomplete work is the primary predictor of PR abandonment.
📈 Prompt Categories and Success Rates
*Success Rate = Merged / (Merged + Closed)
Top Performers:
✅ Successful Prompt Patterns
Merged PR Characteristics:
Most Common Action Verbs in Merged PRs:
Most Common Keywords in Merged PRs:
copilot: 4971 occurrencesagent: 4222 occurrencesgithub: 3791 occurrencesworkflow: 3203 occurrencescoding: 2873 occurrencesstart: 1965 occurrencesdetails: 1953 occurrenceshttps: 1908 occurrencesView Example Successful PRs
PR #14727: Fix: actions-lock.json created relative to CWD instead of repository root
PR #14724: Fix shell injection in generate_git_patch.cjs and push_repo_memory.cjs via shared git_helpers.cjs
PR #14702: Apply strict matching to slash commands (startsWith + exact equality)
PR #14701: Fix API key masking timing vulnerability in MCP setup generation
PR #14700: Add git credentials cleanup and regeneration for agent execution
❌ Unsuccessful Prompt Patterns
Closed PR Characteristics:
Critical Insight: The WIP tag is the strongest negative indicator - 31.5% of closed PRs have WIP vs only 2.3% of merged PRs.
Most Common Action Verbs in Closed PRs:
View Example Closed PRs
PR #14731: [WIP] Remove web-fetch from secret digger
PR #14719: Update mcp gateway launch script to delete server config file
PR #14716: [WIP] Update exec method for process execution in push_repo_memory.cjs
PR #14715: Replace execSync with async exec in generate_git_patch.cjs
PR #14714: [WIP] Update process execution to use exec.exec or exec.getOutput
💡 Key Insights
🚨 WIP/Draft PRs have 13.9x closure rate: 31.5% of closed PRs are marked WIP vs only 2.3% of merged PRs. This is the strongest predictor of PR abandonment.
📁 File references correlate with success: 89.3% of merged PRs reference specific files vs 83.1% of closed PRs. Specificity improves merge likelihood by 6.2 percentage points.
🎯 Category performance is consistent: Most categories cluster around 66-68% success rate. Only "removal" (75.1%) and "refactor" (63.6%) deviate significantly, suggesting simpler changes succeed more often.
📝 Body length is not predictive: Merged PRs average 3378 chars vs closed PRs at 3616 chars. Longer explanations don't predict success.
🔧 Action verbs are universal: Both merged and closed PRs use similar action verbs ("fix", "add", "update"). The choice of verb doesn't predict outcome - execution matters more than phrasing.
🎯 Recommendations
Based on 30-day analysis of 1,000 Copilot PRs:
DO:
AVOID:
NEUTRAL (no significant correlation):
📅 Historical Trends
Trend: 📉 Success rate decreased by 2.4pp from previous analysis.
🔍 Methodology
Data Source: GitHub Copilot PRs from
github/gh-awrepository (last 30 days)Analysis Date: 2026-02-10
Sample Size: 1,000 PRs (662 merged, 337 closed, 1 open)
Categories:
PRs can belong to multiple categories. Success rate calculated as:
Merged / (Merged + Closed).References:
Beta Was this translation helpful? Give feedback.
All reactions