[prompt-analysis] Copilot PR Prompt Analysis - 2026-01-26 #11858
Closed
Replies: 1 comment
-
|
This discussion was automatically closed because it expired on 2026-02-02T12:03:20.434Z. |
Beta Was this translation helpful? Give feedback.
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
-
Summary
Analysis Period: Last 30 days
Total PRs: 1,000 | Merged: 653 (65.3%) | Closed (not merged): 344 (34.4%) | Open: 3
Overall Success Rate: 65.5% (of completed PRs)
Prompt Categories and Success Rates
*Success Rate = Merged / (Merged + Closed)
Note: PRs can belong to multiple categories, so totals may exceed the number of PRs analyzed.
Prompt Analysis
✅ Successful Prompt Patterns
Common characteristics in merged PRs:
View Example Successful Prompts
Bug Fix (64.8% success rate):
PR Fix firewall-escape workflow post-issue job syntax error #11822 (95 words): Fix firewall-escape workflow post-issue job syntax error
PR Fix context7 MCP server configuration: correct tool names and description #11821 (17 words): Fix context7 MCP server configuration: correct tool names and description
Feature Addition (67.2% success rate):
PR Fix cache-memory configuration in chroma.md shared import #11820 (696 words): Fix cache-memory configuration in chroma.md shared import
PR Add github-token input to setup-cli action #11810 (166 words): Add github-token input to setup-cli action
Refactoring (70.3% success rate - HIGHEST):
PR Improve and restructure campaigns documentation with separate Creating Campaigns page #11831 (46 words): Improve and restructure campaigns documentation with separate Creating Campaigns
PR Fix cache-memory configuration in chroma.md shared import #11820 (696 words): Fix cache-memory configuration in chroma.md shared import
❌ Unsuccessful Prompt Patterns
Common characteristics in closed PRs:
View Example Unsuccessful Prompts
Bug Fix (64.8% success rate):
PR Fix Scout workflow failure by removing Context7 MCP dependency #11814 (85 words): Fix Scout workflow failure by removing Context7 MCP dependency → CLOSED
PR [WIP] Fix Dependabot bundler issues requiring token #11783 (8 words): [WIP] Fix Dependabot bundler issues requiring token → CLOSED
Feature Addition (67.2% success rate):
PR Add CLI command for campaign project creation with views and custom fields #11745 (18 words): Add CLI command for campaign project creation with views and custom fields → CLOSED
PR Fix MCP Inspector timeout by removing unnecessary server imports #11728 (578 words): Fix MCP Inspector timeout by removing unnecessary server imports → CLOSED
Key Insights
🎯 Conciseness Matters: Merged PRs have prompts that are ~25% shorter (147 words vs 183 words). Clear, focused prompts perform better than verbose explanations.
📈 Refactoring Has Highest Success: Refactoring PRs have a 70.3% success rate, suggesting that well-defined improvement tasks are easier for Copilot to execute successfully.
🔑 Keyword Indicators: Prompts containing "testify", "registry", "ansi", "bool", "errorf" have 100% merge rates in our dataset, while prompts heavy on "failure", "failed", "security" have lower success rates.
📊 Consistent Performance: The success rate has been stable at 64-67% over the past week, indicating predictable performance patterns.
🚀 Testing Prompts Perform Well: Testing-related PRs have 65.7% success rate, close to the overall average, showing Copilot handles test generation effectively.
Recommendations
Based on today's analysis:
Historical Trends
Trend Observation: Success rates remain stable in the 64-67% range, with minor fluctuations. Today's rate of 65.5% is above the recent average.
Methodology
This analysis examined 1,000 Copilot-generated PRs from the last 30 days:
Data Sources: GitHub Pull Request API, Copilot PR metadata
Analysis Run: §21356801815
Beta Was this translation helpful? Give feedback.
All reactions