[prompt-clustering] Copilot Agent Prompt Clustering Analysis - 2026-02-19

[github-actions[bot]]

Daily NLP-based clustering analysis of copilot agent task prompts from the last 30 days.

Summary

• Analysis Date: 2026-02-19
• Analysis Period: Last 30 days
• Total Tasks Analyzed: 1,000
• Merged/Successful: 690 (69.0%)
• Clusters Identified: 8
• Silhouette Score: 0.2287 (moderate separation — expected for diverse natural language prompts)

Cluster Overview

#	Theme	Tasks	Success Rate	Top Keywords
1	General Task Instructions & Setup	209	73.7%	set, works faster, things coding, work set
2	Feedback & Survey Tasks	205	72.2%	share thoughts, survey, thoughts, share
3	Custom Instructions & Configuration	195	76.4%	custom, make, custom instructions, setting custom
4	Repository & Issue Management	194	46.4%	gh, gh aw, aw, issue
5	Package/Dependency & MCP Updates	67	74.6%	changeset, patch, mcp, v0
6	Testing & Validation	61	77.0%	test, error, lines, files
7	Workflow Debugging & Failures	42	64.3%	failure, workflow, failed, debug
8	Reporting & Documentation	27	92.6%	workflow, formatting, report, issue

Detailed Cluster Analysis

Cluster 1: General Task Instructions & Setup

• Size: 209 tasks (20.9% of total)
• Success Rate: 73.7%
• Top Keywords: set, works faster, things coding, work set, set repo, set things, quality work, let copilot
• Most Common Category: Bug Fixes
• Avg Files Changed: 17.2
• Example Merged PRs:
  • Merge maintenance jobs and add comprehensive logging #11060: Merge maintenance jobs and add comprehensive logging
  • Fix markdown code region balancer treating indented examples as nested fences #11082: Fix markdown code region balancer treating indented examples as nested fences
  • Fix markdown code region balancer: distinguish consecutive blocks from nesting #11091: Fix markdown code region balancer: distinguish consecutive blocks from nesting
• Example Unmerged PRs:
  • [WIP] Fix network configuration for MCP server time #11065: [WIP] Fix network configuration for MCP server time
  • Unable to debug Actions error - logs inaccessible #11114: Unable to debug Actions error - logs inaccessible

Cluster 2: Feedback & Survey Tasks

• Size: 205 tasks (20.5% of total)
• Success Rate: 72.2%
• Top Keywords: share thoughts, survey, thoughts, share, thoughts copilot, love, love input, minute survey
• Most Common Category: Bug Fixes
• Avg Files Changed: 21.2
• Example Merged PRs:
  • chore: Update Sentry MCP server to 0.27.0 #11050: chore: Update Sentry MCP server to 0.27.0
  • Update parent issue template for agentic-workflow failures #11053: Update parent issue template for agentic-workflow failures
  • chore: campaign discovery via label-based approach #11070: chore: campaign discovery via label-based approach
• Example Unmerged PRs:
  • Auto-assign @copilot to workflow sync issues when agent token available #11054: Auto-assign @copilot to workflow sync issues when agent token available
  • [WIP] Update monster agentic workflow to ignore assigned issues #11090: [WIP] Update monster agentic workflow to ignore assigned issues

Cluster 3: Custom Instructions & Configuration

• Size: 195 tasks (19.5% of total)
• Success Rate: 76.4%
• Top Keywords: custom, make, custom instructions, setting custom, tips docs, servers learn, smarter setting, protocol mcp
• Most Common Category: Bug Fixes
• Avg Files Changed: 19.2
• Example Merged PRs:
  • Add interactive engine selection and secret configuration to init command #11064: Add interactive engine selection and secret configuration to init command
  • Improve error messages for invalid target configuration in safe outputs #11066: Improve error messages for invalid target configuration in safe outputs
  • Disallow invalid permissions: read shorthand, require read-all #11077: Disallow invalid permissions: read shorthand, require read-all
• Example Unmerged PRs:
  • Fix create_pull_request in safe-outputs container by installing git #11116: Fix create_pull_request in safe-outputs container by installing git
  • [WIP] Update safe-outputs mcp configuration for node:lts-slime #11117: [WIP] Update safe-outputs mcp configuration for node:lts-slime

Cluster 4: Repository & Issue Management

• Size: 194 tasks (19.4% of total)
• Success Rate: 46.4% ⚠️
• Top Keywords: gh, gh aw, aw, issue, githubnext gh, githubnext, section, workflow
• Most Common Category: CI/CD & Workflows
• Avg Files Changed: 11.4
• Example Merged PRs:
  • Add missing get_repository tool to repos toolset #11067: Add missing get_repository tool to repos toolset
  • Enable safe-input tool tracking for Daily Performance Summary workflow #11074: Enable safe-input tool tracking for Daily Performance Summary workflow
  • Restrict Issue Monster to cookie-labeled issues from automated workflows #11098: Restrict Issue Monster to cookie-labeled issues from automated workflows
• Example Unmerged PRs:
  • Install Go toolchain in daily-cli-performance workflow #11059: Install Go toolchain in daily-cli-performance workflow
  • [WIP] Fix simple workflows and maintenance YAML issues #11084: [WIP] Fix simple workflows and maintenance YAML issues

Cluster 5: Package/Dependency & MCP Updates

• Size: 67 tasks (6.7% of total)
• Success Rate: 74.6%
• Top Keywords: changeset, patch, mcp, v0, type, cli, server, version
• Most Common Category: Dependency Updates
• Avg Files Changed: 66.3
• Example Merged PRs:
  • Convert safe-outputs MCP server to HTTP transport #11120: Convert safe-outputs MCP server to HTTP transport
  • Update CLI Tools: Claude Code 2.1.15, Copilot CLI 0.0.389, Codex 0.88.0 #11160: Update CLI Tools: Claude Code 2.1.15, Copilot CLI 0.0.389, Codex 0.88.0
  • Move safe-output file from /tmp to /opt for read-only agent access #11169: Move safe-output file from /tmp to /opt for read-only agent access
• Example Unmerged PRs:
  • Verify CLI version updates: Copilot 0.0.388, Sandbox Runtime 0.0.29, MCP Gateway v0.0.74 #11085: Verify CLI version updates: Copilot 0.0.388, Sandbox Runtime 0.0.29, MCP Gateway v0.0.74
  • [WIP] Update MCP Gateway to v0.0.76 #11216: [WIP] Update MCP Gateway to v0.0.76

Cluster 6: Testing & Validation

• Size: 61 tasks (6.1% of total)
• Success Rate: 77.0%
• Top Keywords: test, error, lines, files, validation, functions, code, tests
• Most Common Category: Bug Fixes
• Avg Files Changed: 10.8
• Example Merged PRs:
  • Fix ephemerals tests after blockquote prefix requirement in PR #11036 #11058: Fix ephemerals tests after blockquote prefix requirement in PR Fix expiration detection for quoted footers and legacy format #11036
  • Fix TypeScript type error in close_older_issues.cjs - add type guard for error.stack access #11069: Fix TypeScript type error in close_older_issues.cjs
  • Fix test assertions for enhanced logging in close_older_issues #11083: Fix test assertions for enhanced logging in close_older_issues
• Example Unmerged PRs:
  • [WIP] Fix flaky test and update GitHub Actions to latest versions #11217: [WIP] Fix flaky test and update GitHub Actions to latest versions
  • [WIP] Refactor code organization for Go files based on semantic analysis #11679: [WIP] Refactor code organization for Go files based on semantic analysis

Cluster 7: Workflow Debugging & Failures

• Size: 42 tasks (4.2% of total)
• Success Rate: 64.3%
• Top Keywords: failure, workflow, failed, debug, run, issue, ci, section
• Most Common Category: Bug Fixes
• Avg Files Changed: 7.7
• Example Merged PRs:
  • Prevent ANSI escape sequences in compiled workflow YAML files #11068: Prevent ANSI escape sequences in compiled workflow YAML files
  • Add error handling to agent output ingestion for better failure diagnostics #11093: Add error handling to agent output ingestion for better failure diagnostics
  • Pass HTTP MCP server secrets to gateway container #11185: Pass HTTP MCP server secrets to gateway container
• Example Unmerged PRs:
  • Investigation: Code Scanning Fixer transient API error - no fix required #11106: Investigation: Code Scanning Fixer transient API error - no fix required
  • Remove unused Tavily MCP dependency from 4 workflows #11158: Remove unused Tavily MCP dependency from 4 workflows

Cluster 8: Reporting & Documentation

• Size: 27 tasks (2.7% of total)
• Success Rate: 92.6% ✅ (highest of all clusters)
• Top Keywords: workflow, formatting, report, issue, add, markdown, include, update
• Most Common Category: CI/CD & Workflows
• Avg Files Changed: 2.8
• Example Merged PRs:
  • [WIP] Update auto-triage-issues workflow for report formatting #11490: [WIP] Update auto-triage-issues workflow for report formatting
  • [WIP] Normalize header levels in static analysis report #11503: [WIP] Normalize header levels in static analysis report
  • Add progressive disclosure formatting to copilot-cli-deep-research workflow #11538: Add progressive disclosure formatting to copilot-cli-deep-research workflow
• Example Unmerged PRs:
  • Add markdown style guidelines to daily-secrets-analysis workflow #12411: Add markdown style guidelines to daily-secrets-analysis workflow
  • [WIP] Update workflow to normalize report formatting guidelines #12431: [WIP] Update workflow to normalize report formatting guidelines

Rule-Based Category Breakdown (All 1,000 PRs)

Category	Count	% of Total
Bug Fixes	277	27.7%
CI/CD & Workflows	212	21.2%
Dependency Updates	157	15.7%
New Features	151	15.1%
General Maintenance	126	12.6%
Documentation	33	3.3%
Refactoring	32	3.2%
Testing	8	0.8%
Security	4	0.4%

Key Findings

1. Highest Success Rate: Reporting & Documentation tasks (92.6% merge rate, 27 tasks) — well-scoped, small-diff tasks with clear acceptance criteria succeed reliably.
2. Lowest Success Rate: Repository & Issue Management (46.4% merge rate, 194 tasks) — nearly 1 in 2 PRs in this cluster are closed without merging. These prompts tend to involve gh/gh-aw tooling, are categorized as CI/CD workflows, and have fewer changed files on average, suggesting many result in investigations or partial fixes rather than clean solutions.
3. Most Common Task Type: General Task Instructions & Setup (209 tasks, 20.9%) — broad, open-ended prompts dominate the dataset.
4. Stable Overall Merge Rate: At 69.0% today vs 69.2% on 2026-02-14, the overall acceptance rate is consistent week-over-week.
5. Package Updates are Reliable: The Dependency/MCP Updates cluster (67 tasks, 74.6%) has high success with large diffs (avg 66 files changed) — these automated, well-understood tasks rarely fail review.

Recommendations

1. Investigate the Repository & Issue Management cluster: With 46.4% success across 194 tasks, this is the single largest opportunity for improvement. Review closed-without-merge PRs for patterns — are tasks too vague, is the scope too broad, or are there systematic tooling failures?
2. Replicate Reporting & Documentation patterns: These 27 tasks succeed 92.6% of the time with small, focused diffs (~2.8 files). The common pattern is clear outcome expectations (formatting, header normalization, markdown guidelines). Applying this specificity to other clusters may lift their rates.
3. Improve Workflow Debugging prompts: At 64.3%, debugging/failure-investigation tasks underperform. Adding more diagnostic context (log snippets, expected vs. actual behavior) to prompts in this category could help the agent produce actionable fixes rather than investigations.
4. Monitor the Feedback & Survey cluster: 205 PRs (20.5%) are grouped by survey-style language in their PR bodies — these may be prompt boilerplate rather than task-specific signals, which could be skewing the clustering. Consider removing this language from PR bodies.

Historical Trend

Date	Total Tasks	Success Rate	Clusters
2026-02-14	980	69.2%	3
2026-02-19	1,000	69.0%	8

References:

• §22181776692

AI generated by Copilot Agent Prompt Clustering Analysis

• expires on Feb 26, 2026, 12:53 PM UTC

[github-actions[bot]]

🤖 Beep boop! The smoke test agent was here!

Just passing through on run §22183145181 to make sure everything is working properly. All systems nominal! 🚀✨

📰 BREAKING: Report filed by Smoke Copilot

[github-actions[bot]]

This discussion was automatically closed because it expired on 2026-02-26T12:53:49.379Z.

Closed by Workflow