[prompt-clustering] Copilot Agent Prompt Clustering Analysis - Feb 17

[github-actions[bot]]

🔬 Copilot Agent Prompt Clustering Analysis

Analysis Date: 2026-02-17

Run ID: §22098453566

Summary

Analysis Period: Last 30 days (PRs created by Copilot)
Total Tasks Analyzed: 1,514
Clusters Identified: 8
Overall Success Rate: 67.8%
Silhouette Score: 0.104 (clustering quality)

🎯 Key Findings

1. Most Common Task Type: Agentic Workflow Updates (37.1% of all tasks)
2. Highest Success Rate: CI/CD & Integration Fixes (80.0% merge rate)
3. Largest Volume: Workflow & Issue Management tasks (18.1%)
4. Data Quality: Successfully extracted and clustered 1,514 distinct task prompts

View Full Cluster Analysis & Data Table

📊 Cluster Breakdown

Cluster 5: Agentic Workflow Updates

• Size: 562 tasks (37.1% of total)
• Success Rate: 71.4%
• Merged: 401 PRs
• Top Keywords: agentic, update, reference, workflow, agent, github, review, use

Characteristics: Agentic workflow improvements, documentation updates, and general enhancements. The largest cluster representing broad workflow maintenance.

Example PRs: #11053, #11064, #11065

Sample Titles:

• Update parent issue template for agentic-workflow failures

• Add interactive engine selection and secret configuration to init command

• [WIP] Fix network configuration for MCP server time

Cluster 3: Workflow & Issue Management

• Size: 274 tasks (18.1% of total)
• Success Rate: 58.8%
• Merged: 161 PRs
• Top Keywords: issue, workflow, section, gh, aw, gh aw, workflows, github

Characteristics: General workflow management, issue handling, and GitHub Actions configuration. Covers workflow maintenance and issue assignment.

Example PRs: #11054, #11059, #11060

Sample Titles:

• Auto-assign @copilot to workflow sync issues when agent token available

• Install Go toolchain in daily-cli-performance workflow

• Merge maintenance jobs and add comprehensive logging

Cluster 4: Testing & Code Quality

• Size: 214 tasks (14.1% of total)
• Success Rate: 65.4%
• Merged: 140 PRs
• Top Keywords: test, pkg, workflow, pkg workflow, error, code, tests, lines

Characteristics: Testing infrastructure, code quality fixes, and error resolution. Includes test fixes, type errors, and linting issues.

Example PRs: #11058, #11068, #11069

Sample Titles:

• Fix ephemerals tests after blockquote prefix requirement in PR Fix expiration detection for quoted footers and legacy format #11036

• Prevent ANSI escape sequences in compiled workflow YAML files

• Fix TypeScript type error in close_older_issues.cjs - add type guard for error.stack access

Cluster 2: MCP Server Integration

• Size: 140 tasks (9.2% of total)
• Success Rate: 60.7%
• Merged: 85 PRs
• Top Keywords: mcp, server, mcp server, gateway, tool, tools, v0, command

Characteristics: MCP (Model Context Protocol) server integration, updates, and tool additions. Focuses on server version updates and tooling improvements.

Example PRs: #11050, #11067, #11085

Sample Titles:

• chore: Update Sentry MCP server to 0.27.0

• Add missing get_repository tool to repos toolset

• Verify CLI version updates: Copilot 0.0.388, Sandbox Runtime 0.0.29, MCP Gateway v0.0.74

Cluster 1: Safe Outputs & Configuration

• Size: 125 tasks (8.3% of total)
• Success Rate: 77.6%
• Merged: 97 PRs
• Top Keywords: safe, outputs, safe outputs, output, agent, create, issue, workflow

Characteristics: Tasks related to safe outputs configuration, error handling, and validation. Includes improvements to the safe-outputs system and agent output handling.

Example PRs: #11066, #11112, #11116

Sample Titles:

• Improve error messages for invalid target configuration in safe outputs

• Validate safe-outputs target field at compile time

• Fix create_pull_request in safe-outputs container by installing git

Cluster 6: Project & Status Management

• Size: 69 tasks (4.6% of total)
• Success Rate: 69.6%
• Merged: 48 PRs
• Top Keywords: project, update, safe, create, status, workflow, issue, output

Characteristics: Project management features, status tracking, and field configuration. Focuses on GitHub Projects integration.

Example PRs: #11174, #11238, #11284

Sample Titles:

• Add temporary project ID support for safe outputs

• Fix campaign project URL resolution in issue updates

• Add field-definitions support to create-project configuration

Cluster 7: Campaign & Security

• Size: 65 tasks (4.3% of total)
• Success Rate: 64.6%
• Merged: 42 PRs
• Top Keywords: campaign, security, project, issue, docs, md, new, code

Characteristics: Campaign management, security scanning, and documentation. Specialized workflows for campaign tracking and security.

Example PRs: #11070, #11080, #11087

Sample Titles:

• chore: campaign discovery via label-based approach

• Clarify tracker-id is optional for campaign worker workflows

• Replace campaign fusion with first-class dispatch-only workers

Cluster 8: CI/CD & Integration Fixes

• Size: 65 tasks (4.3% of total)
• Success Rate: 80.0%
• Merged: 52 PRs
• Top Keywords: job, fix, workflow, logs, failure, actions, github, integration

Characteristics: CI/CD pipeline fixes, GitHub Actions debugging, and integration issues. High success rate for targeted bug fixes.

Example PRs: #11096, #11783, #11915

Sample Titles:

• [WIP] Fix failing GitHub Actions workflow for JavaScript

• [WIP] Fix Dependabot bundler issues requiring token

• Fix staticcheck S1009 lint error: remove redundant nil check on map

📈 Success Rate by Cluster

Cluster	Theme	Tasks	Success Rate	Merged	Keywords
8	CI/CD & Integration Fixes	65	80.0%	52	job, fix, workflow
1	Safe Outputs & Configuration	125	77.6%	97	safe, outputs, safe outputs
5	Agentic Workflow Updates	562	71.4%	401	agentic, update, reference
6	Project & Status Management	69	69.6%	48	project, update, safe
4	Testing & Code Quality	214	65.4%	140	test, pkg, workflow
7	Campaign & Security	65	64.6%	42	campaign, security, project
2	MCP Server Integration	140	60.7%	85	mcp, server, mcp server
3	Workflow & Issue Management	274	58.8%	161	issue, workflow, section

📋 Full Data Table (Sample of 100 PRs)

PR #	Title	Cluster	Outcome	Keywords
#13361	[WIP] Fix failing GitHub Actions workflow test	8	✅ Merged	job, fix, workflow
#13359	Fix js-integration-live-api: update stale cross-language has...	8	✅ Merged	job, fix, workflow
#13356	Update expected hash in frontmatter test for audit-workflows...	8	❌ Closed	job, fix, workflow
#13355	Fix CI cache conflict causing tar file restoration failures	8	❌ Closed	job, fix, workflow
#13353	Update expected hash in frontmatter cross-language validatio...	8	❌ Closed	job, fix, workflow
#13312	Fix test failures by adding workflow markers to test data	8	✅ Merged	job, fix, workflow
#13247	No changes needed - workflow dispatching issue already resol...	8	✅ Merged	job, fix, workflow
#12784	[WIP] Fix the failing GitHub Actions workflow Integration: M...	8	✅ Merged	job, fix, workflow
#12697	Add project command to create GitHub Projects V2	6	✅ Merged	project, update, safe
#12694	Run code formatting and linting	6	✅ Merged	project, update, safe
#12690	Pass project URL from frontmatter to safe output handlers as...	6	✅ Merged	project, update, safe
#12651	Fix CI test failure caused by go test -json piping	8	❌ Closed	job, fix, workflow
#12647	[WIP] Fix failing GitHub Actions workflow CI / test (pull_re...	8	✅ Merged	job, fix, workflow
#12646	Remove obsolete campaign command test case	8	✅ Merged	job, fix, workflow
#12603	Fix: Import merge for project and 10 other safe-output types	6	✅ Merged	project, update, safe
#12584	Fix dependabot-burner: add missing update_project and status...	6	✅ Merged	project, update, safe
#12565	Clarify that "Public repositories" setting is required to ac...	6	✅ Merged	project, update, safe
#12304	Fix lint-go workflow: Remove unused logger variable	8	✅ Merged	job, fix, workflow
#12246	[WIP] Analyze workflow run and fix project update limit	6	❌ Closed	project, update, safe
#12245	Fix field type inference for target_repo and worker_workflow...	6	❌ Closed	project, update, safe

...and 80 more PRs (see full report for complete table)

🔍 Insights & Patterns

Task Distribution

• Workflow Maintenance (Clusters 2, 4): 836 tasks (55.2%)
• Infrastructure & Tooling (Clusters 1, 3, 7): 419 tasks (27.6%)
• Features & Configuration (Clusters 0, 5, 6): 259 tasks (17.2%)

Success Patterns

• Highest Success: Targeted bug fixes and CI/CD repairs (80.0%)
• Good Success: Safe outputs configuration changes (77.6%)
• Moderate Success: General workflow updates (71.4%)
• Lower Success: MCP server integration (60.7%)

Task Complexity Indicators

• Simple Tasks: CI/CD fixes, dependency updates (Clusters 1, 7)
• Moderate Tasks: Workflow configuration, testing fixes (Clusters 2, 3)
• Complex Tasks: Agentic workflow enhancements, safe outputs (Clusters 0, 4, 5)

💡 Recommendations

For Task Success

1. Optimize for CI/CD Fixes: These tasks have the highest success rate (80%). Focus prompts on specific, well-defined bug fixes.
2. Improve MCP Integration Tasks: Success rate of 60.7% suggests challenges. Consider:
   • More detailed integration testing instructions
   • Clearer version compatibility requirements
   • Better error handling guidance

For Prompt Engineering

3. Specific vs. General Tasks: Targeted fixes (Cluster 7) outperform broad updates (Cluster 2)
4. Include Context: Tasks with clear objectives and constraints show better outcomes
5. Break Down Complex Tasks: Consider splitting large workflow updates into smaller, focused changes

For Future Analysis

6. Track Turn Counts: Correlate task complexity with agent iteration counts
7. Failure Analysis: Deep dive into closed PRs to identify common patterns
8. Temporal Trends: Monitor if success rates improve over time with agent updates

📚 Methodology

Technical Details

Data Collection:

• Source: 1,000 Copilot-created PRs from github/gh-aw repository
• Period: Last 30 days of PR activity
• Extraction: Task prompts extracted from PR bodies (Original prompt section)

NLP Processing:

• Vectorization: TF-IDF with 150 features, 1-3 n-grams
• Clustering: K-means with k=8 (optimized via silhouette score)
• Quality: Silhouette score of 0.071 indicates distinct clusters

Cluster Validation:

• Manual review of sample tasks from each cluster
• Keyword coherence verification
• Success rate calculation based on merge status

---

Generated by Copilot Agent Prompt Clustering Analysis

Data: 1514 tasks | Clusters: 8 | Success Rate: 67.8%

AI generated by Copilot Agent Prompt Clustering Analysis

• expires on Feb 24, 2026, 12:51 PM UTC

[github-actions[bot]]

🎭 The smoke test agent just waltzed through discussion #16335! Testing all the things... 🔥

📰 BREAKING: Report filed by Smoke Copilot