[prompt-clustering] 🔬 Copilot Agent Prompt Clustering Analysis - January 2026

[github-actions[bot]]

Daily NLP-based clustering analysis of copilot agent task prompts to identify patterns, trends, and optimization opportunities.

Summary

Analysis Period: 2025-10-22 to 2026-01-29
Total Tasks Analyzed: 4,389
Clusters Identified: 7
Overall Success Rate: 70.0%

Key Findings

1. 🔒 Security Fixes is the most common task type, representing 41.2% of all tasks (1,810 tasks)
2. 🔒 Security Fixes tasks have the highest success rate at 74.5%
3. 🧪 Testing tasks are the most complex, averaging 29.9 files changed and 4.8 commits

Notable Trend

⚠️ January 2026 shows declining success rate (64.8% vs 76.2% in November 2025) - requires investigation into recent changes or increased task complexity.

View Detailed Analysis

Cluster Breakdown

🔒 Cluster 6: Security Fixes

• Size: 1,810 tasks (41.2% of total)
• Success Rate: 74.5%
• Average Changes: 881 additions, 1137 deletions across 23.5 files
• Average Commits: 3.7
• Average Comments: 2.3
• Top Keywords: update, add, agent, workflow, file, make, fix
• Example PRs: #10000, #10001, #10002

Example task prompt

Please fix this security alert in actions/setup/src/index.js:30
Tool
CodeQL
Rule ID
js/insecure-temporary-file
Query
View source
Description
Temporary files created in the operating system's temporary directory are by default accessible to other users. In some cases, this can lead to information exposure, or in the worst case, to remote code execution.
Recommendation
Use a well-tested library like tmp for creating temporary files. These libraries ensure both that the file is inaccessible to othe
...
``````

</details>

#### 🔒 Cluster 0: Security Fixes

- **Size**: 761 tasks (17.3% of total)
- **Success Rate**: 66.2%
- **Average Changes**: 591 additions, 217 deletions across 11.8 files
- **Average Commits**: 3.1
- **Average Comments**: 0.8
- **Top Keywords**: comments, issuetitle, files, plan, add, githubnext gh, githubnext gh aw
- **Example PRs**: [#10020](https://github.com/githubnext/gh-aw/pull/10020), [#10021](https://github.com/githubnext/gh-aw/pull/10021), [#10125](https://github.com/githubnext/gh-aw/pull/10125)

<details>
<summary><i>Example task prompt</i></summary>

``````
----
*This section details on the original issue you should resolve*
(issue_title[plan] Fix unverified script execution in awf installation (120 occurrences)</issue_title<issue_description## Objective
Replace insecure curl-to-bash pattern with secure download-verify-execute pattern for awf binary installation.
## Context
The daily static analysis scan (zizmor) identified 120 occurrences of unverified script execution across 118 workflows. This is a **Medium severity security vulnerability** that
...
``````

</details)

#### 🐛 Cluster 4: Bug Fixes

- **Size**: 504 tasks (11.5% of total)
- **Success Rate**: 65.3%
- **Average Changes**: 525 additions, 367 deletions across 14.7 files
- **Average Commits**: 2.9
- **Average Comments**: 1.2
- **Top Keywords**: comments, issue, issuetitle, section, workflow, cli, issuedescription
- **Example PRs**: [#10128](https://github.com/githubnext/gh-aw/pull/10128), [#10143](https://github.com/githubnext/gh-aw/pull/10143), [#10172](https://github.com/githubnext/gh-aw/pull/10172)

<details>
<summary><i>Example task prompt</i></summary>

``````
----
*This section details on the original issue you should resolve*
(issue_title[aw] CLI Version Checker failed</issue_title<issue_description## Problem
The agentic workflow **CLI Version Checker** has failed. This typically indicates a configuration or runtime error that requires user intervention.
## Failed Run
- **Workflow:** [CLI Version Checker](#)
- **Failed Run:** https://github.com/githubnext/gh-aw/actions/runs/20897177975
- **Source:** 
## How to investigate
Use the **debug-agentic-wor
...
``````

</details)

#### 🐛 Cluster 5: Bug Fixes

- **Size**: 468 tasks (10.7% of total)
- **Success Rate**: 70.1%
- **Average Changes**: 1236 additions, 358 deletions across 13.3 files
- **Average Commits**: 3.7
- **Average Comments**: 2.2
- **Top Keywords**: agentic, agentic workflow, workflow, agentic workflows, workflows, update, agent
- **Example PRs**: [#10017](https://github.com/githubnext/gh-aw/pull/10017), [#10044](https://github.com/githubnext/gh-aw/pull/10044), [#10063](https://github.com/githubnext/gh-aw/pull/10063)

<details>
<summary><i>Example task prompt</i></summary>

``````
----
*This section details on the original issue you should resolve*
(issue_title[aw] Daily Code Metrics and Trend Tracking Agent failed</issue_title<issue_description## Problem
The agentic workflow **Daily Code Metrics and Trend Tracking Agent** has failed. This typically indicates a configuration or runtime error that requires user intervention.
## Failed Run
- **Workflow:** [Daily Code Metrics and Trend Tracking Agent](#)
- **Failed Run:** https://github.com/githubnext/gh-aw/actions/runs/2089
...
``````

</details)

#### 🔒 Cluster 2: Security Fixes

- **Size**: 332 tasks (7.6% of total)
- **Success Rate**: 61.4%
- **Average Changes**: 1333 additions, 691 deletions across 29.8 files
- **Average Commits**: 4.2
- **Average Comments**: 4.5
- **Top Keywords**: mcp, gateway, server, mcp server, configuration, github, add
- **Example PRs**: [#10162](https://github.com/githubnext/gh-aw/pull/10162), [#10346](https://github.com/githubnext/gh-aw/pull/10346), [#10350](https://github.com/githubnext/gh-aw/pull/10350)

<details>
<summary><i>Example task prompt</i></summary>

``````
Update the MCP gateway spec with the following feature:
- the MCP server should be able to opt of the gate way containerization through a flag
``````

</details>

#### 🧪 Cluster 3: Testing

- **Size**: 284 tasks (6.5% of total)
- **Success Rate**: 66.5%
- **Average Changes**: 3179 additions, 3417 deletions across 29.9 files
- **Average Commits**: 4.8
- **Average Comments**: 4.5
- **Top Keywords**: safe, safe output, output, outputs, create, update, add
- **Example PRs**: [#10161](https://github.com/githubnext/gh-aw/pull/10161), [#10165](https://github.com/githubnext/gh-aw/pull/10165), [#10230](https://github.com/githubnext/gh-aw/pull/10230)

<details>
<summary><i>Example task prompt</i></summary>

``````
Why does this have both separate steps (Update Project, Assign to Agent) but also a mix and many things in Process Safe Outputs? See https://github.com/githubnext/gh-aw/actions/runs/21041168396/job/60503683150
``````

</details>

#### 🔒 Cluster 1: Security Fixes

- **Size**: 230 tasks (5.2% of total)
- **Success Rate**: 73.0%
- **Average Changes**: 871 additions, 4110 deletions across 14.9 files
- **Average Commits**: 5.2
- **Average Comments**: 2.1
- **Top Keywords**: campaign, project, issue, file, fix, create, run
- **Example PRs**: [#10045](https://github.com/githubnext/gh-aw/pull/10045), [#10052](https://github.com/githubnext/gh-aw/pull/10052), [#10096](https://github.com/githubnext/gh-aw/pull/10096)

<details>
<summary><i>Example task prompt</i></summary>

``````
Make the campaign progress board view also add a column "Review required".

Success Rate Comparison

Cluster	Category	Tasks	Success Rate	Avg Files Changed	Avg Commits
🔒 6	Security Fixes	1,810	74.5%	23.5	3.7
🔒 0	Security Fixes	761	66.2%	11.8	3.1
🐛 4	Bug Fixes	504	65.3%	14.7	2.9
🐛 5	Bug Fixes	468	70.1%	13.3	3.7
🔒 2	Security Fixes	332	61.4%	29.8	4.2
🧪 3	Testing	284	66.5%	29.9	4.8
🔒 1	Security Fixes	230	73.0%	14.9	5.2

Sample Tasks by Cluster

PR	Title	Cluster	Outcome	Files	Commits	Keywords
#10000	Fix insecure temporary file creation (CWE-377) with secure f...	🔒 6	✅ Merged	4	5	update, add, agent
#10001	Fix TOCTOU vulnerability in changeset.js file operations	🔒 6	✅ Merged	1	2	update, add, agent
#10002	Add path traversal validation to playground snapshot downloa...	🔒 6	✅ Merged	4	3	update, add, agent
#10005	Add agentic workflow to auto-fix high severity code scanning...	🔒 6	✅ Merged	5	4	update, add, agent
#10008	Enhance AGENTS.md formatting warnings to prevent recurring C...	🔒 6	✅ Merged	5	3	update, add, agent
#10020	Add missing GitHub permissions to workflows using GitHub API...	🔒 0	✅ Merged	9	4	comments, issuetitle, files
#10021	Pin GitHub Actions to commit SHAs in release workflow	🔒 0	✅ Merged	5	4	comments, issuetitle, files
#10125	Fix JavaScript test failures: schema validation and missing ...	🔒 0	✅ Merged	1	3	comments, issuetitle, files
#10523	Standardize metric naming across daily reports with shared g...	🔒 0	✅ Merged	15	4	comments, issuetitle, files
#10524	Fix Issue Arborist workflow: use correct search filter to ex...	🔒 0	✅ Merged	6	4	comments, issuetitle, files
#10128	Use info-level logging for missing agent output files	🐛 4	✅ Merged	2	3	comments, issue, issuetitle
#10143	Fix firewall log artifact upload permission error	🐛 4	✅ Merged	122	5	comments, issue, issuetitle
#10172	Skip permission checks for merge_group events	🐛 4	✅ Merged	4	2	comments, issue, issuetitle
#10178	Fix undici security vulnerability (GHSA-g9mf-h72j-4rw9)	🐛 4	✅ Merged	2	2	comments, issue, issuetitle
#10239	Increase agent-persona-explorer workflow timeout to 10 hours	🐛 4	✅ Merged	2	2	comments, issue, issuetitle

Recommendations

1. Improve MCP/Gateway Tasks: Cluster 2 (MCP-related) has the lowest success rate (61.4%) and highest iteration cycles (4.5 comments). Consider:

   • Providing more detailed prompts for MCP server configuration tasks
   • Including architecture diagrams or examples in task descriptions
   • Breaking down complex MCP gateway changes into smaller subtasks

2. Reduce Iteration Cycles: Tasks averaging 4+ comments indicate unclear requirements. Consider:

   • Adding clearer initial requirements with acceptance criteria
   • Providing better context and examples upfront
   • Including links to relevant documentation or similar PRs

3. Investigate January 2026 Decline: Success rate dropped from 76.2% (Nov 2025) to 64.8% (Jan 2026). Investigate:

   • Recent changes to agent instructions or prompts
   • Increase in task complexity or new task types
   • Infrastructure or tooling changes affecting success

4. Leverage Successful Patterns: Cluster 6 (74.5% success) shows successful characteristics:

   • Well-structured security prompts with CodeQL references
   • Clear problem descriptions with examples
   • Specific file paths and line numbers
   • Concrete recommendations from scanning tools

Temporal Trends

Monthly Activity

Month	Tasks	Success Rate
2025-10	397	73.3%
2025-11	884	76.2%
2025-12	1,348	71.6%
2026-01	1,760	64.8% ⚠️

Task Complexity Distribution

• Simple (0-5 files): 2,355 tasks (53.7%) - 68.2% success rate
• Medium (5-15 files): 1,045 tasks (23.8%) - 73.5% success rate ⭐
• Complex (15-30 files): 236 tasks (5.4%) - 75.0% success rate ⭐
• Very Complex (30+ files): 753 tasks (17.2%) - 68.9% success rate

Insight: Medium and Complex tasks (5-30 files) have better success rates than Simple tasks, suggesting agents perform better with well-scoped, moderately complex tasks than with very simple or very complex ones.

Methodology

This analysis uses NLP clustering (K-means with TF-IDF vectorization) on 4,389 copilot agent task prompts extracted from PRs. Tasks are grouped into 7 semantic clusters based on prompt content, then enriched with success metrics and complexity measures.

Analysis Date: 2026-01-29 06:44 UTC
Workflow Run: §21468085708

AI generated by Copilot Agent Prompt Clustering Analysis

• expires on Feb 5, 2026, 6:47 AM UTC

[github-actions[bot]]

This discussion was automatically closed because it expired on 2026-02-05T06:47:02.424Z.