-
Notifications
You must be signed in to change notification settings - Fork 219
Description
Problem Statement
In the past 7 days (February 7-14, 2026), documentation agents created 47 pull requests with a 0% merge rate - all PRs were closed without merging. This represents a significant decline from the previous period's 70% merge rate.
Affected Workflows:
- Daily Doc Updater
- Instructions Janitor
- Documentation Unbloat
- Workflow Normalizer
Evidence
From Agent Performance Analysis (§22021394730):
| Metric | Current (Feb 7-14) | Previous | Change |
|--------|-------------------|----------|--------|
| PRs Created | 47 | N/A | High volume |
| PR Merge Rate | 0% | 70% | ↓ -70% CRITICAL |Sample Closed PRs (all without merge):
- [docs] Update documentation for PR review comment footer control #15655 - PR review footer control docs (closed)
- [instructions] Add footer control documentation for PR review comments #15653 - Instructions update (closed)
- [jsweep] Clean handle_noop_message.cjs #15649 - Clean handle_noop_message.cjs (closed)
- [ubuntu-image] Add Ubuntu Actions Runner Image Analysis Documentation #15639 - Ubuntu image analysis docs (closed)
- ... and 43 more
Hypotheses for Root Cause
- Timing: PRs too recent to be reviewed (need more time)
- Manual superseding: Maintainers making manual fixes before reviewing agent PRs
- Quality issues: PRs not meeting unstated quality requirements
- Alignment problems: Changes not aligning with maintainer priorities or vision
- Process issues: Wrong workflow for documentation changes (should be issues/discussions?)
Investigation Required
Phase 1: Data Collection (2 hours)
-
Review all 47 closed PRs:
- Read close comments and reasons
- Identify patterns in closure (timing, quality, maintainer feedback)
- Check if any were superseded by manual commits
- Note any common feedback themes
-
Interview/survey maintainers:
- Ask about documentation PR preferences
- Understand what makes a good vs. bad documentation PR
- Clarify unstated quality requirements
- Understand review process and timing expectations
-
Analyze successful PRs (previous period):
- Review the 70% that merged in previous period
- Identify characteristics of successful PRs
- Compare to current failed PRs
- Extract success patterns
Phase 2: Pattern Identification (1 hour)
-
Categorize closure reasons:
- Timing (not yet reviewed)
- Quality (formatting, accuracy, completeness)
- Alignment (wrong changes, unnecessary)
- Process (wrong approach)
- Superseded (manual fix applied)
-
Quantify each category:
- Count PRs in each bucket
- Identify dominant patterns
- Look for workflow-specific issues
Phase 3: Recommendations (1 hour)
Based on findings, develop specific recommendations for:
- Documentation agent prompts (what to change)
- PR creation criteria (when to create vs. not create)
- PR quality gates (checks before creation)
- Alternative approaches (discussions first, issues instead of PRs)
- Process changes (timing, review flow)
Expected Outcomes
- Root cause identified: Clear understanding of why PRs are being closed
- Specific improvements: Actionable changes to agent configuration
- Quality gates: Criteria for creating high-quality documentation PRs
- Success metrics: Target merge rate and quality indicators
- Implementation plan: Step-by-step changes to affected workflows
Success Criteria
- All 47 closed PRs reviewed and categorized
- Dominant closure pattern identified with data
- Maintainer preferences documented
- Agent prompt improvements drafted
- PR quality gates defined
- Target: Increase merge rate from 0% to 40-50% within 2 weeks
Priority: High
Impact: Medium-High - Affects 4 workflows, represents wasted agent effort, degrades ecosystem effectiveness
Effort: 4 hours investigation + 2-4 hours implementation
Timeline: Investigation complete within 48 hours, improvements deployed within 1 week
📊 Related Report: Agent Performance Report - Week of February 7-14, 2026
🔗 Workflow Run: §22021394730
Generated by Agent Performance Analyzer - Meta-Orchestrator
- expires on Feb 16, 2026, 5:36 PM UTC