🎯 Repository Quality Improvement Report - Error Experience Engineering

[github-actions[bot]]

Analysis Date: 2026-01-07
Focus Area: Error Experience Engineering
Strategy Type: Custom
Custom Area: Yes - This focus area addresses the holistic developer experience when errors occur in gh-aw, encompassing error message clarity, recovery paths, debugging support, visual formatting, and proactive error prevention.

Executive Summary

This analysis examines Error Experience Engineering - a custom focus area specifically designed for gh-aw's unique needs as a workflow compilation and execution platform. Unlike traditional "error handling" reviews, this assessment evaluates the complete developer experience when things go wrong, from first error encounter through resolution.

Key Findings: The codebase shows strong foundation with 809 wrapped errors (55.8%) preserving context and 295 validation errors offering suggestions. However, significant gaps exist: only 44 console-formatted error messages vs 178 raw error outputs, 51 anti-pattern fmt.Print calls to stdout, and 80 plain errors.New() calls losing context. With 1,450 error creation sites across 63 validation files (14,642 LOC), systematic improvements to error visibility, actionability, and debugging support would dramatically enhance developer experience.

Full Analysis Report

Focus Area: Error Experience Engineering

Current State Assessment

Metrics Collected:

Metric	Value	Status
Total error creation sites	1,450	ℹ️ Large surface area
Console-formatted errors	44 (3%)	❌ Critical gap
Raw error output (needs formatting)	178	⚠️ Major issue
Error wrapping with context (%w)	809 (55.8%)	✅ Good practice
Plain errors without context	80	⚠️ Context loss
Validation errors with suggestions	295	✅ Strong guidance
Validation LOC	14,642 across 63 files	ℹ️ Complex validation
Commands with proper RunE	10/10	✅ Excellent
Retry mechanisms	80 instances	✅ Resilience
Debug logging for errors	11 instances	❌ Insufficient
Panic recovery (defer recover)	0	⚠️ No safety net
fmt.Print anti-patterns	51	❌ Breaks stderr convention

Findings

Strengths

• Context Preservation: 55.8% of errors use %w wrapping to preserve error chains
• Actionable Validation: 295 validation errors include suggestions with "try", "instead", "should"
• Proper CLI Patterns: All 10 commands use RunE for error handling (no Run anti-patterns)
• Resilience: 80 retry mechanism implementations for transient failures
• Comprehensive Validation: 63 validation files covering security, permissions, network, configuration

Areas for Improvement

• Console Formatting Gap: Only 3% (44/1,450) errors use console.FormatErrorMessage despite AGENTS.md requirement
• Raw Error Output: 178 instances of fmt.Fprintf(os.Stderr, err) without formatting
• Stdout Anti-pattern: 51 instances of fmt.Print(err) violating stderr convention
• Context Loss: 80 plain errors.New() calls losing valuable error chain context
• Debug Visibility: Only 11 debug logs for errors - insufficient for troubleshooting
• No Panic Recovery: Zero defer recover() implementations - crashes are unhandled
• Inconsistent User Experience: Mix of beautifully formatted vs raw error messages

Detailed Analysis

1. Error Visibility & Formatting (Critical Issue)

Problem: Only 3% of error sites use console formatting, creating inconsistent user experience.

Examples of Good Error Formatting:

fmt.Fprintln(os.Stderr, console.FormatErrorMessage(err.Error()))

Examples Found Needing Improvement:

• pkg/cli/access_log.go - Raw error output
• pkg/cli/actions_build_command.go - Raw error output
• pkg/cli/audit_report_render.go - Raw error output
• pkg/workflow/mcp_renderer.go - Uses fmt.Printf for errors
• pkg/workflow/cache.go - Uses fmt.Printf for warnings

Impact: Users see inconsistent error presentations - some beautifully formatted with colors and structure, others as raw text dumps.

2. Error Context & Actionability

Strong Example (from validation):

return fmt.Errorf("tool '%s' uses HTTP transport which is not supported by engine '%s'. " +
    "Only stdio transport is supported. Use a different engine (e.g., copilot) or change the tool " +
    "to use stdio transport. Example:\ntools:\n  %s:\n    type: stdio\n    command: \"node server.js\"", 
    toolName, engine.GetID(), toolName)

Weak Example (context loss):

return errors.New(formattedErr)  // No wrapping, loses error chain

Gap: 80 instances of errors.New() that should wrap underlying errors with %w.

3. Debugging Support

Current State:

• Only 11 instances of debug logging for error paths
• No panic recovery mechanisms (0 defer recover)
• Limited structured error data for automated analysis

Missing Capabilities:

• Stack traces for critical errors
• Error aggregation for pattern detection
• Structured error logging (JSON) for tooling
• Debug mode with verbose error context

4. High-Impact Error Sites

Files with most error returns (top 5):

1. compiler_orchestrator.go: 44 error returns
2. safe_outputs_config_generation.go: 854 LOC validation (from previous analysis)
3. mcp-config.go: 999 LOC with complex error paths
4. runtime_setup.go: 1,016 LOC with setup errors
5. permissions.go: 953 LOC with security errors

5. Anti-Patterns Found

Stdout Errors (51 instances):

// ❌ WRONG - errors to stdout
fmt.Println(err)
fmt.Printf("Error: %v\n", err)

// ✅ CORRECT - errors to stderr with formatting
fmt.Fprintln(os.Stderr, console.FormatErrorMessage(err.Error()))

Raw Error Output (178 instances):

// ❌ WRONG - no console formatting
fmt.Fprintf(os.Stderr, "Error: %v\n", err)

// ✅ CORRECT - console formatted
fmt.Fprintln(os.Stderr, console.FormatErrorMessage(err.Error()))

6. Validation Error Quality

Strengths:

• 295 errors with actionable suggestions
• Examples included in error messages
• Categorized by domain (permissions, network, configuration)

Opportunities:

• Standardize error message templates
• Add error codes for programmatic handling
• Create error catalog documentation
• Implement suggestion system for common mistakes

---

🤖 Tasks for Copilot Agent

NOTE TO PLANNER AGENT: The following tasks are designed for GitHub Copilot agent execution. Please split these into individual work items for Claude to process.

Improvement Tasks

The following code regions and tasks should be processed by the Copilot agent. Each section is marked for easy identification by the planner agent.

---

Task 1: Console Format Error Messages in CLI Commands

Priority: High
Estimated Effort: Medium
Focus Area: Error Visibility & User Experience

Description:
Convert all raw error outputs in CLI command files to use console.FormatErrorMessage() for consistent, professional error presentation. This addresses the critical gap where only 3% of errors use proper formatting despite AGENTS.md requirements.

Acceptance Criteria:

• All error outputs in pkg/cli/*.go use console.FormatErrorMessage()
• No fmt.Println(err) or fmt.Printf(err) direct error outputs remain
• All error output goes to stderr (fmt.Fprintln(os.Stderr, ...))
• Consistent formatting across all CLI commands
• Visual regression tests pass (if applicable)

Code Region: pkg/cli/*_command.go, pkg/cli/audit*.go, pkg/cli/compile_*.go

High-priority files:

• pkg/cli/access_log.go
• pkg/cli/actions_build_command.go
• pkg/cli/add_command.go
• pkg/cli/audit.go
• pkg/cli/audit_report_render.go
• pkg/cli/compile_batch_operations.go
• pkg/cli/compile_helpers.go
• pkg/cli/compile_orchestration.go

Review all CLI command files in pkg/cli/ and convert raw error outputs to use the console formatting system. 

Pattern to replace:
```go
// OLD
fmt.Fprintf(os.Stderr, "Error: %v\n", err)
fmt.Println(err)

// NEW
fmt.Fprintln(os.Stderr, console.FormatErrorMessage(err.Error()))

Ensure all files import "github.com/githubnext/gh-aw/pkg/console" and follow the console formatting guidelines from AGENTS.md.

Focus on these high-priority files first:

• pkg/cli/access_log.go
• pkg/cli/actions_build_command.go
• pkg/cli/add_command.go
• pkg/cli/audit.go
• pkg/cli/compile_*.go files

Verify no errors are printed to stdout - all should go to stderr.

---

#### Task 2: Add Error Context Wrapping to Plain errors.New() Calls

**Priority**: High  
**Estimated Effort**: Medium  
**Focus Area**: Error Context Preservation

**Description:**
Convert 80 instances of errors.New() to properly wrapped errors using fmt.Errorf with %w to preserve error chains. This prevents context loss and enables better error analysis and debugging.

**Acceptance Criteria:**
- [ ] All errors.New() calls in pkg/workflow/ reviewed
- [ ] Context-sensitive errors converted to fmt.Errorf with %w wrapping
- [ ] Error messages include actionable context (what failed, why, what to do)
- [ ] No regression in error handling tests
- [ ] Error chains preserve root cause information

**Code Region:** `pkg/workflow/*.go` (focus on validation files)

```markdown
Audit all instances of errors.New() in pkg/workflow/ and convert to properly wrapped errors:

Pattern to replace:
```go
// OLD - loses context
return errors.New("validation failed")

// NEW - preserves context
return fmt.Errorf("validation failed: %w", underlyingErr)

// OR if no underlying error, add actionable context
return fmt.Errorf("validation failed: field '%s' is required but not provided. Add the field to your workflow frontmatter", fieldName)

Priority files with plain errors.New() usage:

• pkg/workflow/validation files (63 files)
• pkg/workflow/mcp-config.go
• pkg/workflow/safe_outputs_config_generation.go

Ensure wrapped errors use %w (not %v) to preserve error chains for errors.Is() and errors.As() checks.

---

#### Task 3: Implement Debug Logging for Critical Error Paths

**Priority**: Medium  
**Estimated Effort**: Medium  
**Focus Area**: Debugging Support & Observability

**Description:**
Add structured debug logging to critical error paths in workflow compilation, MCP configuration, and validation systems. Currently only 11 debug logs exist for errors - insufficient for troubleshooting production issues.

**Acceptance Criteria:**
- [ ] Debug logging added to top 5 high-impact error sites
- [ ] Logs include structured context (file, operation, input values)
- [ ] Uses existing logger package with appropriate namespaces
- [ ] Logs only appear when DEBUG environment variable matches
- [ ] No performance impact when debug logging disabled

**Code Region:** 
- `pkg/workflow/compiler_orchestrator.go` (44 error returns)
- `pkg/workflow/mcp-config.go` (999 LOC)
- `pkg/workflow/runtime_setup.go` (1,016 LOC)
- `pkg/workflow/permissions.go` (953 LOC)
- `pkg/workflow/safe_outputs_config_generation.go` (854 LOC)

```markdown
Add debug logging to critical error paths in workflow compilation and validation:

Pattern to add:
```go
import "github.com/githubnext/gh-aw/pkg/logger"

var log = logger.New("workflow:compiler_orchestrator")

func someValidation() error {
    log.Printf("Starting validation for workflow: %s", workflowName)
    
    if err := validate(); err != nil {
        log.Printf("Validation failed: %v (context: %+v)", err, contextData)
        return fmt.Errorf("validation failed: %w", err)
    }
    
    log.Printf("Validation completed successfully")
    return nil
}

Focus on these high-impact files:

1. pkg/workflow/compiler_orchestrator.go - Add logs for orchestration steps
2. pkg/workflow/mcp-config.go - Add logs for MCP configuration validation
3. pkg/workflow/runtime_setup.go - Add logs for runtime setup failures
4. pkg/workflow/permissions.go - Add logs for permission validation
5. pkg/workflow/safe_outputs_config_generation.go - Add logs for safe output generation

Follow logger naming convention: "workflow:filename" (e.g., "workflow:compiler_orchestrator")

---

#### Task 4: Create Error Recovery Patterns Documentation

**Priority**: Medium  
**Estimated Effort**: Small  
**Focus Area**: Developer Experience & Documentation

**Description:**
Create comprehensive documentation for error handling patterns, common error scenarios, recovery strategies, and debugging techniques specific to gh-aw. This fills the gap of only having 1 error message doc despite 51 error handling guide mentions.

**Acceptance Criteria:**
- [ ] New document created: specs/error-recovery-patterns.md
- [ ] Documents common error scenarios and solutions
- [ ] Includes error message templates and examples
- [ ] Provides debugging runbook for each error category
- [ ] Links to relevant validation files and handlers
- [ ] Indexed in AGENTS.md Available Skills Reference

**Code Region:** `specs/` directory, `AGENTS.md`

```markdown
Create a comprehensive error handling documentation file at specs/error-recovery-patterns.md covering:

1. **Error Handling Patterns**
   - Console formatting requirements
   - Error wrapping with %w
   - Debug logging standards
   - Panic recovery (when/how to use)

2. **Common Error Scenarios**
   - MCP configuration errors → recovery steps
   - Permission validation errors → suggested fixes
   - Network/firewall errors → debugging steps
   - Workflow compilation errors → resolution guide

3. **Error Message Templates**
   - Validation error template with example
   - Runtime error template with context
   - User-actionable error template
   - System/internal error template

4. **Debugging Runbook**
   - How to enable DEBUG logging
   - Reading error chains with errors.As/Is
   - Analyzing validation failures
   - Troubleshooting MCP server issues

5. **Error Categorization**
   - User errors (fixable by user)
   - Configuration errors (workflow changes)
   - System errors (environment issues)
   - Internal errors (code bugs)

Link this document from AGENTS.md under "Available Skills Reference" > "Development Guidelines"

---

Task 5: Implement Panic Recovery for Critical Operations

Priority: Low
Estimated Effort: Small
Focus Area: Resilience & Stability

Description:
Add panic recovery mechanisms with proper error conversion for critical operations in workflow compilation, MCP server initialization, and CLI orchestration. Currently 0 defer recover() implementations exist, leaving crashes unhandled.

Acceptance Criteria:

• Panic recovery added to workflow compilation entry points
• Recovered panics converted to proper errors with context
• Debug logging captures panic details (stack trace)
• User sees graceful error message instead of crash
• Tests verify panic recovery behavior

Code Region:

• pkg/workflow/compiler_orchestrator.go
• pkg/workflow/mcp-config.go
• pkg/cli/compile_command.go
• pkg/cli/run_command.go

Add panic recovery to critical operation entry points with proper error handling:

Pattern to implement:
```go
import "github.com/githubnext/gh-aw/pkg/logger"

var log = logger.New("workflow:compiler")

func CriticalOperation() (err error) {
    defer func() {
        if r := recover(); r != nil {
            log.Printf("Panic recovered: %v\nStack: %s", r, debug.Stack())
            err = fmt.Errorf("internal error during compilation: %v. This is a bug - please report it", r)
        }
    }()
    
    // Critical operation that might panic
    return doWork()
}

Add panic recovery to these critical entry points:

1. pkg/workflow/compiler_orchestrator.go - CompileWorkflow()
2. pkg/workflow/mcp-config.go - GenerateMCPConfig()
3. pkg/cli/compile_command.go - RunCompile()
4. pkg/cli/run_command.go - RunWorkflow()

Ensure recovered panics:

• Are logged with stack traces for debugging
• Return user-friendly error messages
• Don't silently swallow panics (always log)
• Include "this is a bug" messaging for users to report

---

## 📊 Historical Context

<details>
<summary><b>Previous Focus Areas</b></summary>

| Date | Focus Area | Type | Custom | Key Outcomes |
|------|------------|------|--------|--------------|
| 2026-01-06 | Testing | Standard | N | Identified 2.24:1 test ratio, 320 skipped tests, 20 critical files without tests |
| 2026-01-05 | Engine Configuration Best Practices | Custom | Y | Found 32.6% workflows missing engine config, 15 invalid values, documented patterns |
| 2026-01-07 | **Error Experience Engineering** | **Custom** | **Y** | **Console formatting gap (3%), 178 raw errors, 80 context-loss errors, 0 panic recovery** |

</details>

---

## 🎯 Recommendations

### Immediate Actions (This Week)
1. **Task 1**: Console format all CLI error messages - High priority, affects all users immediately
2. **Task 2**: Add error context wrapping - High priority, improves debugging capabilities

### Short-term Actions (This Month)
1. **Task 3**: Implement debug logging for critical paths - Enables better troubleshooting
2. **Task 4**: Create error recovery patterns documentation - Standardizes approach across codebase

### Long-term Actions (This Quarter)
1. **Task 5**: Add panic recovery mechanisms - Improves stability and resilience
2. Create error catalog with error codes for programmatic handling
3. Implement structured error logging (JSON) for automated analysis
4. Build error pattern detection system for proactive monitoring

---

## 📈 Success Metrics

Track these metrics to measure improvement in **Error Experience Engineering**:

- **Console Formatting Coverage**: 3% → 100% (all error outputs use console.FormatErrorMessage)
- **Error Context Preservation**: 55.8% → 95% (reduce errors.New from 80 to <20)
- **Debug Logging Coverage**: 11 instances → 50+ instances (critical error paths)
- **Panic Recovery**: 0 → 4+ critical entry points protected
- **Documentation**: 1 doc → 4 docs (error recovery patterns, error catalog, debugging guide, runbooks)

**Validation Commands**:
```bash
# Console formatting coverage
grep -r "console.FormatErrorMessage" --include="*.go" pkg/ cmd/ | wc -l

# Error wrapping ratio
grep -r "fmt.Errorf.*%w" --include="*.go" pkg/ cmd/ | wc -l
grep -r "errors.New" --include="*.go" pkg/ cmd/ | wc -l

# Debug logging for errors
grep -r "log.Printf.*err" --include="*.go" pkg/ cmd/ | wc -l

# Panic recovery
grep -r "defer.*recover" --include="*.go" pkg/ | wc -l

---

Next Steps

1. Planner Agent: Split the 5 tasks above into individual work items for Claude execution
2. Priority Order: Execute Task 1 → Task 2 → Task 3 → Task 4 → Task 5
3. Validation: Run make test and make lint after each task
4. Documentation: Update specs/error-recovery-patterns.md as part of Task 4
5. Re-evaluation: Measure success metrics after all tasks completed (target: 2 weeks)
6. Next Analysis: 2026-01-08 - Focus area will be selected based on diversity algorithm

---

Custom Focus Area Rationale:

Error Experience Engineering was selected as a custom focus area because gh-aw is a workflow compilation and execution platform where errors are the primary interface during development. Unlike typical applications where errors are edge cases, gh-aw developers constantly encounter validation errors, configuration errors, and compilation errors. The quality of these error messages directly impacts:

• Time to resolution: How quickly developers fix workflow issues
• Learning curve: How easily new users understand workflow requirements
• Debugging efficiency: How effectively issues can be diagnosed
• User confidence: How much users trust the platform's guidance

This analysis revealed a significant gap between the excellent error handling foundation (55.8% wrapped errors, 295 actionable validations) and the inconsistent user experience (only 3% console formatted). By treating "error experience" holistically - covering visibility, actionability, debugging, and recovery - we can transform errors from frustrating blockers into helpful guides.

The 5 tasks prioritize immediate user-facing improvements (console formatting) → structural improvements (context preservation) → operational improvements (debug logging) → documentation → resilience (panic recovery), providing a clear path to dramatically better developer experience.

---

Generated by Repository Quality Improvement Agent
Next analysis: 2026-01-08 - Focus area will be selected based on diversity algorithm

AI generated by Repository Quality Improvement Agent

[pelikhan]

/plan

[github-actions[bot]]

🎉 Yo ho ho! Plan Command found the treasure and completed successfully! ⚓💰