🎯 Repository Quality Improvement Report - Validation Message Clarity & Developer Guidance

[github-actions[bot]]

Analysis Date: 2026-01-22
Focus Area: Validation Message Clarity & Developer Guidance
Strategy Type: Custom
Custom Area: Yes - Repository-specific focus on how validation errors guide developers to fix workflow configuration issues across 87 validation files and 22,311 LOC of validation logic

Executive Summary

gh-aw's extensive validation system (87 files, 22,311 LOC) provides comprehensive error detection across workflow compilation, but validation message quality and developer guidance varies significantly. While some validators provide excellent actionable guidance with examples (engine_validation.go, mcp_config_validation.go, github_toolset_validation_error.go), many validation errors lack the clarity and context needed for developers to quickly resolve issues.

Key Finding: Only 15.2% (5/33) of validation error tests validate actual message content, indicating validation messages are not treated as a first-class concern. With 127 fmt.Errorf calls creating user-facing errors, this represents a significant opportunity to improve developer experience through clearer, more actionable validation messages with consistent examples, documentation references, and fix suggestions.

The specialized GitHubToolsetValidationError demonstrates the gold standard for validation errors: structured output, sorted details, actionable fix suggestions, and comprehensive context. However, this pattern is rarely replicated across other validators, leading to inconsistent developer experience.

Full Analysis Report

Focus Area: Validation Message Clarity & Developer Guidance

Current State Assessment

The repository has a sophisticated validation architecture with 87 validation-related files totaling 22,311 lines of code. Validation is deeply integrated into workflow compilation, MCP configuration, safe outputs, and CLI commands.

Metrics Collected:

Metric	Value	Status
Total validation files	87	✅
Total validation LOC	22,311	✅
Validation test files	51	✅
Validation test functions	225	✅
fmt.Errorf error sites	127	⚠️
errors.New error sites	5	✅
ValidationError types	83	⚠️
Error messages with "should" guidance	194	⚠️
Error messages with "must" guidance	60	⚠️
Error messages with examples	37	❌
Error messages with URL references	13	❌
Console formatted validation errors	0	❌
Tests validating specific error content	5 (15.2%)	❌
Tests with generic error checks	28 (84.8%)	⚠️
Error message constants/templates	0	❌

Findings

Strengths

• Comprehensive validation coverage: 87 validation files covering all major workflow aspects (compilation, MCP, safe outputs, engines, permissions)
• Excellent testing: 51 test files with 225 test functions provide strong validation coverage
• Best-in-class examples: engine_validation.go, mcp_config_validation.go, and github_toolset_validation_error.go demonstrate excellent error messages with examples, actionable guidance, and fix suggestions
• Specialized error types: GitHubToolsetValidationError provides structured, sorted, actionable error output - a model for other validators
• Contextual error wrapping: 61 errors use %s/%v/%d formatting to include context (though many could use %w wrapping)

Areas for Improvement

1. Inconsistent Error Message Quality (HIGH SEVERITY)

• Only 29% (37/127) of validation errors include examples
• Only 10% (13/127) reference documentation
• 0 validation errors use console formatting despite CLI context
• Error message quality varies dramatically between validators

2. Insufficient Test Validation of Error Content (HIGH SEVERITY)

• Only 15.2% (5/33) error assertions validate actual message content
• 84.8% (28/33) tests use generic assert.Error without checking message quality
• No systematic testing of error message clarity, examples, or actionable guidance
• Validation messages can degrade without detection

3. No Error Message Style Guide (MEDIUM SEVERITY)

• No validation-specific error message documentation in skills/ directory
• No templates or patterns for common validation error scenarios
• Developers creating new validators lack guidance on message quality
• Inconsistent structure across validators (some multi-line, some single-line)

4. Zero Message Reusability (MEDIUM SEVERITY)

• 0 error message constants or templates despite common patterns
• Each validator reinvents error formatting
• Common scenarios (missing field, invalid type, configuration conflict) lack reusable message components
• Opportunity to create validation error message library

5. Limited Actionable Guidance (MEDIUM SEVERITY)

• Only 60 instances of "must" guidance (directive)
• Only 194 instances of "should" guidance (recommendation)
• Many errors describe the problem without explaining how to fix it
• Missing "What to do next" or "Common causes" sections in error output

Detailed Analysis

Gold Standard Examples

engine_validation.go demonstrates excellent error messages:

return fmt.Errorf("invalid engine: %s. Valid engines are: copilot, claude, codex, custom. Example: engine: copilot", engineID)

✅ Strengths: Lists valid options, provides concrete example, clear error description

mcp_config_validation.go provides comprehensive guidance:

return fmt.Errorf("tool '%s' mcp configuration missing required property '%s'. Example:\nmcp-servers:\n  %s:\n    %s: \"value\"", toolName, propertyName, toolName, propertyName)

✅ Strengths: Multi-line YAML example, contextual placeholders, clear structure

github_toolset_validation_error.go sets the bar:

• Structured error type with detailed context
• Sorted output for consistency
• Actionable fix suggestion with complete YAML example
• Clear explanation of why error occurred
• Maps toolsets to specific tools requiring them

Representative of Common Patterns (Need Improvement)

Many validation files follow simpler patterns:

return fmt.Errorf("field '%s' has invalid value: %v", field, value)

❌ Weaknesses: No example, no valid options listed, no fix suggestion, generic message

Testing Gaps

Most validation tests use generic error checking:

err := ValidateWorkflow(input)
assert.Error(t, err) // ❌ Doesn't validate message quality

Better pattern (rare, only 5 instances):

err := ValidateWorkflow(input)
assert.Error(t, err)
assert.Contains(t, err.Error(), "Example:")
assert.Contains(t, err.Error(), "copilot")

CLI Integration Gap

Validation errors flow through CLI commands but never use console formatting:

• 0 console.FormatErrorMessage calls in validation files
• 22 direct stderr writes bypass console formatting system
• Users see plain text errors instead of styled, accessible output
• Inconsistent with CLI command error handling patterns

---

🤖 Tasks for Copilot Agent

NOTE TO PLANNER AGENT: The following tasks are designed for GitHub Copilot agent execution. Please split these into individual work items for Claude to process. Each task focuses on improving validation message clarity, consistency, and developer guidance.

Improvement Tasks

Task 1: Create Validation Error Message Style Guide

Priority: High
Estimated Effort: Medium
Focus Area: Documentation

Description:
Create a comprehensive style guide (skills/validation-error-messages.md) documenting best practices for writing clear, actionable validation error messages. This guide should:

1. Define the anatomy of an excellent validation error (problem statement, valid options, example, fix suggestion)
2. Provide templates for common scenarios (missing field, invalid type, configuration conflict, permission error)
3. Include before/after examples from actual gh-aw validators
4. Establish testing requirements (every error message must have test validating content)
5. Document the gold standard examples (engine_validation.go, github_toolset_validation_error.go)
6. Explain when to use specialized error types vs. simple fmt.Errorf

Acceptance Criteria:

• New file skills/validation-error-messages.md created
• Includes 5+ before/after examples from actual codebase
• Provides reusable templates for 7+ common error scenarios
• Documents testing requirements for validation errors
• References GitHubToolsetValidationError as the gold standard
• Includes decision tree for when to create specialized error types

Code Region: skills/validation-error-messages.md (new file)

Create a comprehensive validation error message style guide at skills/validation-error-messages.md

This skill should help developers write clear, actionable validation error messages across the 87 validation files in gh-aw. Focus on:

1. **Message Anatomy**: Define what makes an excellent validation error
   - Problem statement (what's wrong)
   - Context (which field, which value)
   - Valid options (what's allowed)
   - Concrete example (how to fix it)
   - Documentation reference (optional, for complex cases)

2. **Templates for Common Scenarios**:
   - Missing required field
   - Invalid field type
   - Invalid field value (with valid options)
   - Configuration conflict (multiple fields)
   - Permission/capability not available
   - Deprecated field usage
   - Complex validation failure (multiple issues)

3. **Before/After Examples** from actual codebase:
   - Good: engine_validation.go line 28
   - Good: mcp_config_validation.go line 47
   - Gold standard: github_toolset_validation_error.go (entire file)
   - Needs improvement: Find 3 examples that could be enhanced

4. **Testing Requirements**:
   - Every validation error message MUST have test checking content
   - Use assert.Contains() to verify examples present
   - Use assert.Contains() to verify valid options listed
   - Target: 80%+ of error tests validate specific content

5. **Specialized Error Types**:
   - When to create custom error type (like GitHubToolsetValidationError)
   - When simple fmt.Errorf is sufficient
   - How to implement Error() method with structured output

6. **Console Formatting Integration**:
   - How validation errors should integrate with console formatting
   - Where formatting happens (validation layer vs CLI layer)
   - Current gap: 0 console.FormatErrorMessage calls in validation

Use real examples from the codebase to illustrate patterns.

---

Task 2: Enhance Top 5 Validators with Gold Standard Error Messages

Priority: High
Estimated Effort: Large
Focus Area: Code Quality

Description:
Improve error messages in the 5 most critical validation files that currently lack comprehensive guidance. Target files:

1. pkg/workflow/safe_outputs_target_validation.go (complex validation, critical feature)
2. pkg/workflow/secrets_validation.go (security-sensitive, high user confusion risk)
3. pkg/workflow/template_injection_validation.go (security-critical, complex rules)
4. pkg/workflow/strict_mode_validation.go (workflow behavior changes, needs clear guidance)
5. pkg/workflow/docker_validation.go (container configuration, many options)

For each file:

• Add concrete YAML examples to all error messages
• List valid options where applicable
• Include fix suggestions for common mistakes
• Add documentation references for complex validations
• Ensure multi-line formatting for complex examples

Acceptance Criteria:

• 100% of error messages in these 5 files include examples
• All error messages list valid options (where applicable)
• Complex errors use multi-line formatting with proper indentation
• Added tests validating error message content (assert.Contains for examples)
• Each file has before/after examples documented in commit message
• Run make agent-finish successfully after changes

Code Region:

• pkg/workflow/safe_outputs_target_validation.go
• pkg/workflow/secrets_validation.go
• pkg/workflow/template_injection_validation.go
• pkg/workflow/strict_mode_validation.go
• pkg/workflow/docker_validation.go

Enhance validation error messages in 5 critical validation files to match the gold standard set by engine_validation.go and github_toolset_validation_error.go

For each of these files:
1. pkg/workflow/safe_outputs_target_validation.go
2. pkg/workflow/secrets_validation.go
3. pkg/workflow/template_injection_validation.go
4. pkg/workflow/strict_mode_validation.go
5. pkg/workflow/docker_validation.go

**Transformation Pattern**:

BEFORE:
```go
return fmt.Errorf("invalid value: %v", value)
```

AFTER:
```go
return fmt.Errorf("invalid value for field 'x': %v. Valid values are: a, b, c. Example:\nfield: a", value)
```

**Requirements**:
- Every error must include concrete example
- List valid options when there's a constrained set
- Use multi-line formatting for YAML examples
- Add fix suggestions for common mistakes
- Include documentation references for complex validations

**Testing**:
- Add test cases validating error message content
- Use assert.Contains(t, err.Error(), "Example:")
- Use assert.Contains(t, err.Error(), "Valid values")

Refer to skills/validation-error-messages.md for templates and patterns.
Run `make agent-finish` before committing.

---

Task 3: Implement Validation Error Message Testing Standard

Priority: High
Estimated Effort: Large
Focus Area: Testing

Description:
Systematically improve validation error tests to verify message content, not just error presence. Current state: Only 15.2% (5/33) tests validate actual error messages.

Target improvements:

1. Add error message content assertions to existing tests lacking them
2. Create test helper function assertValidationError(t, err, requirements) that checks:
   • Error is non-nil
   • Contains "Example:" or similar guidance
   • Contains expected keywords (field names, valid options)
   • Contains fix suggestions
3. Update top 10 validation test files to use new standard
4. Document pattern in specs/testing.md

Acceptance Criteria:

• New test helper assertValidationError() in validation_helpers_test.go
• Helper validates: error presence, example inclusion, keyword presence
• At least 50 tests updated to use new helper or equivalent assertions
• Target: 60%+ of validation error tests validate message content (up from 15.2%)
• New section in specs/testing.md documenting validation error testing
• All updated tests pass with make test-unit

Code Region:

• pkg/workflow/validation_helpers_test.go (add helper)
• pkg/workflow/*_validation_test.go (update tests)
• pkg/cli/*_validation_test.go (update tests)
• specs/testing.md (add documentation)

Improve validation error testing to treat error messages as first-class concerns requiring explicit validation

**Goal**: Increase error message content validation from 15.2% to 60%+ of tests

**Step 1: Create Test Helper**

Add to pkg/workflow/validation_helpers_test.go:

```go
// assertValidationError verifies a validation error contains expected guidance
func assertValidationError(t *testing.T, err error, checks ValidationErrorChecks) {
    t.Helper()
    require.Error(t, err, checks.Context)
    
    if checks.RequireExample {
        assert.Contains(t, err.Error(), "Example:", "%s: error should include example", checks.Context)
    }
    
    if len(checks.RequireKeywords) > 0 {
        for _, keyword := range checks.RequireKeywords {
            assert.Contains(t, err.Error(), keyword, "%s: error should mention '%s'", checks.Context, keyword)
        }
    }
    
    if checks.RequireValidOptions {
        assert.Contains(t, err.Error(), "Valid", "%s: error should list valid options", checks.Context)
    }
}

type ValidationErrorChecks struct {
    Context           string   // Test context for assertions
    RequireExample    bool     // Error must include "Example:"
    RequireKeywords   []string // Error must contain these keywords
    RequireValidOptions bool   // Error must list valid options
}
```

**Step 2: Update Existing Tests**

Priority files to update (highest impact):
1. pkg/workflow/engine_validation_test.go (already good, use as reference)
2. pkg/workflow/mcp_config_validation_test.go
3. pkg/workflow/safe_outputs_target_validation_test.go
4. pkg/workflow/secrets_validation_test.go
5. pkg/workflow/template_injection_validation_test.go

Pattern:
```go
// OLD
assert.Error(t, err)

// NEW
assertValidationError(t, err, ValidationErrorChecks{
    Context: "missing required field",
    RequireExample: true,
    RequireKeywords: []string{"required", "field"},
})
```

**Step 3: Document in specs/testing.md**

Add new section: "### Testing Validation Errors"

Run `make test-unit` to verify all changes.

---

Task 4: Create Reusable Validation Error Message Components

Priority: Medium
Estimated Effort: Medium
Focus Area: Code Organization

Description:
Extract common validation error message patterns into reusable components to ensure consistency and reduce duplication. Create a new file pkg/workflow/validation_messages.go with:

1. Message builder helpers for common scenarios
2. Standard format strings for recurring patterns
3. Example generators for common field types
4. Documentation reference formatters

Benefits:

• Ensures consistent error message structure
• Reduces code duplication across 87 validation files
• Makes it easier to improve all error messages at once
• Simplifies creating new validators

Acceptance Criteria:

• New file pkg/workflow/validation_messages.go created
• At least 7 reusable message builders implemented
• Functions cover: missing field, invalid type, invalid value, configuration conflict, example formatting
• At least 3 existing validators refactored to use new helpers
• Documentation in file explaining when to use each helper
• Tests for message builders in validation_messages_test.go
• Run make agent-finish successfully

Code Region:

• pkg/workflow/validation_messages.go (new file)
• pkg/workflow/validation_messages_test.go (new file)
• Update 3 existing validators to demonstrate usage

Create reusable validation error message components in pkg/workflow/validation_messages.go

**Objective**: Provide consistent, reusable error message patterns to improve consistency across 87 validation files

**Message Builders to Implement**:

```go
package workflow

// MissingRequiredFieldError creates an error for a missing required field
func MissingRequiredFieldError(fieldName, example string) error

// InvalidFieldTypeError creates an error for incorrect field type
func InvalidFieldTypeError(fieldName string, expected, actual string, example string) error

// InvalidFieldValueError creates an error for invalid field value with valid options
func InvalidFieldValueError(fieldName string, value any, validOptions []string, example string) error

// ConfigurationConflictError creates an error for mutually exclusive fields
func ConfigurationConflictError(fields []string, resolution string, example string) error

// FormatYAMLExample formats a YAML example with proper indentation
func FormatYAMLExample(yamlString string) string

// FormatValidOptions formats a list of valid options for error messages
func FormatValidOptions(options []string) string

// WithDocReference adds a documentation reference to an error
func WithDocReference(err error, docURL string) error
```

**Example Usage**:

```go
// Instead of:
return fmt.Errorf("missing field 'engine'")

// Use:
return MissingRequiredFieldError("engine", "engine: copilot")
```

**Refactor Examples**:
1. Update pkg/workflow/runtime_validation.go to use new helpers
2. Update pkg/workflow/npm_validation.go to use new helpers
3. Update pkg/workflow/pip_validation.go to use new helpers

**Testing**:
- Test each message builder returns expected format
- Test YAML example formatting preserves structure
- Test valid options formatting with various inputs

Run `make agent-finish` after implementation.

---

Task 5: Integrate Console Formatting with Validation Error Output

Priority: Medium
Estimated Effort: Medium
Focus Area: User Experience

Description:
Bridge the gap between validation errors (currently plain text) and CLI console formatting. Currently 0 validation errors use console formatting, leading to inconsistent user experience.

Approach:

1. Validation layer keeps plain errors (testable, reusable)
2. CLI command layer wraps validation errors with console formatting
3. Create helper in pkg/cli/validation_output.go to format validation errors
4. Update CLI commands to use helper when displaying validation errors

Design principle: Separation of concerns - validation logic generates structured errors, CLI layer formats for human consumption.

Acceptance Criteria:

• New file pkg/cli/validation_output.go with FormatValidationError helper
• Helper detects error types (ValidationError, standard error) and formats appropriately
• At least 5 CLI commands updated to use new formatting helper
• Validation errors now displayed with console.FormatErrorMessage styling
• Complex validation errors (like GitHubToolsetValidationError) preserve structure
• Tests verify formatting doesn't alter error content
• Run make agent-finish successfully

Code Region:

• pkg/cli/validation_output.go (new file)
• pkg/cli/compile_command.go (update)
• pkg/cli/run_command.go (update)
• pkg/cli/mcp_inspect_command.go (update)
• pkg/cli/audit_command.go (update)

Integrate console formatting with validation error output to provide consistent, styled error display

**Architecture**:
- Validation layer: Plain text errors (testable, reusable, framework-agnostic)
- CLI layer: Console formatting wrapper (user-facing, styled, accessible)

**Implementation**:

Create pkg/cli/validation_output.go:

```go
package cli

import (
    "fmt"
    "os"
    "strings"
    
    "github.com/githubnext/gh-aw/pkg/console"
)

// FormatValidationError formats validation errors for console output
// Preserves structured error content while applying console styling
func FormatValidationError(err error) string {
    if err == nil {
        return ""
    }
    
    errMsg := err.Error()
    
    // Detect multi-line structured errors (like GitHubToolsetValidationError)
    if strings.Contains(errMsg, "\n\n") || strings.HasPrefix(errMsg, "ERROR:") {
        // Preserve structure, apply styling to header only
        return console.FormatErrorMessage(errMsg)
    }
    
    // Standard single-line or simple multi-line error
    return console.FormatErrorMessage(errMsg)
}

// PrintValidationError prints a validation error to stderr with formatting
func PrintValidationError(err error) {
    if err == nil {
        return
    }
    fmt.Fprintln(os.Stderr, FormatValidationError(err))
}
```

**Update CLI Commands**:

Pattern:
```go
// OLD
if err := ValidateWorkflow(config); err != nil {
    fmt.Fprintln(os.Stderr, err)
    return err
}

// NEW
if err := ValidateWorkflow(config); err != nil {
    PrintValidationError(err)
    return err
}
```

**Commands to Update**:
1. pkg/cli/compile_command.go - RunCompileWorkflow
2. pkg/cli/run_command.go - RunRunWorkflow
3. pkg/cli/mcp_inspect_command.go - inspect operations
4. pkg/cli/audit_command.go - validation failures
5. pkg/cli/compile_validation.go - all validation error prints

**Testing**:
- Verify styling applied correctly
- Verify structured errors preserve formatting
- Verify error content unchanged

Run `make agent-finish` after implementation.

---

📊 Historical Context

Previous Focus Areas

Date	Focus Area	Type	Custom	Key Outcomes
2026-01-05	Engine Configuration Best Practices	Custom	Y	Identified 32.6% workflows missing engine config, created decision guide
2026-01-06	Testing	Standard	N	Found 2.24:1 test ratio, 320 skipped tests, 20 critical files without tests
2026-01-07	Error Experience Engineering	Custom	Y	Only 3% errors use console formatting, 178 raw error outputs identified
2026-01-08	Security	Standard	N	98.6% workflows with permissions, 99.8% actions pinned, no CodeQL/gosec
2026-01-16	Workflow Compilation Performance	Custom	Y	177 workflows sequential, 4,787 validation LOC per compile, no caching
2026-01-20	Command Interface Consistency	Custom	Y	Only 12% commands use RunX pattern, 0% have 3+ examples, 71% lack Config
2026-01-21	Dependencies	Standard	N	278 dependencies, 48.2% on v0.x, zero vulnerability scanning, Dependabot inactive

---

🎯 Recommendations

Immediate Actions (This Week)

1. Create validation error message style guide (skills/validation-error-messages.md) - Priority: High
2. Enhance top 5 critical validators with gold standard error messages - Priority: High

Short-term Actions (This Month)

1. Implement validation error testing standard - Increase from 15.2% to 60%+ tests validating message content
2. Create reusable validation error message components - Reduce duplication, ensure consistency
3. Integrate console formatting with validation errors - Provide consistent CLI user experience

Long-term Actions (This Quarter)

1. Audit all 87 validation files for error message quality using new style guide
2. Create validation error message linter to enforce standards in CI
3. Add validation error examples to documentation showing before/after improvements

---

📈 Success Metrics

Track these metrics to measure improvement in Validation Message Clarity & Developer Guidance:

• Error Message Quality: 29% → 80%+ errors include examples
• Documentation References: 10% → 40% errors reference docs where appropriate
• Test Coverage of Messages: 15.2% → 60%+ tests validate error content
• Console Formatting: 0% → 100% CLI validation errors use console formatting
• Message Reusability: 0 → 7+ reusable message builder functions
• Developer Time to Fix: Track time from validation error to successful compile (baseline TBD)

---

Next Steps

1. Planner Agent: Split the 5 tasks above into individual work items for Claude
2. Priority Order: Task 1 (style guide) → Task 2 (enhance validators) → Task 3 (testing) → Task 4 (reusability) → Task 5 (console formatting)
3. Validation: Each task includes acceptance criteria and make agent-finish requirement
4. Tracking: Monitor test coverage of error messages as key success indicator
5. Next Analysis: 2026-01-23 - Focus area will be selected based on diversity algorithm (target: custom area to maintain 60%+ rate)

---

References:

• Workflow Run §21250100692

---

Generated by Repository Quality Improvement Agent
Strategy: Custom Focus Area (60% target - 7/8 runs = 87.5% custom rate)
Next analysis: 2026-01-23 - Diversity algorithm will prioritize either custom area or standard category

AI generated by Repository Quality Improvement Agent

• expires on Jan 29, 2026, 1:30 PM UTC

[github-actions[bot]]

This discussion was automatically closed because it expired on 2026-01-29T13:30:55.859Z.