🎯 Repository Quality Improvement: Validation Error Experience

[github-actions[bot]]

🎯 Repository Quality Improvement Report - Validation Error Experience

Analysis Date: 2026-01-19
Focus Area: Validation Error Experience
Strategy Type: Custom
Custom Area: Yes - This focus area is specific to gh-aw's core function as a workflow compiler with extensive validation

Executive Summary

gh-aw has invested significantly in validation infrastructure (83 validation files, 20,492 LOC, 205 test cases), demonstrating a strong commitment to workflow correctness. The repository includes sophisticated error types (ValidationError, ConfigurationError, OperationError) with structured fields for suggestions and context. However, analysis reveals that while the validation infrastructure is excellent, the user-facing error experience has inconsistencies that could impact developer productivity when workflows fail validation.

Key Findings:

• ✅ Strong foundation: 863 error wrapping usages, dedicated error quality tests, helper functions
• ⚠️ Only 1 validation error with explicit suggestions out of 88 ValidationError usages (1% suggestion rate)
• ⚠️ Inconsistent console formatting: 42 uses of console.FormatErrorMessage vs 1,122 direct fmt.Fprintln calls in CLI
• ⚠️ Only 42% of errors are multi-line (37/88), suggesting some may be too terse
• ⚠️ Many validation errors bypass structured types, using plain fmt.Errorf instead of ValidationError

The validation system is technically sound, but error message consistency, actionability, and discoverability could be significantly improved to reduce user friction when debugging failed workflows.

Full Analysis Report

Focus Area: Validation Error Experience

Current State Assessment

gh-aw's validation system is one of the most comprehensive in the repository, reflecting the critical importance of ensuring workflows compile correctly before execution. The analysis examined error message quality, consistency, and user-facing presentation across 83 validation files.

Metrics Collected:

Metric	Value	Status
Validation files	83	✅ Excellent
Total validation LOC	20,492	✅ Comprehensive
Validation test files	48	✅ Well-tested
Test cases	205	✅ Strong coverage
Error wrapping usage	863	✅ Good practice
ValidationError usages	88	✅ Infrastructure exists
Errors with suggestions	1	❌ 1% rate - Critical gap
Errors with examples	12	⚠️ 14% rate - Could improve
Console-formatted CLI errors	42	⚠️ Low vs 1,122 direct prints
Multi-line errors	37 (42%)	⚠️ Many may be terse
Errors with docs links	84	✅ Good documentation

Findings

Strengths

1. Sophisticated Error Infrastructure

   • Well-designed error types (ValidationError, ConfigurationError, OperationError) with dedicated fields for suggestions and context
   • Helper functions for common validations (ValidateRequired, ValidateMaxLength, ValidateInList, etc.)
   • Error wrapping used extensively (863 instances) for good error context chains

2. Quality Testing

   • Dedicated error_message_quality_test.go that validates error messages contain examples and aren't too vague
   • 48 validation test files with 205 test cases
   • Tests check for specific error content patterns

3. MCP Validation Leadership

   • MCP-related validation errors consistently include examples, valid values, and proper formatting
   • Example: "tool '%s' mcp configuration property '%s' must be a string, got %T. Example:\nmcp-servers:\n %s:\n %s: \"my-value\""

4. Troubleshooting Documentation

   • Dedicated troubleshooting docs exist (copilot-schema-validation-error.md, copilot-schema-validation-deep-analysis.md)

Areas for Improvement

1. Suggestion Underutilization (Critical)

   • Problem: Despite ValidationError having a Suggestion field, only 1 out of 88 usages (1%) provides explicit suggestions
   • Impact: Users receive errors describing what's wrong but not how to fix it
   • Root Cause: Many validation errors use plain fmt.Errorf instead of structured ValidationError type

2. Console Formatting Inconsistency

   • Problem: CLI has 42 console.FormatErrorMessage calls but 1,122 direct fmt.Fprintln(os.Stderr calls
   • Impact: Validation errors may not be visually consistent or properly styled for terminal output
   • Example: Some validation errors are printed directly without console formatting, reducing readability

3. Error Terseness

   • Problem: Only 42% of errors (37/88) are multi-line, suggesting many may lack sufficient detail
   • Impact: Single-line errors may not provide enough context for users to understand and fix issues
   • Example: Errors like "invalid engine" without listing valid options

4. Bypass of Structured Error Types

   • Problem: Many validation errors use fmt.Errorf directly instead of NewValidationError() or similar constructors
   • Impact: Loses benefits of structured error fields (Field, Value, Reason, Suggestion, Timestamp)
   • Example: return fmt.Errorf("invalid mount at index %d", i) instead of using ValidationError

5. README Validation Guidance

   • Problem: README has only 1 mention of "validation" or "error"
   • Impact: New users may not know how to interpret or debug validation errors
   • Recommendation: Add "Troubleshooting Validation Errors" section to README

Detailed Analysis

Error Message Quality Patterns

Excellent Example (MCP Validation):

return fmt.Errorf("tool '%s' mcp configuration 'type' must be a string, got %T. Valid types per MCP Gateway Specification: stdio, http. Note: 'local' is accepted for backward compatibility and treated as 'stdio'. Example:\nmcp-servers:\n  %s:\n    type: \"stdio\"\n    command: \"node server.js\"", toolName, mcpType, toolName)

✅ Shows what's wrong, why it's wrong, valid values, and a concrete example

Room for Improvement (Sandbox Validation):

return fmt.Errorf("invalid mount at index %d: mode must be 'ro' (read-only) or 'rw' (read-write), got '%s' in '%s'", i, mode, mount)

⚠️ Good but could benefit from structured ValidationError with suggestion field

Inconsistent Pattern:

// Some files use structured errors
return NewValidationError(field, value, reason, suggestion)

// Others use plain fmt.Errorf
return fmt.Errorf("error: %w", err)

Test Coverage vs Error Quality

The error_message_quality_test.go file demonstrates awareness of error quality importance, testing that errors:

• Contain "Example:" sections
• Are not too vague (>30 chars, >5 words)
• Show actual vs expected values
• Include valid options for enums

However, these quality standards aren't consistently applied across all 83 validation files.

---

🤖 Tasks for Copilot Agent

NOTE TO PLANNER AGENT: The following tasks are designed for GitHub Copilot agent execution. Please split these into individual work items for Claude to process.

Improvement Tasks

The following code regions and tasks should be processed by the Copilot agent. Each section is marked for easy identification by the planner agent.

---

Task 1: Standardize Validation Errors to Use Structured Types

Priority: High
Estimated Effort: Large
Focus Area: Validation Error Infrastructure

Description:
Convert validation errors across the codebase from plain fmt.Errorf to structured error types (ValidationError, ConfigurationError, OperationError). This ensures all validation errors include proper suggestions, context, and formatting.

Acceptance Criteria:

• Identify all validation errors using plain fmt.Errorf in validation files
• Convert to appropriate structured error type (ValidationError, ConfigurationError, or OperationError)
• Ensure all structured errors include actionable suggestions in the Suggestion field
• Update error messages to include examples where helpful
• Maintain backward compatibility with existing error message content
• All tests pass after conversion

Code Region: pkg/workflow/*validation*.go

Review all validation files in pkg/workflow/ that end with `validation.go` or contain validation logic. For each error that uses `fmt.Errorf` or plain `errors.New`, convert it to the appropriate structured error type:

1. For field-level validation errors: Use `NewValidationError(field, value, reason, suggestion)`
2. For configuration errors: Use `NewConfigurationError(configKey, value, reason, suggestion)`
3. For operation failures: Use `NewOperationError(operation, entityType, entityID, cause, suggestion)`

Ensure every structured error includes:
- A clear reason explaining what's wrong
- An actionable suggestion for how to fix it (this is critical - currently only 1/88 errors have suggestions)
- Examples in the suggestion when helpful (especially for enum values, format requirements)

Priority order:
1. Start with sandbox_validation.go (has many plain fmt.Errorf calls)
2. Then safe_outputs_domains_validation.go
3. Then remaining validation files

Example transformation:
```go
// BEFORE
return fmt.Errorf("invalid mount at index %d: mode must be 'ro' or 'rw', got '%s'", i, mode)

// AFTER
return NewValidationError(
    fmt.Sprintf("sandbox.mounts[%d].mode", i),
    mode,
    "mount mode must be 'ro' (read-only) or 'rw' (read-write)",
    "Change the mount mode to either 'ro' or 'rw'. Example:\nsandbox:\n  mounts:\n    - \"/host/path:/container/path:ro\"",
)

---

#### Task 2: Add Console Formatting to CLI Validation Error Display

**Priority**: High  
**Estimated Effort**: Medium  
**Focus Area**: User Interface Consistency

**Description:**
Ensure all validation errors displayed in CLI commands use consistent console formatting (`console.FormatErrorMessage`). Currently, only 42 errors use console formatting while 1,122 use direct `fmt.Fprintln`, leading to inconsistent visual presentation.

**Acceptance Criteria:**
- [ ] All validation errors in pkg/cli/ commands use `console.FormatErrorMessage`
- [ ] Replace direct `fmt.Fprintln(os.Stderr, err)` with `fmt.Fprintln(os.Stderr, console.FormatErrorMessage(err.Error()))`
- [ ] Maintain error message content and context
- [ ] Visual consistency across all CLI commands
- [ ] Test CLI output with sample validation errors

**Code Region:** `pkg/cli/*.go`

```markdown
Review all CLI command files in pkg/cli/ and identify where validation errors are printed to users. Standardize error display to use console formatting:

**Pattern to find and fix:**
```go
// BEFORE - Plain error printing
if err != nil {
    fmt.Fprintln(os.Stderr, err)
    return err
}

// AFTER - Console formatted
if err != nil {
    fmt.Fprintln(os.Stderr, console.FormatErrorMessage(err.Error()))
    return err
}

Priority files:

1. compile_command.go (main entry point for validation errors)
2. run_command.go
3. validate_command.go (if exists)
4. All other *_command.go files

Testing:
After changes, test with intentionally invalid workflows to verify error messages display with proper formatting (colors, structure, readability).

---

#### Task 3: Add Validation Error Troubleshooting Section to README

**Priority**: Medium  
**Estimated Effort**: Small  
**Focus Area**: Documentation and User Guidance

**Description:**
The README currently has minimal guidance on validation errors (1 mention). Add a comprehensive "Troubleshooting Validation Errors" section that helps users understand, interpret, and resolve common validation failures.

**Acceptance Criteria:**
- [ ] Add "Troubleshooting Validation Errors" section to README.md
- [ ] Include common validation error categories with examples
- [ ] Link to detailed troubleshooting documentation
- [ ] Provide quick debugging steps
- [ ] Include validation error format explanation
- [ ] Add examples of fixing common errors

**Code Region:** `README.md`

```markdown
Add a comprehensive "Troubleshooting Validation Errors" section to README.md after the "Quick Setup" or "Usage" section. Structure it as follows:

# Troubleshooting Validation Errors

When compiling workflows, gh-aw performs extensive validation to catch errors early. If your workflow fails validation, error messages follow this format:

[timestamp] Validation failed for field 'field-name'

Value: actual-value
Reason: explanation of what's wrong
Suggestion: how to fix it (with example if applicable)

## Common Validation Errors

### 1. Engine Configuration Errors
**Error**: `invalid engine 'xyz'`  
**Fix**: Use a valid engine: `copilot`, `claude`, `codex`, or `custom`  
**Example**:
```yaml
---
engine: copilot
---

2. MCP Server Configuration Errors

Error: tool 'xyz' missing required property 'command'
Fix: Specify either command or container for stdio-type MCP servers
Example:

---
mcp-servers:
  my-tool:
    command: "node server.js"
    args: ["--port", "3000"]
---

3. Safe Outputs Domain Validation

Error: invalid domain pattern
Fix: Use specific domain patterns (wildcards allowed)
Example:

---
safe-outputs:
  allowed-domains:
    - "*.github.com"
    - "https://api.example.com"
---

Detailed Troubleshooting

For in-depth troubleshooting guides, see:

• docs/troubleshooting/errors.md
• docs/troubleshooting/copilot-schema-validation-error.md

Debugging Tips

1. Run with verbose mode (if available): gh aw compile --verbose workflow.md
2. Check the error timestamp: Multiple errors may be related
3. Look for the Suggestion field: This tells you exactly how to fix the issue
4. Review examples: Error messages often include correct usage examples
5. Check workflow frontmatter: Most validation errors relate to frontmatter configuration

---

#### Task 4: Create Validation Error Catalog Documentation

**Priority**: Medium  
**Estimated Effort**: Medium  
**Focus Area**: Documentation and Error Discoverability

**Description:**
Create comprehensive documentation that catalogs all validation errors, their meanings, causes, and solutions. This serves as a reference guide for users debugging workflow validation failures.

**Acceptance Criteria:**
- [ ] Create `docs/troubleshooting/validation-error-catalog.md`
- [ ] Document all validation error categories (MCP, Safe Outputs, Sandbox, Engine, etc.)
- [ ] Include error message patterns, causes, and solutions for each
- [ ] Add search-friendly formatting
- [ ] Link from main troubleshooting docs
- [ ] Include examples of valid vs invalid configurations

**Code Region:** `docs/troubleshooting/` (new file: `validation-error-catalog.md`)

```markdown
Create a new file `docs/troubleshooting/validation-error-catalog.md` that comprehensively documents all validation errors by category:

# Validation Error Catalog

This catalog documents all validation errors that can occur during workflow compilation, organized by category.

## Table of Contents

1. [Engine Configuration Errors](#engine-configuration-errors)
2. [MCP Server Configuration Errors](#mcp-server-configuration-errors)
3. [Safe Outputs Validation Errors](#safe-outputs-validation-errors)
4. [Sandbox Configuration Errors](#sandbox-configuration-errors)
5. [Network Permissions Errors](#network-permissions-errors)
6. [Template and Expression Errors](#template-and-expression-errors)

---

## Engine Configuration Errors

### Invalid Engine Type

**Error Pattern**: `invalid engine 'xyz'`

**Cause**: The specified engine is not one of the supported engines.

**Supported Values**: `copilot`, `claude`, `codex`, `custom`

**Solution**:
```yaml
---
engine: copilot  # ✅ Valid
---

Invalid:

---
engine: gpt-4  # ❌ Invalid - use 'codex' or 'custom'
---

---

MCP Server Configuration Errors

Missing Required Property

Error Pattern: tool 'xyz' missing required property 'command'

Cause: MCP server configuration is missing a required field.

Solution: Specify either command (for host execution) or container (for containerized execution).

---
mcp-servers:
  my-server:
    command: "node server.js"  # ✅ Valid
    args: ["--port", "3000"]
---

Type Property Errors

Error Pattern: tool 'xyz' mcp configuration 'type' must be a string, got int

Cause: The type field must be a string value.

Valid Types: stdio, http, local (deprecated, treated as stdio)

Solution:

---
mcp-servers:
  my-server:
    type: "stdio"  # ✅ Must be string, not numeric
    command: "node server.js"
---

[Continue with all validation error categories...]

For each category:

1. Show error pattern (regex-like or example)
2. Explain what causes it
3. List valid values if applicable
4. Provide valid example
5. Show common mistakes (invalid examples)
6. Link to related documentation if exists

---

#### Task 5: Enhance Error Message Quality Tests

**Priority**: Low  
**Estimated Effort**: Small  
**Focus Area**: Test Coverage and Quality Assurance

**Description:**
Expand `error_message_quality_test.go` to cover more validation error types and enforce stricter quality standards. Currently tests cover ~10 scenarios; expand to cover all major validation categories.

**Acceptance Criteria:**
- [ ] Add test cases for all major validation categories (sandbox, safe-outputs, network, etc.)
- [ ] Enforce 100% suggestion rate for ValidationError types
- [ ] Add assertion that structured error types are used (not plain fmt.Errorf)
- [ ] Test that all errors include examples where appropriate
- [ ] Ensure errors reference documentation when available
- [ ] Increase test coverage from ~10 scenarios to 30+ scenarios

**Code Region:** `pkg/workflow/error_message_quality_test.go`

```markdown
Expand the error message quality test suite to comprehensively validate error quality across all validation categories:

**Add test cases for:**

1. **Sandbox Validation Errors**
   - Invalid mount syntax
   - Invalid mount mode
   - Missing source or destination
   - Feature flag not enabled

2. **Safe Outputs Domain Errors**
   - Empty domain
   - Invalid protocol
   - Wildcard-only pattern
   - Protocol mismatch

3. **Network Permission Errors**
   - Invalid domain pattern
   - Empty allowed domains
   - Conflicting permissions

4. **Template Validation Errors**
   - Invalid expression syntax
   - Template injection detection
   - Unsafe variable usage

5. **Runtime Configuration Errors**
   - Invalid runtime version
   - Unsupported runtime
   - Version format errors

**Enhanced quality assertions:**
```go
func TestValidationErrorQuality(t *testing.T) {
    tests := []struct {
        name                   string
        testFunc               func() error
        shouldContain          []string
        shouldHaveSuggestion   bool  // NEW: Enforce suggestions
        shouldHaveExample      bool  // NEW: Enforce examples
        shouldUseTStructuredError bool  // NEW: Check for ValidationError type
    }{
        // Test cases...
    }

    for _, tt := range tests {
        t.Run(tt.name, func(t *testing.T) {
            err := tt.testFunc()
            require.Error(t, err)

            // NEW: Check if error is structured type
            if tt.shouldUseStructuredError {
                _, isValidationErr := err.(*ValidationError)
                _, isConfigErr := err.(*ConfigurationError)
                _, isOpErr := err.(*OperationError)
                assert.True(t, isValidationErr || isConfigErr || isOpErr,
                    "Error should use structured type (ValidationError, ConfigurationError, or OperationError)")
            }

            // NEW: Check for suggestions
            if tt.shouldHaveSuggestion {
                assert.Contains(t, err.Error(), "Suggestion:",
                    "Error should include actionable suggestion")
            }

            // NEW: Check for examples
            if tt.shouldHaveExample {
                assert.Contains(t, err.Error(), "Example:",
                    "Error should include example of correct usage")
            }

            // Existing quality checks...
        })
    }
}

Goal: Increase test coverage from 10 to 30+ validation error scenarios, ensuring all errors meet quality standards.

---

## 📊 Historical Context

<details>
<summary><b>Previous Focus Areas</b></summary>

| Date | Focus Area | Type | Custom | Key Outcomes |
|------|------------|------|--------|--------------|
| 2026-01-19 | Validation Error Experience | Custom | Yes | First quality improvement run - identified validation error UX as high-impact custom area |

**Statistics:**
- Total runs: 1
- Custom rate: 100% (1/1)
- Reuse rate: 0% (0/1)
- Unique areas explored: 1

</details>

---

## 🎯 Recommendations

### Immediate Actions (This Week)
1. **Task 1**: Standardize validation errors to use structured types - Priority: High
2. **Task 2**: Add console formatting to CLI validation error display - Priority: High

### Short-term Actions (This Month)
1. **Task 3**: Add validation error troubleshooting section to README - Priority: Medium
2. **Task 4**: Create validation error catalog documentation - Priority: Medium

### Long-term Actions (This Quarter)
1. **Task 5**: Enhance error message quality tests - Priority: Low
2. Monitor validation error metrics: Track suggestion rate, console formatting adoption, user feedback on error clarity

---

## 📈 Success Metrics

Track these metrics to measure improvement in **Validation Error Experience**:

- **Suggestion Rate**: 1% → **80%+** (errors with actionable suggestions)
- **Console Formatting Rate**: 3.7% (42/1,122) → **90%+** (CLI errors with console formatting)
- **Multi-line Error Rate**: 42% → **70%+** (errors with sufficient detail)
- **Structured Error Adoption**: Unknown → **80%+** (validation errors using ValidationError/ConfigurationError/OperationError types)
- **Documentation Coverage**: README mentions → **Comprehensive troubleshooting section**

---

## Next Steps

1. Review and prioritize the tasks above
2. Assign tasks to Copilot agent via planner agent
3. Track progress on improvement items
4. Re-evaluate this focus area in 2-3 months
5. Next analysis: 2026-01-20 - Focus area will be selected based on diversity algorithm (60% custom, 30% standard, 10% reuse)

---

**References:**
- [§21139101051](https://github.com/githubnext/gh-aw/actions/runs/21139101051) - This workflow run

---

*Generated by Repository Quality Improvement Agent*  
*Next scheduled analysis: 2026-01-20 - Focus area TBD based on diversity algorithm*


> AI generated by [Repository Quality Improvement Agent](https://github.com/githubnext/gh-aw/actions/runs/21139101051)
> - [x] expires <!-- gh-aw-expires: 2026-01-26T13:30:04.725Z --> on Jan 26, 2026, 1:30 PM UTC

[github-actions[bot]]

This discussion was automatically closed because it expired on 2026-01-26T13:30:04.725Z.