[Code Quality] Implement validation error message testing standard

[github-actions]

Description

Systematically improve validation error tests to verify message content, not just error presence. Currently only 15.2% (5/33) tests validate actual error messages, meaning validation message quality can degrade without detection.

Current Situation

Testing Metrics:

• 225 validation test functions across 51 test files
• Only 5 tests (15.2%) validate error message content
• 28 tests (84.8%) use generic assert.Error without checking messages
• No systematic testing of error message clarity, examples, or actionable guidance
• Validation messages can degrade without detection in CI

Current Pattern (84.8% of tests):

err := ValidateWorkflow(input)
assert.Error(t, err) // ❌ Doesn't validate message quality

Better Pattern (rare, only 15.2%):

err := ValidateWorkflow(input)
assert.Error(t, err)
assert.Contains(t, err.Error(), "Example:")
assert.Contains(t, err.Error(), "copilot")

Proposed Solution

Create a test helper function that validates error message quality:

// pkg/workflow/validation_helpers_test.go

// assertValidationError verifies a validation error contains expected guidance
func assertValidationError(t *testing.T, err error, checks ValidationErrorChecks) {
    t.Helper()
    require.Error(t, err, checks.Context)
    
    if checks.RequireExample {
        assert.Contains(t, err.Error(), "Example:", "%s: error should include example", checks.Context)
    }
    
    if len(checks.RequireKeywords) > 0 {
        for _, keyword := range checks.RequireKeywords {
            assert.Contains(t, err.Error(), keyword, "%s: error should mention '%s'", checks.Context, keyword)
        }
    }
    
    if checks.RequireValidOptions {
        assert.Contains(t, err.Error(), "Valid", "%s: error should list valid options", checks.Context)
    }
}

type ValidationErrorChecks struct {
    Context           string   // Test context for assertions
    RequireExample    bool     // Error must include "Example:"
    RequireKeywords   []string // Error must contain these keywords
    RequireValidOptions bool   // Error must list valid options
}

Implementation Plan

Phase 1: Create Test Helper (Day 1)

• Add assertValidationError() to pkg/workflow/validation_helpers_test.go
• Add comprehensive tests for the helper itself
• Document usage pattern in helper docstring

Phase 2: Update High-Priority Tests (Days 2-3)
Priority files to update (highest impact):

1. pkg/workflow/engine_validation_test.go (already good, use as reference)
2. pkg/workflow/mcp_config_validation_test.go
3. pkg/workflow/safe_outputs_target_validation_test.go
4. pkg/workflow/secrets_validation_test.go
5. pkg/workflow/template_injection_validation_test.go

Phase 3: Documentation (Day 3)

• Add new section "### Testing Validation Errors" to scratchpad/testing.md
• Document the pattern and helper usage
• Include examples of before/after test code

Example Transformation

Before:

func TestInvalidEngine(t *testing.T) {
    err := ValidateEngine("invalid")
    assert.Error(t, err)
}

After:

func TestInvalidEngine(t *testing.T) {
    err := ValidateEngine("invalid")
    assertValidationError(t, err, ValidationErrorChecks{
        Context: "invalid engine",
        RequireExample: true,
        RequireKeywords: []string{"copilot", "claude", "codex"},
        RequireValidOptions: true,
    })
}

Success Criteria

• New test helper assertValidationError() in validation_helpers_test.go
• Helper validates: error presence, example inclusion, keyword presence
• At least 50 tests updated to use new helper or equivalent assertions
• Target: 60%+ of validation error tests validate message content (up from 15.2%)
• New section in scratchpad/testing.md documenting validation error testing
• All updated tests pass with make test-unit
• Run make agent-finish successfully

Expected Outcomes

• Error message quality becomes a first-class testing concern
• Validation message regressions caught in CI
• Clear testing patterns for new validators
• Improved developer confidence in error message quality

Success Metrics

Track improvement in Error Message Test Coverage:

• Baseline: 15.2% (5/33 tests validate message content)
• Target: 60%+ (at least 50 tests validate message content)
• Measurement: Count tests using assert.Contains() for error message validation

Priority

High - Prevents validation message quality regression

Estimated Effort

Large (2-3 days)

Dependencies

Should be implemented after or in parallel with:

• Issue about creating validation error message style guide (defines what to test for)

Source

Extracted from Repository Quality Improvement Report #11292 - Validation Message Clarity & Developer Guidance (Task 3)

AI generated by Discussion Task Miner - Code Quality Improvement Agent

• expires on Feb 12, 2026, 9:09 PM UTC