[Schema Consistency] Runtime Behavior vs Schema Contracts - Jan 29, 2026

[github-actions[bot]]

Summary

• Inconsistencies Found: 11 findings across all categories
• Categories Analyzed: Schema, Parser, Documentation, Workflows
• Strategy Used: Runtime Behavior vs Schema Contracts Analysis (Strategy 11 - NEW)
• New Strategy: YES (radically different approach focusing on runtime behavior)
• Critical Issues: 2
• Moderate Issues: 5
• Documentation/Enhancement Opportunities: 4

Analysis Approach (Strategy 11)

This analysis used a completely new strategy not attempted before: Runtime Behavior vs Schema Contracts. Unlike previous strategies that focused on field enumeration, type analysis, or documentation completeness, this strategy examines what actually happens when code executes versus what the schema promises.

Key Innovation: Instead of comparing static definitions, this strategy traces execution paths to find:

• Silent failures (fields ignored without error)
• Type coercion mismatches
• Partial implementations
• Validation gaps
• Undocumented runtime behaviors

Critical Issues

1. Dual Field Names: `timeout-minutes` vs `timeout_minutes` ⚠️

Impact: HIGH - Schema pollution and potential confusion

Details:

• Schema defines BOTH timeout-minutes and timeout_minutes as separate top-level properties
• Runtime handles both for backwards compatibility (compiler_orchestrator_workflow.go:149-151)

Evidence:

$ grep timeout /tmp/gh-aw/cache-memory/analysis/schema_fields.txt
timeout-minutes
timeout_minutes

Location:

• Schema: pkg/parser/schemas/main_workflow_schema.json (both fields present)
• Runtime: pkg/workflow/compiler_orchestrator_workflow.go:149-151

Recommendation:

• Remove timeout_minutes from schema (keep only timeout-minutes for consistency)
• Runtime can continue supporting both for backwards compatibility
• Document the dual support in migration/backwards-compatibility notes

2. Schema `required` Field Not Enforced: `on` Field ⚠️

Impact: MEDIUM - Schema contract violation

Details:

• Schema marks on field as required: "required": ["on"]
• Runtime does NOT enforce this - accepts workflows without on field
• Runtime generates default trigger configuration when missing (compiler_orchestrator_workflow.go:139)

Evidence:

# Schema declares it required
$ jq -r '.required[]? // empty' pkg/parser/schemas/main_workflow_schema.json
on

# But runtime provides defaults
$ grep -A5 "No 'on:' setting found" pkg/workflow/*.go
# (Runtime generates default without error)

Location:

• Schema: pkg/parser/schemas/main_workflow_schema.json (line with "required")
• Runtime: pkg/workflow/compiler_orchestrator_workflow.go:139

Recommendation:

• Remove on from schema's required array (align with actual runtime behavior)
• OR enforce validation if on should truly be required
• Update documentation to clarify that on has sensible defaults

Moderate Issues

3. Pass-Through YAML Sections Without Validation 📋

Impact: MEDIUM - Limited validation of user input

Details:
Several schema fields are extracted as raw YAML sections and passed through to GitHub Actions without validation:

• concurrency
• container
• environment
• env
• runs-on
• services

Evidence:

// pkg/workflow/compiler_orchestrator_workflow.go
workflowData.Concurrency = c.extractTopLevelYAMLSection(frontmatter, "concurrency")
workflowData.Container = c.extractTopLevelYAMLSection(frontmatter, "container")
workflowData.Environment = c.extractTopLevelYAMLSection(frontmatter, "environment")
workflowData.Env = c.extractTopLevelYAMLSection(frontmatter, "env")

Location: pkg/workflow/compiler_orchestrator_workflow.go:139-296

Recommendation:

• Document that these fields pass through to GitHub Actions
• Consider adding basic structure validation if schema defines specific shapes
• OR mark in schema that these are "pass-through" fields with minimal validation

4. Automatic Firewall Enablement (Undocumented Behavior) 🔐

Impact: MEDIUM - Security feature not clearly documented in schema

Details:

• Runtime automatically enables firewall for copilot and claude engines when network restrictions exist
• This behavior is NOT documented in schema descriptions
• Happens silently without user configuration

Evidence:

// pkg/workflow/compiler_orchestrator_engine.go:191-194
enableFirewallByDefaultForCopilot(engineSetting, networkPermissions, sandboxConfig)
enableFirewallByDefaultForClaude(engineSetting, networkPermissions, sandboxConfig)

Location: pkg/workflow/compiler_orchestrator_engine.go:191-194, pkg/workflow/firewall.go:81-96

Recommendation:

• Add note in schema's engine field description about automatic firewall enablement
• Document in network.firewall section that firewall may be auto-enabled
• Consider adding to workflow output/logs when firewall is auto-enabled

5. `max-patch-size` Type Coercion with Silent Truncation 🔢

Impact: LOW-MEDIUM - Silent data loss

Details:

• Schema likely defines max-patch-size as integer
• Runtime accepts float values and silently truncates them
• Logs truncation but doesn't error

Evidence:

// pkg/workflow/safe_outputs_config.go:325
safeOutputsConfigLog.Printf("max-patch-size: float value %.2f truncated to integer %d", v, intVal)

Location: pkg/workflow/safe_outputs_config.go:307-325

Recommendation:

• Schema should explicitly document that floats are truncated
• OR enforce integer-only validation
• Consider warning users in docs about precision loss

6. `cache` Field Indirect Access Pattern 💾

Impact: LOW - Inconsistent access pattern

Details:

• Most fields accessed via frontmatter["fieldname"]
• cache field accessed via separate topLevel map
• Pattern is inconsistent with other fields

Evidence:

// pkg/workflow/cache.go:247
cacheConfig, exists := topLevel["cache"]

Location: pkg/workflow/cache.go:247

Recommendation:

• Document why cache uses different access pattern
• Consider standardizing access patterns for consistency
• OR keep as-is if there's architectural reason

7. Limited Usage of `bots` and `roles` Fields 🤖

Impact: LOW - Potential incomplete implementation

Details:

• Schema defines bots and roles fields
• Very limited references in production code (2 occurrences each)
• Mostly used in role validation checks

Evidence:

Field: bots
  Production code: 2 references
  Test code: 0 references
Field: roles  
  Production code: 2 references
  Test code: 1 reference
``````

**Location**: `pkg/workflow/role_checks.go:77`

**Recommendation**:
- Verify if `bots` and `roles` are fully implemented
- If these are newer/experimental features, mark them as such in schema
- Add more comprehensive tests if features are production-ready

</details>

### Documentation/Enhancement Opportunities

<details>
<summary><b>8. `maxItems` Constraints Runtime Validation ✅</b></summary>

**Impact**: INFO - Validation appears present

**Details:**
- Schema defines 20 `maxItems` constraints for arrays
- Runtime has 19 references to max/exceed/too-many checks
- Good coverage but worth verifying specific constraints

**Evidence:**
``````
Schema maxItems count: 20
Runtime max-related checks: 19

Recommendation:

• Audit each schema maxItems to ensure corresponding runtime validation
• Document any intentionally unvalidated maxItems
• Consider adding schema reference comments in validation code

9. `engine` Default Value - Correctly Implemented ✅

Impact: POSITIVE - No issue found

Details:

• Schema declares engine default as "copilot"
• Runtime correctly applies this default when field is missing
• Good alignment between schema contract and implementation

Evidence:

// pkg/workflow/compiler_orchestrator_engine.go:158
defaultEngine := c.engineRegistry.GetDefaultEngine()
engineSetting = defaultEngine.GetID()
log.Printf("No 'engine:' setting found, defaulting to: %s", engineSetting)

Location: pkg/workflow/compiler_orchestrator_engine.go:156-167

Status: ✅ Working as expected

10. Pass-Through Fields Could Use Schema Documentation Tag 📝

Impact: LOW - Documentation clarity

Details:
Many fields are pass-through to GitHub Actions YAML:

• concurrency, container, environment, env, runs-on, services, network, run-name

Recommendation:

• Add "$comment": "Pass-through field: forwarded to GitHub Actions without validation" in schema
• Or create a "x-passthrough": true extension property
• Helps users understand validation boundaries

11. Silent Field Ignores via `exists` Checks 🤫

Impact: INFO - Design pattern observation

Details:

• Many fields use if exists { process } else { skip } pattern
• No error when field is missing
• This is intentional for optional fields but not always obvious from schema

Evidence:

// Common pattern throughout pkg/workflow/*
if engine, exists := frontmatter["engine"]; exists {
    // process engine
}
// No else clause - silently skip if missing

Recommendation:

• Ensure schema marks these fields as not required (omit from required array)
• Consider schema comment explaining optional processing
• Document expected behavior when field is omitted

Recommendations by Priority

Immediate Actions

1. Fix Critical: Remove duplicate timeout_minutes from schema - keeps clean API surface
2. Fix Critical: Remove on from required array OR enforce validation - align schema with runtime
3. Document: Add automatic firewall enablement to schema descriptions - security transparency

Short-term Improvements

4. Document pass-through fields in schema with comments
5. Verify bots and roles implementation completeness
6. Audit maxItems constraints against runtime validation

Long-term Enhancements

7. Consider standardizing field access patterns (cache outlier)
8. Add runtime warnings when type coercion occurs (like max-patch-size)
9. Create schema extension for pass-through fields

Strategy Performance

• Strategy Used: Runtime Behavior vs Schema Contracts (Strategy 11)
• Findings: 11 total (2 critical, 5 moderate, 4 informational)
• Effectiveness: VERY HIGH - discovered runtime behaviors not found by previous strategies
• Should Reuse: YES - excellent for finding schema/runtime contract violations
• Innovation: First strategy to trace execution paths rather than compare static definitions

Strategy Strengths

✅ Found runtime behaviors not visible in static analysis
✅ Discovered silent failures and type coercion issues
✅ Identified undocumented automatic behaviors (firewall enablement)
✅ Validated schema contracts against actual implementation

Strategy Weaknesses

⚠️ Requires deeper code reading (slower than grep-based approaches)
⚠️ May miss issues only visible in edge cases
⚠️ Relies on tracing execution paths (could miss rarely-used code paths)

Next Steps

• Complete schema consistency analysis
• Fix critical schema issues (timeout field duplication, 'on' required status)
• Update schema documentation for undocumented runtime behaviors
• Verify maxItems validation coverage
• Add strategy 11 to proven strategies cache

---

Analysis Date: January 29, 2026
Strategy: Runtime Behavior vs Schema Contracts (NEW)
Repository: githubnext/gh-aw
Workflow Run: §21498944402

AI generated by Schema Consistency Checker

• expires on Feb 5, 2026, 11:58 PM UTC