[Schema Consistency] Schema Consistency Check - 3 Critical Documentation Gaps Found

[github-actions[bot]]

🔍 Schema Consistency Check - 2026-01-15

Overview

Executed Strategy-004: Cross-Schema Consistency & Type Analysis comparing main_workflow_schema.json, included_file_schema.json, and mcp_config_schema.json. Found 3 critical schema inconsistencies that reveal undocumented design decisions in the security model between main workflows and included files.

Key Finding: All three critical issues appear to be intentional security restrictions for included files, but they lack any schema documentation ($comments) or reference documentation explaining the design philosophy.

Full Report

Critical Issues

1. MCP Tool Configuration Structure Mismatch 🔴

Severity: Critical - Security Model Documentation Gap
Location: tools.additionalProperties structure differs between schemas

Main schema provides full MCP configuration with 11 properties:

• command, args, env, mode, type, version
• toolsets, url, headers, container, entrypointArgs

Included schema provides simplified configuration with only 2 properties:

• mcp (reference to MCP server)
• allowed (permission list)

Impact:

• Main workflows can configure MCP tools with full detail
• Included files have restricted MCP configuration (security model)
• No documentation explaining this intentional difference
• Users may be confused why MCP configuration works differently

Recommendation:

// Add to included_file_schema.json tools.additionalProperties
"$comment": "Simplified MCP configuration for included files. Full MCP configuration (command, args, env, etc.) is only available in main workflow files for security reasons. Included files use mcp references and allowed lists."

Files:

• pkg/parser/schemas/main_workflow_schema.json:tools.additionalProperties
• pkg/parser/schemas/included_file_schema.json:tools.additionalProperties

2. engine.command Property Missing from Included Schema 🔴

Severity: Critical - Capability Restriction Undocumented
Location: $defs.engine_config.oneOf[1].properties

The engine.command property exists in main schema but is completely absent from included schema.

Main schema definition:

"command": {
  "type": "string",
  "description": "Custom executable path for the AI engine CLI. When specified, the workflow will skip the standard installation steps and use this command instead..."
}

Included schema: Property does not exist (12 properties vs 11 in included)

Impact:

• Included files cannot specify custom engine commands
• Security restriction is undocumented
• No error message guides users if they try to use this
• Compiler behavior unclear

Recommendation:

1. Add $comment to included schema explaining restriction
2. Document in docs/reference/frontmatter.md which engine properties work in included vs main
3. Add compiler validation to provide helpful error if user attempts to use restricted property

Files:

• pkg/parser/schemas/main_workflow_schema.json:$defs.engine_config
• pkg/parser/schemas/included_file_schema.json:$defs.engine_config

3. Required Field Inconsistency: 'on' Field 🔴

Severity: Critical - Semantic Difference Undocumented
Location: Top-level required array

Main schema: required: ['on'] - trigger field is mandatory
Included schema: required: [] - no required fields

Analysis:

• Main workflows must specify trigger conditions (on field required)
• Included files have no required fields at top level
• This is intentional - included files use applyTo patterns, not triggers
• However, design rationale is not documented in schema

Impact:

• Schema validation behavior differs significantly
• IDE autocomplete won't guide users on semantic difference
• Migration path from main to included unclear

Recommendation:

// Add to included_file_schema.json
"$comment": "Included files use applyTo patterns to determine when they apply, so the 'on' trigger field is not required. Main workflow files must specify 'on' to define trigger conditions.",
"required": []

Files:

• pkg/parser/schemas/main_workflow_schema.json:required
• pkg/parser/schemas/included_file_schema.json:required

Architectural Insights

Design Philosophy: Main vs Included

The analysis reveals a clear but undocumented design pattern:

Main workflows (38 properties):

• Full trigger control: on, command, if
• Workflow orchestration: jobs, concurrency, imports
• Security features: github-token, roles, strict
• Rich configuration: cache, sandbox, features

Included files (15 properties):

• Application scope: applyTo, inputs
• Reusable components: subset of tools, permissions, steps
• Security-restricted: no github-token, no full engine config, simplified MCP

Property Breakdown:

• Common (13): description, engine, mcp-servers, metadata, network, permissions, runtimes, safe-inputs, safe-outputs, secret-masking, services, steps, tools
• Only in Main (25): bots, cache, command, concurrency, container, env, environment, features, github-token, if, imports, jobs, labels, name, on, post-steps, roles, run-name, runs-on, sandbox, source, strict, timeout-minutes, timeout_minutes, tracker-id
• Only in Included (2): applyTo, inputs

Problem: This design philosophy exists in practice but is not documented anywhere.

additionalProperties Discipline

Positive finding: Both schemas maintain excellent discipline:

• Main: 146 instances of additionalProperties: false
• Included: 41 instances of additionalProperties: false

This prevents typos and invalid properties through strict validation.

Documentation Gaps

1. Missing Schema Design Documentation

Needed: docs/src/content/docs/reference/schema-design.md

Should explain:

• Security model: why main has full capabilities, included is restricted
• When to use main vs included files
• Property availability matrix (table format)
• Migration guidance

2. Missing Inline Schema Documentation

Needed: $comment fields in schemas explaining:

• Why tools.additionalProperties differs (security)
• Why engine.command is restricted (security)
• Why required fields differ (semantic model)

3. Missing Reference Documentation

Needed: Update docs/src/content/docs/reference/frontmatter.md

Add section: "Main Workflows vs Included Files"

• Table showing which properties are available in each
• Explanation of security restrictions
• Examples of valid and invalid configurations

Recommendations

High Priority (Immediate)

• Add $comment to included schema explaining tools.additionalProperties structure
• Add $comment explaining why engine.command is restricted
• Add $comment explaining required field difference
• Verify compiler behavior when restricted properties are used in included files

Medium Priority (This Sprint)

• Create schema-design.md documentation page
• Update frontmatter.md with main vs included comparison table
• Add validation tests for intentional schema differences
• Add compiler error messages for restricted property usage

Low Priority (Backlog)

• Consider schema versioning to track compatibility
• Create IDE snippets that guide users to correct schema type
• Evaluate if engine.command restriction should be relaxed
• Add JSON Schema extension for security annotations

Positive Findings ✅

1. $ref Integrity: All $ref references valid across all three schemas - no broken links
2. $defs Clean: No unused definitions - all $defs are actively used
3. Metadata Complete: All schemas have $id, title, description
4. Tool Scope Aligned: Both main and included have identical 12 tools
5. Permissions Identical: Both use same 2 oneOf structure
6. MCP $defs Consistent: stdio_mcp_tool and http_mcp_tool are identical between schemas

Strategy Performance

• Strategy: Cross-Schema Consistency & Type Analysis (Strategy-004)
• Selection: Proven strategy (day 015 mod 10 = 5, use proven)
• Findings: 3 critical + 2 architectural insights
• Effectiveness: Very High
• Success Count: 14 runs (increased from 13)
• Should Reuse: YES - continues to find real issues

Comparison to Previous Run (2026-01-11):

• Previous: Found tools structure + safe-outputs differences
• This run: Confirmed tools structure (with details) + found engine.command + found required field inconsistency
• Trend: Strategy maintains very-high effectiveness

Analysis Methodology

1. Loaded three schemas: main (38 props), included (15 props), mcp (0 props)
2. Compared structures: Properties, $defs, required fields, additionalProperties
3. Deep-dived differences: tools.additionalProperties, engine_config, top-level fields
4. Validated $ref integrity: All references valid, no broken links
5. Checked metadata: All schemas properly documented at top level
6. Analyzed $defs: Compared shared definitions for consistency

Tools Used:

• Python JSON parsing and recursive comparison
• Property enumeration and set operations
• Deep dictionary comparison algorithms

Summary Statistics

• Total Issues: 5 (3 critical + 2 architectural)
• Documentation Gaps: 3 critical schema documentation issues
• Security Model: Undocumented but intentional design
• Positive Findings: 6 areas of excellent consistency

Next Action: Add inline $comment documentation to schemas explaining intentional differences and update reference documentation with main vs included comparison guide.

References:

• §21050370404

AI generated by Schema Consistency Checker

[pelikhan]

/plan

[github-actions[bot]]

🎉 Yo ho ho! Plan Command found the treasure and completed successfully! ⚓💰