[CI Failure Doctor] 🏥 CI Failure Investigation - Runtime-import Validation Regression (Run #30032)

[github-actions]

🏥 CI Failure Investigation - Run #30032

Summary

🚨 CRITICAL: ALL CI BUILDS BLOCKED - PR #10312 introduced compile-time validation for runtime-import expressions that breaks lock file compilation, preventing any PRs from being merged.

Failure Details

• Run: #21072294659
• Commit: 75ce58e28a9d3cb12b500deec165c21d659d19d7
• PR: Add compile-time validation for runtime-import expressions #10312 - "Add compile-time validation for runtime-import expressions"
• Trigger: push to main branch
• Date: 2026-01-16 15:49:23Z

Root Cause Analysis

PR #10312 added a new validation step validateRuntimeImportFiles() in the compiler that validates GitHub Actions expressions in runtime-imported markdown files at compile time. This validation is failing when rebuilding lock files, causing the CI build job to fail.

Files Changed in PR #10312

1. pkg/workflow/compiler.go (+19 lines)

   • Added validateRuntimeImportFiles() call during compilation
   • Calculates workspace directory by going up from .github/workflows/file.md

2. pkg/workflow/expression_validation.go (+117 lines)

   • New validateRuntimeImportFiles() function
   • New extractRuntimeImportPaths() helper function

3. pkg/workflow/runtime_import_validation_test.go (+323 lines)

   • Test coverage for the new validation

Suspected Issue

Two workflows use runtime-import to include files from .github/agentics/:

1. code-simplifier.md: {{#runtime-import agentics/code-simplifier.md}}
2. repo-audit-analyzer.md: {{#runtime-import agentics/repo-audit-analyzer.md}}

These imported files contain GitHub Actions expressions like:

• ${{ github.repository }} ✅ In AllowedExpressions list
• ${{ github.workspace }} ✅ In AllowedExpressions list
• ${{ inputs.repository || 'FStarLang/FStar' }} ❓ Should be allowed via inputs.* pattern

Possible root causes:

1. Path Resolution Bug: The validation may not be correctly building the absolute path to .github/agentics/ files
2. Expression Validation False Positive: The logical OR operator (||) in expressions may not be handled correctly
3. Workspace Directory Calculation: The compiler's workspace directory calculation may be incorrect in CI environment

Failed Jobs and Errors

Build Job (FAILED)

• Job ID: 60604929311
• Failed Step: "Rebuild lock files" (step Weekly Research Report: AI Workflow Automation Landscape and Strategic Opportunities - August 2025 #9)
• Impact: Lock file compilation halted

Subsequent Jobs (CANCELLED)

• fuzz (60604929427)
• Integration: CMD Tests (60604929900)

Note: Unable to access full error logs without GitHub API authentication.

Investigation Findings

Validation Logic Flow

// From compiler.go
workflowDir := filepath.Dir(markdownPath)     // .github/workflows
githubDir := filepath.Dir(workflowDir)        // .github
workspaceDir := filepath.Dir(githubDir)       // repo root

validateRuntimeImportFiles(content, workspaceDir)

// From expression_validation.go  
normalizedPath := "agentics/code-simplifier.md"  // After removing .github/, ./
githubFolder := filepath.Join(workspaceDir, ".github")
absolutePath := filepath.Join(githubFolder, normalizedPath)
// Expected: /home/runner/work/gh-aw/gh-aw/.github/agentics/code-simplifier.md

Files That Should Exist

✅ .github/agentics/code-simplifier.md - EXISTS (confirmed)
✅ .github/agentics/repo-audit-analyzer.md - EXISTS (confirmed)

Expressions Found in Imported Files

From code-simplifier.md:

- **Repository**: ${{ github.repository }}
- **Workspace**: ${{ github.workspace }}

From repo-audit-analyzer.md:

${{ inputs.repository || 'FStarLang/FStar' }}

All these expressions should be valid according to the allowlist.

Recommended Actions

🔥 IMMEDIATE (Fix CI)

1. Reproduce Locally:

   git checkout 75ce58e28a9d3cb12b500deec165c21d659d19d7
   make build
   make recompile  # Should fail on code-simplifier.md or repo-audit-analyzer.md

2. Get Actual Error Message: The CI logs will show the exact validation error

3. Quick Fix Options:

   • Option A: Temporarily disable runtime-import validation (comment out the call)
   • Option B: Skip validation for files in .github/agentics/ subdirectory
   • Option C: Fix the specific validation issue once identified

🔧 ROOT CAUSE FIX

Based on the actual error message, fix one of:

1. Path Resolution: Ensure workspaceDir calculation works in CI environment
2. Expression Validation: Handle logical operators in expressions (||, &&)
3. Inputs Pattern Matching: Verify inputs.* regex matches inputs.repository

✅ VERIFY FIX

# After fix
make build
make recompile  # Must succeed for ALL workflows
make test       # Ensure no regressions

Prevention Strategies

Pre-Commit Checklist for Expression Validation Changes

# MANDATORY before committing ANY validation changes
make build              # Rebuild binary with new validation
make recompile          # Compile ALL 178 workflow files
make test               # Run full test suite

Add Integration Tests

Create test that compiles workflows with actual runtime-import files:

func TestCompileWorkflowsWithRuntimeImports(t *testing.T) {
    workflows := []string{
        ".github/workflows/code-simplifier.md",
        ".github/workflows/repo-audit-analyzer.md",
    }
    
    for _, wf := range workflows {
        err := compiler.CompileWorkflow(wf)
        require.NoError(t, err, "Failed to compile %s", wf)
    }
}

CI Enhancement

Add step to compile each workflow individually to catch failures early:

- name: Test compile all workflows
  run: |
    for workflow in .github/workflows/*.md; do
      echo "Compiling $workflow"
      ./gh-aw compile "$workflow" || exit 1
    done

AI Team Self-Improvement

CRITICAL: Test compilation of ALL workflows before merging validation changes

Pre-Merge Validation Change Checklist

When modifying any validation code in pkg/workflow/*_validation.go:

1. ✅ Rebuild binary: make build
2. ✅ Test compile all 178 workflows: make recompile
3. ✅ Verify NO compilation errors or warnings
4. ✅ Run full test suite: make test
5. ✅ Test with workflows that use features being validated
6. ✅ Check both positive cases (valid) and negative cases (invalid)

Example Validation Test Pattern

# WRONG - Only testing with synthetic test data
go test ./pkg/workflow -run TestValidateExpression

# CORRECT - Testing with real workflows
make build && make recompile && make test

Red Flags for Validation Changes

🚨 DO NOT MERGE if:

• make recompile fails or shows errors
• Any workflow fails to compile that previously worked
• New validation breaks existing valid workflows
• Tests only use synthetic data, not real workflow files

Path Calculation Pitfalls

When calculating file paths in validation:

// ❌ WRONG - May break in different environments
workspaceDir := os.Getwd()

// ✅ CORRECT - Calculate relative to markdown file
workflowDir := filepath.Dir(markdownPath)
githubDir := filepath.Dir(workflowDir)
workspaceDir := filepath.Dir(githubDir)

// 🧪 TEST - Verify path calculation with debug logging
log.Printf("markdownPath=%s, workspaceDir=%s", markdownPath, workspaceDir)

Expression Validation Gotchas

GitHub Actions expressions support complex syntax:

${{ inputs.value || 'default' }}          # Logical OR with default
${{ github.event.name == 'push' }}       # Comparison operators  
${{ contains(github.ref, 'refs/tags/') }} # Function calls
${{ github.event.*.id }}                 # Wildcards (some contexts)

Always test validation with:

• Real workflow markdown files
• Complex expressions with operators
• Edge cases (empty, null, multiline)
• Optional runtime-imports ({{#runtime-import? ...}})

Historical Context

This is the first time runtime-import validation has been added. Previous similar failures:

• 2026-01-15 Run #21046376185: JavaScript test failures after PR claiming to fix tests (PR Fix JavaScript test failures: schema validation and missing template files #10125)
  • Pattern: "Fix" PR actually broke validation
  • Lesson: Test the fix, don't just trust PR title

Severity Assessment

• Severity: 🔴 CRITICAL
• Impact: ALL CI builds blocked, cannot merge any PRs
• Urgency: IMMEDIATE fix required
• Is Flaky: No - deterministic failure
• Is Recurring: No - new regression from PR Add compile-time validation for runtime-import expressions #10312

Links

• Workflow Run: https://github.com/githubnext/gh-aw/actions/runs/21072294659
• PR: Add compile-time validation for runtime-import expressions #10312
• Commit: 75ce58e

---

Investigation Status: ⏳ Awaiting CI log access to identify exact error message

Next Steps:

1. Developer with CI access should check build logs
2. Reproduce issue locally with make recompile
3. Apply appropriate fix from Recommended Actions
4. Test fix compiles all workflows successfully
5. Add integration test to prevent regression

AI generated by CI Failure Doctor

To add this workflow in your repository, run gh aw add githubnext/agentics/workflows/ci-doctor.md@ea350161ad5dcc9624cf510f134c6a9e39a6f94d. See usage guide.

[github-actions]

🔄 Duplicate Failure Detected

Run #30043 (#21073622967) failed with the same issue.

Details

• Commit: 5a8843b - "Fix doc build: add required title field to campaign example files (Fix doc build: add required title field to campaign example files #10331)"
• Failed Job: build (job ID: 60609661645)
• Failed Step: "Rebuild lock files"
• Timestamp: 2026-01-16 16:35:13 UTC

Analysis

This commit is only 1 commit after the problematic commit 75ce58e (PR #10312). The failure persists because the underlying runtime-import validation issue has not been fixed yet.

Impact: CI remains completely blocked - no PRs can be merged until this is resolved.

Commit Timeline

75ce58e (PR #10312) ← Runtime-import validation added (BROKE CI)
   ↓
5a8843b (PR #10331) ← Current failure (STILL BROKEN)

---

🤖 Duplicate detected by CI Failure Doctor

AI generated by CI Failure Doctor

[github-actions]

🔄 Duplicate Failure Detected

Run #30046 (#21074242627) failed with the same issue.

Details

• Commit: 945d8e9 - "fix: links in campaign examples (fix links in campaign examples #10339)"
• Failed Job: build (job ID: 60611857676)
• Failed Step: "Rebuild lock files"
• Timestamp: 2026-01-16 16:59:04Z
• Files Added: 566 markdown files (massive merge/migration commit)

Analysis

This is the 3rd consecutive failure of the same pattern since PR #10312 introduced runtime-import validation:

1. Run #30032 (original) - validation introduced
2. Run #30043 - attempted fix for doc build
3. Run #30046 (this run) - massive commit adding campaign workflows

The root cause remains unchanged: make recompile fails during lock file compilation when validating runtime-imported files.

Impact

• All PRs blocked - cannot merge until this is resolved
• 178 workflow markdown files now in repository
• Priority: CRITICAL - main branch CI is broken

Recommended Action

The fix for PR #10312's validation logic needs to be prioritized and merged ASAP to unblock the CI pipeline.

AI generated by CI Failure Doctor