[smoke-detector] 🔍 Smoke Test Investigation - Smoke Codex Run #49: Agent Output Artifact Missing in Staged Mode

[github-actions]

🔍 Smoke Test Investigation - Run #49

Summary

The Smoke Codex workflow failed because the create_issue job could not find the expected agent_output.json artifact. The agent job completed successfully in staged mode, but the artifact upload step failed with a warning that no files were found at the expected path /tmp/gh-aw/safe-outputs/outputs.jsonl.

Failure Details

• Run: 18840299097
• Run Number: 49
• Commit: b88a647
• Branch: main
• Trigger: schedule
• Duration: 2.9 minutes
• Failed Jobs: create_issue (4s)
• Workflow: Smoke Codex

Root Cause Analysis

Primary Issue

The workflow ran in staged mode (GH_AW_SAFE_OUTPUTS_STAGED=true), where the agent is expected to:

1. ✅ Complete its task successfully
2. ✅ Use safe-outputs tools or generate output
3. ✅ Create /tmp/gh-aw/safe-outputs/outputs.jsonl
4. ❌ Have the artifact uploaded for downstream jobs

However, the artifact upload step could not find any files to upload, causing the create_issue job to fail when attempting to download the non-existent artifact.

Error Chain

1. Artifact Upload Warning (agent job)

##[warning]No files were found with the provided path: /tmp/gh-aw/safe-outputs/outputs.jsonl
No artifacts will be uploaded.

2. Artifact Download Failure (create_issue job)

##[error]Unable to download artifact(s): Artifact not found for name: agent_output.json
Please ensure that your artifact is not expired and the artifact was uploaded using a compatible version of toolkit/upload-artifact.

3. File Not Found (create_issue job)

##[error]Error reading agent output file: ENOENT: no such file or directory, open '/tmp/gh-aw/safe-outputs/agent_output.json'

Why Did This Happen?

This failure indicates one of the following scenarios:

1. Agent didn't use safe-outputs tools: The Codex agent may have completed successfully but didn't call the safe_outputs_create_issue tool, similar to issue [smoke-detector] 🔍 Smoke Test Investigation - Smoke GenAIScript Run #57: Agent Does Not Use Safe-Outputs MCP Tools #2307 (GenAIScript)

2. Safe-outputs file not created: Even if the agent intended to use safe-outputs, the file /tmp/gh-aw/safe-outputs/outputs.jsonl was never written

3. Staged mode behavior: In staged mode, the agent may behave differently and not create the expected output files

4. Workflow configuration issue: The artifact upload step expects outputs.jsonl, but the create_issue job looks for agent_output.json - there may be a transformation step that failed

Failed Jobs and Errors

Job Sequence

1. ✅ activation - succeeded (4s)
2. ✅ agent - succeeded (1.3m)
3. ✅ detection - succeeded (24s)
4. ❌ create_issue - failed (4s)
5. ⏭️ missing_tool - skipped

Key Observations

• Agent job succeeded (1.3m runtime)
• Detection job succeeded (24s) - this job also depends on artifacts but succeeded
• Only create_issue job failed
• Workflow ran in staged mode where issues are not actually created but previewed

Investigation Findings

Agent Execution

• Status: Success
• Duration: 1.3 minutes
• Turns: 1
• Error Count: 1
• Warning Count: 0

Staged Mode Configuration

GH_AW_SAFE_OUTPUTS_STAGED=true
GH_AW_WORKFLOW_NAME=Smoke Codex
GH_AW_AGENT_OUTPUT=/tmp/gh-aw/safe-outputs/agent_output.json

In staged mode, the workflow should:

• Run the agent normally
• Generate output to preview
• Not actually create GitHub issues
• Still upload artifacts for the create_issue job to preview

Artifact Details

The workflow attempts to upload artifacts from the agent job:

• Expected source path: /tmp/gh-aw/safe-outputs/outputs.jsonl
• Artifact name: agent_output.json
• Result: No files found, no artifact uploaded

Recommended Actions

High Priority

• Investigate agent logs for this specific run

  # Download and examine the agent-stdio.log
  # Look for:
  # - Was the safe-outputs MCP server loaded?
  # - Did the agent attempt to use safe_outputs_create_issue?
  # - Were there any errors creating outputs.jsonl?

• Check safe-outputs MCP configuration for Codex

  • Verify the MCP server is properly configured in the Codex workflow
  • Confirm the safe-outputs tools are available to the Codex agent
  • Check if there are Codex-specific issues with MCP integration

• Add validation step before artifact upload

  - name: Validate Safe Outputs Created
    run: |
      if [ -f "/tmp/gh-aw/safe-outputs/outputs.jsonl" ]; then
        echo "✓ outputs.jsonl created"
      else
        echo "⚠️ outputs.jsonl not found - agent may not have used safe-outputs tools"
        echo "This is expected if the agent task didn't require creating issues"
        # Don't fail, just warn
      fi

• Make artifact upload conditional

  - name: Upload Safe Outputs
    if: hashFiles('/tmp/gh-aw/safe-outputs/outputs.jsonl') != ''
    uses: actions/upload-artifact@v4
    with:
      name: agent_output.json
      path: /tmp/gh-aw/safe-outputs/outputs.jsonl

Medium Priority

• Enhance agent prompt for staged mode

  You are running in STAGED MODE. You MUST still use the safe_outputs_create_issue 
  tool to create your output, even though the issue won't actually be created. 
  This is required for the workflow to complete successfully.
  

• Make downstream jobs handle missing artifacts gracefully

  create_issue:
    needs: [agent, detection]
    runs-on: ubuntu-latest
    continue-on-error: true  # Don't fail workflow if artifact missing
    steps:
      - name: Download artifact
        uses: actions/download-artifact@v4
        with:
          name: agent_output.json
        continue-on-error: true
      
      - name: Check artifact exists
        run: |
          if [ ! -f "$GH_AW_AGENT_OUTPUT" ]; then
            echo "No agent output artifact found - agent may not have required creating issues"
            exit 0
          fi

• Add detection job check

  • The detection job succeeded, so understand why it didn't catch this issue
  • May need to enhance detection logic to identify missing output files

Low Priority

• Compare with successful Codex runs

  • Look at recent successful Smoke Codex workflow runs
  • Identify what's different about this failure
  • Check if this is a new pattern or recurring issue

• Document expected behavior in staged mode

  • Clarify whether agents MUST create safe-outputs in staged mode
  • Define clear success criteria for staged mode workflows
  • Add troubleshooting guide for artifact-related failures

Prevention Strategies

1. Explicit Output Requirements

   • Make it clear in prompts when safe-outputs tools are required
   • Add validation that verifies expected outputs were created
   • Fail fast if required outputs are missing

2. Graceful Degradation

   • Don't fail the entire workflow if artifacts are missing
   • Distinguish between "agent failed" vs "agent succeeded but no output needed"
   • Add conditional logic for optional outputs

3. Better Error Messages

   • When artifact download fails, explain possible causes
   • Provide troubleshooting steps
   • Link to documentation about safe-outputs and staged mode

4. Monitoring and Alerts

   • Track artifact upload success rates
   • Alert on patterns of missing artifacts
   • Monitor staged mode workflow health

Historical Context

Similar Past Failures

Issue	Engine	Pattern	Status
#2307	GenAIScript	Agent doesn't use safe-outputs	Closed
#2280	Copilot	MCP config malformed	Closed
#2143	OpenCode	Agent doesn't use safe-outputs	Closed

This appears to be a recurring pattern across different engines where agents complete successfully but don't use safe-outputs tools, particularly in staged mode.

Pattern Classification

• Pattern ID: CODEX_AGENT_NO_ARTIFACT_STAGED_MODE
• Category: Workflow Configuration - Staged Mode Issue
• Severity: Medium
• Is Flaky: Unknown (need more data)
• First Occurrence: This investigation (2025-10-27)

Technical Details

Workflow Configuration

tools:
  safe-outputs:
    staged: true
    expected-outputs:
      create_issue:
        min: 1
        max: 1

Artifact Upload Configuration

- name: Upload Safe Outputs
  uses: actions/upload-artifact@v4
  with:
    name: agent_output.json
    path: /tmp/gh-aw/safe-outputs/outputs.jsonl

Artifact Download Configuration

- name: Download artifact
  uses: actions/download-artifact@v4
  with:
    name: agent_output.json
    path: /tmp/gh-aw/safe-outputs/

Environment Variables

GH_AW_SAFE_OUTPUTS_STAGED=true
GH_AW_WORKFLOW_NAME=Smoke Codex
GH_AW_AGENT_OUTPUT=/tmp/gh-aw/safe-outputs/agent_output.json

Related Issues

• [smoke-detector] 🔍 Smoke Test Investigation - Smoke GenAIScript Run #57: Agent Does Not Use Safe-Outputs MCP Tools #2307 - GenAIScript agent doesn't use safe-outputs (closed)
• [smoke-detector] 🔍 Smoke Test Investigation - Smoke Copilot: Safe-Outputs MCP Crashes Due to Malformed Config JSON #2280 - Copilot safe-outputs MCP crashes (closed)
• [smoke-detector] 🔍 Smoke Test Investigation - Smoke OpenCode Run #18722224746: Agent Does Not Use Safe-Outputs MCP Tools #2143 - OpenCode agent doesn't use safe-outputs (closed)

Next Steps

1. Immediate: Download and analyze agent-stdio.log from run 18840299097
2. Short-term: Add validation and conditional logic to prevent this failure mode
3. Long-term: Enhance agent prompts and MCP tool reliability across all engines

---

Investigation Metadata:

• Investigator: Smoke Detector (automated investigator)
• Investigation Run: 18840378169
• Pattern Database: /tmp/gh-aw/cache-memory/patterns/codex_no_artifact_staged.json
• Investigation Record: /tmp/gh-aw/cache-memory/investigations/2025-10-27-18840299097.json

AI generated by Smoke Detector - Smoke Test Failure Investigator

[pelikhan]

/q investigate run

[github-actions]

Agentic Q triggered by this issue comment.

[github-actions]

Q Workflow Optimization Investigation - Live Data Analysis

I've analyzed recent workflow runs (last 7 days, 10 runs) using live data from the gh-aw MCP server. Here's my report:

📊 Summary Statistics (Live Data)

• Total Runs Analyzed: 10
• Total Duration: 1.3 hours
• Total Tokens: 7.8M tokens
• Total Cost: $4.93
• Total Turns: 471
• Total Errors: 217
• Total Warnings: 63
• Total Missing Tools: 1

🔍 Critical Issues Found

1. Missing Tool: GitHub Workflow Runs API Access

Affected Workflow: Daily Firewall Logs Collector and Reporter (run #18823842600)

Error Evidence (from live logs):

Permission denied and could not request permission from user

Reason: Need to list workflow runs, download artifacts, and access run metadata for firewall log analysis. Current permissions block gh CLI and API calls.

Recommended Fix: Add proper GitHub token with workflow read permissions to the daily-firewall-report.md workflow:

permissions:
  actions: read      # Required for gh CLI workflow access
  contents: read

2. Copilot CLI 404 Errors (Multiple Workflows)

Affected Workflows:

• CLI Version Checker (4 warnings)
• Tidy
• Mergefest
• Daily Firewall Report

Error Pattern:

Request to GitHub API at https://api.enterprise.githubcopilot.com/agents/swe/custom-agents/githubnext/gh-aw?exclude_invalid_config=true failed with status 404

Impact: These are warnings but indicate configuration issues with custom agent loading. Currently non-blocking but should be investigated.

💰 Cost Analysis

High-Cost Workflows (from 10 sample runs):

1. Safe Output Health Monitor: $1.62 (21.1m, 127 turns, claude)
2. Documentation Unbloat: $1.36 (9.7m, 132 turns, claude)
3. Agentic Workflow Audit: $1.03 (20.3m, 108 turns, claude)
4. Schema Consistency Checker: $0.92 (8.4m, 104 turns, claude)

Observation: Claude workflows with high turn counts are the most expensive. Consider adding max-turns limits.

📈 Tool Usage Patterns

Most Used Tools (from 10 runs):

1. github: 69 calls across 4 runs
2. TodoWrite: 33 calls across 4 runs
3. Read: 26 calls across 4 runs
4. Edit: 19 calls in 1 run
5. Write: 15 calls across 2 runs

MCP Server Tools:

• gh-aw_logs: 9 calls (successful log analysis)
• safeoutputs: 11 calls (5 runs)
• safeoutputs_create_discussion: 3 calls (3 runs)

🚨 Error Patterns

Most Common Errors:

1. MCP Tool Response Truncation (multiple runs):

   MCP tool "logs" response (167098 tokens) exceeds maximum allowed tokens (25000)
   

   Impact: Log analysis workflows hitting token limits
   Fix: Add pagination parameters to gh aw logs calls

2. Tool Use Errors (Claude workflows):

   • File write attempts before reading
   • JQ syntax errors in bash commands
   • Path resolution issues

✅ Positive Findings

• Safe-outputs system working well: 11 successful safe-output operations
• gh-aw MCP integration successful: Logs and audit tools working properly
• No systemic failures: Most runs completing successfully

🎯 Recommendations

Immediate Actions:

1. Fix Daily Firewall Report Permissions

   • Add actions: read permission
   • Verify GitHub token has appropriate scopes

2. Add MCP Pagination

   • Update log analysis workflows to use pagination
   • Example: gh aw logs -c 5 instead of large batch fetches

3. Investigate Copilot 404s

   • Check if custom agent configuration is properly set up
   • May need to update agent endpoint URLs

Long-term Optimizations:

1. Cost Control

   • Add max-turns: 50 to high-cost Claude workflows
   • Consider using copilot engine for simpler tasks

2. Error Handling

   • Add retry logic for GitHub API calls
   • Improve error messages for permission issues

3. Tool Efficiency

   • Reduce repetitive MCP calls by caching results
   • Use batch operations where possible

📁 Data Source

All findings are based on live workflow logs downloaded from:

• Location: /tmp/gh-aw/aw-mcp/logs/
• Runs Analyzed: 18840939610, 18823850355, 18823838827, 18823835040, 18823785776, 18823842600, 18823746243, 18824363133, 18816898777, 18816855563
• Analysis Method: gh-aw MCP server logs tool with structured output parsing

---

Next Steps: I can create a pull request with fixes for the identified issues. Let me know if you'd like me to proceed with specific fixes!

AI generated by Q for #2604

[pelikhan]

/plan apply daily firewall suggestion

[github-actions]

✅ Agentic Plan Command completed successfully.