[smoke-detector] 🔍 Smoke Test Investigation - Run #63: Anthropic API Overload in Detection Job

[github-actions]

🔍 Smoke Test Investigation - Run #63

Summary

Smoke Claude workflow failed in the detection job due to a transient Anthropic API 500 error: "Overloaded". The main agent job completed successfully and created the expected issue output, but the secondary threat detection analysis failed when the API service was temporarily overloaded.

Failure Details

• Run: 18907236182
• Workflow: Smoke Claude
• Commit: 8f8cfbd
• Trigger: schedule
• Duration: 4.7 minutes
• Date: 2025-10-29 12:09:23 UTC

Root Cause Analysis

Primary Issue

The detection job failed with an Anthropic API 500 error indicating the service was overloaded:

API Error: 500 {"type":"error","error":{"type":"api_error","message":"Overloaded"},"request_id":null}

This occurred when the detection agent attempted to read /tmp/gh-aw/threat-detection/prompt.txt using the Read tool. The API call failed with "Streaming fallback triggered" before returning the overload error.

Why This Matters

• The main agent job succeeded (created issue output, 16 turns, 162k tokens, ~$0.22)
• The create_issue job succeeded
• Only the detection job failed due to this transient API error
• This is a new failure pattern - first occurrence of "Overloaded" error in our investigation history

Is This a Real Problem?

This is a transient infrastructure issue with the Anthropic API, not a code bug. However, it represents a workflow robustness issue:

1. The detection job is a secondary analysis step
2. Its failure caused the entire smoke test to fail
3. This could mask real issues in future runs

Failed Jobs and Errors

Failed: detection (2.7m)

Error Type: api_error - Anthropic API Overloaded
HTTP Status: 500
Message: "Overloaded"
Context: Failed while attempting to read threat detection prompt file
Is Transient: Yes ✅

Succeeded Jobs

• ✅ pre_activation (4s)
• ✅ activation (4s)
• ✅ agent (1.4m) - Main job completed successfully
• ✅ create_issue (7s) - Issue created successfully

Investigation Findings

Agent Performance

• Turns: 16
• Tokens Used: 162,803
• Estimated Cost: $0.22
• Task: Review last 5 merged PRs and create summary issue
• Result: Task completed successfully ✅

Detection Job Analysis

• Started threat detection analysis
• Attempted to read /tmp/gh-aw/threat-detection/prompt.txt
• Hit Anthropic API overload before completing first read
• Duration: 2.7 minutes (includes wait time for API)
• 4 turns attempted before failure

Pattern Analysis

Pattern ID: ANTHROPIC_API_OVERLOADED (New Pattern)

• First Occurrence: This run (2025-10-29)
• Severity: Medium (transient, not code-related)
• Category: AI Engine - API Failure
• Flaky: Yes - depends on Anthropic API load

Related Patterns Found:

• Similar to OPENCODE_ANTHROPIC_API_ERROR (AI_APICallError) seen in run 18893290104
• Different from typical OpenCode "no safe-outputs" patterns

Recommended Actions

🔴 High Priority

• Add retry logic to detection job: Implement exponential backoff retry for Anthropic API 500 errors
  • Initial delay: 30 seconds
  • Max retries: 3
  • Doubles delay on each retry
  • Prevents: Transient API failures causing workflow failure

🟡 Medium Priority

• Make detection job non-blocking: Add continue-on-error: true to detection job or make it optional

  • The detection job is a secondary analysis
  • Main agent success should be the primary signal
  • Prevents: False negatives in smoke tests masking real issues

• Add rate limit handling: Implement monitoring and graceful handling of Anthropic API rate limits

  • Track frequency of overload errors
  • Identify peak load times (scheduled runs may create spikes)
  • Prevents: Cascading failures during high load periods

🟢 Low Priority

• Implement fallback detection: Consider simpler static analysis when API unavailable
  • Prevents: Complete detection failure

Prevention Strategies

Immediate Actions

1. ✅ Pattern documented: Created /tmp/gh-aw/cache-memory/patterns/anthropic_api_overloaded.json
2. ✅ Investigation saved: Created /tmp/gh-aw/cache-memory/investigations/2025-10-29-18907236182.json
3. Next: Implement retry logic for detection job

Long-term Improvements

• Consider scheduled workflow staggering to reduce concurrent API load
• Monitor Anthropic API status before running detection jobs
• Implement circuit breaker pattern for repeated API failures

Historical Context

Similar Past Failures

• Run 18893290104 (2025-10-29): OpenCode with AI_APICallError calling Anthropic API
  • Different error: 401 vs 500
  • Different engine: OpenCode vs Claude
  • Same root: Anthropic API issues

Pattern Comparison

This is the first occurrence of the "Overloaded" error specifically. Previous Anthropic API failures were:

• Authentication errors (401)
• Generic API call errors
• Not "Overloaded" capacity issues

Frequency

• Anthropic API failures: Rare but recurring (3 occurrences in past week across different engines)
• This specific error: First occurrence

Related PR Context

PR #2717: "docs: document zizmor URL links and verbose Docker command output"

• Merged: Just before workflow run
• Content: Documentation updates
• Relevance: ❌ Not related to failure
• Assessment: Failure was due to external Anthropic API overload, not code changes

---

Investigation Metadata

• Pattern: ANTHROPIC_API_OVERLOADED (New)
• Investigator: smoke-detector
• Investigation Run: 18907360485
• Investigation Files:
  • /tmp/gh-aw/cache-memory/investigations/2025-10-29-18907236182.json
  • /tmp/gh-aw/cache-memory/patterns/anthropic_api_overloaded.json

AI generated by Smoke Detector - Smoke Test Failure Investigator

[pelikhan]

/q investigate why this url was redacted.

Investigation Run: 18907360485

[github-actions]

Agentic Q triggered by this issue comment.

[github-actions]

Investigation Complete: URL Redaction Bug Found 🔍

Root Cause Analysis

I've investigated run 18907360485 and found the root cause of why https://github.com/githubnext/gh-aw/actions/runs/18907236182 was redacted to (redacted).

The Bug

The URL sanitization regex in pkg/workflow/js/sanitize_output.cjs (line 117) has a character class escaping bug:

// CURRENT (BUGGY):
/\bhttps:\/\/([^\/\s\])}'"<>&\x00-\x1f]+)/gi
                    ^^
            This ] is NOT properly escaped!

The issue is that inside a character class [...], the sequence \] does NOT properly escape the closing bracket. Instead, it prematurely closes the character class at the ] character.

This means the regex is actually being interpreted as:

• [^\/\s\] - Match anything NOT a slash, whitespace, or backslash (character class closes here!)
• )}'"<>&\x00-\x1f]+) - Match the literal characters )}'"<>& plus hex range

Why It Affects Markdown Links

When the smoke detector agent creates investigation reports with markdown links in this format:

[18907236182](https://github.com/githubnext/gh-aw/actions/runs/18907236182)

The URL is immediately followed by a ) character. Because the regex doesn't properly capture the domain when a ) follows the URL, the domain matching fails and the URL gets redacted.

The Fix

The ] character should be properly escaped in the character class:

// FIXED:
/\bhttps:\/\/([^\/\s\])}'"<>&\x00-\x1f]+)/gi
//                   ^^^
//         Add backslash to escape the ]

Actually, the better fix is to use a different escaping approach. Inside character classes, to include a literal ], you should:

1. Place it first in the class: []...]
2. Or escape it properly with double backslash in the source

Recommended Fix

function sanitizeUrlDomains(s) {
  // Fixed regex: properly escape ] inside character class
  s = s.replace(/\bhttps:\/\/([^\/<>&\s\])}'"]+)/gi, (match, domain) => {
    const hostname = domain.split(/[\/:\?#]/)[0].toLowerCase();
    const isAllowed = allowedDomains.some(allowedDomain => {
      const normalizedAllowed = allowedDomain.toLowerCase();
      return hostname === normalizedAllowed || hostname.endsWith("." + normalizedAllowed);
    });
    return isAllowed ? match : "(redacted)";
  });
  return s;
}

Evidence

From run 18907360485's output (agent_output.json):

{
  "title": "🔍 Smoke Test Investigation - Run #63: Anthropic API Overload in Detection Job",
  "body": "# 🔍 Smoke Test Investigation - Run #63\n\n## Summary\n...\n\n## Failure Details\n- **Run**: [18907236182]((redacted))\n..."
}

The URL should have been preserved since github.com is in the default allowed domains list.

Impact

This bug affects:

• ✅ Smoke detector reports - Investigation reports with markdown links
• ✅ Any workflow output - That includes GitHub Actions run URLs in markdown link format
• ✅ Issue/PR creation - When agents create links to runs, commits, or other GitHub resources

Next Steps

I'll create a pull request with:

1. Fix for the regex pattern in sanitize_output.cjs
2. New test case to catch this specific markdown link format
3. Validation that GitHub URLs in markdown links are preserved

---

Investigation Run: 18907360485

AI generated by Q for #2730