[smoke-detector] 🚨 CRITICAL: GenAIScript Failure Persists After Model Fix - 7th Consecutive Failure (API Key or Service Issue) #2272
Description
🚨 CRITICAL RECURRING FAILURE - 7th Consecutive Occurrence
Summary
The Smoke GenAIScript workflow has FAILED AGAIN with the EXACT SAME ERROR, despite the model configuration being fixed from gpt-4.1 to gpt-4o in previous investigations. This is the 7th consecutive failure since 2025-10-22. The pattern has EVOLVED - the model name is now correct, but the error persists, pointing to a different root cause.
Failure Details
- Run: #18771194604
- Commit: 4fcfdc4 - "Add support for silently ignoring description and applyTo fields in frontmatter (Add support for silently ignoring description and applyTo fields in frontmatter #2266)"
- Trigger: schedule (automated smoke test)
- Duration: 3.1 minutes
- Failed Job: detection (59 seconds)
- Status: ❌ FAILED
Root Cause Analysis - Pattern Evolution
🔄 Pattern Has Changed
Previous Issues (#2157, #2204, #2207, #2227): Invalid model name openai:gpt-4.1
Current Configuration: ✅ Valid model name openai:gpt-4o (confirmed in .github/workflows/shared/genaiscript.md:6)
Result: ❌ Same error persists
This definitively rules out model configuration as the root cause. The failure has evolved from a configuration issue to what appears to be an authentication, service availability, or code bug issue.
Error Chain (Unchanged Despite Fix)
- GenAIScript attempts to use model
openai:gpt-4o(valid model) ✅ - OPENAI_API_KEY validation passes ✅
- GenAIScript calls OpenAI API ❓
- Something fails - API returns error/null/unexpected response ❌
- GenAIScript crashes:
TypeError: Cannot read properties of undefined (reading 'text')❌ - Detection job fails with exit code 255 ❌
Stack Trace (Identical to All Previous Occurrences)
2025-10-24T06:09:40.6320337Z 2025-10-24T06:09:40.631Z genaiscript:error {
2025-10-24T06:09:40.6320853Z name: 'TypeError',
2025-10-24T06:09:40.6321311Z message: "Cannot read properties of undefined (reading 'text')",
2025-10-24T06:09:40.6321918Z stack: "TypeError: Cannot read properties of undefined (reading 'text')\\n" +
2025-10-24T06:09:40.6322786Z ' at githubActionSetOutputs ((redacted))\\n' +
2025-10-24T06:09:40.6324255Z ' at async Command.runScriptWithExitCode ((redacted))'
2025-10-24T06:09:40.6324970Z }
2025-10-24T06:09:40.6325324Z Cannot read properties of undefined (reading 'text')
Investigation Findings
Possible Root Causes (Now That Config Is Fixed)
-
Invalid/Expired OPENAI_API_KEY 🔴 MOST LIKELY
- The secret may exist but be invalid/expired
- API returning 401/403 authentication errors
- GenAIScript doesn't properly handle auth failures
-
OpenAI API Service Issues 🟡 POSSIBLE
- Rate limiting (429 errors)
- Service degradation or outages
- Timeout issues
-
GenAIScript Bug 🟡 POSSIBLE
- Poor error handling in
githubActionSetOutputs - Doesn't check for undefined before accessing
.text - Mishandles certain API error responses
- Poor error handling in
-
MCP Configuration Issue 🟢 UNLIKELY (but noted)
Failed to load MCP configuration: MCP configuration file not found: /tmp/gh-aw/mcp-config/mcp-servers.jsonThis is a separate warning - MCP config is optional for this workflow
Configuration Status ✅
# .github/workflows/shared/genaiscript.md:6
GH_AW_AGENT_MODEL_VERSION: "openai:gpt-4o" ✅ CORRECTThe model configuration was successfully updated and is now valid. The workflow also includes:
- ✅ API key validation step (passes)
- ✅ GenAIScript version 2.5.1 installed
- ✅ Proper environment variable setup
Failed Jobs and Errors
Job Execution Summary
- ✅ activation - succeeded (2s)
- ✅ agent - succeeded (1.4m) - Agent completed successfully
- ❌ detection - FAILED (59s) - Threat detection crashed
- ✅ create_issue - succeeded (3s)
- ⏭️ missing_tool - skipped
Failure Timeline - 7 Consecutive Occurrences
| # | Run ID | Date/Time (UTC) | Hours Since Prev | Model Config | Status |
|---|---|---|---|---|---|
| 1 | 18727962258 | 2025-10-22 19:45:52 | - | gpt-4.1 ❌ |
Issue #2157 created |
| 2 | 18733557489 | 2025-10-23 00:19:22 | ~5.5h | gpt-4.1 ❌ |
Issue #2157 closed |
| 3 | 18739169072 | 2025-10-23 06:07:04 | ~6.2h | gpt-4.1 ❌ |
Issue #2204 created |
| 4 | 18747816413 | 2025-10-23 12:08:41 | ~6.6h | gpt-4.1 ❌ |
Issue #2207 created |
| 5 | 18757658104 | 2025-10-23 18:06:57 | ~6.0h | gpt-4.1 ❌ |
Issue #2227 created |
| 6 | 18765594567 | 2025-10-24 00:17:56 | ~6.2h | gpt-4o ✅ |
Model fixed! But still failed |
| 7 | 18771194604 | 2025-10-24 06:06:47 | ~6.1h | gpt-4o ✅ |
This failure |
Pattern: Failing every ~6 hours on scheduled runs
Duration: Over 34 hours of continuous failures
Failure Rate: 100% since first occurrence
Model Fix: Occurred between run #5 and #6, but failures continued
Recommended Actions
🔴 CRITICAL - Immediate Investigation
-
Verify OPENAI_API_KEY secret in repository settings
- Check if the key exists
- Test the key manually with a simple OpenAI API call
- Rotate key if invalid/expired
- Ensure proper permissions are set
-
Check OpenAI API Status
- Visit (redacted)
- Check for service degradation or outages
- Review any rate limiting or quota issues
-
Enable Detailed Logging
- Capture actual OpenAI API responses
- Log HTTP status codes and error messages
- Add verbose debugging to GenAIScript execution
🟡 HIGH PRIORITY - GenAIScript Error Handling
-
File upstream bug with GenAIScript
- Repository: https://github.com/microsoft/genaiscript
- Issue:
githubActionSetOutputsdoesn't handle undefined results - Request: Add null checks before accessing
.textproperty - Request: Better error messages for API failures
-
Add Retry Logic
- Implement exponential backoff for transient failures
- Add circuit breaker pattern for persistent failures
🟢 MEDIUM PRIORITY - Alternative Solutions
-
Consider switching providers
- Option A: GitHub Models (
github:gpt-4o) - Option B: Azure OpenAI (if configured)
- Option C: Disable threat detection until issue resolved
- Option A: GitHub Models (
-
Add Health Check
- Pre-flight API key validation with test call
- Fail fast with clear error message
- Skip smoke test if API unavailable
🔵 LOW PRIORITY - Documentation
- Update troubleshooting guide with API key validation steps
- Document OpenAI API dependency and alternatives
- Add monitoring for API key expiration
Prevention Strategies
- Implement API Key Health Checks - Test API keys before workflows run
- Better Error Handling - Fix GenAIScript to handle API failures gracefully
- Monitoring & Alerts - Monitor OpenAI API status and API key validity
- Retry Logic - Add exponential backoff for transient API failures
- Provider Alternatives - Configure fallback AI providers
- Scheduled Key Rotation - Proactively rotate API keys before expiration
Impact Assessment
Severity: 🔴 CRITICAL
- All GenAIScript smoke tests failing continuously for 34+ hours
- Threat detection non-functional
- 7 consecutive failures with no resolution
- Model fix applied but issue persists
- Pattern evolution indicates deeper systemic issue
Urgency: 🔴 IMMEDIATE
- Issue extends beyond configuration to authentication/service layer
- Requires investigation of API key validity
- May require provider change if OpenAI unreliable
Scope:
- Affects: All workflows using GenAIScript with OpenAI
- Frequency: Every scheduled smoke test run (~6 hours)
- Duration: Ongoing since 2025-10-22 19:45 UTC (34+ hours)
- Wasted CI Minutes: ~24.5 minutes (7 failures x 3.5 min average)
Historical Context
From investigation database (/tmp/gh-aw/cache-memory/investigations/):
{
"pattern_signature": "GENAISCRIPT_INVALID_MODEL → GENAISCRIPT_API_OR_OUTPUT_ERROR",
"pattern_evolution": true,
"first_occurrence": "2025-10-22T19:45:52Z",
"recurrence_count": 7,
"failure_rate": "100%",
"model_fix_applied": "2025-10-24 between run 5 and 6",
"failures_after_fix": 2,
"is_flaky": false,
"external_dependency": "OpenAI API",
"persistence_across_releases": true
}Related Issues
- [smoke-detector] 🚨 CRITICAL: GenAIScript Invalid Model (gpt-4.1) - 5th Consecutive Failure Post-v0.24.0 #2227 - 5th occurrence (closed as "not_planned") - Model config issue identified
- [smoke-detector] 🚨 CRITICAL RECURRING: GenAIScript Invalid Model (gpt-4.1) - 3rd Occurrence #2204 - 3rd occurrence (closed as "completed") - Model config issue identified
- [smoke-detector] Comment on #2157 #2207 - 4th occurrence (closed as "completed") - Model config issue identified
- [smoke-detector] 🔍 Smoke Test Investigation - GenAIScript Invalid Model Name (gpt-4.1) #2157 - Original investigation (closed as "not_planned") - First to identify invalid model
- [smoke-detector] 🔍 Smoke Test Investigation - GenAIScript OPENAI_API_KEY Missing #2142 - Similar GenAIScript error (missing API key) - Different root cause
Next Steps
This issue requires immediate attention from someone with access to:
- Repository secrets (to verify OPENAI_API_KEY)
- OpenAI account/billing (to check API key status)
- Alternative AI provider configuration
The model configuration has been fixed, but failures persist. This is now an authentication, service, or code bug issue - not a configuration issue.
Investigation Metadata
- Investigator: Smoke Detector (Failure Investigation Agent)
- Investigation Run: #18771256728
- Pattern:
GENAISCRIPT_API_OR_OUTPUT_ERROR(evolved fromGENAISCRIPT_INVALID_MODEL) - Investigation Record:
/tmp/gh-aw/cache-memory/investigations/2025-10-24-18771194604.json - Created: 2025-10-24T06:12:00Z
🤖 AI generated by Smoke Detector - Smoke Test Failure Investigator
This is an automated investigation of recurring smoke test failures.
AI generated by Smoke Detector - Smoke Test Failure Investigator