[smoke-detector] 🔍 Smoke Test Investigation - GenAIScript Invalid Model Name (gpt-4.1)

# 🔍 Smoke Test Investigation - Run #18727962258

## Summary
The Smoke GenAIScript workflow failed in the `detection` job due to an **invalid OpenAI model name** configured in the GenAIScript engine. The workflow is using `openai:gpt-4.1` which is not a valid OpenAI model, causing GenAIScript to crash with a TypeError when attempting to access properties of an undefined result.

## Failure Details
- **Run**: [#18727962258](https://github.com/githubnext/gh-aw/actions/runs/18727962258)
- **Commit**: [9d6dacf](https://github.com/githubnext/gh-aw/commit/9d6dacff1d9753a0dc011a997192d9399ec86018) - "Update dictation prompt instructions with documentation-based glossary (#2152)"
- **Trigger**: workflow_dispatch
- **Duration**: 3.9 minutes
- **Failed Job**: detection (1.1 minutes)
- **Workflow**: Smoke GenAIScript

## Root Cause Analysis

### Primary Issue: Invalid Model Name

The GenAIScript engine configuration in `.github/workflows/shared/genaiscript.md:6` specifies:
```yaml
GH_AW_AGENT_MODEL_VERSION: "openai:gpt-4.1"
```

**Problem**: `gpt-4.1` is **not a valid OpenAI model name**. 

Valid OpenAI model names include:
- `gpt-4o` (latest GPT-4 optimized model)
- `gpt-4-turbo`
- `gpt-4`
- `gpt-3.5-turbo`

### Error Chain

1. GenAIScript attempts to resolve model `openai:gpt-4.1`
2. OpenAI API returns an error or null because the model doesn't exist
3. GenAIScript continues execution with undefined result
4. When trying to set GitHub Action outputs, it attempts to read `.text` property from undefined
5. **Crash**: `TypeError: Cannot read properties of undefined (reading 'text')`

### Stack Trace
```
2025-10-22T19:49:26.0407893Z Cannot read properties of undefined (reading 'text')
2025-10-22T19:49:26.0408573Z TypeError: Cannot read properties of undefined (reading 'text')
2025-10-22T19:49:26.0409433Z     at githubActionSetOutputs ((redacted))
2025-10-22T19:49:26.0410525Z     at async Command.runScriptWithExitCode ((redacted))
2025-10-22T19:49:26.0549645Z ##[error]Process completed with exit code 255.
```

## Failed Jobs and Errors

### Job Execution Summary
1. ✅ **activation** - succeeded (4s)
2. ✅ **agent** - succeeded (1.9m)
3. ❌ **detection** - **failed** (1.1m)
4. ✅ **create_issue** - succeeded (7s)
5. ⏭️ **missing_tool** - skipped

### Detection Job Failure
The detection job runs GenAIScript for threat detection analysis. From the logs:

```
2025-10-22T19:49:25.9916102Z   model: 'openai:gpt-4.1',
2025-10-22T19:49:25.9918993Z 2025-10-22T19:49:25.986Z genaiscript:host:node alias: large.model = openai:gpt-4.1 (source: script)
2025-10-22T19:49:25.9920566Z 2025-10-22T19:49:25.987Z genaiscript:modelalias large: openai:gpt-4.1 (script)
2025-10-22T19:49:25.9921343Z 2025-10-22T19:49:25.987Z genaiscript:models resolving model for 'openai:gpt-4.1'
```

GenAIScript successfully validates OPENAI_API_KEY but fails when attempting to use the invalid model.

## Investigation Findings

### Configuration Location
The invalid model is configured in:
- **File**: `.github/workflows/shared/genaiscript.md`
- **Line**: 6
- **Variable**: `GH_AW_AGENT_MODEL_VERSION`

### Affected Workflows
All workflows that import `shared/genaiscript.md` are affected:
- `smoke-genaiscript.md` - Currently failing (this investigation)
- Any other workflows using the GenAIScript shared configuration

### Why This Wasn't Caught Earlier
- OPENAI_API_KEY validation passes (the key exists)
- GenAIScript doesn't validate model names during initialization
- Error only occurs when attempting to use the model
- Poor error handling in GenAIScript's output setting code

### Related Historical Context
Similar issue was reported in #2142, but that was caused by missing OPENAI_API_KEY. This is a related but distinct issue - the API key exists but the model name is invalid.

**Pattern Database**: This is the first occurrence of the `GENAISCRIPT_INVALID_MODEL` pattern.

## Recommended Actions

### 🔴 CRITICAL - Immediate Fix
- [ ] **Update model name in `.github/workflows/shared/genaiscript.md:6`**
  ```yaml
  # Change from:
  GH_AW_AGENT_MODEL_VERSION: "openai:gpt-4.1"
  
  # Change to (recommended):
  GH_AW_AGENT_MODEL_VERSION: "openai:gpt-4o"
  ```

### 🟡 HIGH PRIORITY - GenAIScript Error Handling
- [ ] **Fix GenAIScript's githubActionSetOutputs function** (upstream issue)
  - Add null/undefined checks before accessing `.text` property
  - Provide meaningful error message when model resolution fails
  - File issue with GenAIScript project: https://github.com/microsoft/genaiscript

### 🟢 MEDIUM PRIORITY - Prevention
- [ ] **Add model name validation** in GenAIScript configuration
- [ ] **Document valid model names** in engine configuration comments
- [ ] **Add pre-flight check** to validate model exists before running workflow
- [ ] **Create smoke test** specifically for engine configuration validation

### 🔵 LOW PRIORITY - Documentation
- [ ] Update GenAIScript engine documentation with list of valid models
- [ ] Add troubleshooting guide for model configuration errors
- [ ] Document the relationship between model names and required API keys

## Prevention Strategies

### 1. Configuration Validation
Add a validation step at the beginning of GenAIScript workflows:
```yaml
- name: Validate model configuration
  run: |
    VALID_MODELS="gpt-4o gpt-4-turbo gpt-4 gpt-3.5-turbo"
    MODEL_NAME="${GH_AW_AGENT_MODEL_VERSION#openai:}"
    if [[ ! " $VALID_MODELS " =~ " $MODEL_NAME " ]]; then
      echo "Error: Invalid model '$MODEL_NAME'"
      echo "Valid models: $VALID_MODELS"
      exit 1
    fi
```

### 2. Schema Validation
Use JSON schema or similar to validate workflow configuration files before they're used.

### 3. Better Error Messages
Enhance GenAIScript to detect invalid model names and provide suggestions:
```
Error: Model 'gpt-4.1' not found
Did you mean: gpt-4o, gpt-4-turbo, gpt-4?
```

### 4. Smoke Test Coverage
Add specific smoke tests for each engine configuration to catch these issues before they affect production workflows.

## Technical Details

### Debug Logs Analysis
From the detection job logs, we can see GenAIScript attempting to resolve the model:
```
2025-10-22T19:49:25.9922308Z 2025-10-22T19:49:25.987Z genaiscript:models candidate openai:gpt-4.1
2025-10-22T19:49:25.9922980Z 2025-10-22T19:49:25.987Z genaiscript:models resolving openai:gpt-4.1
2025-10-22T19:49:25.9923735Z 2025-10-22T19:49:25.989Z genaiscript:config:env parsing token for 'openai:gpt-4.1:'
2025-10-22T19:49:25.9924493Z 2025-10-22T19:49:25.989Z genaiscript:config:env processing openai
```

The model resolution appears to succeed internally, but the actual API call fails because the model doesn't exist.

### MCP Configuration Note
```
Failed to load MCP configuration: MCP configuration file not found: /tmp/gh-aw/mcp-config/mcp-servers.json
```
This is a separate (minor) issue - MCP configuration is optional for this workflow.

## Impact Assessment

**Severity**: 🔴 **HIGH**
- All GenAIScript-based workflows are failing
- Threat detection is not running (security impact)
- Smoke tests cannot validate GenAIScript engine functionality

**Urgency**: 🔴 **IMMEDIATE**
- Simple one-line fix required
- Blocking critical functionality

**Scope**: 
- Affects: All workflows using `shared/genaiscript.md`
- Not affected: OpenCode, Claude, and other engine workflows

## Reproduction Steps

1. Configure GenAIScript with model: `openai:gpt-4.1`
2. Ensure OPENAI_API_KEY is set (so validation passes)
3. Run any GenAIScript workflow
4. Observe failure when GenAIScript attempts to use the invalid model
5. See TypeError when accessing undefined result

## Related Issues

- #2142 - Similar GenAIScript error with missing OPENAI_API_KEY (different root cause)

---

## Investigation Metadata

- **Investigator**: Smoke Detector (Investigation Agent)
- **Investigation Run**: [#18728054085](https://github.com/githubnext/gh-aw/actions/runs/18728054085)
- **Pattern**: `GENAISCRIPT_INVALID_MODEL` (first occurrence)
- **Pattern File**: `/tmp/gh-aw/cache-memory/patterns/genaiscript_invalid_model.json`
- **Investigation Record**: `/tmp/gh-aw/cache-memory/investigations/2025-10-22-18727962258.json`
- **Created**: 2025-10-22T19:52:00Z



> AI generated by [Smoke Detector - Smoke Test Failure Investigator](https://github.com/githubnext/gh-aw/actions/runs/18728054085)




> AI generated by [Smoke Detector - Smoke Test Failure Investigator](https://github.com/githubnext/gh-aw/actions/runs/18728054085)

[smoke-detector] 🔍 Smoke Test Investigation - GenAIScript Invalid Model Name (gpt-4.1) #2157

Description

🔍 Smoke Test Investigation - Run #18727962258

Summary

Failure Details

Root Cause Analysis

Primary Issue: Invalid Model Name

Error Chain

Stack Trace

Failed Jobs and Errors

Job Execution Summary

Detection Job Failure

Investigation Findings

Configuration Location

Affected Workflows

Why This Wasn't Caught Earlier

Related Historical Context

Recommended Actions

🔴 CRITICAL - Immediate Fix

🟡 HIGH PRIORITY - GenAIScript Error Handling

🟢 MEDIUM PRIORITY - Prevention

🔵 LOW PRIORITY - Documentation

Prevention Strategies

1. Configuration Validation

2. Schema Validation

3. Better Error Messages

4. Smoke Test Coverage

Technical Details

Debug Logs Analysis

MCP Configuration Note

Impact Assessment

Reproduction Steps

Related Issues

Investigation Metadata

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions