Fail Claude smoke runs when startup exits before structured log output by Copilot · Pull Request #16240 · github/gh-aw

Copilot · 2026-02-16T23:22:33Z

Smoke-Claude could report success even when Claude failed during startup, because the parser only failed runs for MCP startup failures or max-turns and did not treat “no structured Claude entries” as an execution failure. This allowed silent startup/configuration regressions (e.g., .claude.json write failures) to evade smoke detection.

What changed
- Added a Claude-specific failure guard in the shared log parser bootstrap: if Claude parsing yields no structured logEntries, mark the run failed.
- This closes the gap where Claude exits early with no tool activity and no MCP/max-turns signal.
Behavioral impact
- Smoke-Claude now fails fast on silent pre-tool failures instead of emitting a misleading “parsed successfully” path.
- Existing failure paths (MCP launch failures, max-turns limit) remain unchanged.
Targeted test coverage
- Added coverage in log_parser_bootstrap.test.cjs for the Claude + empty logEntries failure path.
- Added integration coverage in parse_claude_log.test.cjs to assert failure on unstructured/non-JSON Claude output.

// actions/setup/js/log_parser_bootstrap.cjs
if (parserName === "Claude" && (!logEntries || logEntries.length === 0)) {
  core.setFailed(
    "Claude execution failed: no structured log entries were produced. This usually indicates a startup or configuration error before tool execution."
  );
}

💬 We'd love your input! Share your thoughts on Copilot coding agent in our 2 minute survey.

Changeset

Type: patch
Description: Fail Claude log parsing when no structured entries appear so a noisy execution failure surfaces instead of silently succeeding.

Generated by Changeset Generator

Co-authored-by: Mossaka <5447827+Mossaka@users.noreply.github.com>

github-actions · 2026-02-17T00:01:31Z

📰 BREAKING: Smoke Copilot is now investigating this pull request. Sources say the story is developing...

github-actions · 2026-02-17T00:01:31Z

🧪 Smoke Temporary ID is now testing temporary ID functionality...

github-actions · 2026-02-17T00:01:34Z

🧪 Smoke Project is now testing project operations...

github-actions · 2026-02-17T00:01:35Z

✨ The prophecy is fulfilled... Smoke Codex has completed its mystical journey. The stars align. 🌟

github-actions · 2026-02-17T00:01:36Z

❌ Smoke Copilot SDK failed. Please review the logs for details.

github-actions · 2026-02-17T00:01:38Z

🎬 THE END — Smoke Claude MISSION: ACCOMPLISHED! The hero saves the day! ✨

github-actions · 2026-02-17T00:03:51Z

Agent Container Tool Check

Tool	Status	Version
bash	✅	5.2.21
sh	✅	available
git	✅	2.52.0
jq	✅	1.7
yq	✅	4.52.2
curl	✅	8.5.0
gh	✅	2.86.0
node	✅	20.20.0
python3	✅	3.12.3
go	✅	1.24.13
java	✅	21.0.10
dotnet	✅	10.0.102

Result: 12/12 tools available ✅

🔧 Tool validation by Agent Container Smoke Test for issue #16240

github-actions · 2026-02-17T00:04:13Z

✅ Smoke Temporary ID completed successfully. Temporary ID validation passed.

github-actions · 2026-02-17T00:04:49Z

✅ Smoke Project completed successfully. All project operations validated.

github-actions · 2026-02-17T00:05:50Z

🤖 Beep boop! The smoke test agent just teleported through discussion #16225! 🚀

I left a trail of binary breadcrumbs while testing the Copilot engine. If you hear mysterious clicking sounds, that's just me running more tests in the background!

🎭 Smoke test agent, signing off with style!

📰 BREAKING: Report filed by Smoke Copilot for issue #16240

github-actions · 2026-02-17T00:06:01Z

Smoke Test Results: PASS ✅

[instructions] Add --fail-fast flag to compilation options #16244: [instructions] Add --fail-fast flag
🧹 Remove old package functionality in workflow addition #16246: 🧹 Remove old package functionality

10/11 tests passed (Serena MCP not available)

@Mossaka @Copilot

📰 BREAKING: Report filed by Smoke Copilot for issue #16240

github-actions

Solid implementation of Claude execution validation. The guardrail catches silent failures early.

📰 BREAKING: Report filed by Smoke Copilot for issue #16240

github-actions · 2026-02-17T00:06:02Z

actions/setup/js/log_parser_bootstrap.test.cjs

+          } finally {
+            fs.unlinkSync(logFile);
+            fs.rmdirSync(tmpDir);
+          }


Test coverage for the zero-entries guardrail ensures silent failures are caught.

github-actions · 2026-02-17T00:06:02Z

actions/setup/js/log_parser_bootstrap.cjs

+    // Claude-specific guardrail: if no structured log entries were parsed, treat as execution failure.
+    // This catches silent startup failures where Claude exits before producing JSON tool activity.
+    if (parserName === "Claude" && (!logEntries || logEntries.length === 0)) {
+      core.setFailed("Claude execution failed: no structured log entries were produced. This usually indicates a startup or configuration error before tool execution.");


Good safeguard! This prevents silent failures where Claude exits before tool execution.

github-actions · 2026-02-17T00:06:31Z

📰 VERDICT: Smoke Copilot has concluded. All systems operational. This is a developing story. 🎤

github-actions · 2026-02-17T00:06:39Z

Smoke test (Codex)
PRs: [instructions] Add --fail-fast flag to compilation options; Fix indentation in compiler_activation_jobs.go
GitHub MCP: ✅
Serena MCP: ✅
Playwright: ✅
File write: ✅
Bash cat: ✅
Build: ✅
Overall: PASS

🔮 The oracle has spoken through Smoke Codex for issue #16240

github-actions · 2026-02-17T00:09:05Z

Smoke Test: Claude - Run 22081144175

Core Tests (1-10): ✅✅✅✅✅✅✅✅✅✅

PR Review Tests (11-17): ✅✅✅⚠️✅⚠️⚠️

Overall Status: PARTIAL

All critical tests passed. 3 tests appropriately skipped due to environmental constraints.

💥 [THE END] — Illustrated by Smoke Claude for issue #16240

github-actions

💥 Automated smoke test review - all systems nominal!

The documentation updates look great! Version tracking and quality verification metadata are properly maintained.

💥 [THE END] — Illustrated by Smoke Claude for issue #16240

Copilot

Pull request overview

This PR adds a Claude-specific failure guard to catch silent startup failures where Claude exits before producing any structured log entries. Previously, smoke-Claude could report success even when Claude failed during startup because the parser only failed runs for MCP startup failures or max-turns limits.

Changes:

Added a check in log_parser_bootstrap.cjs that fails Claude runs when no structured log entries are produced
Added unit test coverage in log_parser_bootstrap.test.cjs for the new failure path
Added integration test in parse_claude_log.test.cjs to verify failure on unstructured Claude output

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.

File	Description
actions/setup/js/log_parser_bootstrap.cjs	Added Claude-specific guardrail (lines 212-216) to fail runs when no structured log entries are parsed
actions/setup/js/log_parser_bootstrap.test.cjs	Added test case (lines 65-80) to verify Claude runs fail when no structured log entries are produced
actions/setup/js/parse_claude_log.test.cjs	Added integration test (lines 441-444) to verify failure on unstructured Claude output

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-02-17T00:14:50Z

actions/setup/js/log_parser_bootstrap.cjs

+    if (parserName === "Claude" && (!logEntries || logEntries.length === 0)) {
+      core.setFailed("Claude execution failed: no structured log entries were produced. This usually indicates a startup or configuration error before tool execution.");
+    }


The success message "Claude log parsed successfully" is logged on line 182 before the failure check on line 214. This creates a confusing situation where Claude logs indicate success, then immediately fail the run. The check for empty logEntries should happen earlier, before any success messages are logged, or the success message should be conditional to avoid this contradiction.

Initial plan

962ed77

Copilot AI assigned Copilot and Mossaka Feb 16, 2026

Copilot started work on behalf of Mossaka February 16, 2026 23:22 View session

fix: fail Claude runs with no structured log entries

6c73cde

Co-authored-by: Mossaka <5447827+Mossaka@users.noreply.github.com>

Copilot AI changed the title ~~[WIP] Investigate issue with smoke-claude detection~~ Fail Claude smoke runs when startup exits before structured log output Feb 16, 2026

Copilot AI requested a review from Mossaka February 16, 2026 23:31

Copilot finished work on behalf of Mossaka February 16, 2026 23:31

Mossaka added the smoke label Feb 17, 2026

Mossaka marked this pull request as ready for review February 17, 2026 00:01

Copilot AI review requested due to automatic review settings February 17, 2026 00:01

Merge branch 'main' into copilot/investigate-smoke-claude-issue

c82d9d7

Copilot started reviewing on behalf of Mossaka February 17, 2026 00:01 View session

This was referenced Feb 17, 2026

[smoke-temporary-id] Test Parent: Temporary ID Validation #16249

Closed

[smoke-temporary-id] Sub-Issue 2: Test Different ID Length #16250

Closed

[smoke-temporary-id] Sub-Issue 1: Test Temporary ID References #16251

Closed

Add changeset [skip-ci]

2bf7c41

github-actions bot mentioned this pull request Feb 17, 2026

Smoke Test: Copilot - 22081144182 #16253

Closed

github-actions bot added smoke-copilot and removed smoke labels Feb 17, 2026

github-actions bot reviewed Feb 17, 2026

View reviewed changes

github-actions bot added the smoke-codex label Feb 17, 2026

github-actions bot mentioned this pull request Feb 17, 2026

Smoke Test: Claude - 22081144175 #16254

Closed

github-actions bot reviewed Feb 17, 2026

View reviewed changes

Mossaka merged commit f75f2dd into main Feb 17, 2026
1 check passed

Mossaka deleted the copilot/investigate-smoke-claude-issue branch February 17, 2026 00:11

Copilot AI reviewed Feb 17, 2026

View reviewed changes

Comments

Conversation

Copilot AI commented Feb 16, 2026 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changeset

Uh oh!

github-actions bot commented Feb 17, 2026

Uh oh!

github-actions bot commented Feb 17, 2026

Uh oh!

github-actions bot commented Feb 17, 2026

Uh oh!

github-actions bot commented Feb 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Feb 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Feb 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Feb 17, 2026

Agent Container Tool Check

Uh oh!

github-actions bot commented Feb 17, 2026

Uh oh!

github-actions bot commented Feb 17, 2026

Uh oh!

github-actions bot commented Feb 17, 2026

Uh oh!

github-actions bot commented Feb 17, 2026

Uh oh!

github-actions bot left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot Feb 17, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions bot Feb 17, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Feb 17, 2026

Uh oh!

github-actions bot commented Feb 17, 2026

Uh oh!

github-actions bot commented Feb 17, 2026

Smoke Test: Claude - Run 22081144175

Uh oh!

github-actions bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI Feb 17, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Copilot AI commented Feb 16, 2026 •

edited by github-actions bot

Loading

github-actions bot commented Feb 17, 2026 •

edited

Loading

github-actions bot commented Feb 17, 2026 •

edited

Loading

github-actions bot commented Feb 17, 2026 •

edited

Loading