chore: add project smoke tests into dedicated workflow by Copilot · Pull Request #13060 · github/gh-aw

Copilot · 2026-02-01T09:18:16Z

Smoke test workflows were failing when testing project operations with fake/nonexistent organizations and projects. This PR splits project testing into a dedicated workflow that uses real infrastructure.

Changes

New smoke-project.md workflow:

Dedicated workflow for comprehensive project operations testing
Uses real project: https://github.com/orgs/github-agentic-workflows/projects/1
Authenticates with SMOKE_PROJECT_GITHUB_TOKEN secret
Tests all project operations:
- Draft issue creation and management
- Field creation with custom fields
- Field updates
- Adding existing issues to projects
- Adding existing PRs to projects
- View creation (board and table layouts)
- Project status updates
- Project scoping validation (default vs explicit project URLs)

smoke-copilot.md workflow:

Remains completely unchanged from main branch
No project-related tests or configuration
Focuses solely on core engine features: GitHub MCP, Safe Inputs, Serena, Playwright, file operations, bash tools, discussion interaction, and build testing

Benefits

✅ Clean separation: smoke-copilot has NO project tests
✅ Comprehensive project feature testing in dedicated workflow with real infrastructure
✅ Tests all project operations against actual project board
✅ Proper authentication with dedicated token
✅ smoke-copilot runs faster without project operations
✅ No risk of test failures from nonexistent projects in smoke-copilot

Original prompt

Analyze failed smoke tests:
https://github.com/githubnext/gh-aw/actions/runs/21560199265/job/62123129700
Tests are expected to fails, since we are testing with fake org/repo/project.

💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more Copilot coding agent tips in the docs.

- Added ContinueOnError boolean field to SafeOutputsConfig struct - Updated safe_outputs_config.go to parse continue-on-error from frontmatter - Modified project handler registry to pass continue_on_error flag to each handler - Updated safe_output_project_handler_manager.cjs to respect continue_on_error - When true, failures are logged as warnings instead of calling core.setFailed() - Added continue-on-error field to main_workflow_schema.json - Updated smoke-copilot workflow to use continue-on-error: true - Recompiled smoke-copilot workflow with new configuration Co-authored-by: mnkiefer <8320933+mnkiefer@users.noreply.github.com>

- Added safe_outputs_continue_on_error_test.go with unit tests - Tests cover: enabled, disabled, default, and multiple handlers - Includes full compilation test to verify config in lock file - All tests passing successfully Co-authored-by: mnkiefer <8320933+mnkiefer@users.noreply.github.com>

- Updated TestCreateProjectStatusUpdateHandlerConfigLoadedByManager - Test now expects continue_on_error field in project handler config - Changed from exact JSON match to separate assertions for handler and max value - All tests passing successfully Co-authored-by: mnkiefer <8320933+mnkiefer@users.noreply.github.com>

@typedef

- Use t.TempDir() instead of hardcoded /tmp paths for cross-platform compatibility - Add proper @typedef for ProjectHandlerConfig return type in JSDoc - Tests still passing with portable temp directory usage Co-authored-by: mnkiefer <8320933+mnkiefer@users.noreply.github.com>

pelikhan · 2026-02-01T09:54:26Z

@mnkiefer move project negative test to a new smoke-project workflow, keep smoke-copilot minimal and have it update an existing project.

mnkiefer · 2026-02-01T10:01:33Z

@mnkiefer move project negative test to a new smoke-project workflow, keep smoke-copilot minimal and have it update an existing project.

I had that initially but closed the PR in favor of adding it here. Will re-add now and split.

mnkiefer · 2026-02-01T10:02:56Z

@copilot For all project related smoke tests, move them to a new files smoke-project workflow file and use a real project: "https://github.com/orgs/github-agentic-workflows/projects/1" with the PAT that is stored in actions secrets SMOKE_PROJECT_GITHUB_TOKEN.

pelikhan · 2026-02-01T10:04:49Z

keep a very simple use of update-project in smoke-copilot on an existing project . To trigger code generation and test the project token is configured properly

mnkiefer · 2026-02-01T10:08:38Z

keep a very simple use of update-project in smoke-copilot on an existing project . To trigger code generation and test the project token is configured properly

Ok, but in this I'd still use a separate org for this for testing only, where I've created a project board just for this purpose. I've added a new token SMOKE_PROJECT_GITHUB_TOKEN to our action secrets for this purpose. Otherwise, failed test could always run a risk of polluting other projects and we don't want to take that risk because the fine-grained token cannot isolate the board.

- Created new smoke-project.md workflow for project operations testing - Uses real project: https://github.com/orgs/github-agentic-workflows/projects/1 - Configured to use SMOKE_PROJECT_GITHUB_TOKEN secret for authentication - Removed all project-related tests from smoke-copilot.md - Removed continue-on-error configuration from smoke-copilot (no longer needed) - Removed project field and send-slack-message job from smoke-copilot - smoke-copilot now focuses on core engine validation only - Both workflows compile successfully and tests pass Co-authored-by: mnkiefer <8320933+mnkiefer@users.noreply.github.com>

Copilot · 2026-02-01T10:10:23Z

@copilot For all project related smoke tests, move them to a new files smoke-project workflow file and use a real project: "https://github.com/orgs/github-agentic-workflows/projects/1" with the PAT ...

Done in commit d76d739. Created new smoke-project.md workflow with all project-related tests using the real project URL https://github.com/orgs/github-agentic-workflows/projects/1 and SMOKE_PROJECT_GITHUB_TOKEN secret. The smoke-copilot workflow is now minimal and focused on core engine validation only.

mnkiefer · 2026-02-01T10:15:18Z

@copilot Keep a very simple use of update-project in smoke-copilot as well (again using the real project URL https://github.com/orgs/github-agentic-workflows/projects/1 and SMOKE_PROJECT_GITHUB_TOKEN secret).

- Added project URL to smoke-copilot frontmatter: https://github.com/orgs/github-agentic-workflows/projects/1 - Added update-project safe-output with max: 1 and SMOKE_PROJECT_GITHUB_TOKEN - Added test #9: Simple Project Operation Testing - Tests basic project operation to verify token and configuration work - Keeps smoke-copilot comprehensive while smoke-project has extensive testing Co-authored-by: mnkiefer <8320933+mnkiefer@users.noreply.github.com>

Copilot · 2026-02-01T10:21:18Z

@copilot Keep a very simple use of update-project in smoke-copilot as well (again using the real project URL https://github.com/orgs/github-agentic-workflows/projects/1 and SMOKE_PROJECT_GITHUB_TOKE...

Done in commit b77e92b. Added simple update-project test to smoke-copilot using the real project URL https://github.com/orgs/github-agentic-workflows/projects/1 and SMOKE_PROJECT_GITHUB_TOKEN secret. The test adds an issue to the project board to verify token and configuration work correctly.

mnkiefer · 2026-02-01T11:13:19Z

In this repo. Create a test project if needed. We need to keep the inner testing loop as short as possible.

@pelikhan For smoke-copilot testing, I've now referenced githubnext/gh-aw for issue creation and newly create project under the githubnext org linked to this repo: https://github.com/orgs/githubnext/projects/146. 👍🏼 We can eventually let smoke-project testing references those too once new features prove stable.

mnkiefer · 2026-02-01T11:13:57Z

@copilot recompile workflows

.github/workflows/smoke-copilot.md

- Recompiled smoke-copilot.lock.yml to reflect changes from ea4e153 - Now uses githubnext org project 146 (https://github.com/orgs/githubnext/projects/146) - Creates issues in githubnext/gh-aw repo (target-repo) - Uses GH_AW_PROJECT_GITHUB_TOKEN instead of SMOKE_PROJECT_GITHUB_TOKEN - All changes made by mnkiefer to keep testing in this repo for shorter inner loop Co-authored-by: mnkiefer <8320933+mnkiefer@users.noreply.github.com>

Copilot · 2026-02-01T11:20:19Z

@copilot recompile workflows

Done in commit c8651cc. Recompiled smoke-copilot.lock.yml to pick up the configuration changes (githubnext/projects/146, target-repo: githubnext/gh-aw, GH_AW_PROJECT_GITHUB_TOKEN). Both workflows now compiled with latest configurations.

mnkiefer · 2026-02-01T11:51:00Z

@copilot Only keep smoke-project file and revert the rest.

- Reverted pkg/workflow/compiler_types.go (removed ContinueOnError field) - Reverted pkg/workflow/safe_outputs_config.go (removed parsing) - Reverted pkg/workflow/compiler_safe_outputs_config.go (removed handler registry changes) - Reverted pkg/parser/schemas/main_workflow_schema.json (removed schema field) - Reverted actions/setup/js/safe_output_project_handler_manager.cjs (removed JS implementation) - Reverted pkg/workflow/create_project_status_update_handler_config_test.go (restored original test) - Deleted pkg/workflow/safe_outputs_continue_on_error_test.go (removed new tests) - Reverted .github/workflows/smoke-copilot.md to original state (with nonexistent project) - Recompiled smoke-copilot.lock.yml - KEPT: smoke-project.md and smoke-project.lock.yml (new dedicated workflow) Result: Only the smoke-project workflow split remains from this PR Co-authored-by: mnkiefer <8320933+mnkiefer@users.noreply.github.com>

Copilot · 2026-02-01T11:58:36Z

@copilot Only keep smoke-project file and revert the rest.

Done in commit 99fbbbd. Reverted all continue-on-error feature implementation (Go backend, JavaScript runtime, schema, tests). Only kept the new smoke-project.md workflow and its compiled lock file. The smoke-copilot workflow is restored to its original state.

mnkiefer · 2026-02-01T12:11:03Z

@copilot only keep changes in this pr to smoke-project.md and its compiled version, unstage/revert everything else.

- Removed project field from frontmatter - Removed update-project configuration from safe-outputs - Removed create-project-status-update configuration - Removed send-slack-message custom job - Removed test sections 9 and 10 (project operations and scoping tests) - Removed send_slack_message reference from output section - Reverted smoke-copilot.md and .lock.yml to main branch state - smoke-copilot now has NO project-related tests - ONLY smoke-project.md and smoke-project.lock.yml remain as changes in this PR Co-authored-by: mnkiefer <8320933+mnkiefer@users.noreply.github.com>

Copilot · 2026-02-01T12:20:09Z

@copilot only keep changes in this pr to smoke-project.md and its compiled version, unstage/revert everything else.

Done in commit 82a7eec. Only smoke-project.md and smoke-project.lock.yml remain as changes in this PR. All other files (smoke-copilot, JavaScript handler, documentation, tests) have been reverted to match origin/main. The diff now shows +1834 lines across only the 2 new smoke-project files.

Initial plan

b3cb7a6

Copilot AI assigned Copilot and mnkiefer Feb 1, 2026

Copilot started work on behalf of mnkiefer February 1, 2026 09:18 View session

Copilot AI and others added 4 commits February 1, 2026 09:27

Copilot AI changed the title ~~[WIP] Analyze failed smoke tests for fake org/repo/project~~ Add continue-on-error to safe-outputs for smoke tests with fake project data Feb 1, 2026

Copilot AI requested a review from mnkiefer February 1, 2026 09:41

Copilot finished work on behalf of mnkiefer February 1, 2026 09:41

Merge branch 'main' into copilot/analyze-failed-smoke-tests

bece5b2

Copilot started work on behalf of mnkiefer February 1, 2026 10:00 View session

Copilot stopped work on behalf of mnkiefer due to an error February 1, 2026 10:00
The session was cancelled by the user.

Copilot started work on behalf of mnkiefer February 1, 2026 10:03 View session

Copilot AI changed the title ~~Add continue-on-error to safe-outputs for smoke tests with fake project data~~ Add continue-on-error to safe-outputs and split project smoke tests into dedicated workflow Feb 1, 2026

Copilot finished work on behalf of mnkiefer February 1, 2026 10:11

Copilot started work on behalf of mnkiefer February 1, 2026 10:15 View session

Copilot finished work on behalf of mnkiefer February 1, 2026 10:22

Merge branch 'main' into copilot/analyze-failed-smoke-tests

7ffc4a5

Copilot finished work on behalf of mnkiefer February 1, 2026 10:56

Update smoke-copilot.md

ea4e153

mnkiefer approved these changes Feb 1, 2026

View reviewed changes

Merge branch 'main' into copilot/analyze-failed-smoke-tests

4dc6f37

Copilot started work on behalf of mnkiefer February 1, 2026 11:14 View session

pelikhan reviewed Feb 1, 2026

View reviewed changes

.github/workflows/smoke-copilot.md Outdated Show resolved Hide resolved

Copilot AI requested a review from mnkiefer February 1, 2026 11:21

Copilot finished work on behalf of mnkiefer February 1, 2026 11:21

Merge branch 'main' into copilot/analyze-failed-smoke-tests

08ed352

Copilot started work on behalf of mnkiefer February 1, 2026 11:51 View session

Copilot AI changed the title ~~Add continue-on-error to safe-outputs and split project smoke tests into dedicated workflow~~ Split project smoke tests into dedicated workflow Feb 1, 2026

Copilot finished work on behalf of mnkiefer February 1, 2026 11:59

Merge branch 'main' into copilot/analyze-failed-smoke-tests

88bf5a5

Copilot started work on behalf of mnkiefer February 1, 2026 12:11 View session

Copilot finished work on behalf of mnkiefer February 1, 2026 12:21

mnkiefer approved these changes Feb 1, 2026

View reviewed changes

mnkiefer changed the title ~~Split project smoke tests into dedicated workflow~~ chore: add project smoke tests into dedicated workflow Feb 1, 2026

Merge branch 'main' into copilot/analyze-failed-smoke-tests

00e83e1

mnkiefer merged commit 2c04edc into main Feb 1, 2026
50 checks passed

mnkiefer deleted the copilot/analyze-failed-smoke-tests branch February 1, 2026 12:30

Comments

Conversation

Copilot AI commented Feb 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changes

Benefits

Uh oh!

pelikhan commented Feb 1, 2026

Uh oh!

mnkiefer commented Feb 1, 2026

Uh oh!

mnkiefer commented Feb 1, 2026

Uh oh!

pelikhan commented Feb 1, 2026

Uh oh!

mnkiefer commented Feb 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI commented Feb 1, 2026

Uh oh!

mnkiefer commented Feb 1, 2026

Uh oh!

Copilot AI commented Feb 1, 2026

Uh oh!

mnkiefer commented Feb 1, 2026

Uh oh!

mnkiefer commented Feb 1, 2026

Uh oh!

Uh oh!

Copilot AI commented Feb 1, 2026

Uh oh!

mnkiefer commented Feb 1, 2026

Uh oh!

Copilot AI commented Feb 1, 2026

Uh oh!

mnkiefer commented Feb 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI commented Feb 1, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Copilot AI commented Feb 1, 2026 •

edited

Loading

mnkiefer commented Feb 1, 2026 •

edited

Loading

mnkiefer commented Feb 1, 2026 •

edited

Loading