Add build setup to Daily Syntax Error Quality Check workflow

[Copilot]

The Daily Syntax Error Quality Check workflow requires the gh-aw compiler binary to test error message quality, but the binary was not available in PATH during agent execution.

Changes

Added steps: section to workflow frontmatter that executes before the agent job:

steps:
  - name: Set up Go
    uses: actions/setup-go@v5
    with:
      go-version-file: go.mod
      cache: true
      
  - name: Build gh-aw
    run: |
      make build
      
  - name: Verify gh-aw installation
    run: |
      ./gh-aw --version
      echo "gh-aw binary is ready at ./gh-aw"

The Go setup uses go-version-file: go.mod to match the ci.yml build pattern, ensuring the workflow uses the correct Go version (1.25.0) specified in go.mod.

The agent can now execute ./gh-aw compile commands to introduce syntax errors and capture compiler output for quality analysis.

Files Modified

• .github/workflows/daily-syntax-error-quality.md - Added build steps in frontmatter
• .github/workflows/daily-syntax-error-quality.lock.yml - Regenerated by compiler

Original prompt

---

This section details on the original issue you should resolve

<issue_title>[syntax-error-quality] Daily Syntax Error Quality Check - Setup Required</issue_title>
<issue_description>### 🔍 Workflow Analysis Status

Analysis Date: 2026-02-12
Status: ⚠️ Setup Required
Workflow: Daily Syntax Error Quality Check

---

Executive Summary

The Daily Syntax Error Quality Check workflow is designed to test compiler error message quality by introducing syntax errors into workflows and evaluating the error output. However, the current workflow environment lacks the necessary gh-aw compiler binary to perform the analysis.

Key Issues:

• gh-aw compiler binary not available in workflow PATH
• Cannot build binary due to permission restrictions
• Unable to compile test workflows and capture error messages
• Quality analysis cannot be completed without compiler access

---

Required Setup

To enable this workflow to function correctly, the following setup is required:

1. Add Workflow Setup Steps

The workflow needs to include steps that build or install the gh-aw binary before the agent runs:

steps:
  - name: Setup Go
    uses: actions/setup-go@v5
    with:
      go-version: '1.23'
      
  - name: Build gh-aw
    run: |
      cd /home/runner/work/gh-aw/gh-aw
      make build
      
  - name: Add to PATH
    run: |
      echo "$GITHUB_WORKSPACE" >> $GITHUB_PATH

2. Grant Build Permissions

Ensure the workflow has permissions to:

• Read repository contents
• Build Go binaries
• Write to temporary directories

3. Verify Binary Access

Add a verification step to ensure the binary is accessible:

steps:
  - name: Verify gh-aw installation
    run: |
      which gh-aw || echo "gh-aw not in PATH"
      ./gh-aw --version || echo "gh-aw binary not executable"

---

Analysis Design

Once the setup is complete, the workflow should execute this analysis process:

Phase 1: Select Test Workflows

Select 3 diverse workflows for testing:

1. Simple workflow (< 100 lines, minimal config)
2. Medium workflow (100-300 lines, moderate config)
3. Complex workflow (> 300 lines, many tools/features)

Criteria:

• Avoid daily-* and test workflows
• Choose different complexity levels
• Prefer different engines, tools, safe-outputs configurations

Phase 2: Introduce Syntax Errors

Create 3 test cases with different error categories:

Category A: Frontmatter Syntax Errors

engine copilot  # Missing colon after key

Category B: Configuration Errors

engine: copiilot  # Typo in engine name

Category C: Semantic Errors

tools:
  github:
    lockdown: true
    toolsets: [default]  # Conflicting: lockdown with toolsets

Phase 3: Compile and Capture Output

For each test case:

./gh-aw compile test-workflow.md 2>&1 | tee test-output.txt

Phase 4: Evaluate Error Quality

Score error messages across 5 dimensions:

Dimension	Weight	Criteria
Clarity	25 pts	Clear, non-technical language; obvious what went wrong
Actionability	25 pts	Specific suggestions, steps to fix, documentation links
Context	20 pts	Shows problematic code, highlights error location
Examples	15 pts	Provides correct usage examples
Consistency	15 pts	IDE-parseable format (file:line:column), consistent terminology

Quality Threshold: Average score ≥ 70 across all test cases

Phase 5: Create Issue with Findings

Only create an issue if:

• Average score < 70 across all test cases, OR
• Any individual test case scores < 55, OR
• Critical pattern issues are identified

---

Example Test Workflows

View Test Case Examples

Test 1: Frontmatter Syntax Error (Simple Workflow)

File: test-1-frontmatter-syntax.md

---
description: Test workflow
on:
  issues:
    types: [opened]
engine copilot  # ERROR: Missing colon
tools:
  github:
    toolsets: [default]
---

# Test Workflow
``````

**Expected Error Output**:
``````
test-1-frontmatter-syntax.md:8:7: error: invalid YAML syntax
  8 | engine copilot
    |       ^ expected ':' after key

Valid engines: copilot, claude, codex, custom

Correct usage:
  engine: copilot

Test 2: Invalid Engine Name (Medium Workflow)

File: test-2-invalid-engine.md

---
description: Test workflow
on:
  workflow_run:
    workflows: [CI]
engine: copiilot  # ERROR: Typo in engine name
tools:
  github:
    toolsets: [default]
---

# Test Workflow
``````

**Expected Error Output**:
``````
test-2-invalid-engine.md:6:9: error: invalid engine 'copiilot'

Valid engines: copilot, claude, codex, custom

Did you mean: copilot?

Correct usage:
  engine: copilot

Test 3: Conflicting Configuration (Complex Workflow)

File: `test-3-conflicting-c...

• Fixes [syntax-error-quality] Daily Syntax Error Quality Check - Setup Required #15074

---

💬 We'd love your input! Share your thoughts on Copilot coding agent in our 2 minute survey.

[github-actions]

🔍 PR Triage Results

Category: chore | Risk: low | Priority: 40/100

Scores Breakdown

• Impact: 20/50 - Workflow configuration improvement that enables Daily Syntax Error Quality Check to function properly
• Urgency: 8/30 - Not WIP, fixes critical quality check workflow, should be merged soon
• Quality: 12/20 - CI pending, good description with clear purpose, workflow-only changes are low risk

📋 Recommended Action: fast_track

Explanation: This PR adds necessary build setup steps to enable the quality check workflow to compile and validate error messages. It's a straightforward workflow configuration fix with low risk.

Why Fast Track:

• ✅ Not marked as WIP (ready for review)
• ✅ Clear, focused changes (adds Go setup and build steps)
• ✅ Low risk (workflow configuration only)
• ✅ Enables important quality checks

Next Steps:

1. Wait for CI to pass
2. Quick review of workflow changes
3. Merge if CI green

---

Triaged by PR Triage Agent on 2026-02-12

AI generated by PR Triage Agent

[pelikhan]

1.25 @copilot see ci.yml build

[Copilot]

Updated to match ci.yml build pattern. Changed from hardcoded go-version: "1.23" to go-version-file: go.mod to use the correct Go version (1.25.0) specified in go.mod. Also updated step name to "Set up Go" for consistency. (ca229d2)

[Copilot]

Pull request overview

Adds a build/setup phase to the “Daily Syntax Error Quality Check” agentic workflow so the gh-aw binary is available during the run, aligning the workflow with the repository’s compiler-dependent analysis tasks.

Changes:

• Added frontmatter steps: to set up Go, build gh-aw, and verify the binary is runnable.
• Regenerated the compiled lock workflow to include the new setup steps / updated frontmatter hash.
• Updated the actions lock entries to include a pin for actions/setup-go@v5.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 5 comments.

File	Description
.github/workflows/daily-syntax-error-quality.md	Adds Go setup + gh-aw build/verify steps before the agent executes.
.github/workflows/daily-syntax-error-quality.lock.yml	Regenerated compiled workflow reflecting the new frontmatter steps.
.github/aw/actions-lock.json	Adds actions/setup-go@v5 entry used by the regenerated lock workflow.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

actions/setup-go@v5 was added to the action lock entries. If the workflow is updated to use actions/setup-go@v6 like the rest of the repo, this extra v5 entry should be unnecessary; consider removing it (via re-running the lock regeneration) to keep the lockfile minimal and consistent.

Suggested change

	"actions/setup-go@v5": {
	"repo": "actions/setup-go",
	"version": "v5",
	"sha": "40f1582b2485089dde7abd97c1529aa768e1baff"
	},

[Copilot]

This repo’s workflow frontmatter generally uses actions/setup-go@v6 (and CI uses setup-go v6) with go-version-file: go.mod. Using @v5 here is inconsistent and also forces an extra actions-lock.json entry; switching to @v6 should align with the rest of the workflows and the existing lock for v6.2.0.

[Copilot]

The agent instructions later cd /tmp/... and run ./gh-aw compile ...; after that cd, ./gh-aw won’t exist unless the binary is copied there or added to PATH. Consider adding a step to add $GITHUB_WORKSPACE to $GITHUB_PATH (and then invoke gh-aw), or adjust the instructions/commands to use an absolute path like $GITHUB_WORKSPACE/gh-aw.

[Copilot]

There are trailing spaces on otherwise-blank lines in the new steps: block (after cache: true and after make build). Please remove the trailing whitespace to avoid churn in future diffs and to satisfy editors/linters that trim whitespace.

Suggested change

	- name: Build gh-aw
	run: |
	make build
	- name: Build gh-aw
	run: |
	make build

[Copilot]

The generated lock workflow now pins actions/setup-go at v5, while most other lock workflows in this repo pin v6.2.0 and CI uses setup-go v6. If you switch the source workflow to actions/setup-go@v6, please regenerate the lock so it pins the consistent v6 SHA here too.