feat: add manual trigger support to fuzzer fix automation

[joseph-isaacs]

Add Manual Trigger Support to Fuzzer Fix Automation

This PR adds workflow_dispatch support to the fuzzer fix automation workflow, allowing manual triggering on specific issues.

Motivation

The fuzzer fix automation workflow currently only triggers automatically when:

1. An issue is opened with the fuzzer label
2. The fuzzer label is added to an existing issue

However, there are cases where manual triggering is useful:

• Testing the workflow before automatic triggers happen
• Re-running analysis on existing issues
• Retroactive fixes for issues created before the workflow existed (like Fuzzing Crash: ListViewArray size/offset type mismatch after compression #5322)

Changes

Added workflow_dispatch Trigger

workflow_dispatch:
  inputs:
    issue_number:
      description: 'Issue number to analyze and fix'
      required: true
      type: number

Fetch Issue Details for Manual Runs

When manually triggered, the workflow fetches issue details via gh issue view:

• Issue number
• Issue title
• Issue body
• Issue labels

Support Both Trigger Modes

The workflow now handles both automatic and manual triggers:

• Automatic: Uses github.event.issue.*
• Manual: Uses inputs.issue_number and fetched issue data

Usage

Manual Trigger via GitHub UI

1. Go to Actions → Fuzzer Fix Automation
2. Click "Run workflow"
3. Enter issue number (e.g., 5322)
4. Click "Run workflow"

Manual Trigger via CLI

gh workflow run fuzzer-fix-automation.yml \
  --repo vortex-data/vortex \
  -f issue_number=5322

Testing

Successfully tested by manually triggering on issue #5322:

• Workflow started: https://github.com/vortex-data/vortex/actions/runs/19333592253
• Issue details fetched correctly
• Claude analysis in progress

Notes

• Automatic triggers still work exactly as before
• No changes to the Claude analysis logic
• Backward compatible with existing behavior

---

Related: #5322 (test issue)
Depends on: #5319 (already merged)

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 86.26%. Comparing base (9d938cd) to head (40c8089).
⚠️ Report is 3 commits behind head on develop.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[codspeed-hq]

CodSpeed Performance Report

Merging #5323 will not alter performance

_{Comparing ji/cluade-code-fuzzz-issue (40c8089) with develop (9d938cd)¹}

Summary

✅ 1425 untouched
⏩ 567 skipped²

Footnotes

1. No successful run was found on develop (5405737) during the generation of this report, so 9d938cd was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩

2. 567 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

[0ax1]

@claude review

[0ax1]

Hah, what is even going on here: if: always() && needs.report-io-fuzz-failures.outputs.issue_number != ''

[claude]

Claude finished @0ax1's task —— View job

---

Code Review for PR #5323

Reviewing manual trigger support for fuzzer fix automation

---

Summary

This PR successfully adds manual trigger support to the fuzzer fix automation workflow. The implementation is generally solid, but I have some observations and recommendations.

Key Changes

1. .github/workflows/fuzzer-fix-automation.yml

✅ Strengths:

• Clean implementation of workflow_dispatch alongside workflow_call
• Both triggers use identical inputs, ensuring consistency
• Proper validation of issue labels (must have fuzzer label)
• Good progressive commenting strategy with detailed templates for Claude
• Comprehensive error handling and validation steps

🔍 Observations:

1. Conditional logic (lines 23-25):

   if: |
     github.event_name == 'workflow_call' ||
     github.event_name == 'workflow_dispatch'

   This is redundant since the workflow only has these two triggers. The job will only run when one of these events occurs anyway. You could simplify to just if: always() or remove the condition entirely.

2. Issue fetch approach (lines 40-50):
   The workflow fetches issue details using gh issue view for both manual and automatic triggers. This is fine, but creates an implicit dependency that the issue must exist and be accessible. Consider adding explicit error handling if the issue doesn't exist.

3. Claude tool restrictions (line 362):

   --allowedTools "Read,Write,Edit,Glob,Grep,Bash(cargo:*),Bash(gh issue comment:*),..."

   The Bash(RUST_BACKTRACE=*:*) pattern might be overly permissive. Consider if you really need to allow arbitrary commands with RUST_BACKTRACE prefix.

4. Missing workflow_dispatch validation:
   When manually triggered, there's no check that the provided issue_number is actually a valid integer. The GitHub Actions type: number helps, but additional validation could be useful.

2. .github/workflows/fuzz.yml

✅ Strengths:

• Clean integration of the fix automation workflow
• Proper use of workflow_call to invoke the automation

🤔 Addressing the review comment (line 131):

if: always() && needs.report-io-fuzz-failures.outputs.issue_number != ''

The reviewer (@0ax1) asked "what is even going on here?" This line is actually correct and necessary:

• always() - Ensures this job runs even if previous jobs failed (which they will if crashes are found)
• needs.report-io-fuzz-failures.outputs.issue_number != '' - Only runs if the reporting job successfully created/found an issue

However, there's a potential issue: This condition doesn't check if report-io-fuzz-failures actually succeeded. If that job fails, outputs.issue_number will be empty, but the condition might still try to evaluate it. A more robust version would be:

if: always() && needs.report-io-fuzz-failures.result == 'success' && needs.report-io-fuzz-failures.outputs.issue_number != ''

Recommendations

High Priority

1. Add explicit result checking in fuzz.yml:131:

   if: always() && needs.report-io-fuzz-failures.result == 'success' && needs.report-io-fuzz-failures.outputs.issue_number != ''

2. Add error handling for missing issues in fuzzer-fix-automation.yml:

   - name: Fetch issue details
     id: fetch_issue
     continue-on-error: true
     env:
       GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
     run: |
       if ! ISSUE_DATA=$(gh issue view ${{ inputs.issue_number }} --repo ${{ github.repository }} --json number,title,body,labels); then
         echo "❌ Issue #${{ inputs.issue_number }} not found or not accessible"
         exit 1
       fi
       # ... rest of the script

Medium Priority

3. Simplify redundant conditional - Remove or simplify the job-level if condition in fuzzer-fix-automation.yml:23-25

4. Review Claude tool permissions - Consider whether Bash(RUST_BACKTRACE=*:*) is too permissive

Low Priority

5. Add workflow_dispatch testing documentation - Consider adding a note in the PR or workflow comments about testing the workflow manually

Testing Notes

The PR description mentions successful manual testing on issue #5322, which is good. Consider:

• Testing what happens when an invalid issue number is provided
• Testing what happens when an issue without the fuzzer label is provided
• Testing the workflow_call path from the fuzz.yml integration

Overall Assessment

✅ Approve with minor suggestions

The implementation is sound and achieves the stated goals. The manual trigger support is well-designed and the integration is clean. The main improvements would be:

1. More robust error checking in the conditional logic
2. Better error handling for missing/invalid issues

The code is production-ready as-is, but would benefit from the suggested hardening.

---

Re: Review comment on line 131: The logic is correct but could be more robust. See "Addressing the review comment" section above.

---

[0ax1]

@claude is this secure, in particular in terms of token handling?

[claude]

Claude finished @0ax1's task —— View job

---

Security Analysis: Token Handling

I've reviewed the token handling in both workflow files. Here's my comprehensive security assessment:

Todo List

• Read fuzzer-fix-automation.yml workflow
• Read fuzz.yml workflow
• Analyze token usage patterns
• Review Claude tool permissions
• Assess security implications
• Provide security assessment

---

Summary: ✅ Generally Secure with Minor Concerns

The token handling is largely secure, but there are a few areas worth noting and one moderate concern with Claude's bash tool permissions.

---

Token Usage Analysis

1. GITHUB_TOKEN (Automatic Token)

Used in:

• fuzzer-fix-automation.yml lines 43, 55, 93, 123, 201, 242, 367
• fuzz.yml lines 30, 126, 254 (passed to other workflows)

Scoped Permissions:

permissions:
  contents: write
  pull-requests: write
  issues: write
  id-token: write

Security Assessment: ✅ Secure

• Uses GitHub's automatically-provided GITHUB_TOKEN with explicit, minimal permissions
• Properly scoped to only what's needed for the workflow
• Token is automatically scoped to the repository and expires after the job
• Good practice: Explicit permissions declaration prevents over-privilege

2. CLAUDE_CODE_OAUTH_TOKEN (Secret)

Used in:

• fuzzer-fix-automation.yml line 241
• Passed via secrets parameter to Claude Code Action

Security Assessment: ✅ Secure

• Stored as a GitHub secret (encrypted at rest)
• Only exposed to the Claude Code Action
• Not logged or exposed in workflow output
• Secret is not accessible via bash commands in the workflow

3. AWS Credentials (R2 Storage)

Used in:

• fuzz.yml lines 45-46, 98-99, 226-227 (corpus storage)

Security Assessment: ✅ Secure

• Stored as GitHub secrets
• Used only for accessing fuzzer corpus storage
• Properly scoped to specific S3 bucket operations
• Not passed to Claude or exposed in logs

---

Claude Code Action Security

Tool Restrictions (Line 362)

--allowedTools "Read,Write,Edit,Glob,Grep,Bash(cargo:*),Bash(gh issue comment:*),Bash(gh run download:*),Bash(curl:*),Bash(find:*),Bash(ls:*),Bash(cat:*),Bash(RUST_BACKTRACE=*:*)"

Security Assessment: ⚠️ Moderate Concern

✅ Good Security Practices:

1. Whitelisted tools - Only specific tools are allowed (Read, Write, Edit, etc.)
2. Restricted bash commands - Using pattern matching to limit what can be executed
3. No arbitrary shell access - Claude cannot run arbitrary commands
4. Safe git operations - No git push/commit allowed, preventing unauthorized code changes
5. Controlled GitHub API access - Limited to gh issue comment and gh run download

⚠️ Potential Concerns:

1. Bash(RUST_BACKTRACE=*:*) - This pattern is overly permissive:

   • Allows ANY command if prefixed with RUST_BACKTRACE=<value>
   • Could potentially be exploited: RUST_BACKTRACE=1:rm -rf / (hypothetically)
   • Recommendation: Replace with specific allowed commands:

     Bash(RUST_BACKTRACE=*:cargo:*)
     Bash(RUST_BACKTRACE=*:ls:*)
     Bash(RUST_BACKTRACE=*:cat:*)

   • Or use just the specific pattern needed for fuzzer runs

2. Bash(curl:*) - Allows arbitrary HTTP requests:

   • Claude could make external network requests
   • Could potentially exfiltrate data (though unlikely given the context)
   • Risk level: Low (Claude is trusted, but defense-in-depth suggests limiting this)
   • Recommendation: Consider if curl is actually needed, or restrict to specific domains if possible

3. Bash(find:*) - Allows filesystem traversal:

   • Could be used to discover sensitive files
   • Risk level: Low (filesystem is limited to repo checkout)
   • Mitigation: Acceptable since it's scoped to repo files only

---

Token Exposure Risks

✅ No Direct Exposure

• GITHUB_TOKEN is only passed to:
  • Standard GitHub actions (checkout, setup)
  • gh CLI commands (safe, designed for this)
  • Claude Code Action (safe, official action)

✅ Secrets Not Accessible to Claude

• Claude Code Action receives github_token and claude_code_oauth_token
• AWS secrets are NOT passed to Claude
• Claude cannot access other secrets

✅ Output Safety

• show_full_output: true doesn't expose tokens
• GitHub Actions automatically redacts secrets in logs

---

Potential Attack Vectors

1. Malicious Issue Content

Risk: An attacker creates an issue with malicious content that Claude processes

Mitigations in place:

• Issue must have fuzzer label (line 100)
• Issue content is only used for analysis, not execution
• Claude's bash permissions are restricted
• No eval or dynamic code execution of issue content

Assessment: ✅ Low risk

2. Malicious Crash Files

Risk: Crash file contains exploitable content

Mitigations in place:

• Crash files come from trusted artifact storage
• Only processes artifacts from the same repository's workflow runs
• Crash files are used as fuzzer input, not executed directly

Assessment: ✅ Low risk

3. Claude Code Action Compromise

Risk: Claude Code Action itself is malicious

Mitigations in place:

• Using official Anthropic action: anthropics/claude-code-action@v1
• Action runs with limited permissions
• Cannot push code changes (no git push allowed)
• Cannot access AWS secrets

Assessment: ✅ Low risk (trusting Anthropic's official action)

---

Recommendations

High Priority

1. Fix overly permissive RUST_BACKTRACE pattern:

   # Change from:
   Bash(RUST_BACKTRACE=*:*)
   
   # To:
   Bash(RUST_BACKTRACE=*:cargo:*)

Medium Priority

2. Consider restricting curl:

   • Document why curl is needed
   • If possible, restrict to specific domains or remove if not needed

3. Add token validation logging:

   • Consider adding a step that logs (without exposing) that tokens are properly set
   • Helps detect configuration issues

Low Priority

4. Principle of least privilege for permissions:
   • Current permissions look good, but periodically review if all are needed
   • Consider if id-token: write is actually required

---

Conclusion

Overall Security Rating: ✅ Good with Minor Improvements Needed

The workflow demonstrates good security practices:

• ✅ Explicit permission scoping
• ✅ Proper secret handling
• ✅ Restricted tool access for Claude
• ✅ No arbitrary code execution
• ✅ Tokens properly isolated

Main concern: The Bash(RUST_BACKTRACE=*:*) pattern should be tightened to prevent potential abuse.

Recommendation: This PR is safe to merge as-is, but I'd suggest addressing the RUST_BACKTRACE pattern before or shortly after merging to harden security further.

---

[0ax1]

✅ But worthwhile to double check security aspects.