Add integration test for adding workflows from githubnext/agentics#16421
Add integration test for adding workflows from githubnext/agentics#16421
Conversation
Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
.github/workflows/ci.yml
Outdated
| path: conformance-output.txt | ||
| retention-days: 7 | ||
|
|
||
| agentics-workflows-integration: |
.github/workflows/ci.yml
Outdated
| retention-days: 7 | ||
|
|
||
| agentics-workflows-integration: | ||
| name: Test Adding Agentics Workflows |
Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
|
@copilot the add command seemed to have failed but still marked as success See https://github.com/github/gh-aw/actions/runs/22111194750/job/63908108300#step:11:1 Make sure to pass the GitHub token to the step |
Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
Fixed in f47d490. Added Warning Firewall rules blocked me from connecting to one or more addresses (expand for details)I tried to connect to the following addresses, but was blocked by firewall rules:
If you need me to access, download, or install something from one of these locations, you can either:
|
There was a problem hiding this comment.
Pull request overview
This PR adds a new integration test job to the CI workflow that validates the gh aw add command can successfully download and add workflows from the githubnext/agentics repository. The main fix is adding the GH_TOKEN environment variable to authenticate GitHub API requests, which was causing failures in workflow downloads.
Changes:
- Added
integration-addjob to ci.yml that clones the githubnext/agentics repository and attempts to add all its workflows one by one - Set
GH_TOKEN: ${{ github.token }}environment variable in the "Add workflows one by one" step to fix authentication issues - Implemented detailed logging and summary reporting of success/failure counts for each workflow addition attempt
Comments suppressed due to low confidence (1)
.github/workflows/ci.yml:2138
- The workflow count calculation will produce incorrect results if there are no workflows.
wc -lcounts the number of newlines, so ifWORKFLOWSis empty, it will still return 0 (which is correct), but if there's one workflow without a trailing newline, it might return 0 instead of 1. Additionally, if the variable is empty, this could lead to counting issues. Consider using a more reliable method:WORKFLOW_COUNT=$(echo "$WORKFLOWS" | grep -c . || echo "0")which counts non-empty lines, or handle the empty case explicitly before this line.
WORKFLOW_COUNT=$(echo "$WORKFLOWS" | wc -l)
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| FAILURE_COUNT=0 | ||
|
|
||
| # Read workflow list | ||
| while IFS= read -r workflow; do |
There was a problem hiding this comment.
The workflow list may contain empty lines if there are any empty entries from the sed command or if the file list has issues. When iterating with while IFS= read -r workflow, empty lines will still be processed, potentially causing gh-aw add to be called with an empty string or just the repository path. Add a check to skip empty lines: add [ -z "$workflow" ] && continue right after the do statement on line 2161.
| while IFS= read -r workflow; do | |
| while IFS= read -r workflow; do | |
| [ -z "$workflow" ] && continue |
| echo "Processing workflow: $workflow" | ||
|
|
||
| # Try to add the workflow using gh aw add | ||
| if ./gh-aw add "githubnext/agentics/$workflow" --force 2>&1 | tee /tmp/add-${workflow}.log; then |
There was a problem hiding this comment.
The tee command always returns 0 (success) regardless of the exit code of the piped command before it. This means the if condition will always evaluate to true, and failures won't be caught properly. To fix this, you should set PIPEFAIL before the loop: add set -o pipefail at the beginning of the script block (after the cd command). This ensures that the exit code of the piped command is preserved even when using tee.
| WORKFLOWS=$(ls *.md | sed 's/\.md$//') | ||
|
|
There was a problem hiding this comment.
The workflow list parsing doesn't handle edge cases properly. If the /tmp/agentics/workflows directory is empty or contains no .md files, the ls *.md command will fail with an error, causing the entire step to fail. Consider adding error handling: WORKFLOWS=$(ls *.md 2>/dev/null | sed 's/\.md$//' || echo "") and then check if WORKFLOWS is empty before proceeding. Additionally, add a conditional check after line 2143 to handle the case where no workflows are found.
This issue also appears on line 2138 of the same file.
| WORKFLOWS=$(ls *.md | sed 's/\.md$//') | |
| WORKFLOWS=$(ls *.md 2>/dev/null | sed 's/\.md$//' || echo "") | |
| # Handle case where no workflows are found | |
| if [ -z "$WORKFLOWS" ]; then | |
| echo "No workflow files (.md) found in /tmp/agentics/workflows" | |
| echo "workflow_count=0" >> $GITHUB_OUTPUT | |
| exit 0 | |
| fi | |
| env: | ||
| GH_TOKEN: ${{ github.token }} | ||
| run: | | ||
| cd /home/runner/work/gh-aw/gh-aw |
There was a problem hiding this comment.
The hardcoded path /home/runner/work/gh-aw/gh-aw is fragile and assumes a specific repository name and GitHub Actions runner environment. This will break if the repository is forked or the runner environment changes. Instead, use $GITHUB_WORKSPACE which is automatically set by GitHub Actions to the correct workspace directory, or use a relative path by staying in the workspace directory from the start.
| cd /home/runner/work/gh-aw/gh-aw | |
| cd "$GITHUB_WORKSPACE" |
| if ./gh-aw add "githubnext/agentics/$workflow" --force 2>&1 | tee /tmp/add-${workflow}.log; then | ||
| echo "✅ Successfully added: $workflow" | ||
| echo "| $workflow | ✅ Success | Added successfully |" >> $GITHUB_STEP_SUMMARY | ||
| SUCCESS_COUNT=$((SUCCESS_COUNT + 1)) | ||
| else | ||
| EXIT_CODE=$? | ||
| echo "❌ Failed to add: $workflow (exit code: $EXIT_CODE)" | ||
|
|
||
| # Extract error message from log | ||
| ERROR_MSG=$(tail -5 /tmp/add-${workflow}.log | tr '\n' ' ' | cut -c1-100) |
There was a problem hiding this comment.
The workflow names are used directly in the log file path /tmp/add-${workflow}.log without sanitization. If a workflow name contains special characters like spaces, slashes, or other shell metacharacters, this could cause issues or security vulnerabilities. While the githubnext/agentics repository likely has well-formed workflow names, it's a best practice to sanitize the workflow name before using it in file paths. Consider using parameter expansion to remove or replace problematic characters: SAFE_NAME=$(echo "$workflow" | tr -c '[:alnum:]-_' '_').
GH_TOKENenvironment variable to "Add workflows one by one" stepgh aw addcommand failed without GitHub tokengh aw add💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more Copilot coding agent tips in the docs.