Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1,197 changes: 1,197 additions & 0 deletions .github/workflows/agent-runner-parity.lock.yml

Large diffs are not rendered by default.

135 changes: 135 additions & 0 deletions .github/workflows/agent-runner-parity.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,135 @@
---
description: Agent-Runner Environment Parity Test
on:
schedule: every 6h
workflow_dispatch:
pull_request:
types: [labeled]
names: ["test-parity"]
reaction: "rocket"
permissions:
contents: read
issues: read
pull-requests: read
name: Agent-Runner Parity
engine: copilot
network:
allowed:
- defaults
- github
tools:
bash:
- "*"
safe-outputs:
add-comment:
max: 1
messages:
run-started: "🚀 Testing agent-runner environment parity..."
run-success: "✅ Environment parity test completed successfully"
run-failure: "❌ Environment parity test failed: {status}"
timeout-minutes: 10
strict: true
---

# Agent-Runner Environment Parity Test

This workflow validates that the agent container environment has parity with the GitHub Actions runner environment for essential tools, libraries, and environment variables.

## Test Categories

### 1. Utilities Accessibility (10+ utilities)

Verify the following utilities are accessible via `which`:
- `jq` - JSON processor
- `curl` - HTTP client
- `git` - Version control
- `wget` - File downloader
- `tar` - Archive utility
- `gzip` - Compression utility
- `unzip` - Archive extractor
- `sed` - Stream editor
- `awk` - Pattern processor
- `grep` - Text search
- `find` - File finder
- `xargs` - Argument builder

**Command**: `which <utility>`

### 2. Runtime Availability (4 runtimes)

Verify the following runtimes are available and can execute:
- `node --version` - Node.js runtime
- `python3 --version` - Python interpreter
- `go version` - Go compiler
- `ruby --version` - Ruby interpreter

### 3. Environment Variables (5+ variables)

Check that essential environment variables are set:
- `JAVA_HOME` - Java installation directory
- `ANDROID_HOME` - Android SDK directory
- `GOROOT` - Go installation root
- `PATH` - Executable search path
- `HOME` - User home directory

**Command**: `printenv <VAR>` or `echo $<VAR>`

### 4. Shared Library Linking

Use `ldd` to verify shared libraries can be loaded for key binaries:
- `/usr/bin/python3`
- `/usr/bin/node`
- `/usr/bin/git`
- `/usr/bin/curl`

**Command**: `ldd <binary>` (check for "not found" errors)

## Testing Instructions

For each test category:

1. Run the verification commands using bash
2. Record which items **PASS** ✅ and which **FAIL** ❌
3. For failures, include the error message or missing item

## Reporting

Create a summary report with:
- Total tests run
- Pass/Fail counts by category
- List of failed items (if any)
- Overall status (PASS if all tests pass, FAIL otherwise)

**Keep the report concise** - only list failures in detail.

## Example Output Format

```
🔍 Agent-Runner Environment Parity Test Results

Utilities: 12/12 ✅
Runtimes: 4/4 ✅
Environment Variables: 5/5 ✅
Shared Libraries: 4/4 ✅

Overall: PASS ✅
```

Or if there are failures:

```
🔍 Agent-Runner Environment Parity Test Results

Utilities: 11/12 ❌
- Missing: unzip

Runtimes: 4/4 ✅
Environment Variables: 4/5 ❌
- Missing: ANDROID_HOME

Shared Libraries: 4/4 ✅

Overall: FAIL ❌
```

## Post a summary comment with the results using safe-outputs add-comment
7 changes: 7 additions & 0 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -50,6 +50,13 @@ test:
test-unit:
go test -v -timeout=3m -tags '!integration' -run='^Test' ./...

# Test agent-runner environment parity
.PHONY: test-parity
test-parity:
@echo "Running agent-runner environment parity tests..."
go test -v -timeout=3m -tags 'integration' -run '^TestAgentRunnerParity' ./pkg/workflow
@echo "✓ Agent parity tests completed"

# Update golden test files
.PHONY: update-golden
update-golden:
Expand Down
110 changes: 90 additions & 20 deletions pkg/cli/templates/create-agentic-workflow.md

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Loading