Skip to content

Smoke tests: split compaction test and use debug build #6984

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
alexhancock merged 4 commits into from
Feb 8, 2026
Merged

Smoke tests: split compaction test and use debug build #6984

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
9beaaae
Smoke tests: split compaction test and use debug build
jamadeo Feb 5, 2026
43f0cff
update binary paths in scripts
jamadeo Feb 5, 2026
174e25d
Compaction test uses python
jamadeo Feb 5, 2026
0472ba7
Update .github/workflows/pr-smoke-test.yml
jamadeo Feb 6, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
67 changes: 48 additions & 19 deletions .github/workflows/pr-smoke-test.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -45,7 +45,7 @@ jobs:
- '!documentation/**'

build-binary:
name: Build Release Binary
name: Build Binary
runs-on: ubuntu-latest
needs: changes
if: needs.changes.outputs.code == 'true' || github.event_name == 'workflow_dispatch'
Expand All @@ -65,15 +65,15 @@ jobs:
- name: Cache Rust dependencies
uses: Swatinem/rust-cache@v2

- name: Build Release Binary for Smoke Tests
- name: Build Binary for Smoke Tests
run: |
cargo build --release
cargo build --bin goose

- name: Upload Binary for Smoke Tests
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6.0.0
with:
name: goose-binary
path: target/release/goose
path: target/debug/goose
Comment on lines +70 to +76
Copy link

Copilot AI Feb 5, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The workflow now builds and uploads a debug binary from target/debug/goose, but the test scripts (test_compaction.sh, test_providers.sh, test_subrecipes.sh, test_mcp.sh) all have hardcoded paths to target/release/goose. Since SKIP_BUILD=1 is set, these scripts won't rebuild the binary but will look for it in the wrong location, causing the tests to fail.

The scripts need to be updated to use target/debug instead of target/release, or an environment variable like GOOSE_BIN should be passed to override the default path.

Copilot uses AI. Check for mistakes.
retention-days: 1

smoke-tests:
Expand All @@ -90,10 +90,10 @@ jobs:
uses: actions/download-artifact@37930b1c2abaa49bbe596cd826c3c89aef350131 # v7.0.0
with:
name: goose-binary
path: target/release
path: target/debug

- name: Make Binary Executable
run: chmod +x target/release/goose
run: chmod +x target/debug/goose

- name: Run Smoke Tests with Provider Script
env:
Expand Down Expand Up @@ -152,17 +152,6 @@ jobs:
run: |
bash scripts/test_subrecipes.sh

- name: Run Compaction Tests
env:
ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
GOOSE_PROVIDER: anthropic
GOOSE_MODEL: claude-sonnet-4-5-20250929
HOME: /tmp/goose-home
GOOSE_DISABLE_KEYRING: 1
SKIP_BUILD: 1
run: |
bash scripts/test_compaction.sh

smoke-tests-code-exec:
name: Smoke Tests (Code Execution)
runs-on: ubuntu-latest
Expand All @@ -177,10 +166,10 @@ jobs:
uses: actions/download-artifact@37930b1c2abaa49bbe596cd826c3c89aef350131 # v7.0.0
with:
name: goose-binary
path: target/release
path: target/debug

- name: Make Binary Executable
run: chmod +x target/release/goose
run: chmod +x target/debug/goose

- name: Run Provider Tests (Code Execution Mode)
env:
Expand All @@ -200,3 +189,43 @@ jobs:
mkdir -p $HOME/.local/share/goose/sessions
mkdir -p $HOME/.config/goose
bash scripts/test_providers.sh --code-exec

compaction-tests:
name: Compaction Tests
runs-on: ubuntu-latest
needs: build-binary
steps:
- name: Checkout Code
uses: actions/checkout@8e8c483db84b4bee98b60c0593521ed34d9990e8 # v6.0.1
with:
ref: ${{ github.event.inputs.branch || github.ref }}

- name: Download Binary
uses: actions/download-artifact@37930b1c2abaa49bbe596cd826c3c89aef350131 # v7.0.0
with:
name: goose-binary
path: target/debug

- name: Make Binary Executable
run: chmod +x target/debug/goose

- name: Set up Python
uses: actions/setup-python@v5
with:
python-version: '3.12'

- name: Install uv
uses: astral-sh/setup-uv@61cb8a9741eeb8a550a1b8544337180c0fc8476b # v7.2.0

- name: Run Compaction Tests
env:
ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
GOOSE_PROVIDER: anthropic
GOOSE_MODEL: claude-sonnet-4-5-20250929
HOME: /tmp/goose-home
GOOSE_DISABLE_KEYRING: 1
SKIP_BUILD: 1
run: |
mkdir -p $HOME/.local/share/goose/sessions
mkdir -p $HOME/.config/goose
bash scripts/test_compaction.sh
4 changes: 2 additions & 2 deletions scripts/test_compaction.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,15 +14,15 @@ fi

if [ -z "$SKIP_BUILD" ]; then
echo "Building goose..."
cargo build --release --bin goose
cargo build --bin goose
echo ""
else
echo "Skipping build (SKIP_BUILD is set)..."
echo ""
fi

SCRIPT_DIR=$(pwd)
GOOSE_BIN="$SCRIPT_DIR/target/release/goose"
GOOSE_BIN="$SCRIPT_DIR/target/debug/goose"

Comment on lines 24 to 26
Copy link

Copilot AI Feb 6, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hard-coding GOOSE_BIN to target/debug/goose means SKIP_BUILD requires a debug build to exist; consider supporting an env override or auto-detecting debug vs release so developers can reuse an existing release build when running this script locally.

Copilot uses AI. Check for mistakes.
# Apply provider/model overrides if set
if [ -n "$COMPACTION_PROVIDER" ]; then
Expand Down
4 changes: 2 additions & 2 deletions scripts/test_mcp.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,15 +3,15 @@ set -e

if [ -z "$SKIP_BUILD" ]; then
echo "Building goose..."
cargo build --release --bin goose
cargo build --bin goose
echo ""
else
echo "Skipping build (SKIP_BUILD is set)..."
echo ""
fi

SCRIPT_DIR=$(pwd)
GOOSE_BIN="$SCRIPT_DIR/target/release/goose"
GOOSE_BIN="$SCRIPT_DIR/target/debug/goose"

Comment on lines +14 to 15
Copy link

Copilot AI Feb 6, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Setting GOOSE_BIN to target/debug/goose makes SKIP_BUILD mode dependent on a debug build being present; consider letting callers override the binary path (or falling back to target/release) to keep the script usable outside CI.

Suggested change
GOOSE_BIN="$SCRIPT_DIR/target/debug/goose"
# Allow callers to override the goose binary, and fall back to common build outputs.
if [ -z "$GOOSE_BIN" ]; then
if [ -x "$SCRIPT_DIR/target/debug/goose" ]; then
GOOSE_BIN="$SCRIPT_DIR/target/debug/goose"
elif [ -x "$SCRIPT_DIR/target/release/goose" ]; then
GOOSE_BIN="$SCRIPT_DIR/target/release/goose"
else
# Preserve previous default even if it does not exist yet.
GOOSE_BIN="$SCRIPT_DIR/target/debug/goose"
fi
fi

Copilot uses AI. Check for mistakes.
TEST_PROVIDER=${GOOSE_PROVIDER:-anthropic}
TEST_MODEL=${GOOSE_MODEL:-claude-haiku-4-5-20251001}
Expand Down
4 changes: 2 additions & 2 deletions scripts/test_providers.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@ fi

if [ -z "$SKIP_BUILD" ]; then
echo "Building goose..."
cargo build --release --bin goose
cargo build --bin goose
echo ""
else
echo "Skipping build (SKIP_BUILD is set)..."
Expand Down Expand Up @@ -247,7 +247,7 @@ run_test() {
(
export GOOSE_PROVIDER="$provider"
export GOOSE_MODEL="$model"
cd "$testdir" && "$SCRIPT_DIR/target/release/goose" run --text "Immediately use the shell tool to run 'ls'. Do not ask for confirmation." --with-builtin "$BUILTINS" 2>&1
cd "$testdir" && "$SCRIPT_DIR/target/debug/goose" run --text "Immediately use the shell tool to run 'ls'. Do not ask for confirmation." --with-builtin "$BUILTINS" 2>&1
Comment on lines 249 to +250
Copy link

Copilot AI Feb 6, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With SKIP_BUILD enabled this assumes target/debug/goose exists; consider supporting a GOOSE_BIN override or falling back to a release binary so local runs don’t unexpectedly break when only a release build is present.

Copilot uses AI. Check for mistakes.
) > "$output_file" 2>&1

# Check result
Expand Down
10 changes: 5 additions & 5 deletions scripts/test_subrecipes.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ fi

if [ -z "$SKIP_BUILD" ]; then
echo "Building goose..."
cargo build --release --bin goose
cargo build --bin goose
echo ""
else
echo "Skipping build (SKIP_BUILD is set)..."
Expand All @@ -17,7 +17,7 @@ fi
SCRIPT_DIR=$(pwd)

# Add goose binary to PATH so subagents can find it when spawning
export PATH="$SCRIPT_DIR/target/release:$PATH"
export PATH="$SCRIPT_DIR/target/debug:$PATH"
Copy link

Copilot AI Feb 6, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With SKIP_BUILD set, this hard-codes the debug target dir; if a user (or another workflow) only has a release build available, the script will fail—consider allowing a GOOSE_BIN/GOOSE_TARGET_DIR override or falling back to target/release when target/debug/goose is missing.

Suggested change
export PATH="$SCRIPT_DIR/target/debug:$PATH"
if [ -n "$GOOSE_BIN" ]; then
GOOSE_DIR="$(dirname "$GOOSE_BIN")"
elif [ -n "$GOOSE_TARGET_DIR" ]; then
GOOSE_DIR="$GOOSE_TARGET_DIR"
elif [ -x "$SCRIPT_DIR/target/debug/goose" ]; then
GOOSE_DIR="$SCRIPT_DIR/target/debug"
elif [ -x "$SCRIPT_DIR/target/release/goose" ]; then
GOOSE_DIR="$SCRIPT_DIR/target/release"
else
echo "Error: goose binary not found. Set GOOSE_BIN or GOOSE_TARGET_DIR, or build goose (debug or release)." >&2
exit 1
fi
export PATH="$GOOSE_DIR:$PATH"

Copilot uses AI. Check for mistakes.

# Set default provider and model if not already set
# Use fast model for CI to speed up tests
Expand Down Expand Up @@ -76,7 +76,7 @@ RESULTS=()
check_recipe_output() {
local tmpfile=$1
local mode=$2

# Check for unified subagent tool invocation (new format: "─── subagent |")
if grep -q "─── subagent" "$tmpfile"; then
echo "✓ SUCCESS: Subagent tool invoked"
Expand All @@ -85,7 +85,7 @@ check_recipe_output() {
echo "✗ FAILED: No evidence of subagent tool invocation"
RESULTS+=("✗ Subagent tool invocation ($mode)")
fi

# Check that both subrecipes were called (shown as "subrecipe: <name>" in output)
if grep -q "subrecipe:.*file_stats\|file_stats.*subrecipe" "$tmpfile" && grep -q "subrecipe:.*code_patterns\|code_patterns.*subrecipe" "$tmpfile"; then
echo "✓ SUCCESS: Both subrecipes (file_stats, code_patterns) found in output"
Expand All @@ -98,7 +98,7 @@ check_recipe_output() {

echo "Running recipe with parallel subrecipes..."
TMPFILE=$(mktemp)
if (cd "$TESTDIR" && "$SCRIPT_DIR/target/release/goose" run --recipe project_analyzer_parallel.yaml --no-session 2>&1) | tee "$TMPFILE"; then
if (cd "$TESTDIR" && "$SCRIPT_DIR/target/debug/goose" run --recipe project_analyzer_parallel.yaml --no-session 2>&1) | tee "$TMPFILE"; then
Copy link

Copilot AI Feb 6, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This invocation assumes the binary is always at target/debug/goose; when SKIP_BUILD=1 and only a release build exists locally, the test will error—consider deriving the path from an env var or auto-detecting debug vs release based on what exists.

Copilot uses AI. Check for mistakes.
echo "✓ SUCCESS: Recipe completed successfully"
RESULTS+=("✓ Recipe exit code")
check_recipe_output "$TMPFILE" "parallel"
Expand Down
Loading