feat: computer controller overhaul, adding peekaboo by michaelneale · Pull Request #7342 · block/goose

michaelneale · 2026-02-19T03:56:19Z

It's been a minute, time for the computer controller to get an overhaul. on macos, the peekaboo tool is excellent (and in use by openclaw), this swiches out some of it to be similar, and tidies up some tools.

Add Peekaboo CLI integration to the computer controller extension, following the OpenClaw pattern of shelling out to the `peekaboo` binary. New MCP tools (macOS only, conditional on `peekaboo` being installed): - peekaboo_see: Annotated screenshots with UI element IDs (B1, T2, L3) - peekaboo_click: Click elements by ID from see output, or by coordinates - peekaboo_type: Type text into focused UI elements - peekaboo_app: Launch, quit, switch, list applications - peekaboo_hotkey: Press keyboard shortcuts (cmd+c, cmd+shift+t, etc.) The see → click → type workflow: 1. peekaboo_see captures an annotated screenshot with element overlays 2. peekaboo_click targets elements by their assigned ID 3. peekaboo_type enters text into the focused field Key design decisions: - Tools shell out to `peekaboo` CLI (not MCP server) — proven by OpenClaw - Snapshot state persists via filesystem cache (~/.peekaboo/snapshots/) - Tools always registered but return install instructions if peekaboo missing - Annotated screenshots returned as base64 images (same pattern as screen_capture) - SystemAutomation trait extended with has_peekaboo() for detection Also includes: - Documentation page for Peekaboo integration (built-in + MCP server option) - Updated macOS instructions describing the see→click→type workflow - Platform abstraction for Peekaboo availability detection

On macOS, the computer_control tool's description now explicitly directs the agent to use peekaboo_see/peekaboo_click/peekaboo_type for all visual UI interaction, keeping computer_control only for AppleScript app scripting and system settings. Changes: - Replaced the verbose macOS computer_control description with a minimal one that steers toward Peekaboo for UI tasks - Updated macOS instructions to remove computer_control from the tools listing and position Peekaboo as the primary UI automation approach - Consolidated Windows/Linux/fallback computer_control variants into two: macOS (minimal) and non-macOS (full) - Removed unused MacOSAutomation::run_peekaboo() (dead code) - Fixed clippy string_slice lint in peekaboo_see truncation - Ran cargo fmt

…o-install Peekaboo is now a separate builtin extension rather than being embedded in the Computer Controller. On macOS, it auto-installs via Homebrew (brew install steipete/tap/peekaboo) on first tool use if the binary isn't found on PATH. Changes: - New crates/goose-mcp/src/peekaboo/mod.rs: standalone PeekabooServer with all 5 tools (see, click, type, app, hotkey) and auto-install logic - Stripped all Peekaboo code from computercontroller/mod.rs (~300 lines removed) - Restored computercontroller to its original platform-specific tool descriptions - Removed has_peekaboo() from SystemAutomation trait and MacOSAutomation - Registered peekaboo as builtin extension in lib.rs (cfg macos only) - Added Peekaboo to bundled-extensions.json, CLI configure menu, MCP server runner, goose-cli, and goose-server - Updated documentation to reflect standalone extension with auto-install - Extension is macOS-only via #[cfg(target_os = "macos")] Auto-install flow: 1. User enables Peekaboo extension and calls any peekaboo_* tool 2. ensure_peekaboo() checks if binary exists on PATH 3. If missing, resolves brew (PATH, /opt/homebrew/bin, /usr/local/bin) 4. Runs brew install steipete/tap/peekaboo with error handling 5. Falls back to manual install instructions if brew unavailable

…docs Replace the 5 rigid tool wrappers (peekaboo_see, peekaboo_click, peekaboo_type, peekaboo_app, peekaboo_hotkey) with a single `peekaboo` tool that passes through to the full Peekaboo CLI. This gives the agent access to all 30+ commands (click, type, press, hotkey, paste, scroll, drag, window, menu, menubar, dock, clipboard, dialog, space, etc.) without needing individual wrappers for each. The agent learns CLI syntax from comprehensive instructions embedded in the extension (matching the approach used by OpenClaw's peekaboo skill). Key features: - Single tool: `peekaboo(command: "see --app Safari --annotate")` - Auto-injects --path and --json-output for see/image commands - Auto-injects --json for list/query commands - Returns annotated screenshots as base64 images for see/image commands - capture_screenshot param for verifying UI state after actions - Shell-words parsing for proper quote handling in commands - Full CLI reference in extension instructions Added shell-words dependency for command string parsing.

…n automation_script On macOS, the computer_control tool is now the Peekaboo CLI passthrough instead of AppleScript. This means when users ask to move windows, click UI elements, or interact visually, the agent uses Peekaboo directly through the existing computer_control tool — no separate extension needed. Changes to computer_control on macOS: - Params changed from {script, save_output} to {command, capture_screenshot} - command takes peekaboo CLI strings e.g. 'see --app Safari --annotate' - Description says 'Peekaboo CLI for GUI automation' not 'AppleScript' - Full peekaboo command reference embedded in macOS extension instructions - Auto-installs Peekaboo via Homebrew on first use - Auto-injects --path/--json-output for see/image, --json for list commands - Returns annotated screenshots as base64 images - capture_screenshot param takes a fresh screenshot after actions Changes to automation_script: - Description now mentions AppleScript (via osascript) as an option - Example added: osascript -e 'tell app ...' - AppleScript is available here for app scripting / system settings Other platforms (Windows, Linux) are unchanged. The standalone peekaboo extension remains available for users who want it as a separate extension. The CC integration means it works out of the box for CC users.

…ript leak to Linux automation_script was cfg(not(windows)) which leaked AppleScript mentions to Linux users. Now split into: - cfg(macos): Shell, Ruby, AppleScript (via osascript) - cfg(not(windows,macos)): Shell scripts only - cfg(windows): PowerShell/Batch (unchanged) Also cleaned up macOS instructions to not repeat AppleScript in both the instructions block and the tool description.

… unchanged Restored 3 missing lines in Linux computer_control description: - File and system management - Process management and monitoring - System settings and configurations Verified: Windows instructions and tool descriptions are identical to the original pre-Peekaboo code. Linux instructions block is identical. Linux automation_script slightly simplified (removed macOS-specific Ruby mention). All Peekaboo/macOS changes are properly cfg-gated.

…kill depth Expanded the macOS instructions block with: - Explicit 5-step quickstart (see → click → type → press → type) - Full command reference: capture, swipe, space added - Richer parameter docs: --snapshot, --screen-index, --pid, --window-index, --no-auto-focus, --format, --capture-engine, --duration, --profile - Multi-step examples inline with each command - Tips section: permissions check, multi-monitor, paste vs type - Hint to use 'peekaboo <cmd> --help' for self-discovery Also improved the tool description to show core workflow + key commands at a glance, not just a one-liner. Adapted from OpenClaw's SKILL.md but restructured for Goose's tool approach (command param, not shell execution). Windows and Linux instructions are unchanged.

github-actions · 2026-02-19T03:59:49Z

PR Preview Action v1.8.1
🚀 View preview at https://block.github.io/goose/pr-preview/pr-7342/
Built to branch `gh-pages` at 2026-02-19 04:09 UTC. Preview will be ready when the GitHub Pages deployment is complete.

…uter_control on macOS Peekaboo is no longer a separate builtin extension. On macOS, the existing computer_control tool IS the Peekaboo passthrough — same tool name, same extension (computercontroller), zero new config. Removed: - PeekabooServer struct and MCP server impl (was duplicate of CC) - McpCommand::Peekaboo variant - Peekaboo entries in cli.rs, main.rs, configure.rs, bundled-extensions.json - PEEKABOO_INSTRUCTIONS const (instructions are in CC now) Kept: - crates/goose-mcp/src/peekaboo/mod.rs as helper module (just 3 public functions: is_peekaboo_installed, resolve_brew, auto_install_peekaboo) - CC imports these helpers for auto-install on macOS CLI, server, configure menu, bundled-extensions.json are now identical to main — the only external-facing change is that computer_control behaves differently on macOS.

Copilot

Pull request overview

This PR adds a new macOS-only Peekaboo-based automation path and wires it into goose’s MCP/builtin extension system, alongside UI/CLI surfacing and documentation for the new extension.

Changes:

Add a new PeekabooServer MCP server (macOS-only) plus CLI/server command routing.
Update Computer Controller’s macOS computer_control to pass through Peekaboo commands and optionally return screenshots.
Surface Peekaboo in Desktop bundled extensions + CLI configuration, and add new documentation.

Reviewed changes

Copilot reviewed 11 out of 12 changed files in this pull request and generated 12 comments.

Show a summary per file

File	Description
ui/desktop/src/components/settings/extensions/bundled-extensions.json	Adds `peekaboo` to the bundled extensions list for Desktop.
documentation/docs/mcp/peekaboo-mcp.md	New docs page describing the Peekaboo extension and setup.
crates/goose-server/src/main.rs	Adds `mcp peekaboo` server command handling with macOS gating.
crates/goose-mcp/src/peekaboo/mod.rs	Implements the new Peekaboo MCP server with auto-install + screenshot return.
crates/goose-mcp/src/mcp_server_runner.rs	Adds `Peekaboo` to `McpCommand` parsing/printing.
crates/goose-mcp/src/lib.rs	Registers Peekaboo as a builtin extension on macOS.
crates/goose-mcp/src/computercontroller/platform/mod.rs	Adds an attribute to `SystemAutomation`.
crates/goose-mcp/src/computercontroller/mod.rs	Reworks macOS `computer_control` to use Peekaboo passthrough + screenshot capture.
crates/goose-mcp/Cargo.toml	Adds `shell-words` dependency for command parsing.
crates/goose-cli/src/commands/configure.rs	Adds Peekaboo to the interactive “enable builtin extension” list.
crates/goose-cli/src/cli.rs	Adds `mcp peekaboo` command handling with macOS gating.
Cargo.lock	Locks the newly added `shell-words` dependency.

Comments suppressed due to low confidence (1)

crates/goose-mcp/src/computercontroller/platform/mod.rs:21

#[allow(dead_code)] on SystemAutomation suppresses dead-code warnings across the whole trait, but the trait and its methods are referenced from ComputerControllerServer; this attribute likely isn’t needed and can hide real unused-code issues.

#[allow(dead_code)]
pub trait SystemAutomation: Send + Sync {
    fn execute_system_script(&self, script: &str) -> std::io::Result<String>;
    fn get_shell_command(&self) -> (&'static str, &'static str); // (shell, arg)
    fn get_temp_path(&self) -> std::path::PathBuf;

crates/goose-mcp/src/computercontroller/mod.rs

crates/goose-mcp/src/peekaboo/mod.rs

crates/goose-cli/src/commands/configure.rs

ui/desktop/src/components/settings/extensions/bundled-extensions.json

documentation/docs/mcp/peekaboo-mcp.md

crates/goose-mcp/src/computercontroller/mod.rs

crates/goose-mcp/src/peekaboo/mod.rs

documentation/docs/mcp/peekaboo-mcp.md

Peekaboo is just an implementation detail of computer_control on macOS, not a separate extension that needs its own doc page.

blackgirlbytes

oh wow looks like it works way better. approving to unblock the @block/goose-devrel requirement

* main: (46 commits) chore(deps): bump hono from 4.11.9 to 4.12.0 in /ui/desktop (#7369) Include 3rd-party license copy for JavaScript/CSS minified files (#7352) docs for reasoning env var (#7367) docs: update skills detail page to reference Goose Summon extension (#7350) fix(apps): restore MCP app sampling support reverted by #6933 (#7366) feat: TUI client of goose-acp (#7362) docs: agent variable (#7365) docs: pass env vars to shell (#7361) docs: update sandbox topic (#7336) feat: add local inference provider with llama.cpp backend and HuggingFace model management (#6933) Docs: claude code uses stream-json (#7358) Improve link confirmation modal (#7333) fix(ci): deflake smoke tests for Google models (#7344) feat: add Cerebras provider support (#7339) fix: skip whitespace-only text blocks in Anthropic message (#7343) fix(goose-acp): heap allocations (#7322) Remove trailing space from links (#7156) fix: detect low balance and prompt for top up (#7166) feat(apps): add support for MCP apps to sample (#7039) Typescript SDK for ACP extension methods (#7319) ...

Copilot

Pull request overview

Copilot reviewed 5 out of 6 changed files in this pull request and generated 1 comment.

crates/goose-mcp/src/peekaboo/mod.rs

* origin/main: (21 commits) feat(ui): show token counts directly for "free" providers (#7383) Update creator note (#7384) Remove display_name from local model API and use model ID everywhere (#7382) fix(summon): stop MOIM from telling models to sleep while waiting for tasks (#7377) Completely pointless ascii art (#7329) feat: add Neighborhood extension to the Extensions Library (#7328) feat: computer controller overhaul, adding peekaboo (#7342) Add blog post: Gastown Explained: How to Use Goosetown for Parallel Agentic Engineering (#7372) docs: type-to-search goose configure lists (#7371) docs: search conversation history (#7370) fix: stderr noise (#7346) chore(deps): bump hono from 4.11.9 to 4.12.0 in /ui/desktop (#7369) Include 3rd-party license copy for JavaScript/CSS minified files (#7352) docs for reasoning env var (#7367) docs: update skills detail page to reference Goose Summon extension (#7350) fix(apps): restore MCP app sampling support reverted by #6933 (#7366) feat: TUI client of goose-acp (#7362) docs: agent variable (#7365) docs: pass env vars to shell (#7361) docs: update sandbox topic (#7336) ... # Conflicts: # Cargo.lock

alexhancock

Nice - this looks great. I look forward to using it.

…extension * origin/main: (79 commits) Post release checklist as a comment on release PRs (#7307) unique api key (#7391) fix: use correct colors for download progress bar (#7390) Add local model settings access from bottom bar model menu (#7378) Change Recipe Security Scanner API key (#7387) switch Ask AI Discord bot from openrouter to anthropic (#7386) feat(ui): show token counts directly for "free" providers (#7383) Update creator note (#7384) Remove display_name from local model API and use model ID everywhere (#7382) fix(summon): stop MOIM from telling models to sleep while waiting for tasks (#7377) Completely pointless ascii art (#7329) feat: add Neighborhood extension to the Extensions Library (#7328) feat: computer controller overhaul, adding peekaboo (#7342) Add blog post: Gastown Explained: How to Use Goosetown for Parallel Agentic Engineering (#7372) docs: type-to-search goose configure lists (#7371) docs: search conversation history (#7370) fix: stderr noise (#7346) chore(deps): bump hono from 4.11.9 to 4.12.0 in /ui/desktop (#7369) Include 3rd-party license copy for JavaScript/CSS minified files (#7352) docs for reasoning env var (#7367) ...

* origin/main: (28 commits) [docs] Add Excalidraw MCP App Tutorial (#7401) Post release checklist as a comment on release PRs (#7307) unique api key (#7391) fix: use correct colors for download progress bar (#7390) Add local model settings access from bottom bar model menu (#7378) Change Recipe Security Scanner API key (#7387) switch Ask AI Discord bot from openrouter to anthropic (#7386) feat(ui): show token counts directly for "free" providers (#7383) Update creator note (#7384) Remove display_name from local model API and use model ID everywhere (#7382) fix(summon): stop MOIM from telling models to sleep while waiting for tasks (#7377) Completely pointless ascii art (#7329) feat: add Neighborhood extension to the Extensions Library (#7328) feat: computer controller overhaul, adding peekaboo (#7342) Add blog post: Gastown Explained: How to Use Goosetown for Parallel Agentic Engineering (#7372) docs: type-to-search goose configure lists (#7371) docs: search conversation history (#7370) fix: stderr noise (#7346) chore(deps): bump hono from 4.11.9 to 4.12.0 in /ui/desktop (#7369) Include 3rd-party license copy for JavaScript/CSS minified files (#7352) ... # Conflicts: # ui/desktop/src/components/bottom_menu/CostTracker.tsx

* origin/main: (215 commits) docs: voice dictation updates (#7396) [docs] Add Excalidraw MCP App Tutorial (#7401) Post release checklist as a comment on release PRs (#7307) unique api key (#7391) fix: use correct colors for download progress bar (#7390) Add local model settings access from bottom bar model menu (#7378) Change Recipe Security Scanner API key (#7387) switch Ask AI Discord bot from openrouter to anthropic (#7386) feat(ui): show token counts directly for "free" providers (#7383) Update creator note (#7384) Remove display_name from local model API and use model ID everywhere (#7382) fix(summon): stop MOIM from telling models to sleep while waiting for tasks (#7377) Completely pointless ascii art (#7329) feat: add Neighborhood extension to the Extensions Library (#7328) feat: computer controller overhaul, adding peekaboo (#7342) Add blog post: Gastown Explained: How to Use Goosetown for Parallel Agentic Engineering (#7372) docs: type-to-search goose configure lists (#7371) docs: search conversation history (#7370) fix: stderr noise (#7346) chore(deps): bump hono from 4.11.9 to 4.12.0 in /ui/desktop (#7369) ... # Conflicts: # ui/desktop/package-lock.json # ui/desktop/package.json # ui/desktop/src/components/GooseSidebar/AppSidebar.tsx

michaelneale added 8 commits February 18, 2026 18:43

michaelneale requested a review from a team as a code owner February 19, 2026 03:56

Copilot AI review requested due to automatic review settings February 19, 2026 03:56

Copilot started reviewing on behalf of michaelneale February 19, 2026 03:56 View session

michaelneale requested a review from alexhancock February 19, 2026 04:00

Copilot AI reviewed Feb 19, 2026

View reviewed changes

michaelneale changed the title ~~feat: computer controller overall, adding peekaboo~~ feat: computer controller overhaul, adding peekaboo Feb 19, 2026

remove peekaboo-mcp.md doc page, not needed

ffd68e3

Peekaboo is just an implementation detail of computer_control on macOS, not a separate extension that needs its own doc page.

michaelneale requested review from Copilot and removed request for Copilot February 19, 2026 04:18

Copilot started reviewing on behalf of michaelneale February 19, 2026 04:19 View session

blackgirlbytes approved these changes Feb 19, 2026

View reviewed changes

Copilot AI review requested due to automatic review settings February 19, 2026 23:33

Copilot started reviewing on behalf of michaelneale February 19, 2026 23:34 View session

Copilot AI reviewed Feb 19, 2026

View reviewed changes

crates/goose-mcp/src/peekaboo/mod.rs Show resolved Hide resolved

michaelneale added this pull request to the merge queue Feb 20, 2026

Merged via the queue into main with commit 62d2aa8 Feb 20, 2026
26 checks passed

michaelneale deleted the peekaboo-integration branch February 20, 2026 01:53

alexhancock reviewed Feb 20, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Comments

feat: computer controller overhaul, adding peekaboo#7342

feat: computer controller overhaul, adding peekaboo#7342
michaelneale merged 11 commits intomainfrom
peekaboo-integration

michaelneale commented Feb 19, 2026

Uh oh!

github-actions bot commented Feb 19, 2026 •

edited

Loading

Built to branch `gh-pages` at 2026-02-19 04:09 UTC.
Preview will be ready when the GitHub Pages deployment is complete.

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

blackgirlbytes left a comment

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

alexhancock left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Comments

Conversation

michaelneale commented Feb 19, 2026

Uh oh!

github-actions bot commented Feb 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Built to branch gh-pages at 2026-02-19 04:09 UTC. Preview will be ready when the GitHub Pages deployment is complete.

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

blackgirlbytes left a comment

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

alexhancock left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

github-actions bot commented Feb 19, 2026 •

edited

Loading

Built to branch `gh-pages` at 2026-02-19 04:09 UTC.
Preview will be ready when the GitHub Pages deployment is complete.