feat: add mcp-ui support for Goose (backend logic, interfaces, frontend components)#3432
feat: add mcp-ui support for Goose (backend logic, interfaces, frontend components)#3432mgd1984 wants to merge 31 commits intoblock:feature/diff-viewer-sidecarfrom
Conversation
This commit implements comprehensive MCP UI capabilities, enabling Goose to render interactive web components from MCP servers instead of static JSON responses. Backend Changes: - Enhanced MCP protocol with UICapabilities in mcp-core - Added UI capability negotiation in mcp-client - Implemented call_tool_with_ui() for UI-aware tool calls - Added UI capability tracking in ExtensionManager - Updated MCP server router to support UI capabilities Frontend Changes: - Added @mcp-ui/client dependency for standards-compliant UI rendering - Created UIResourceRenderer component for interactive UI components - Enhanced ToolCallWithResponse to detect and render UI resources - Added proper Accept headers for content negotiation - Updated message stream handling for UI resources Features: - Automatic UI capability detection and negotiation - Interactive component rendering with @mcp-ui/client - Backward compatibility with existing text/JSON responses - Secure iframe-based UI rendering with proper sandboxing - Support for all MCP UI resource types (remote-dom, HTML, etc.) This enables rich, interactive experiences for MCP servers like phantasmo-toolkit that can now display product catalogs, design previews, and interactive tools directly in the Goose interface.
- Remove promotional language and emojis - Make content more concise and technical - Focus on implementation details rather than marketing claims - Structure as proper technical documentation - Fix TypeScript error in UIResourceRenderer
- Integrate UIResourceRenderer directly in main message body - UI resources now appear as first-class message content, not just tool outputs - Add proper type filtering for MessageContent to Content conversion - UI resources render alongside text and images in assistant messages
- UI resources now only appear in main message body, not in tool results - Prevents double rendering of interactive UI components - Remove unused imports from ToolCallWithResponse
- UI resources are typically returned as tool results, not message content - Check both message content and tool responses for UI resources - Check tool responses from subsequent messages for tool calls in current message - This ensures UI resources show up in main message body instead of tool results
|
@iandouglas - thanks for organizing the Goose hacknight during Toronto Tech Week (and for the OpenRouter tokens!) Would be great to have your esteemed human eyes review my AI-generated code. It's not fully baked yet, but I suspect the juice is worth the squeeze - https://x.com/tobi/status/1942628740909130167?s=61 🍻 |
- Added new entries to .gitignore for workspace and example files - Updated Accept headers in SSE and StreamableHttp transport implementations to support application/vnd.mcp-ui.remote-dom+javascript for improved content negotiation
- Add @mcp-ui/client dependency for standards-compliant UI rendering - Implement UIResourceRenderer component with smart mimeType detection - Add UI resource detection and scanning logic in GooseMessage component - Support Remote DOM, HTML, and URI-list resource types with proper iframe sandboxing - Integrate UI actions (tool calls, intents, prompts) with Goose chat flow - Add backend UI capabilities declaration in MCP client - Maintain backward compatibility with existing text/image responses - Clean up verbose debug logging while preserving essential diagnostics Successfully renders interactive UI components from MCP servers like Phantasmo product catalog with full user interaction support.
|
@mgd1984 would you mind updating to main - we now have the new GUI merged, and would be good to look at this next if possible - looks like small adjustments (was a big change, but hopefully still works!) |
|
@mgd1984 oh yeah - and that is really jack BTW - keen to see MCP UI happen! |
Resolved merge conflicts in: - ui/desktop/package.json: Merged dependencies from both branches, kept newer versions, added @mcp-ui/client - ui/desktop/package-lock.json: Regenerated based on resolved package.json - ui/desktop/src/components/GooseMessage.tsx: Kept UI resources rendering from feature branch - ui/desktop/src/components/ToolCallWithResponse.tsx: Accepted upstream version All merge conflicts have been resolved and the branch is ready for development.
|
@mgd1984 do you have a test server which shows some MCP ui that you use? |
|
@michaelneale - thanks for all the feedback. Much appreciated.
Goose.MCP-UI.Demo.remote-dom.mp4 |
|
yeah I am liking this - wonder how we can unify efforts with @aharvard? I am also curious about the remoteDom script - and, crucially, component libraries. I expect in goose we will want to provide goose L&F components so when it uses that, it will show up nicely in goose? Also - there is the concept of a visual "sidecar" in goose - I like the idea of MCP UI stuff showing in that side car (to the right) vs inline, but would need to play with it to see if it makes sense. The reason is - we can use that to extend goose itself for nice things like automatic graph/chart visualisation in parallel to the main chat output - but this is all really just ideas in heads so far, but lets keep going! On security front: there is good discussion in that other PR, but I wonder if at a high level, given these are MCPs, the risk isn't from the MCP itself (if you are installing a bad MCP - you have already lost and would attack at execution level directly), perhaps the concern is more content injection/prompt injection (similarly in that case, there are a plenthora of tools for an injected prompt to try to use - this is another one of those but I am not sure it would be the worst one) so I am still confused as to what the security concern is in the case of a desktop implementation. I understand untrusted/browser would be very different though. |
Upgrades @mcp-ui/client library to latest version for improved UI resource rendering capabilities and bug fixes
- Add automatic UI resource detection in extension manager - Implement content conversion for flattened UI resources - Update provider formats to handle UI resource metadata - Add UI-aware content processing in toolshim module Enables backend to detect when interactive content has been returned as text and converts it to proper UI resource format
Updates message content types to properly handle UI resource content, enabling frontend components to detect and process interactive MCP UI resources
- Add comprehensive UI content detection in tool responses - Implement automatic JSON parsing for structured UI resources - Add JavaScript and HTML content recognition and rendering - Support both direct UI resources and text-embedded UI content - Provide robust fallback handling for various content formats Enables automatic detection and rendering of interactive MCP UI content from tool responses while maintaining backward compatibility Note: Contains some eslint warnings for complex MCP protocol types that will be addressed in follow-up commit
- Update context management to handle resource content types - Enhance message streaming hooks for UI resource processing - Add UI resource handling to GooseMessage component - Update main.ts and index.html for UI rendering support Provides necessary frontend infrastructure changes to support MCP UI resource detection, processing, and rendering throughout the application Note: Contains eslint warnings for generated API types that will be addressed in follow-up commit
- Replace 'any' type with proper interface for toolCall prop - Add eslint-disable comments for complex MCP resource types - Ensures clean linting while maintaining type safety Resolves remaining linting issues from MCP-UI integration
- Fix mimeType typing in UIResourceRenderer to avoid ReactNode type errors - Remove HTMLIFrameElement reference causing no-undef error - Add eslint-disable comments for complex generated API types - Ensure proper string conversion for display values All linting and type checking now passes cleanly
|
Love the sidecar concept (although, wingmate miiiight be a better name...jus' sayin' 😉). It feels similar to ChatGPT's Canvas or Claude's Artifact from a UX/UI perspective, both of which were both big usability improvements. Catching up on @Kvadratni's #3493 to see how the As for a "Goosified L&F" via remoteDom components, I found this on Shopify's remote-dom repo that seems like it might be helpful:
Further reading: https://shopify.engineering/remote-rendering-ui-extensibility
|
This commit combines two major enhancements to create a unified interactive UI system: 🎯 MCP-UI Integration: - Add @mcp-ui/client dependency for rendering interactive UI components - Implement UIResourceRenderer component with secure iframe sandboxing - Add UI resource detection and automatic UI/text content switching - Update Accept headers to support MCP-UI content types (remote-dom, HTML, URI lists) - Enhance Content Security Policy to allow iframe rendering with ui: scheme 🔧 Checkpoint & Sidecar System (from PR block#3493): - Add SidecarLayout component for side-by-side UI rendering - Implement checkpoint system with automatic file backup and versioning - Add intelligent diff generation with file-type aware context sizing - Create RestoreModal and MessageRestoreLink for file restoration workflow - Add list_checkpoints and restore_checkpoint tools to developer router 🔗 Unified Architecture: - Sidecar can display both diff content AND interactive MCP-UI components - Enhanced message filtering to support tool request/response matching - Resource type system handles both UI resources and checkpoint data - Integrated debug logging for MCP-UI resource detection This creates a comprehensive interactive AI agent interface that combines: - Rich, interactive UI components via MCP-UI standard - File versioning and diff management via checkpoint system - Side-by-side layout for optimal workflow management Resolves: Unifies MCP-UI integration efforts with checkpoint/sidecar functionality
🎯 Sidecar UI Resource Integration: - Add UIResourceViewer component to SidecarLayout for seamless UI display - Implement showUIResource/hideUIResource methods with proper state management - Add placeholder cards in chat when UI resources move to sidecar - Enhance iframe styling with full-space utilization and proper backgrounds 🔒 Type Safety & Stability: - Add getResourceText and isTextResource helper functions - Replace unsafe (resource as any).text patterns with proper type guards - Fix infinite loop issues by replacing useEffect with useRef pattern - Implement showInSidecarOnce helper to prevent resource re-rendering 🎨 UI/UX Improvements: - Remove padding constraints limiting iframe display area - Add absolute positioning for optimal space usage - Inject CSS rules ensuring 100% iframe dimensions - Improve visual feedback with 'Opened in side panel' indicators ⚡ Performance & Reliability: - Eliminate continuous re-render cycles causing message streams - Add proper resource content validation and error handling - Optimize resource detection and processing workflow
- Resolved module refactoring from sub_recipe_execution_tool to subagent_execution_tool - Fixed SidecarLayout implementation with Monaco diff viewer and UI resource support - Updated CSP in index.html to support MCP UI functionality - Added missing ToolCallWithResponse exports for diff content detection - Resolved import conflicts and mutual file additions - All TypeScript compilation passes successfully
- Fix SidecarLayout to use UIResourceRenderer instead of placeholder - Add adaptive sidecar width (600px for UI resources, 500px for diffs) - Improve UIResourceRenderer styling with proper iframe dimensions - Add conditional overflow handling for different content types - Include CSS styling for iframe content with rounded corners and shadows - Ensure UI resources render properly without truncation Resolves UI resource display issues in the sidecar panel.
Kvadratni
force-pushed
the
feature/diff-viewer-sidecar
branch
from
b7df644 to
aa57f48
Compare
|
Update: merged with feature/diff-viewer-sidecar branch and integrated MCP-UI rendering into the existing Key changes:
Re: security - Using iframe sandboxing similar to @aharvard's approach, but integrated into the existing sidecar rather than inline chat. |
|
This is awesome @mgd1984! Super exciting. Trying to follow up on some threads -
Please LMK if there's anything you need. Happy to contribute to the effort. |
|
@mgd1984 thanks! Great to see it taking shape :) As for the As Ido mentioned - we're happy to help. For testing - there's a sample mcp-ui server we built to demonstrate(but it's fully functioning), it's built on top of the existing Shopify MCP server and enriches it with MCP-UI components. It's returning iframes but can be used to test some of the flows: https://mcpstorefront.com Liad.Yosef.s.Video.-.Jul.10.2025-VEED.2.mp4 |
- Update SidecarLayout to increase width for UI resources to 800px. - Add logging to ToolCallWithResponse for detecting proper UI resources and skipping fallback processing. - Modify ToolResultView to conditionally skip fallback UI detection if proper UI resources are present. - Implement detection and extraction for @mcp-ui/server format in UIResourceRenderer, improving resource handling and logging. These changes improve the rendering and processing of UI resources, ensuring a more robust integration with the new UI format.
|
Thanks for the resources @idosal and the demo/sample server @liady! I went back and did some further digging into the PRs, tweets, docs, and demos in the space - really trying to get the lay of the land here. The domain is still quite nascent, but it feels like some common terms & tools are starting to emerge. Whether we call it Generative UI/GenUI, Invisible UX, Agent-native, or something Karpathy coins and we all jump on, the challenge seems to be in making the abstractions concrete enough to actually build with, yet flexible and familiar enough to be composable and adoptable by devs. To make matters more complex, all of the above must be done while models with larger context windows and better tool use continue to drop. That said, devs have to start somewhere and React components seem to be a popular choice that many projects are converging on. Vercel literally includes it in their definition:
By piggy-backing on existing MCP primitives, What would be great for developers is to have a solid synthesis of the emerging alternatives, so that we can pick wisely. I feel like this is @kentcdodds' superpower. re: the sample mcp-ui server - very cool and meta ("MCP-ify your Shopify store"). I tried to plug it in, but was still having problems rendering UIResources. Most of my testing has been with an MCP server that I deployed to Cloudflare, but it's quite complex (uses Trying to isolate the issue, hope to have a working demo to share soon. |
Kvadratni
force-pushed
the
feature/diff-viewer-sidecar
branch
3 times, most recently
from
3b7ad8a to
b4b6bfb
Compare
|
@mgd1984 ,super cool! I agree that we need to explore the ideal generalization, while in parallel already provide concrete starting point for developers. |
|
I think we have this now in a different version thanks to all people involved? closing for now, feel free to reopen if wrong |
7 participants
Adds support for MCP-UI inside of Goose.
This PR introduces support for interactive UI components in the Goose framework by extending the Model Context Protocol (MCP) to include UI resource capabilities. The changes span the Rust backend and the TypeScript frontend, adding new capabilities for detecting, rendering, and handling UI resources while maintaining backward compatibility with existing functionality.
Note: MCP-UI is not currently part of the official MCP Specification (or any popular LLM/agent host...yet), but offers a compelling vision for how to extend its capabilities. As an extensible, open-source agent framework, Goose felt like a natural fit for MCP-UI. h/t @idosal
Backend Enhancements (Rust)
ui_capable_extensionsto theExtensionManagerfor tracking extensions that support UI resources and implemented methods to query and manage these capabilities (supports_ui,extension_supports_ui) incrates/goose/src/agents/extension_manager.rs. [1] [2] [3]UICapabilitiesstruct incrates/mcp-core/src/protocol.rsto represent server-side UI resource support, including supported formats. This is integrated intoServerCapabilities. [1] [2]McpClientincrates/mcp-client/src/client.rsto declare and detect UI capabilities during initialization and to handle UI-aware tool calls (call_tool_with_ui). [1] [2] [3] [4]Frontend Enhancements (TypeScript)
@mcp-ui/clientlibrary for rendering UI components in an iframe-based sandboxed environment. Updatedpackage.jsonandpackage-lock.jsonto include this dependency. [1] [2]UIResourceRenderercomponent in the frontend to detect and render UI resources, and modified theToolCallWithResponsecomponent to support UI-aware interactions.Documentation
goose-mcp-ui-integration.md, a detailed document explaining the implementation, usage, and technical details of the MCP UI integration in Goose.