Replace custom JSON handler with StreamableHTTPHandler in mcp-gateway

[Copilot]

The mcp-gateway was using a custom JSON POST API instead of the standard MCP streamable-http protocol, preventing proper client connection via StreamableClientTransport.

Changes

Architecture shift: Each backend MCP server now gets its own proper streamable-http endpoint via go-sdk's StreamableHTTPHandler:

• Proxy server pattern: Query backend tools/resources/prompts at startup, dynamically register them on a proxy mcp.Server, forward calls to backend session
• Replace custom handlers: /mcp/{server} endpoints now use mcp.NewStreamableHTTPHandler instead of manual JSON-RPC parsing
• Preserve compatibility: /health and /servers endpoints unchanged

Code reduction: Removed ~150 lines of custom protocol implementation (handleInitialize, handleListTools, handleCallTool, etc.) in favor of go-sdk's built-in handling.

Usage

Agents connect using standard MCP protocol:

{
  "mcpServers": {
    "gh-aw": {
      "url": "http://host.docker.internal:8080/mcp/gh-aw"
    }
  }
}

Client code:

transport := &mcp.StreamableClientTransport{
    Endpoint: "http://host.docker.internal:8080/mcp/gh-aw",
}
session, _ := client.Connect(ctx, transport, nil)
tools, _ := session.ListTools(ctx, &mcp.ListToolsParams{})

Ready for awf firewall integration once host.docker.internal is whitelisted in Squid proxy.

Warning

Firewall rules blocked me from connecting to one or more addresses (expand for details)

I tried to connect to the following addresses, but was blocked by firewall rules:

• https://api.github.com/user
  • Triggering command: /usr/bin/gh gh api user --jq .login (http block)

If you need me to access, download, or install something from one of these locations, you can either:

• Configure Actions setup steps to set up my environment, which run before the firewall is enabled
• Add the appropriate URLs or hosts to the custom allowlist in this repository's Copilot coding agent settings (admins only)

Original prompt

---

This section details on the original issue you should resolve

<issue_title>Integrate mcp-gateway (awmg) with awf firewall using MCP streamable-http protocol</issue_title>
<issue_description>## Summary

Enable MCP servers running on the host to be accessible by agents running inside the awf (aw-firewall) container through the mcp-gateway, using the standard MCP streamable-http protocol.

Current State

mcp-gateway (awmg) currently:

• Exposes custom HTTP endpoints: /health, /servers, /mcp/{server} (POST)
• Uses simple JSON-RPC over HTTP, NOT the standard MCP streamable-http protocol
• Only supports command/stdio transport for backend MCP servers
• Has config rewriting that generates http://host.docker.internal:PORT/mcp/{server} URLs

awf (firewall) currently:

• Runs agents in isolated containers on dedicated network (172.30.0.0/24)
• Routes HTTP traffic through Squid proxy with domain whitelisting
• Allows localhost traffic (stdio MCP servers work automatically)
• Supports host.docker.internal for accessing host services

The Problem

The current gateway's /mcp/{server} endpoint is a custom JSON POST API, not a proper MCP streamable-http endpoint. An MCP client using StreamableClientTransport (which is the standard way to connect via HTTP) won't be able to communicate with it correctly.

Proposed Architecture

┌─────────────────────────────────────────────────────────────────┐
│                        HOST MACHINE                              │
│                                                                  │
│  ┌──────────────────────────────────────────────────────────┐   │
│  │              mcp-gateway (awmg)                           │   │
│  │                                                           │   │
│  │   HTTP Server (:8080)                                     │   │
│  │   └── /mcp  ← StreamableHTTPHandler (NEW)                │   │
│  │       └── routes to appropriate backend MCP server        │   │
│  │                                                           │   │
│  │   Backend MCP Sessions (stdio):                           │   │
│  │   ├── gh-aw mcp-server                                    │   │
│  │   ├── filesystem server                                   │   │
│  │   └── other MCP servers...                                │   │
│  └──────────────────────────────────────────────────────────┘   │
│                           ▲                                      │
│                           │ host.docker.internal:8080            │
│  ┌────────────────────────┼─────────────────────────────────┐   │
│  │        awf container   │                                  │   │
│  │                        │                                  │   │
│  │   Agent (Copilot/Claude/Codex)                            │   │
│  │   └── MCP Client                                          │   │
│  │       └── StreamableClientTransport                       │   │
│  │           Endpoint: http://host.docker.internal:8080/mcp  │   │
│  │                                                           │   │
│  └───────────────────────────────────────────────────────────┘   │
└─────────────────────────────────────────────────────────────────┘

Required Changes

1. Gateway: Add StreamableHTTPHandler endpoint

The gateway needs to expose a proper MCP streamable-http endpoint. The go-sdk already has StreamableHTTPHandler available (used in mcp_server.go:762-794):

handler := mcp.NewStreamableHTTPHandler(func(req *http.Request) *mcp.Server {
    // Route based on request path or header to appropriate backend
    return server
}, &mcp.StreamableHTTPOptions{
    SessionTimeout: 2 * time.Hour,
})

The gateway would need to:

1. Create a virtual "aggregated" MCP server that combines tools from all backends
2. Expose this via StreamableHTTPHandler on /mcp
3. When tools are called, route to the appropriate backend MCP session

2. Tool Aggregation Strategy

Two approaches to consider:

Option A - Namespace prefixing: Tools from different servers get prefixed (e.g., gh-aw:list_files, filesystem:read_file)

Option B - Flat namespace: All tools exposed without prefix, conflicts resolved by priority

3. Agent Configuration

The agent inside awf would use a single MCP config entry:

{
  "mcpServers": {
    "gateway": {
      "url": "http://host.docker.internal:8080/mcp"
    }
  }
}

4. Firewall Whitelist

host.docker.internal needs to be whitelisted in the Squid proxy configuration for the awf container.

Benefits

1. Security: MCP servers run on host with full access; agent runs in restricted container
2. Simplicity: Agent sees single MCP endpoint with all tools aggregated
3. Standard Protocol: Uses proper MCP streamable-http, not custom API
4. Existing Patterns: Both StreamableClientTransport and StreamableHTTPHandler are already used in the codebase

Implementation Scope

Main work in `pkg/cli/mc...

• Fixes Integrate mcp-gateway (awmg) with awf firewall using MCP streamable-http protocol #7429

---

💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more Copilot coding agent tips in the docs.