Codex smoke test fails: HTTP_PROXY points to port 3128 but explicit proxy is unreachable from agent container

[Mossaka]

Summary

Codex (Rust/reqwest) honors the HTTP_PROXY=http://172.30.0.10:3128 environment variable set by the firewall, but port 3128 is unreachable from the agent container in intercept mode. This causes all MCP connections and API calls to fail with Connection refused (os error 111).

Copilot and Claude are unaffected because they don't route traffic through HTTP_PROXY — their requests go through ports 80/443 which get transparently DNAT-redirected to port 3129.

Tracking issue: github/gh-aw#13603

Evidence

From smoke-codex run 21724483561 on PR #13970:

1. HTTP_PROXY env vars injected (port 3128)

[entrypoint]   HTTP_PROXY=http://172.30.0.10:3128
[entrypoint]   HTTPS_PROXY=http://172.30.0.10:3128

2. Intercept mode redirects 80/443 → port 3129

[iptables] Squid proxy: squid-proxy:3128 (intercept: 3129)
[iptables] Redirect HTTP (80) and HTTPS (443) to Squid intercept port...
DNAT  tcp  dpt:80   to:172.30.0.10:3129
DNAT  tcp  dpt:443  to:172.30.0.10:3129

3. Codex (reqwest) routes through HTTP_PROXY → Connection refused

codex_rmcp_client::auth_status: OAuth discovery requests failed for http://172.30.0.1:80/mcp/safeinputs:
  error sending request for url (http://172.30.0.1/.well-known/oauth-authorization-server)
  Caused by:
    0: client error (Connect)
    1: tcp connect error
    2: Connection refused (os error 111)

4. All MCP transport workers die

ERROR rmcp::transport::worker: worker quit with fatal: Transport channel closed,
  when Client(reqwest::Error {
    kind: Request,
    url: "http://172.30.0.1/mcp/safeoutputs",
    source: hyper_util::client::legacy::Error(Connect,
      ConnectError("tcp connect error", 172.30.0.10:3128,
        Os { code: 111, kind: ConnectionRefused, message: "Connection refused" }))
  })

All 6 MCP servers fail identically: safeinputs, safeoutputs, playwright, serena, tavily, github.

Root Cause Analysis

1. docker-manager.ts lines 328-329 set proxy env vars to port 3128:

   HTTP_PROXY: `http://${networkConfig.squidIp}:${SQUID_PORT}`,   // 3128
   HTTPS_PROXY: `http://${networkConfig.squidIp}:${SQUID_PORT}`,  // 3128

2. Squid is configured with both ports — 3128 (explicit) and 3129 (intercept):

   http_port 3128 ssl-bump ...
   http_port 3129 intercept
   

3. Healthcheck passes because it runs inside the Squid container (nc -z localhost 3128 && nc -z localhost 3129).

4. But from the agent container, port 3128 on 172.30.0.10 refuses connections. The intercept port (3129) works because traffic on 80/443 is DNAT-redirected there by iptables.

5. Codex uses Rust's reqwest which respects HTTP_PROXY — so it tries to route all traffic through 172.30.0.10:3128 instead of going through 80/443 (which would be intercepted). Since 3128 is unreachable, every request fails.

Suggested Fixes

Option A: Fix port 3128 exposure to the agent container network
Ensure the explicit proxy port is actually reachable from the agent container. This may require Docker Compose network/port configuration changes.

Option B: Unset HTTP_PROXY/HTTPS_PROXY in intercept mode
Since intercept mode uses DNAT on 80/443, the proxy env vars are unnecessary and actively harmful for clients like reqwest that honor them. Don't set them when intercept mode is enabled.

Option C: Point HTTP_PROXY to port 3129
Change the env vars to use the intercept port instead:

HTTP_PROXY: `http://${networkConfig.squidIp}:${SQUID_INTERCEPT_PORT}`,  // 3129

(May not work since intercept mode expects relative URLs, not absolute proxy URLs.)

Option B seems safest — intercept mode should not need explicit proxy env vars at all.

[github-actions]

👋 Hello! This issue appears to be related to #519, which was recently closed.

Connection:

• HTTP traffic NAT'd to Squid fails with 'Invalid URL - Missing hostname' #519 fixed Squid to use intercept mode (port 3129) for NAT'd traffic from 80/443
• Codex smoke test fails: HTTP_PROXY points to port 3128 but explicit proxy is unreachable from agent container #523 identifies that HTTP_PROXY/HTTPS_PROXY still point to port 3128, which is unreachable from the agent container
• Both issues affect Codex smoke tests and involve the dual-port Squid configuration (3128 explicit, 3129 intercept)

Key difference:

• HTTP traffic NAT'd to Squid fails with 'Invalid URL - Missing hostname' #519: NAT'd traffic was failing because Squid wasn't in intercept mode
• Codex smoke test fails: HTTP_PROXY points to port 3128 but explicit proxy is unreachable from agent container #523: Explicit proxy traffic (via HTTP_PROXY env var) fails because port 3128 is unreachable

This appears to be a follow-up issue revealing that the fix for #519 didn't address the HTTP_PROXY configuration. Your suggested Option B (unset HTTP_PROXY/HTTPS_PROXY in intercept mode) makes sense since intercept mode handles traffic transparently.

This is an automated message from the issue duplication detector.

AI generated by Issue Duplication Detector