[Bug]:playwright.connectOverCDP can't connect to the browser via a proxy

[hiczlf]

Version

1.51.0

Steps to reproduce

In the macOS environment, the proxy is configured in the command line. The proxy settings are as follows:

env |grep proxy
https_proxy=http://127.0.0.1:7890
http_proxy=http://127.0.0.1:7890
all_proxy=socks5://127.0.0.1:7890

Chromium is started in advance with the option --remote-debugging-port=9222.

run the following code:

  const browser = await chromium.connectOverCDP("http://127.0.0.1:9222/");

Expected behavior

I can successfully connect to the browser.

Actual behavior

will output:

browserType.connectOverCDP: Unexpected status 400 when connecting to http://127.0.0.1:9222/json/version/.
This does not look like a DevTools server, try connecting via ws://.
Call log:
  - <ws preparing> retrieving websocket url from http://127.0.0.1:9222/
  - <ws preparing> fetching http://127.0.0.1:9222/json/version/

    at main (/Users/lf/code/learn/ln-patchright/cdp.mjs:7:34)
    at /Users/lf/code/learn/ln-patchright/cdp.mjs:19:1 {
  name: 'Error'
}

Additional context

I've taken a look at the code. I think there is a bug in the final processing when the endpointURL is http under the proxy configuration. The specific code is here. https://github.com/microsoft/playwright/blob/main/packages/playwright-core/src/server/utils/network.ts#L52

export function httpRequest(params: HTTPRequestParams, onResponse: (r: http.IncomingMessage) => void, onError: (error: Error) => void) {
  const parsedUrl = url.parse(params.url);
  let options: https.RequestOptions = {
    ...parsedUrl,
    agent: parsedUrl.protocol === 'https:' ? httpsHappyEyeballsAgent : httpHappyEyeballsAgent,
    method: params.method || 'GET',
    headers: params.headers,
  };
  if (params.rejectUnauthorized !== undefined)
    options.rejectUnauthorized = params.rejectUnauthorized;

  const timeout = params.timeout ?? NET_DEFAULT_TIMEOUT;

  const proxyURL = getProxyForUrl(params.url);
  if (proxyURL) {
    const parsedProxyURL = url.parse(proxyURL);
    if (params.url.startsWith('http:')) {
      options = {
        path: parsedUrl.href,
        host: parsedProxyURL.hostname,
        port: parsedProxyURL.port,
       #!!!! Here, the Host request header will be lost, which is very important for HTTP requests and proxy servers.
        headers: options.headers,
        method: options.method
      };
    } else {
      (parsedProxyURL as any).secureProxy = parsedProxyURL.protocol === 'https:';

      options.agent = new HttpsProxyAgent(parsedProxyURL);
      options.rejectUnauthorized = false;
    }
  }

I can successfully connect to the browser in the same environment using the following code.

  const browser = await chromium.connectOverCDP("http://127.0.0.1:9222/", {
    headers: {
      Host: "127.0.0.1:9222",
    },
  });

Environment

System:
    OS: macOS 15.3.1
    CPU: (12) arm64 Apple M2 Max
    Memory: 101.30 MB / 32.00 GB
  Binaries:
    Node: 22.12.0 - ~/.local/state/fnm_multishells/28246_1741934992754/bin/node
    Yarn: 1.22.22 - ~/.local/state/fnm_multishells/28246_1741934992754/bin/yarn
    npm: 10.9.0 - ~/.local/state/fnm_multishells/28246_1741934992754/bin/npm
    pnpm: 9.5.0 - /opt/homebrew/bin/pnpm
  IDEs:
    VSCode: 1.91.1 - /Applications/Visual Studio Code.app/Contents/Resources/app/bin/code
  Languages:
    Bash: 3.2.57 - /bin/bash
  npmPackages:
    playwright: /Users/lf/code/source/playwright/packages/playwright => 1.52.0-next
    playwright-core: /Users/lf/code/source/playwright/packages/playwright-core => 1.52.0-next

[agg23]

What proxy are you using? Can you examine the response and make sure the proxy is adding X-Forwarded-Host?

[hiczlf]

@agg23 Actually, it has little to do with X-Forwarded-Host。I use clashx pro as proxy, but it's quite difficult to see the specific headers. So I used mitmproxy as a proxy and compared the differences between the request headers sent by curl and Playwright.

curl

http_proxy="http://127.0.0.1:8080"   curl http://127.0.0.1:9222/json/version/
//request headers by curl
GET http://127.0.0.1:9222/json/version/ HTTP/1.1
Host: 127.0.0.1:9222
User-Agent: curl/8.7.1
Accept: */*
Proxy-Connection: Keep-Alive

playwright

// call connectOverCDP("http://127.0.0.1:9222/") in cdp.mjs
http_proxy="http://127.0.0.1:8080" all_proxy="" https_proxy="" node cdp.mjs
//request headers by playwright
GET http://127.0.0.1:9222/json/version/ HTTP/1.1
User-Agent: Playwright/1.52.0-next (arm64; macOS 15.3) node/22.12
Host: 127.0.0.1:8080
Connection: keep-alive

As you can see, the "Host" fields in the two request headers are different. The "Host" field should be the host corresponding to the target url, rather than the host of the proxy server.

Some proxy servers will encounter errors in this situation, such as ClashX Pro.