How to debug HTTP_PROXY config?

[pat-s]

I am deploying code-server in an environment with an HTTP proxy in a container. Env vars HTTP_PROXY and HTTPS_PROXY are set in the container.

However, when installing extensions, I am getting the error

ERROR: getaddinfo ENOTFOUND open-vsx.org

The nameservers in the container are internal ones and only resolve internal addresses. So far, so good. However, I would expect code-server to utilize the proxy for DNS resolution (which is able to resolve public DNS records).
Yet even when starting code-server with --log debug I don't see any proof that the env vars are actually honored and picked up during the attempt to install an extension.

Note: I am able to manually download any .vsix extension from open-vsx.org in the code-server terminal with the env vars set. So the proxy is working in general. What is not working is the attempt to install extensions when clicking on the "Install extension XY" in the extension overview.

I've already tried to set the proxy explicitly in the user settings in code-server, same result.

Is there any additional configuration I need to do to tell code-server to use the proxy for DNS resolution? Or any way I can ensure the proxy is actually picked up?

[code-asher]

Just to make sure I am on the same page:

1. Running code-server --install-extension <extension id> in the terminal: works
2. Searching extensions in the sidebar: works
3. Clicking install in the sidebar or in the extension details: broken

My first guess is that the environment variables are set in your shell, but not in code-server's environment. Can you share more details on how you are running code-server and setting environment variables in the container?

I believe you can also set the proxy in the settings, so that might be worth a shot just to see if it is an environment variable issue.

[code-asher]

I tried 4.22.1 just in case it has a bug but that worked for me as well. Here is what I got (minus the router debug lines). I used a nonexistent domain this time, just wanted to make sure it gets set even in that case.

coder@dev27:~/coder$ printenv | grep -i proxy
VSCODE_PROXY_URI=https://{{port}}--dev--dev27--code-asher--apps.dev.coder.com
coder@dev27:~/coder$ code-server --version
4.22.1 760d1318e98945d05133c6121f99541ca7a39bf8 with Code 1.87.2
coder@dev27:~/coder$ http_proxy=http://does-not-exist DEBUG=* code-server --install-extension vscodevim.vim
Installing extensions...
  https-proxy-agent Creating new HttpsProxyAgent instance: 'http://does-not-exist/' +0ms
  https-proxy-agent Creating `net.Socket`: { ALPNProtocols: [ 'http/1.1' ], host: 'does-not-exist', port: 80, auth: null, rejectUnauthorized: true, keepAlive: true } +5ms                                                  
  https-proxy-agent:parse-proxy-response onerror Error: getaddrinfo ENOTFOUND does-not-exist     at GetAddrInfoReqWrap.onlookup [as oncomplete] (node:dns:108:26) { errno: -3008, code: 'ENOTFOUND', syscall: 'getaddrinfo', hostname: 'does-not-exist' } +0ms                                                                             
Error while installing extensions: getaddrinfo ENOTFOUND does-not-exist
getaddrinfo ENOTFOUND does-not-exist

Sorry this is being so difficult, I wish I knew the solution. If I had a way to reproduce I would step through the code and find the problem spot.

[pat-s]

Thanks for the help!

I digged a bit more and here's what I found:

It is likely a DNS resolution issue in the first place. I am not yet sure what code-server's part in that one is.

-> The local nameservers in that specific environment do not resolve public domains. If I declare them explicitly in /etc/hosts (besides more intermediate microsoft.xxx.yyy domains), I get over this step and finally get to a point where clicking on the "install" button does not immediately error but just runs infinite (yet this is nothing more than a debug solution as code-server is running in a k8s pod).

Inspecting the logs then shows some TLS issue. This is strange again as I have set NODE_EXTRA_CA_CERTS to the bundle containing all certs (this bundle is working for many other applications).

So right now I see it as follows:

• code-server is not directly honoring the proxy for DNS resolution, i.e. it first uses the local nameservers and does not directly go through the proxy
• There's an issue with the TLS certs which I am not sure yet which side is "to blame". This bundle is working fine with other apps and AFAIK, the only thing needed is to set NODE_EXTRA_CA_CERTS to the CA cert bundle for code-server to honor these?

For now, I don't know what to next. The people responsible for the DNS tell me it is an application issue and the local nameservers will never resolve public domains. The application should directly talk to the HTTP_PROXY for DNS resolution.
Their claim is backed by the fact that other applications running in this environment do so when they have to resolve public domains.

For now, I have to tell users that they need to install every extension via manual downloads + install. Which is of course a pain, the same goes for updating. Hence, I am still interested in a long-term solution to get this working, otherwise it is likely that code-server will have to be removed again.

[code-asher]

That is very interesting. 🤔 Hmm so maybe I can reproduce if I have a DNS that is only accessible through the proxy. Honestly I am not too sure how to set that up; maybe I can do something with iptables.

the only thing needed is to set NODE_EXTRA_CA_CERTS to the CA cert bundle for code-server to honor these?

I believe this is exactly right, although maybe there is some caveat I am not aware of.

I looked around the proxy agent for issues related to DNS but did not find anything. https://github.com/search?q=repo%3ATooTallNate%2Fproxy-agents+dns&type=issues

Maybe if we can come up with a simple Node repro we can post there. Oh they have an example actually:

import * as https from 'https';
import { ProxyAgent } from 'proxy-agent';

// The correct proxy `Agent` implementation to use will be determined
// via the `http_proxy` / `https_proxy` / `no_proxy` / etc. env vars
const agent = new ProxyAgent();

// The rest works just like any other normal HTTP request
https.get('https://jsonip.com', { agent }, (res) => {
  console.log(res.statusCode, res.headers);
  res.pipe(process.stdout);
});

I used the Node bundled with my copy of code-server:

$ mkdir test
$ vim test.mjs # paste contents above
$ npm install proxy-agent
$ NODE_EXTRA_CA_CERTS=$HOME/.mitmproxy/mitmproxy-ca-cert.pem DEBUG=* https_proxy=http://localhost:8085 ~/.local/lib/code-server-4.22.1/lib/node test.mjs

One difference I noticed is that I have to use https_proxy where with code-server I could get by with only http_proxy, but I think this is because VS Code essentially sets https_proxy equal to http_proxy if not set, so probably not related to the issue.

Hopefully it reproduces here, but if not the problem must be in code-server or VS Code...

---

Another thing we might try is downloading the official VS Code, and then using code --install-extension to see if it has the same issues. That could narrow down whether code-server specifically has messed something up in the way it wraps VS Code or if it is in VS Code itself.

If it is in VS Code itself, we could try opening an issue in https://github.com/microsoft/vscode where they probably understand the inner workings a lot better than I do. If not, then, at least it narrows things down.

It is also possible to use their web version directly without the code-server wrapper by running code --serve-web with the official VS Code; note this will download their web server though, it is not bundled with the VS Code distribution.

[pat-s]

Good news! I got is solved.

Turns out it was a wrong specification of NODE_EXTRA_CA_CERTS! 🤯️
Background: I assumed that /etc/ssl/certs/ca-bundle.crt is automatically read, also based on your comment here. And that NODE_EXTRA_CA_CERTS only adds additional certs.

Turns out that in this case, setting NODE_EXTRA_CA_CERTS=/etc/ssl/certs/ca-bundle.crt solved the issue directly as all required certs were in there and no important ones were in the bundle which I initially referenced via NODE_EXTRA_CA_CERTS.

Take-away: maybe it is useful to add that piece of information to the docs somewhere?

Overall: thanks for your great patience and tremendous help here: In the end the one-liner to efficiently debug the issue lead me to find the issue and made it easier to swap different CA bundles.

[code-asher]

Ahhhhhhh very interesting! I believe VS Code meant to read certificates installed on those system paths but I bet there is a bug there; I made a note to dig into it later. Sorry that ended up being misleading! Definitely agree this needs to be documented.

Really glad you figured it out!!