HTTPS request not working with proxy

[chowey]

I have a squid proxy on my local computer (127.0.0.1:3128). It is configured to require no authentication. If I try to use an https.request, I get an error like:

{ [Error: socket hang up] code: 'ECONNRESET' }

The following code reproduces this issue (provided of course that you have a squid proxy set up on port 3128):

var https = require('https');

var options = {
  hostname: '127.0.0.1',
  port: 3128,
  path: 'https://github.com/',
  headers: {
    Host: 'github.com',
    'User-Agent': 'Node.js/0.6.6'
  },
  method: 'GET'
};

var req = https.request(options, function (res) {
  console.log('statusCode: ', res.statusCode);
  console.log('headers: ', res.headers);

  res.on('data', function (chunk) {
    process.stdout.write(chunk);
  });
}).on('error', function (e) {
  console.error(e);
}).end();

It is definitely an error with node itself. If the problem were with squid, it would just result in responses with statusCode 501.

I am running v0.6.6 on Windows 7.

[koichik]

I think that you should use CONNECT method with HTTP (not HTTPS) if you want to use end-to-end HTTPS through a proxy. Unfortunately, Node does not support it well.

[chowey]

Okay, thanks for the pointer.

I played some more with this. I made a simple tcp echo server, which just passes along requests to the squid proxy, like so:

var net = require('net');

net.createServer(function (upstream) {
  var downstream = net.connect(3128, function () {
    upstream.on('data', function (data) {
      console.log(data.toString());
      downstream.write(data);
    });
    upstream.on('end', function () {
      downstream.end();
    });
  });
  downstream.on('data', function (data) {
    upstream.write(data);
  });
  downstream.on('end', function () {
    upstream.end();
  });
}).listen(3000);

This logs the request so I can see what request headers my browser sends for https through a proxy. All the browser does is send the following headers (for https://www.google.com):

CONNECT www.google.com:443 HTTP/1.1
User-Agent: Mozilla/5.0 (Windows NT 6.1; rv:9.0.1) Gecko/20100101 Firefox/9.0.1
Proxy-Connections: keep-alive
Host: www.google.com

Logically then I could reproduce these in node like so:

var http = require('http');

var options = {
  hostname: '127.0.0.1',
  port: 3128,
  path: 'www.google.com:443',
  headers: {
    'User-Agent': 'Node.js/0.6.6',
    'Proxy-Connections': 'keep-alive',
    'Host': 'www.google.com'
  },
  method: 'CONNECT'
};

var req = http.request(options, function (res) {
  console.log('statusCode: ', res.statusCode);
  console.log('headers: ', res.headers);

  res.on('data', function (chunk) {
    process.stdout.write(chunk);
  });
}).on('error', function (e) {
  console.error(e);
}).end();

It gives me a statusCode 411 from squid now. Per squid, it received the following headers:

CONNECT / HTTP/1.1
User-Agent: Node.js/0.6.6
Proxy-Connections: keep-alive
Host: www.google.com
Transfer-Encoding: chunked

It seems to me that the path got truncated from www.google.com:443 to just /.

Why is node nuking my path for the CONNECT method?

[koichik]

@chowey - Probably, squid displays a path without host/port. Actually, Node sends valid request line:

CONNECT www.google.com:443 HTTP/1.1

The problem may be Transfer-Encoding. You can prevent it:

-    'Host': 'www.google.com'
+    'Host': 'www.google.com',
+    `Content-Length`: 0

result:

$ ./node /tmp/2474.js
statusCode:  200
headers:  {}

I think that Node should not add Transfer-Encoding if CONNECT method is used. And should provide a socket to the user like Upgrade. Then, TLS and HTTPS modules should be able to accept the socket from the user... We need a lot of work :-<

[chowey]

I figured you might use options.createConnection to provide the custom socket. But in the https module:

78  options.createConnection = createConnection;

they just overwrite any createConnection I provide. It isn't like that for the http module.

If they did something instead like:

78  options.createConnection = options.createConnection || createConnection;

then I could provide a custom socket. My createConnection formula might be something like:

function createConnection(port, host, options) {
  var socket = net.connect(port, host, function () {
    socket.write('CONNECT www.google.com:443 HTTP/1.1\r\n' +
        'User-Agent: Node.js/0.6.6\r\n' +
        'Proxy-Connections: keep-alive\r\n' +
        'Host: www.google.com\r\n\r\n',
    function () {
      return socket;
    });
  });
}

(probably this function needs more work, but you get the idea.)

[koichik]

I have noticed that http-parser handles CONNECT method request as a Upgrade request. So, we can handle it using 'upgrade' listener not 'request' listener on http.Server.

[koichik]

@bnoordhuis - Can you review koichik/node@47adb2e and koichik/node@a635acb?

[bnoordhuis]

@koichik: I don't really like the concept, it adds a lot of extra API for a relatively minor issue. What are the alternatives?

[koichik]

@bnoordhuis - Thanks for the suggestion, how about like this:

var req = https.get({
  host: 'github.com',
  ...
  agent: new https.Agent({ // HTTPS
    httpTunnel: {          // over HTTP
      host: 'localhost',
      port: 3128
    }
  }
}, function(res) {
  ...
});

Or, should not Agent support tunneling?

[bnoordhuis]

That looks much simpler.

should not Agent support tunneling?

I'm undecided.

Yes, because HTTP is a first class citizen in Node.

No, because tunneling is complex. You start with simple CONNECT support, next thing you know you'll need to add various authentication methods, SOCKS, DNS-over-SOCKS, workarounds for proxy bugs, etc.

@isaacs, @piscisaureus, @ry: What do you think?

[chowey]

If I can add my $0.02, if the authentication methods, etc. etc. can be done in userland, then they can be done in userland.

The problem with CONNECT support was it could not be done in userland because the node core's http/https modules did not really permit it.

I don't know if the Agent support is really too much "fluff" above the node core, but I don't think it will create a snowball effect of needing to support proxy bugs, etc. Because all that complexity stuff, if the core API is solid and adaptable, can be pushed off to userland.

[koichik]

@chowey - Sorry for the delay, with current master you can use HTTPS via proxy like this:

var http = require('http');
var https = require('https');

var connectReq = http.request({ // establishing a tunnel
  host: 'localhost',
  port: 3128,
  method: 'CONNECT',
  path: 'github.com:443',
}).on('connect', function(res, socket, head) {
  // should check res.statusCode here
  var req = https.get({
    host: 'github.com',
    socket: socket, // using a tunnel
    agent: false    // cannot use a default agent
  }, function(res) {
    res.setEncoding('utf8');
    res.on('data', console.log);
  });
}).end();

So, Agent support may not be MUST...