Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

HTTP/HTTPS client requests throwing EPROTO #3692

Closed
brandonros opened this issue Nov 6, 2015 · 128 comments
Closed

HTTP/HTTPS client requests throwing EPROTO #3692

brandonros opened this issue Nov 6, 2015 · 128 comments
Labels
http Issues or PRs related to the http subsystem.

Comments

@brandonros
Copy link

Error: write EPROTO
    at Object.exports._errnoException (util.js:874:11)
    at exports._exceptionWithHostPort (util.js:897:20)
    at WriteWrap.afterWrite (net.js:763:14)

My code is littered with both HTTP and HTTPS requests. I have no idea what is causing this error but it is occurring in many places since I've upgraded to 4.2.1 from 0.12.7. Please tell me what information I need to provide to help debug this.

@Fishrock123
Copy link
Contributor

@brandonros I'm suspecting it has to do with the first-class IPv6 support Node v4 has. cc @mscdex maybe.

Edit: added IPv6

@Fishrock123 Fishrock123 added the http Issues or PRs related to the http subsystem. label Nov 6, 2015
@mscdex
Copy link
Contributor

mscdex commented Nov 6, 2015

What OS and OS version is this on?

@brandonros
Copy link
Author

@mscdex

Linux Nigel 3.13.0-43-generic #72-Ubuntu SMP Mon Dec 8 19:35:44 UTC 2014 i686 i686 i686 GNU/Linux
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 14.04.3 LTS
Release:    14.04
Codename:   trusty

@brandonros
Copy link
Author

@Fishrock123 @mscdex Any other information I can provide? Would love to get this fixed quickly. I'm sure it has to be something on my end, otherwise a huge portion of people would be experiencing it.

The code used to work without issue, but it started happening a) when I changed machines and b) when I switched versions.

I installed with nvm if that matters.

@mscdex
Copy link
Contributor

mscdex commented Nov 8, 2015

Is it happening with the same server(s)? If so, are these public servers? Otherwise, if you can, it might be helpful to see what types of addresses the servers are resolving to (e.g. IPv4, IPv6) and what types of addresses are assigned to the outgoing network interface the requests are going out.

@brandonros
Copy link
Author

@mscdex It is happening on the same offending server. It seems to happen on both requests made to localhost, and external APIs. All IPv4.

@trotyl
Copy link

trotyl commented Nov 9, 2015

Also encountered on OS X (My computer) and Ubuntu (Travis-CI) and Windows Server (AppVeyor) when using npm package 'request', the error trace is exactly same as described by @brandonros

@evanlucas
Copy link
Contributor

It sounds like you are trying to use port 80 for https?

Reproduced via:

require('https').get('https://google.com:80', function(res) { console.log(res.statusCode) }).on('error', console.log)

@brandonros
Copy link
Author

The reason that isn't the problem in my case is because it only happens
semi-frequently on otherwise valid endpoints.

On Sunday, November 8, 2015, Evan Lucas notifications@github.com wrote:

It sounds like you are trying to use port 80 for https?

Reproduced via:

require('https').get('https://google.com:80', function(res) { console.log(res.statusCode) }).on('error', console.log)


Reply to this email directly or view it on GitHub
#3692 (comment).

@evanlucas
Copy link
Contributor

It will be difficult for us to help debug the issue without being able to see some of the code. Is there any that you could share?

@brandonros
Copy link
Author

I went to dumb down some code I have, and it really just boils down to any HTTP or HTTPS request. Nothing else.

@trotyl Do you think it has anything to do with DNS or nginx rewrites? I'm really stretching here. I can't come up with anything else.

@trotyl
Copy link

trotyl commented Nov 9, 2015

It appears on some certain urls and randomly, I cannot even cause it to appear manually, and I have used a 'retry(100)' of RxJS to get rid of this problem. But it seems not a good solution.

Some url(s) on my crawler of my university site may cause the problem at some low possibility, I do have no idea where it came from as error trace directly begin from 'net.js', But what I can guess seems to be DNS or HTTPS or something relevant.

The url I have problem with seems to be 'https://uis.uestc.edu.cn/amserver/UI/Login' with method 'POST' with form 'IDToken0&IDToken=2012019050020&IDToken2=811073&IDButton=Submit&goto=aHR0cDovL3BvcnRhbC51ZXN0Yy5lZHUuY24vbG9naW4ucG9ydGFs&encoded=true&gx_charset=UTF-8', (the id and password is not mine but valid)

@brandonros
Copy link
Author

Upgrading to 5.0, changing machines, and switching to 64-bit seems to have fixed the problem. I don't want to jinx anything, but I have not seen the error in 3 days.

@brandonros
Copy link
Author

The error is still happening. I am installing longjohn on all of my servers to get more information.

@trotyl Any update?

@brandonros
Copy link
Author

How is this for a stack trace?

Error: write EPROTO
    at Object.exports._errnoException (util.js:860:11)
    at exports._exceptionWithHostPort (util.js:883:20)
    at WriteWrap.afterWrite (net.js:763:14)
---------------------------------------------
    at TLSSocket.Readable.on (_stream_readable.js:665:33)
    at tickOnSocket (_http_client.js:486:10)
    at onSocketNT (_http_client.js:502:5)
    at doNTCallback2 (node.js:450:9)
    at process._tickDomainCallback (node.js:405:17)
---------------------------------------------
    at ClientRequest.onSocket (_http_client.js:494:11)
    at Agent.addRequest (_http_agent.js:143:9)
    at new ClientRequest (_http_client.js:139:16)
    at Object.exports.request (http.js:31:10)
    at Object.exports.request (https.js:173:15)
    at Object.ebay.request (/home/brandon/foo/integrations/ebay.js:34:18)
    at Object.ebay.get_orders_api_call (/home/brandon/foo/integrations/ebay.js:122:7)
    at Object.ebay.get_orders (/home/brandon/foo/integrations/ebay.js:138:7)
    at handle_job (/home/brandon/foo/sync_server/app.js:124:29)
    at /home/brandon/foo/sync_server/app.js:214:12
    at process._tickDomainCallback (node.js:420:9)
    at /home/brandon/foo/api_server/lib/sync_server.js:77:11
    at _fulfilled (/home/brandon/foo/api_server/node_modules/q/q.js:834:54)
    at self.promiseDispatch.done (/home/brandon/foo/api_server/node_modules/q/q.js:863:30)
    at Promise.promise.promiseDispatch (/home/brandon/foo/api_server/node_modules/q/q.js:796:13)
    at /home/brandon/foo/api_server/node_modules/q/q.js:604:44
    at runSingle (/home/brandon/foo/api_server/node_modules/q/q.js:137:13)
    at flush (/home/brandon/foo/api_server/node_modules/q/q.js:125:13)
    at doNTCallback0 (node.js:428:9)
    at process._tickDomainCallback (node.js:398:13)
---------------------------------------------
    at requestTick (/home/brandon/foo/api_server/node_modules/q/q.js:197:21)
    at Function.nextTick (/home/brandon/foo/api_server/node_modules/q/q.js:180:13)
    at /home/brandon/foo/api_server/node_modules/q/q.js:603:15
    at Array.reduce (native)
    at /home/brandon/foo/api_server/node_modules/q/q.js:263:21
    at become (/home/brandon/foo/api_server/node_modules/q/q.js:602:9)
    at deferred.resolve (/home/brandon/foo/api_server/node_modules/q/q.js:618:9)
    at IncomingMessage.<anonymous> (/home/brandon/foo/api_server/lib/sync_server.js:17:13)
    at emitNone (events.js:72:20)
    at IncomingMessage.emit (events.js:166:7)
    at endReadableNT (_stream_readable.js:905:12)
    at doNTCallback2 (node.js:450:9)
    at process._tickDomainCallback (node.js:405:17)
---------------------------------------------
    at IncomingMessage.Readable.on (_stream_readable.js:665:33)
    at Object.sync_server.read_body (/home/brandon/foo/api_server/lib/sync_server.js:15:11)
    at ClientRequest.<anonymous> (/home/brandon/foo/api_server/lib/sync_server.js:45:22)
    at emitOne (events.js:77:13)
    at ClientRequest.emit (events.js:169:7)
    at HTTPParser.parserOnIncomingClient (_http_client.js:421:21)
    at HTTPParser.parserOnHeadersComplete (_http_common.js:88:23)
    at Socket.socketOnData (_http_client.js:311:20)
    at emitOne (events.js:77:13)
    at Socket.emit (events.js:169:7)
    at readableAddChunk (_stream_readable.js:146:16)
    at Socket.Readable.push (_stream_readable.js:110:10)
    at TCP.onread (net.js:523:20)
---------------------------------------------
    at Object.sync_server.request (/home/brandon/foo/api_server/lib/sync_server.js:44:6)
    at /home/brandon/foo/api_server/lib/sync_server.js:73:23
    at _fulfilled (/home/brandon/foo/api_server/node_modules/q/q.js:834:54)
    at self.promiseDispatch.done (/home/brandon/foo/api_server/node_modules/q/q.js:863:30)
    at Promise.promise.promiseDispatch (/home/brandon/foo/api_server/node_modules/q/q.js:796:13)
    at /home/brandon/foo/api_server/node_modules/q/q.js:604:44
    at runSingle (/home/brandon/foo/api_server/node_modules/q/q.js:137:13)
    at flush (/home/brandon/foo/api_server/node_modules/q/q.js:125:13)
    at doNTCallback0 (node.js:428:9)
    at process._tickDomainCallback (node.js:398:13)
---------------------------------------------
    at requestTick (/home/brandon/foo/api_server/node_modules/q/q.js:197:21)
    at Function.nextTick (/home/brandon/foo/api_server/node_modules/q/q.js:180:13)
    at /home/brandon/foo/api_server/node_modules/q/q.js:603:15
    at Array.reduce (native)
    at /home/brandon/foo/api_server/node_modules/q/q.js:263:21
    at become (/home/brandon/foo/api_server/node_modules/q/q.js:602:9)
    at deferred.resolve (/home/brandon/foo/api_server/node_modules/q/q.js:618:9)
    at [object Object].<anonymous> (/home/brandon/foo/api_server/node_modules/q/q.js:1859:22)
    at Timer.listOnTimeout (timers.js:92:15)
---------------------------------------------
    at /home/brandon/foo/api_server/node_modules/q/q.js:1858:9
    at _fulfilled (/home/brandon/foo/api_server/node_modules/q/q.js:834:54)
    at self.promiseDispatch.done (/home/brandon/foo/api_server/node_modules/q/q.js:863:30)
    at Promise.promise.promiseDispatch (/home/brandon/foo/api_server/node_modules/q/q.js:796:13)
    at /home/brandon/foo/api_server/node_modules/q/q.js:857:14
    at runSingle (/home/brandon/foo/api_server/node_modules/q/q.js:137:13)
    at flush (/home/brandon/foo/api_server/node_modules/q/q.js:125:13)
    at doNTCallback0 (node.js:428:9)
    at process._tickDomainCallback (node.js:398:13)
---------------------------------------------
    at requestTick (/home/brandon/foo/api_server/node_modules/q/q.js:197:21)
    at Function.nextTick (/home/brandon/foo/api_server/node_modules/q/q.js:180:13)
    at /home/brandon/foo/api_server/node_modules/q/q.js:603:15
    at Array.reduce (native)
    at /home/brandon/foo/api_server/node_modules/q/q.js:263:21
    at become (/home/brandon/foo/api_server/node_modules/q/q.js:602:9)
    at deferred.resolve (/home/brandon/foo/api_server/node_modules/q/q.js:618:9)
    at IncomingMessage.<anonymous> (/home/brandon/foo/api_server/lib/sync_server.js:17:13)
    at emitNone (events.js:72:20)
    at IncomingMessage.emit (events.js:166:7)
    at endReadableNT (_stream_readable.js:905:12)
    at doNTCallback2 (node.js:450:9)
    at process._tickDomainCallback (node.js:405:17)
---------------------------------------------
    at IncomingMessage.Readable.on (_stream_readable.js:665:33)
    at Object.sync_server.read_body (/home/brandon/foo/api_server/lib/sync_server.js:15:11)
    at ClientRequest.<anonymous> (/home/brandon/foo/api_server/lib/sync_server.js:45:22)
    at emitOne (events.js:77:13)
    at ClientRequest.emit (events.js:169:7)
    at HTTPParser.parserOnIncomingClient (_http_client.js:421:21)
    at HTTPParser.parserOnHeadersComplete (_http_common.js:88:23)
    at Socket.socketOnData (_http_client.js:311:20)
    at emitOne (events.js:77:13)
    at Socket.emit (events.js:169:7)
    at readableAddChunk (_stream_readable.js:146:16)
    at Socket.Readable.push (_stream_readable.js:110:10)
    at TCP.onread (net.js:523:20)
---------------------------------------------
    at Object.sync_server.request (/home/brandon/foo/api_server/lib/sync_server.js:44:6)
    at Object.sync_server.request_refresh (/home/brandon/foo/api_server/lib/sync_server.js:99:21)
    at /home/brandon/foo/api_server/controllers/sync.js:131:22
    at _fulfilled (/home/brandon/foo/api_server/node_modules/q/q.js:834:54)
    at self.promiseDispatch.done (/home/brandon/foo/api_server/node_modules/q/q.js:863:30)
    at Promise.promise.promiseDispatch (/home/brandon/foo/api_server/node_modules/q/q.js:796:13)
    at /home/brandon/foo/api_server/node_modules/q/q.js:604:44
    at runSingle (/home/brandon/foo/api_server/node_modules/q/q.js:137:13)
    at flush (/home/brandon/foo/api_server/node_modules/q/q.js:125:13)
    at doNTCallback0 (node.js:428:9)
    at process._tickDomainCallback (node.js:398:13)
---------------------------------------------
    at requestTick (/home/brandon/foo/api_server/node_modules/q/q.js:197:21)
    at Function.nextTick (/home/brandon/foo/api_server/node_modules/q/q.js:180:13)
    at /home/brandon/foo/api_server/node_modules/q/q.js:603:15
    at Array.reduce (native)
    at /home/brandon/foo/api_server/node_modules/q/q.js:263:21
    at become (/home/brandon/foo/api_server/node_modules/q/q.js:602:9)
    at deferred.resolve (/home/brandon/foo/api_server/node_modules/q/q.js:618:9)
    at _fulfilled (/home/brandon/foo/lib/node_modules/q/q.js:834:54)
    at self.promiseDispatch.done (/home/brandon/foo/lib/node_modules/q/q.js:863:30)
    at Promise.promise.promiseDispatch (/home/brandon/foo/lib/node_modules/q/q.js:796:13)
    at /home/brandon/foo/lib/node_modules/q/q.js:604:44
    at runSingle (/home/brandon/foo/lib/node_modules/q/q.js:137:13)
    at flush (/home/brandon/foo/lib/node_modules/q/q.js:125:13)
    at doNTCallback0 (node.js:428:9)
    at process._tickDomainCallback (node.js:398:13)

I had

var headers = {
    ...
    'Content-Type': 'text/xml',
    'Content-Length': Buffer(body).byteLength
};

But I think this was the culpritL

var headers = {
    ...
    'Content-Type': 'text/xml',
    'Content-Length': body.length
};

@BennyH26
Copy link

I'm also receiving this error intermittently on random servers and URLs (though all are being accessed via HTTPS). @brandonros - Based on other threads, I'm investigating the possibility of this being caused by drop of RC4 support in Node >= v4.0. Any update on your end?

@brandonros
Copy link
Author

@BennyH26 Say this is the case. What can be done to fix this?

Edit: Maybe the reason this is happening intermittently is because we are getting load balanced to different servers?

I'm going to test if these options will fix the problem. I'm not sure if they are correct.

var options = {
    ...
    ciphers: 'ALL',
    secureProtocol: 'TLSv1_method'
};

@jasnell
Copy link
Member

jasnell commented Nov 21, 2015

/cc @nodejs/http

@BennyH26
Copy link

@brandonros I've added the code to my agent options, and will keep you posted if I still receive errors. Intermittent errors are so challenging to solve sometimes... As you suggested, I suspect we are hitting load balanced servers, and their reverse proxy configurations are SLIGHTLY different, thereby negotiating different ciphers. Just a guess though.

@brandonros
Copy link
Author

@BennyH26 What servers/URLs are you hitting? I ran eBay's API endpoint through https://www.ssllabs.com/ssltest/, if that helps.

@BennyH26
Copy link

@brandonros https://api.ebay.com/ws/api.dll Looks like two of the three servers got C grades =]

@indutny
Copy link
Member

indutny commented Nov 21, 2015

@trotyl are you sure about OS X? This seems pretty strange to me, because I don't see a place in kernel that could set errno = EPROTO for TCP syscalls.

@indutny
Copy link
Member

indutny commented Nov 21, 2015

Gosh, nevermind. It is returned by src/tls_wrap.cc, not kernel. My bad!

@indutny
Copy link
Member

indutny commented Nov 21, 2015

This is very likely happened, because there were pending writes on TLS socket and it was destroyed before they were encrypted and sent to the other side.

@indutny
Copy link
Member

indutny commented Nov 21, 2015

I think this kind of error can only happen if ssl_ == nullptr in TLSWrap::DoWrite.

@indutny
Copy link
Member

indutny commented Nov 21, 2015

@brandonros may I ask you to give a try to following patch?

diff --git a/lib/_tls_wrap.js b/lib/_tls_wrap.js
index ef5769c..d517d1b 100644
--- a/lib/_tls_wrap.js
+++ b/lib/_tls_wrap.js
@@ -47,7 +47,11 @@ function onhandshakestart() {
 function onhandshakedone() {
   // for future use
   debug('onhandshakedone');
-  this._finishInit();
+
+  const self = this;
+  setImmediate(function() {
+    self._finishInit();
+  });
 }


@@ -556,6 +560,10 @@ TLSSocket.prototype._releaseControl = function() {
 };

 TLSSocket.prototype._finishInit = function() {
+  // Destroyed before getting there
+  if (this._handle == null)
+    return;
+
   // `newSession` callback wasn't called yet
   if (this._newSessionPending) {
     this._securePending = true;

@indutny
Copy link
Member

indutny commented Nov 21, 2015

Basically, what I think happens here is that ssl_ is destroyed in the middle of DoWrite. Otherwise I can't see how it could happen. I'm trying to make sure that no unsafe JS callbacks will be executed from within DoWrite.

@indutny
Copy link
Member

indutny commented Nov 21, 2015

(It doesn't seem like it could be any other UV_EPROTO case from tls_wrap.cc, because the error would include some OpenSSL strings as well in that case)

@brandonros
Copy link
Author

@indutny Do you think the options I pass as per #3692 (comment) will interfere or solve the issue?

I will try the patch but the error very rarely happens. I guess I could write a test to try to exacerbate the behavior.

@j-langlois
Copy link

j-langlois commented Sep 14, 2016

eBay will add TLS1.2 support to our Production API endpoints on 9/15
https://api.ebay.com
https://svcs.ebay.com

https://ebaydts.com/eBayKBDetails?KBid=5073

@kevinburke
Copy link

@j-langlois IIRC the problem was not the lack of TLS 1.2 support, it was that there were maybe 10 machines in the eBay load balancer, several of them supported TLS 1.2 and several of them did not. The node client couldn't correctly handle the protocol downgrade

@noeltimothy
Copy link

Using the following worked for me.
agentOptions: {
ciphers: 'ALL',
secureProtocol: 'TLSv1_1_method',
},

Using just TLSv1_method, I cannot load https://www.jewishvoice.org

@LunaSquee
Copy link

LunaSquee commented Nov 27, 2017

I can't request my site via HTTPS either.

> { Error: write EPROTO 139966196083584:error:14077410:SSL routines:SSL23_GET_SERVER_HELLO:sslv3 alert handshake failure:s23_clnt.c:772:

    at _errnoException (util.js:1031:13)
    at WriteWrap.afterWrite [as oncomplete] (net.js:873:14) errno: 'EPROTO', code: 'EPROTO', syscall: 'write' }

My nginx ssl configurations are

        ssl_protocols TLSv1 TLSv1.1 TLSv1.2;

        ssl_certificate /***/fullchain.pem;
        ssl_certificate_key /***/privkey.pem;

        ssl_ciphers kEECDH+ECDSA+AES256+AESGCM:kEECDH+AES256+AESGCM:kEDH+AES256+AESGCM:kEECDH+ECDSA+AESGCM:kEECDH+AESGCM:kEDH+AESGCM:!DES-CBC3-SHA:!SHA1:!aNULL:!eNULL:!MEDIUM:!LOW:!kECDH:!DSS:!MD5:!EXP:!PSK:!SRP:!CAMELLIA:!SEED;

        ssl_prefer_server_ciphers on;
        ssl_session_cache shared:SSL:10m;

        ssl_ecdh_curve secp384r1;

Node.js v9.2.0
OpenSSL 1.1.0g 2 Nov 2017
nginx/1.12.2

It has worked before, I don't know when this error started occuring but I've noticed just now

EDIT: I found better help from #16196, setting tls.DEFAULT_ECDH_CURVE to "auto" fixes my issue

tobysimone pushed a commit to tobysimone/nodejs-ebay-api that referenced this issue Dec 7, 2017
As of 3/22/2016, the eBay API has several servers that can only
negotiate TLS v1.0 sessions, and several servers that can negotiate TLS
v1.0, v1.1 and v1.2. Node/OpenSSL get confused by this, and occasionally
attempt to parse a v1.2 response using TLS v1.0 and vice versa. The
error you get back from the request looks something like this:

```
{ [Error: write EPROTO 140113357338496:error:1408F10B:SSL
routines:SSL3_GET_RECORD:wrong version number:../deps/openssl/openssl/ssl/s3_pkt.c:362:
] code: 'EPROTO',
errno: 'EPROTO',
syscall: 'write' }
```

As far as I can tell, this isn't patched yet, in Node or OpenSSL. But
setting the following options forces all connections to be negotiated
with TLS v1.0, effectively fixing the issue.

More reading:

aws/aws-sdk-js#862
nodejs/node#3692
https://www.ssllabs.com/ssltest/analyze.html?d=api.ebay.com

If you know anyone at eBay, please tell them it's a) unacceptable to
have servers that can only negotiate TLS v1.0, and b) unacceptable to
have a SSL certificate that was signed with SHA1, and they should
upgrade both things.
@bnoordhuis
Copy link
Member

I'll close this out, doesn't look like there's anything left to do. The upstream openssl changes were merged last year.

@apachevirus
Copy link

create-react-app failed, reason: write EPROTO 14100:error:14094438:SSL
SSL routines:ssl3_read_bytes:tlsv1 alert internal

any answer please

@WarKitteh
Copy link

Im getting a very similar error using a truffle for Ethereum development. happens in my virtual machine, host machine and MacOS. Not getting a lot of answers from the Truffle Devs. They just shrugged.

✔ Preparing to download box
⠏ DownloadingError: write EPROTO 4569120192:error:1408F10B:SSL routines:ssl3_get_record:wrong version number:../deps/openssl/openssl/ssl/record/ssl3_record.c:252:

    at WriteWrap.afterWrite (net.js:779:14)

@termitek12
Copy link

termitek12 commented Dec 26, 2020

LOL got this error when using simplest nodejs module - request (https://www.npmjs.com/package/request).

It happened when I tried to access this page:
https://pclab.pl/

Error: write EPROTO 140329133451072:error:1414D172:SSL routines:tls12_check_peer_sigalg:wrong signature type:../ssl/t1_lib.c:1145:

I have no idea WTF is going one... and how to fix this.

@ormanlis
Copy link

getting this in v16.14.0

@naatebarber
Copy link

naatebarber commented Nov 30, 2022

Got this error while using the https.request() method to proxy incoming connections on an express server. Problem for me was the host header, stripping it from the proxied request fixed this bug for me.

Might not be the solution for everyone but hopefully this helps.

node v18.11.0

@wuriyanto48
Copy link

hey @naatebarber would appreciate if you could provide an example or snippet,
thank you

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
http Issues or PRs related to the http subsystem.
Projects
None yet
Development

No branches or pull requests