Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Consider disabling server.requestTimeout and server.headersTimeout by default #44228

Closed
flunderpero opened this issue Aug 13, 2022 · 8 comments
Closed
Labels
feature request Issues that request new features to be added to Node.js. http Issues or PRs related to the http subsystem.

Comments

@flunderpero
Copy link

What is the problem this feature will solve?

Node 18 introduced these two settings in order to prevent (D)DoS attacks. That's great!

But we had a very hard few days figuring out that these timeouts result in some 408 reported in our Google Cloud Run services.

I suspect that something triggers these timeouts in the communication between Cloud Run (a managed KNative service) and our Express app. Cloud Run reported these 408 errors with a request duration of a few ms and so the timeout hasn't been reached. And we never saw timeouts with Node 16. (We are currently running everything in our CI, so we have full control over the clients and servers and monitor every failed request.)

Either there is something wrong in the implementation of these timeouts or they in general do not work particularly well together with a reverse proxy.

What is the feature you are proposing to solve the problem?

I don't know, how many Node HTTP servers are exposed to the web without a reverse proxy.

I would want you to consider disabling these timeouts by default and to add some kind of logging or notification when these timeouts happen. We were completely in the dark for a couple of days.

What alternatives have you considered?

No response

@flunderpero flunderpero added the feature request Issues that request new features to be added to Node.js. label Aug 13, 2022
@MoLow MoLow added the http Issues or PRs related to the http subsystem. label Aug 14, 2022
@nicolasburtey
Copy link

I think I'm running into the same issue

@ShogunPanda
Copy link
Contributor

Disable these timeouts again would leave Node unprotected against Slowloris and eventually similar attacks.
If you says 408 is triggered before the actual timeout, then something is very wrong here.
Can you please share a snippet (so no private data or business logic) of the Node server here?
Have you tried running with NODE_DEBUG=http?

@wbulot
Copy link

wbulot commented Nov 23, 2022

Just to add that those new timeouts also cost us a week of hassle.
We operate a file storage platform. Under node 16 no problem, as soon as we switched to node18, large file uploads always ended up in timeout 408 for the client.
This post has put us on the right track.

@ShogunPanda
Copy link
Contributor

I'm sorry to hear about the time wasted.
In order to improve our releasing process, did you check release notes for Node 18 (https://nodejs.org/en/blog/release/v18.0.0/) or not? If yes, were timeout changes not visible enough?

@flunderpero
Copy link
Author

@ShogunPanda This is where I eventually got the clue from. It wouldn't have been a problem at all if the timeouts presented themselves in any way, but all we could find where strange 408 errors. And we did not even exceed those timeouts but they still affected us.

@ShogunPanda
Copy link
Contributor

The last part bothers me a lot. Can you steadily reproduce the problem so I can verify it?

@flunderpero
Copy link
Author

As I wrote in the initial blog post, the errors where very few in between. We encountered them maybe a few times a day, some days went by without an error. I don't know how to reliably reproduce it. All I know is that after disabling the new timeouts we never saw a 408 again.

@ShogunPanda
Copy link
Contributor

I see. I think there is nothing we can do about it. If you find a way to reproduce them, please post it here, but for now I'm closing this issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature request Issues that request new features to be added to Node.js. http Issues or PRs related to the http subsystem.
Projects
None yet
Development

No branches or pull requests

5 participants