IIS-Node hangs if a request is made from the server to itself under load

[PatrickKing]

I've encountered an issue where IIS seems to hang, under certain workloads in IIS-Node.

Specifically, I have a handler on an Express app that results in a second call to the same server being made during the first. If I make between one and three concurrent requests of this endpoint when the app is running with IIS-Node, everything is fine. If I make four or more requests simultaneously, the server hangs for two minutes. Afterward, only one of the four requests will complete successfully, the other three receive HTTP 500.1013 errors from IIS-Node.

When run standalone, my Node app doesn't experience the hang under any amount of load, on Windows or on OSX. The app also performs normally if I make three or less simultaneous requests, under IIS.

In the log files created by my Node app in the failure case, there's a two minute period after the app begins to handle each of the four requests where no logs are written and the app seems to do no processing at all. Afterward, the logs show my app completing all of the four (or more) requests with no errors.

Similarly, I've recorded ETW traces of failed (4 simultaneous request) and successful (3 simultaneous request) cases, and the failed case shows no new ETW events emitted by IIS-Node for two minutes. Search for the times 17:02:13 and 17:04:15 to see the gap in the failure trace.

I think that #405 might be related. Looking over the ETW trace reported there, there is also a gap from 12:28:07 to 12:30:07.

My best guess at what's going on: making a request from the IIS-Node server to itself leads to a race condition, and some internal IIS request management system is timing out and killing the request. I have a workaround for my case, which involves writing a file instead of making the request.

Hope this helps! Working with IIS-Node has been an excellent experience otherwise, thank you for your work.

[cmahrtian]

@PatrickKing Am curious if, since you posted this in January, you managed to confirm your initial hunch re: race condition resulting in an IIS timeout and a killed request. Dealing with a similar issue on my end; the app I'm working on worked just fine when we had a limited number of users concurrently testing the app while in development only, but upon exposing it to a much larger user base in production, it started throwing 500.1013 errors on login.

[PatrickKing]

I never did identify the precise problem. But I did fix my own issue: during one request handled by IIS Node, I was making another request to the same handler, from the Node app to itself. Reading over the code for IIS-Node's handler shows that it uses a mutex. I won't claim to fully understand how the handler code works, but my best guess is that the second request deadlocks, and a two minute IIS request timeout is what breaks the deadlock.

I removed that second request and switched to another approach for internal communication in my app, and this addressed my problem.

So, avoid making requests from the server to itself. If that doesn't apply to you, you might consider load testing the app to find out precisely where the issue occurs, and you could rate limit your server.

[jzwack]

@PatrickKing do you mean communication between functions in your node app (across files), or just HTTP requests back to your own app?