Request Queue Size - determine what's hanging.

[mmahalwy]

I have two servers, both high in traffic and usage. Initially we kept getting the error Request queue full (configured max. size: 100) and crashed every ~12hrs. Now we bumped it up to 200 and at times every 24hrs we have a crash. Below is the passenger-status of both our servers at the time one (the first one) crashed. Note, the latter has a process that was used 4s ago from when passenger-status was run, while the other server (the first) is not getting used.

How do we find out what is causing passenger to hang or it's threads to hang? When doing passenger-status --show=requests it doesn't show enough information to determine what's causing the hang.

Thanks!

First server, down:

root@9c1c94be0901:/home/app/quran# /usr/sbin/passenger-status
Version : 5.0.18
Date    : 2015-09-23 13:36:18 +0000
Instance: IQYEyPZg (nginx/1.8.0 Phusion_Passenger/5.0.18)

----------- General information -----------
Max pool size : 6
App groups    : 1
Processes     : 6
Requests in top-level queue : 0

----------- Application groups -----------
/home/app/quran/public (production):
  App root: /home/app/quran
  Requests in queue: 200
  * PID: 86      Sessions: 1       Processed: 206651   Uptime: 1d 19h 31m 51s
    CPU: 73%     Memory  : 91M     Last used: 1d
  * PID: 93      Sessions: 1       Processed: 34972   Uptime: 1d 19h 31m 51s
    CPU: 70%     Memory  : 95M     Last used: 1d 6
  * PID: 100     Sessions: 1       Processed: 326117   Uptime: 1d 19h 31m 51s
    CPU: 31%     Memory  : 105M    Last used: 4h
  * PID: 20441   Sessions: 1       Processed: 42885   Uptime: 1d 6h 57m 1s
    CPU: 60%     Memory  : 45M     Last used: 18h 42
  * PID: 20595   Sessions: 1       Processed: 50365   Uptime: 1d 6h 51m 38s
    CPU: 49%     Memory  : 85M     Last used: 15h 2
  * PID: 36771   Sessions: 1       Processed: 35793   Uptime: 20h 15m 8s
    CPU: 69%     Memory  : 114M    Last used: 14h 4m 9

Second, working server:

root@760251876d1d:/home/app/quran# /usr/sbin/passenger-status
Version : 5.0.15
Date    : 2015-09-23 13:37:27 +0000
Instance: 6VWPk0L4 (nginx/1.8.0 Phusion_Passenger/5.0.15)

----------- General information -----------
Max pool size : 6
App groups    : 1
Processes     : 6
Requests in top-level queue : 0

----------- Application groups -----------
/home/app/quran/public (production):
  App root: /home/app/quran
  Requests in queue: 0
  * PID: 95      Sessions: 1       Processed: 106886   Uptime: 1d 3h 20m 38s
    CPU: 75%     Memory  : 120M    Last used: 19h
  * PID: 102     Sessions: 1       Processed: 23900   Uptime: 1d 3h 20m 38s
    CPU: 72%     Memory  : 121M    Last used: 19h 4
  * PID: 109     Sessions: 1       Processed: 29189   Uptime: 1d 3h 20m 38s
    CPU: 65%     Memory  : 118M    Last used: 17h 4
  * PID: 11787   Sessions: 0       Processed: 239016   Uptime: 19h 59m 38s
    CPU: 7%      Memory  : 121M    Last used: 1s ago
  * PID: 12013   Sessions: 1       Processed: 5633    Uptime: 19h 51m 19s
    CPU: 76%     Memory  : 113M    Last used: 15h 5m
  * PID: 15265   Sessions: 0       Processed: 32977   Uptime: 17h 44m 11s
    CPU: 1%      Memory  : 116M    Last used: 4s ago

[FooBarWidget]

Try the tips at http://old.blog.phusion.nl/2012/09/21/the-right-way-to-deal-with-frozen-processes-on-unix/

[pakkua80]

@mmahalwy You manage to figure out the steps to nail the culprit ? I am having the same problem here and would love to know how to find out.

[snirbenyosef]

we have got the same problem. request In queue are stuck. and cpu load show 0%.
did someone found a solution ?