worker continue die with gevent loop engine #68
Comments
|
does it happens on the first request or as soon as the server starts ? |
|
As I observed, it happens on the first request starts. After I switch to preforking mode, the web app can run normally. (BTW, the OS is 3.2.0-29-generic #46-Ubuntu 12.04 LTS server) Same code will run normally with gevent loop engine under Centos 2.6.32-220.13.1.el6.x86_64. |
|
I also experience similar crashes on first request after nobody touched the server for some time: uwsgi(uwsgi_backtrace+0x25) [0x4452b5] The system is also on Ubuntu 12.04 LTS using uWSGI 1.4.9 and gevent 1.0rc2. |
|
how you are making mysql gevent friendly ? |
|
You are right, I did not do anything specific to make mysql play nicely with gevent. So, I guess, my comment can be disregarded. |
|
Is this issue still valid? Does it work with 1.9? |
|
Seems I'm running in a similar issue with 1.9, not sure it's the same though *** HARAKIRI ON WORKER 4 (pid: 142962, try: 1) ***
*** backtrace of 142962 ***
/srv/www/app.kwarter.com/current/virtualenv/bin/uwsgi(uwsgi_backtrace+0x25) [0x453795]
/srv/www/app.kwarter.com/current/virtualenv/bin/uwsgi(what_i_am_doing+0x30) [0x4538d0]
/lib/x86_64-linux-gnu/libc.so.6(+0x370b0) [0x7fb4a3cd40b0]
/lib/x86_64-linux-gnu/libc.so.6(epoll_wait+0x33) [0x7fb4a3d974d3]
/srv/www/app.kwarter.com/releases/20130506213646/virtualenv/src/gevent/gevent/core.so(+0x22475) [0x7fb499be6475]
/srv/www/app.kwarter.com/releases/20130506213646/virtualenv/src/gevent/gevent/core.so(ev_run+0x284) [0x7fb499bedde4]
/srv/www/app.kwarter.com/releases/20130506213646/virtualenv/src/gevent/gevent/core.so(+0x2a909) [0x7fb499bee909]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x5e75) [0x7fb4a43098d5]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x81a) [0x7fb4a439d2fa]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(+0xff4a6) [0x7fb4a439d4a6]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyObject_Call+0x4e) [0x7fb4a43cfd8e]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(+0xa106d) [0x7fb4a433f06d]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyObject_Call+0x4e) [0x7fb4a43cfd8e]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyEval_CallObjectWithKeywords+0x47) [0x7fb4a43d0b37]
/srv/www/app.kwarter.com/releases/20130506213646/virtualenv/local/lib/python2.7/site-packages/greenlet.so(+0x3913) [0x7fb4a1dba913]
/srv/www/app.kwarter.com/releases/20130506213646/virtualenv/local/lib/python2.7/site-packages/greenlet.so(+0x3340) [0x7fb4a1dba340]
/srv/www/app.kwarter.com/releases/20130506213646/virtualenv/local/lib/python2.7/site-packages/greenlet.so(+0x3d96) [0x7fb4a1dbad96]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyObject_Call+0x4e) [0x7fb4a43cfd8e]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyEval_CallObjectWithKeywords+0x47) [0x7fb4a43d0b37]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(+0xa0499) [0x7fb4a433e499]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyObject_Call+0x4e) [0x7fb4a43cfd8e]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x1bfe) [0x7fb4a430565e]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x81a) [0x7fb4a439d2fa]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x4a00) [0x7fb4a4308460]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x81a) [0x7fb4a439d2fa]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x4a00) [0x7fb4a4308460]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x81a) [0x7fb4a439d2fa]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(+0xff4a6) [0x7fb4a439d4a6]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyObject_Call+0x4e) [0x7fb4a43cfd8e]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(+0xa106d) [0x7fb4a433f06d]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyObject_Call+0x4e) [0x7fb4a43cfd8e]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyObject_CallMethod+0xea) [0x7fb4a42eaa3a]
/srv/www/app.kwarter.com/current/virtualenv/bin/uwsgi() [0x46fe4a]
/srv/www/app.kwarter.com/current/virtualenv/bin/uwsgi(uwsgi_ignition+0xba) [0x453c0a]
/srv/www/app.kwarter.com/current/virtualenv/bin/uwsgi(uwsgi_start+0x1488) [0x4558e8]
/srv/www/app.kwarter.com/current/virtualenv/bin/uwsgi() [0x41c306]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5) [0x7fb4a3cbeea5]
/srv/www/app.kwarter.com/current/virtualenv/bin/uwsgi() [0x41c331]
*** end of backtrace ***[uwsgi]
socket = /tmp/app.socket
chown-socket = www-data
master = true
processes = 8
harakiri = 30
gevent = 5
gevent-monkey-patch = true
#threaded-logger = true => does not make any difference
stats = 127.0.0.1:9191
max-requests = 10000
wsgi-file = wsgi.py
lazy = true
#enable-threads = true => does not make any difference |
|
no, it is a simple harakiri triggered because your app is blocked. You can get more infos about why your app was blocked adding the tracebacker (by default you only get the C traceback) |
uWSGI 1.4.2
gevent 1.0rc1
INI file:
[uwsgi]
gid = pgy
uid = pgy
socket = 127.0.0.1:3131
master = true
processes = 1
loop = gevent
async = 50
enable-threads = true
harakiri = 60
socket-timeout = 30
disable-logging = true
daemonize = /home/pgy/log/uwsgi/%n.log
virtualenv = /home/pgy/venv
module = runserver
callable = app
uwsgi log:
*** Starting uWSGI 1.4.2 (64bit) on [Thu Nov 29 00:11:56 2012] ***
compiled with version: 4.6.3 on 28 November 2012 10:45:02
os: Linux-3.2.0-29-generic #46-Ubuntu SMP Fri Jul 27 17:03:23 UTC 2012
nodename: pgy-web1
machine: x86_64
clock source: unix
detected number of CPU cores: 8
current working directory: /home/pgy/pgysocial
detected binary path: /usr/local/bin/uwsgi
your processes number limit is 63631
your memory page size is 4096 bytes
*** WARNING: you have enabled harakiri without post buffering. Slow upload could be rejected on post-unbuffered webservers ***
detected max file descriptor number: 10240
lock engine: pthread robust mutexes
uwsgi socket 0 bound to TCP address 127.0.0.1:3131 fd 3
Python version: 2.7.3 (default, Aug 1 2012, 05:25:23) [GCC 4.6.3]
Set PythonHome to /home/pgy/venv
Python main interpreter initialized at 0x241e910
python threads support enabled
your server socket listen backlog is limited to 100 connections
mapped 1117728 bytes (1091 KB) for 50 cores
*** Operational MODE: async ***
WSGI app 0 (mountpoint='') ready in 1 seconds on interpreter 0x241e910 pid: 6932 (default app)
*** uWSGI is running in multiple interpreter mode ***
spawned uWSGI master process (pid: 6932)
spawned uWSGI worker 1 (pid: 6939, cores: 50)
*** running gevent loop engine [addr:0x45d140] ***
!!! uWSGI process 6939 got Segmentation Fault !!!
*** backtrace of 6939 ***
uwsgi(uwsgi_backtrace+0x25) [0x442bb5]
uwsgi(uwsgi_segfault+0x21) [0x442c91]
/lib/x86_64-linux-gnu/libc.so.6(+0x364a0) [0x7fe4251e94a0]
/usr/lib/libpython2.7.so.1.0(PyObject_Call+0x3a) [0x7fe4256b2e6a]
/usr/lib/libpython2.7.so.1.0(PyEval_CallObjectWithKeywords+0x47) [0x7fe4256b37d7]
uwsgi(python_call+0x24) [0x44e6f4]
uwsgi(uwsgi_gevent_current_wsgi_req+0x1d) [0x45e89d]
uwsgi(warn_pipe+0xa) [0x43fdea]
/lib/x86_64-linux-gnu/libc.so.6(+0x364a0) [0x7fe4251e94a0]
/lib/x86_64-linux-gnu/libpthread.so.0(send+0x22) [0x7fe4267cc282]
/usr/lib/libpython2.7.so.1.0(+0x81ea7) [0x7fe4255f3ea7]
/usr/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x4f18) [0x7fe4256033b8]
/usr/lib/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x855) [0x7fe4255ce605]
/usr/lib/libpython2.7.so.1.0(+0x5c7bd) [0x7fe4255ce7bd]
/usr/lib/libpython2.7.so.1.0(PyObject_Call+0x53) [0x7fe4256b2e83]
/usr/lib/libpython2.7.so.1.0(+0xd5047) [0x7fe425647047]
/usr/lib/libpython2.7.so.1.0(PyObject_Call+0x53) [0x7fe4256b2e83]
/usr/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x3fcd) [0x7fe42560246d]
/usr/lib/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x855) [0x7fe4255ce605]
/usr/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x5420) [0x7fe4256038c0]
/usr/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x614b) [0x7fe4256045eb]
/usr/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x614b) [0x7fe4256045eb]
/usr/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x614b) [0x7fe4256045eb]
/usr/lib/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x855) [0x7fe4255ce605]
/usr/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x5420) [0x7fe4256038c0]
/usr/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x614b) [0x7fe4256045eb]
/usr/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x614b) [0x7fe4256045eb]
/usr/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x614b) [0x7fe4256045eb]
/usr/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x614b) [0x7fe4256045eb]
/usr/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x614b) [0x7fe4256045eb]
/usr/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x614b) [0x7fe4256045eb]
/usr/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x614b) [0x7fe4256045eb]
/usr/lib/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x855) [0x7fe4255ce605]
/usr/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x5420) [0x7fe4256038c0]
/usr/lib/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x855) [0x7fe4255ce605]
/usr/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x5420) [0x7fe4256038c0]
/usr/lib/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x855) [0x7fe4255ce605]
/usr/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x5420) [0x7fe4256038c0]
/usr/lib/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x855) [0x7fe4255ce605]
/usr/lib/libpython2.7.so.1.0(+0x5c7bd) [0x7fe4255ce7bd]
/usr/lib/libpython2.7.so.1.0(PyObject_Call+0x53) [0x7fe4256b2e83]
/usr/lib/libpython2.7.so.1.0(+0x1251cf) [0x7fe4256971cf]
/usr/lib/libpython2.7.so.1.0(PyObject_Call+0x53) [0x7fe4256b2e83]
/usr/lib/libpython2.7.so.1.0(PyObject_CallFunctionObjArgs+0xb0) [0x7fe4256f8a40]
/usr/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x17fe) [0x7fe4255ffc9e]
/usr/lib/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x855) [0x7fe4255ce605]
/usr/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x5420) [0x7fe4256038c0]
/usr/lib/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x855) [0x7fe4255ce605]
/usr/lib/libpython2.7.so.1.0(+0x5c7bd) [0x7fe4255ce7bd]
/usr/lib/libpython2.7.so.1.0(PyObject_Call+0x53) [0x7fe4256b2e83]
/usr/lib/libpython2.7.so.1.0(+0x1251cf) [0x7fe4256971cf]
/usr/lib/libpython2.7.so.1.0(PyObject_Call+0x53) [0x7fe4256b2e83]
/usr/lib/libpython2.7.so.1.0(+0x13136c) [0x7fe4256a336c]
/usr/lib/libpython2.7.so.1.0(PyObject_Call+0x53) [0x7fe4256b2e83]
/usr/lib/libpython2.7.so.1.0(PyEval_CallObjectWithKeywords+0x47) [0x7fe4256b37d7]
uwsgi(python_call+0x24) [0x44e6f4]
uwsgi(uwsgi_request_wsgi+0xe1) [0x450a81]
uwsgi(py_uwsgi_gevent_request+0xc2) [0x45eb12]
/usr/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x4f18) [0x7fe4256033b8]
/usr/lib/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x855) [0x7fe4255ce605]
/usr/lib/libpython2.7.so.1.0(+0x5c7bd) [0x7fe4255ce7bd]
/usr/lib/libpython2.7.so.1.0(PyObject_Call+0x53) [0x7fe4256b2e83]
/usr/lib/libpython2.7.so.1.0(+0x1251cf) [0x7fe4256971cf]
/usr/lib/libpython2.7.so.1.0(PyObject_Call+0x53) [0x7fe4256b2e83]
*** end of backtrace ***
DAMN ! worker 1 (pid: 6939) died :( trying respawn ...
Respawned uWSGI worker 1 (new pid: 7008)
The text was updated successfully, but these errors were encountered: