gc refcount related crash when loading web UI under Python 2.6.6

[daubman]

Running under Python 2.6.6 (Centos 6.3) I get the following crash as soon as I try and load the web UI. This seems to work fine using Python 2.7.3 as an altinstall on the same host. I have seen this on two different Centos 6.3 hosts...

--snip--
$ /usr/bin/locust -f some_locust_script.py
/usr/lib/python2.6/site-packages/locust/rpc/init.py:7: UserWarning: WARNING: Using pure Python socket RPC implementation instead of zmq. This will not affect you if your not running locust in distributed mode, but if you are, we recommend you to install the python packages: pyzmq and gevent-zeromq
warnings.warn("WARNING: Using pure Python socket RPC implementation instead of zmq. This will not affect you if your not running locust in distributed mode, but if you are, we recommend you to install the python packages: pyzmq and gevent-zeromq")
[2012-12-18 15:14:52,705] INFO/locust.main: Starting web monitor on port 8089
[2012-12-18 15:14:52,705] INFO/locust.main: Starting Locust 0.6.1
Modules/gcmodule.c:348: visit_decref: Assertion "gc->gc.gc_refs != 0" failed.
refcount was too small
object : <weakref at 0x15b1db8; to 'gevent.core.http_request' at 0x15b1d60>
type : weakref
refcount: 1
address : 0x15b1db8
Aborted
--snip--

[cgbystrom]

That sounds really strange. We've run Locust fine for countless hours under different 2.6.x versions.

1. Are you using a vanilla CentOS 6.3 version?
2. Did you use the default Python interpreter that CentOS was shipped with or did you build it yourself?

Since I haven't experienced this myself I did quick google and for some this issue is caused by too aggressive compilation options when building the Python interpreter.

[Jahaja]

I was able to reproduce the problem using a minimal centos 6.3 installation on a VM. As well as using the python version that's shipped with it.

It seems to be related to gevent. Updating gevent to 1.0rc2 seems to solve the problem. So that should be a work-around for now.

[cgbystrom]

I was able reproduce it as well. @Jahaja's fix works!

Will close this issue as there is a short term workaround and it will be solved when gevent hits 1.0 (Locust will switch).
If you still have issues, please comment.

[daubman]

Would you please provide some more details on the fix, I have tried this and it does not appear to help - here's what I've done:

$ uname -a
Linux ip-10-46-247-69 2.6.32-71.29.1.el6.x86_64 #1 SMP Mon Jun 27 19:49:27 BST 2011 x86_64 x86_64 x86_64 GNU/Linux
$ cat /etc/redhat-release
CentOS release 6.3 (Final)

$ which python
/usr/bin/python
$ python --version
Python 2.6.6

$ yum install c-ares-devel
$ yum install libev-devel
$ yum install zeromq-devel
$ pip install https://github.com/downloads/SiteSupport/gevent/gevent-1.0rc2.tar.gz
$ pip search gevent
gevent - Python network library that uses greenlet and libevent for easy and scalable concurrency
INSTALLED: 1.0rc2
LATEST: 0.13.8
...
$ pip install https://github.com/locustio/locust/archive/master.zip
$ pip install gevent-zeromq
$ pip install pyzmq
$ locust -f my_locust_file.py --master
[2013-01-08 16:07:53,945] INFO/locust.main: Starting web monitor on port 8089
[2013-01-08 16:07:53,947] ERROR/stderr: Traceback (most recent call last):
[2013-01-08 16:07:53,947] ERROR/stderr: File "/usr/bin/locust", line 9, in
[2013-01-08 16:07:53,948] ERROR/stderr:
[2013-01-08 16:07:53,948] ERROR/stderr: load_entry_point('locustio==0.6.1', 'console_scripts', 'locust')()
[2013-01-08 16:07:53,948] ERROR/stderr: File "/usr/lib/python2.6/site-packages/locust/main.py", line 363, in main
[2013-01-08 16:07:53,948] ERROR/stderr:
[2013-01-08 16:07:53,948] ERROR/stderr: runners.locust_runner = MasterLocustRunner(locust_classes, options.hatch_rate, options.num_clients, num_requests=options.num_requests, host=options.host, master_host=options.master_host)
[2013-01-08 16:07:53,948] ERROR/stderr: File "/usr/lib/python2.6/site-packages/locust/runners.py", line 241, in init
[2013-01-08 16:07:53,948] ERROR/stderr:
[2013-01-08 16:07:53,949] ERROR/stderr: self.greenlet.spawn(self.client_listener).link_exception()
[2013-01-08 16:07:53,949] ERROR/stderr: File "/usr/lib64/python2.6/site-packages/gevent/greenlet.py", line 370, in link_exception
[2013-01-08 16:07:53,949] ERROR/stderr:
[2013-01-08 16:07:53,949] ERROR/stderr: self.link(receiver=receiver, SpawnedLink=SpawnedLink)
[2013-01-08 16:07:53,949] ERROR/stderr: File "/usr/lib64/python2.6/site-packages/gevent/greenlet.py", line 355, in link
[2013-01-08 16:07:53,949] ERROR/stderr:
[2013-01-08 16:07:53,949] ERROR/stderr: self.rawlink(SpawnedLink(receiver))
[2013-01-08 16:07:53,950] ERROR/stderr: File "/usr/lib64/python2.6/site-packages/gevent/greenlet.py", line 23, in init
[2013-01-08 16:07:53,950] ERROR/stderr:
[2013-01-08 16:07:53,950] ERROR/stderr: raise TypeError("Expected callable: %r" % (callback, ))
[2013-01-08 16:07:53,950] ERROR/stderr: TypeError
[2013-01-08 16:07:53,950] ERROR/stderr: :
[2013-01-08 16:07:53,950] ERROR/stderr: Expected callable: None
[2013-01-08 16:07:53,950] ERROR/stderr:

Also, from this post, it appears locust's use of gevent is not compatible with 1.0rc2:
https://groups.google.com/forum/?fromgroups=#!topic/gevent/JHbtDU5HtL8

[heyman]

I've just pushed some changes which I think fixes the problems. Could you try to see if works and then I can push a new version to PyPI.

Thanks for reporting!

[jvenkataramanan]

Yes this change seems to fix all the issues. Thanks for getting to it so fast!

[heyman]

That's great to hear! The fix is now in a new 0.6.2 release on PyPI.