hard times in the circus

[ltalirz]

I am getting the impression that the development of circus is slowing down to the point where it starts to become a problem for us:

• still no release with python3.7 support (master branch seems updated, though)
• latest release forces pyzmq<17.0, which conflicts with the whole jupyter notebook stack, even on python2 (all versions of notebook after 5.4.1 (March 2018) require pyzmq>=0.17)
• Out of the top 10 contributors, 9 did not make a single commit last year https://github.com/circus-tent/circus/graphs/contributors

Is there an alternative in sight?

[ltalirz]

mentioning @giovannipizzi @sphuber for comment

[ltalirz]

E.g. just pip install aiida-core[notebook]==1.0.0b2 already has dependency conflicts because of circus
pipenv says: ERROR: ERROR: Could not find a version that matches pyzmq<17.0,>=13.1.0,>=17

[giovannipizzi]

Mmm, that's bad...
Could you try to ping them to see if they can support pyzmq > 17?
We already spent quite a few weeks to transition to circus, and if possible I'd like to avoid investing a few more weeks of time in switching to (yet another) manager of the daemon processes, if this is possible.

[ltalirz]

Could you try to ping them to see if they can support pyzmq > 17?

An issue on this was opened beginning of february but nothing much is happening
circus-tent/circus#1099 (comment)

[giovannipizzi]

I see. It is also true that we are using circus just for a small functionality (namely demonising a process), I think.
We could look, at some point, into other libraries to see if they are easy to replace - just to mention one, https://pypi.org/project/python-daemon/ but I didn't investigate/did a thorough analysis of options yet

[sphuber]

It is annoying, and we already had a problem quite some time ago, when tornado==5.0 was released which also broke circus, and that also has still not been addressed. However, I would be hesitant to invest a lot of time in changing the daemonizer now. Even though it is true that what we use it for seems trivial and not a lot, I spent a lot of time, even after the first implementation to get things to work across the various platforms. I fear we might encounter similar problems with a potential replacement that might seem adequate on the surface.

[ltalirz]

Ok. Of course I also don't want to cause unnecessary work.
So, we stay with circus for the moment and accept the dependency conflict.

Unfortunately, there is not yet a good way of handling such cases in pipenv which we're using in the aiida lab (i.e. to tell it to ignore a certain dependency conflict) - perhaps this suggestion could work, we'll have to try.

[ltalirz]

Closing this now - happy to reopen when there is something we can do

[ltalirz]

Just checked, and there is zero ongoing development in circus, loads of open PRs (but not for the pyzmq/tornado upgrade).
Given that there is ipython 7 now, which requires the newer tornado and pyzqm versions, I don't think we can afford to stay in ipython 5 (forced by circus) for much longer.

I don't understand a whole lot about which features we need from a daemonizer, but it seems python-daemon is a reference implementation of the PEP 3143 “Standard daemon process library” specification, that seems to be maintained and supports python 2.7 and up (see also SO).

@sphuber Does that have what we need?

[ltalirz]

I should add that the only dependencies of python-daemon are docutils and twine.

[sphuber]

On a first look it seems it is just possible to daemonize a single process. There is no single controlling process that can spawn and control multiple like circus now does for us. Even if we were able to launch multiple, it wlll be unlikely then that we get the functionality circus currently gives us, such as increasing/decreasing number of workers and giving us their status in terms of MEM and CPU usage.