supervisorctl start <process> hangs when system clock is set to an earlier time

[andy-ld]

To reproduce:
1)Set your machine's time to the future.
2)start supervisord
3)set your machine's time to the correct time.
4)supervisorctl stop
5)supervisorctl start

issuing 5) will hang until shortly after the time the system thought it was when supervisord was started.

It looks like the process is started immediately, but supervisorctl is blocked on the http response.

this:

diff --git a/supervisor/http.py b/supervisor/http.py
index 947c25e..c365975 100644
--- a/supervisor/http.py
+++ b/supervisor/http.py
@@ -343,7 +343,7 @@ class deferring_http_channel(http_server.http_channel):
             # we called a deferred producer via this channel (see refill_buffer)
             last_writable_check = self.writable_check
             elapsed = now - last_writable_check
-            if elapsed > self.delay:
+            if elapsed > self.delay or elapsed < 0:
                 self.writable_check = now
                 return True
             else:

seems to fix it for me, but I'm not familiar enough with the code to be confident this is the right thing to do.

In older version's of supervisor, when stopProcess() had a non-zero delay time, supervisorctl stop would also hang.

[mnaberez]

4)supervisorctl stop

$ supervisorctl stop
Error: stop requires a process name
stop <name>     Stop a process
stop <gname>:*      Stop all processes in a group
stop <name> <name>  Stop multiple processes or groups
stop all        Stop all processes

[mnaberez]

What's the rest of the supervisorctl stop command you are using?

What version of Supervisor are you using?

[andy-ld]

Sorry, I should have been more clear.

4) supervisorctl stop <process name>
5) supervisorctl start <process name>

where<process name>is some process that supervisord watches. I don't think the type of process matters.

I just did a git clone and tested with the latest code. I also have seen the same issue with 3.2.3 and 3.3.0.

[mnaberez]

Reproduced on Supervisor 3.3.0.

Minimal supervisord.conf:

[supervisord]
logfile = /tmp/supervisord.log

[inet_http_server]
port = 127.0.0.1:9001

[supervisorctl]
serverurl = http://127.0.0.1:9001

[rpcinterface:supervisor]
supervisor.rpcinterface_factory = supervisor.rpcinterface:make_main_rpcinterface

[program:cat]
command = /bin/cat
startsecs = 1
autostart = false

Set date to one day in the future, then start supervisord in the foreground:

$ date
Tue Aug  2 10:54:55 PDT 2016
$ supervisord --version
3.3.0
$ supervisord -n -c /path/to/supervisord.conf
2016-08-02 10:54:57,898 INFO Increased RLIMIT_NOFILE limit to 1024
2016-08-02 10:54:57,928 INFO RPC interface 'supervisor' initialized
2016-08-02 10:54:57,928 CRIT Server 'inet_http_server' running without any HTTP authentication checking
2016-08-02 10:54:57,929 INFO supervisord started with pid 2214

Set date back to normal, then run supervisorctl in another terminal:

$ date
Mon Aug  1 10:55:02 PDT 2016
$ supervisorctl -c /path/to/supervisord.conf
cat                              STOPPED   Not started
supervisor> start cat

supervisorctl will hang on start cat, even though supervisord shows the process as started:

2016-08-01 10:55:05,950 INFO spawned: 'cat' with pid 2229
2016-08-01 10:55:06,955 INFO success: cat entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)

[mnaberez]

Fixed in Supervisor 3.3.1. Thanks for the report!