Command line stuck after CTRL + C

[kinow]

Describe the bug

I stopped a workflow today as I normally do, with CTRL + C, while testing Cylc UI. And noticed it got stuck after several minutes.

Not sure if a) an issue with my local working copy, b) an issue with cylc-uiserver, or c) something else?

But thought worth reporting in case others see something similar.

Release version(s) and/or repository branch(es) affected?

Everything on master, Cylc UI Server, Cylc Flow.

Steps to reproduce the bug

I was running a workflow (families2) that I normally use for testing with families. Just stopped in a random moment with CTRL + C, and got the traceback.

Expected behavior

Exit, leaving some KeyboardInterrupt message. But then the command line would be free and I would be able to run the workflow again.

Screenshots

Additional context

Command line stacktrace:

2020-01-30T15:29:31+13:00 INFO - [bar.20590130T1525+13] status=submitted: (received)started at
	2020-01-30T15:29:31+13:00 for job(01)
2020-01-30T15:29:31+13:00 INFO - [bar.20590130T1525+13] -health check settings: execution timeout=None
^C2020-01-30T15:29:31+13:00 ERROR -
	Traceback (most recent call last):
	  File "/home/kinow/Development/python/workspace/cylc-uiserver/venv/lib/python3.7/site-
	packages/cylc/flow/scheduler.py", line 299, in start
	    self.run()
	  File "/home/kinow/Development/python/workspace/cylc-uiserver/venv/lib/python3.7/site-
	packages/cylc/flow/scheduler.py", line 1601, in run
	    has_updated = self.update_data_structure()
	  File "/home/kinow/Development/python/workspace/cylc-uiserver/venv/lib/python3.7/site-
	packages/cylc/flow/scheduler.py", line 1652, in update_data_structure
	    self.ws_data_mgr.get_publish_deltas())
	  File "/home/kinow/Development/python/workspace/cylc-uiserver/venv/lib/python3.7/site-
	packages/cylc/flow/network/publisher.py", line 99, in publish
	    self.loop.run_until_complete(gather_coros(self.send_multi, items))
	  File "/home/kinow/anaconda3/lib/python3.7/asyncio/base_events.py", line 571, in run_until_complete
	    self.run_forever()
	  File "/home/kinow/anaconda3/lib/python3.7/asyncio/base_events.py", line 539, in run_forever
	    self._run_once()
	  File "/home/kinow/anaconda3/lib/python3.7/asyncio/base_events.py", line 1775, in _run_once
	    handle._run()
	  File "/home/kinow/anaconda3/lib/python3.7/asyncio/events.py", line 88, in _run
	    self._context.run(self._callback, *self._args)
	  File "/home/kinow/Development/python/workspace/cylc-uiserver/venv/lib/python3.7/site-
	packages/cylc/flow/network/publisher.py", line 88, in send_multi
	    [topic, serialize_data(data, serializer)]
	  File "/home/kinow/Development/python/workspace/cylc-uiserver/venv/lib/python3.7/site-
	packages/zmq/sugar/socket.py", line 442, in send_multipart
	    return self.send(msg_parts[-1], flags, copy=copy, track=track)
	  File "/home/kinow/Development/python/workspace/cylc-uiserver/venv/lib/python3.7/site-
	packages/zmq/sugar/socket.py", line 395, in send
	    return super(Socket, self).send(data, flags=flags, copy=copy, track=track)
	  File "zmq/backend/cython/socket.pyx", line 733, in zmq.backend.cython.socket.Socket.send
	  File "zmq/backend/cython/socket.pyx", line 780, in zmq.backend.cython.socket.Socket.send
	  File "zmq/backend/cython/socket.pyx", line 242, in zmq.backend.cython.socket._send_copy
	  File "zmq/backend/cython/checkrc.pxd", line 12, in zmq.backend.cython.checkrc._check_rc
	KeyboardInterrupt
2020-01-30T15:29:31+13:00 CRITICAL - Suite shutting down - 
2020-01-30T15:29:31+13:00 WARNING - qux.20580130T1525+13: orphaned task (running)
2020-01-30T15:29:31+13:00 WARNING - foo.20590130T1525+13: orphaned task (running)
2020-01-30T15:29:31+13:00 WARNING - bar.20590130T1525+13: orphaned task (running)
Traceback (most recent call last):
  File "/home/kinow/Development/python/workspace/cylc-uiserver/venv/bin/cylc-run", line 8, in <module>
    sys.exit(main())
  File "/home/kinow/Development/python/workspace/cylc-uiserver/venv/lib/python3.7/site-packages/cylc/flow/scripts/cylc_run.py", line 22, in main
    scheduler_main(is_restart=False)
  File "/home/kinow/Development/python/workspace/cylc-uiserver/venv/lib/python3.7/site-packages/cylc/flow/scheduler_cli.py", line 290, in main
    return run()  # pylint: disable=E1120
  File "/home/kinow/Development/python/workspace/cylc-uiserver/venv/lib/python3.7/site-packages/cylc/flow/terminal.py", line 153, in wrapper
    wrapped_function(*wrapped_args, **wrapped_kwargs)
  File "/home/kinow/Development/python/workspace/cylc-uiserver/venv/lib/python3.7/site-packages/cylc/flow/scheduler_cli.py", line 314, in run
    return scheduler_cli(parser, options, args, is_restart=False)
  File "/home/kinow/Development/python/workspace/cylc-uiserver/venv/lib/python3.7/site-packages/cylc/flow/scheduler_cli.py", line 280, in scheduler_cli
    scheduler.start()
  File "/home/kinow/Development/python/workspace/cylc-uiserver/venv/lib/python3.7/site-packages/cylc/flow/scheduler.py", line 314, in start
    self.shutdown(exc)
  File "/home/kinow/Development/python/workspace/cylc-uiserver/venv/lib/python3.7/site-packages/cylc/flow/scheduler.py", line 1795, in shutdown
    [(b'shutdown', f'{str(reason)}'.encode('utf-8'))])
  File "/home/kinow/Development/python/workspace/cylc-uiserver/venv/lib/python3.7/site-packages/cylc/flow/network/publisher.py", line 99, in publish
    self.loop.run_until_complete(gather_coros(self.send_multi, items))
  File "/home/kinow/anaconda3/lib/python3.7/asyncio/base_events.py", line 571, in run_until_complete
    self.run_forever()
  File "/home/kinow/anaconda3/lib/python3.7/asyncio/base_events.py", line 539, in run_forever
    self._run_once()
  File "/home/kinow/anaconda3/lib/python3.7/asyncio/base_events.py", line 1775, in _run_once
    handle._run()
  File "/home/kinow/anaconda3/lib/python3.7/asyncio/events.py", line 88, in _run
    self._context.run(self._callback, *self._args)
  File "/home/kinow/Development/python/workspace/cylc-uiserver/venv/lib/python3.7/site-packages/cylc/flow/scheduler.py", line 299, in start
    self.run()
  File "/home/kinow/Development/python/workspace/cylc-uiserver/venv/lib/python3.7/site-packages/cylc/flow/scheduler.py", line 1601, in run
    has_updated = self.update_data_structure()
  File "/home/kinow/Development/python/workspace/cylc-uiserver/venv/lib/python3.7/site-packages/cylc/flow/scheduler.py", line 1652, in update_data_structure
    self.ws_data_mgr.get_publish_deltas())
  File "/home/kinow/Development/python/workspace/cylc-uiserver/venv/lib/python3.7/site-packages/cylc/flow/network/publisher.py", line 99, in publish
    self.loop.run_until_complete(gather_coros(self.send_multi, items))
  File "/home/kinow/anaconda3/lib/python3.7/asyncio/base_events.py", line 571, in run_until_complete
    self.run_forever()
  File "/home/kinow/anaconda3/lib/python3.7/asyncio/base_events.py", line 539, in run_forever
    self._run_once()
  File "/home/kinow/anaconda3/lib/python3.7/asyncio/base_events.py", line 1775, in _run_once
    handle._run()
  File "/home/kinow/anaconda3/lib/python3.7/asyncio/events.py", line 88, in _run
    self._context.run(self._callback, *self._args)
  File "/home/kinow/Development/python/workspace/cylc-uiserver/venv/lib/python3.7/site-packages/cylc/flow/network/publisher.py", line 88, in send_multi
    [topic, serialize_data(data, serializer)]
  File "/home/kinow/Development/python/workspace/cylc-uiserver/venv/lib/python3.7/site-packages/zmq/sugar/socket.py", line 442, in send_multipart
    return self.send(msg_parts[-1], flags, copy=copy, track=track)
  File "/home/kinow/Development/python/workspace/cylc-uiserver/venv/lib/python3.7/site-packages/zmq/sugar/socket.py", line 395, in send
    return super(Socket, self).send(data, flags=flags, copy=copy, track=track)
  File "zmq/backend/cython/socket.pyx", line 733, in zmq.backend.cython.socket.Socket.send
  File "zmq/backend/cython/socket.pyx", line 780, in zmq.backend.cython.socket.Socket.send
  File "zmq/backend/cython/socket.pyx", line 242, in zmq.backend.cython.socket._send_copy
  File "zmq/backend/cython/checkrc.pxd", line 12, in zmq.backend.cython.checkrc._check_rc
KeyboardInterrupt
^CException ignored in: <module 'threading' from '/home/kinow/anaconda3/lib/python3.7/threading.py'>
Traceback (most recent call last):
  File "/home/kinow/anaconda3/lib/python3.7/threading.py", line 1281, in _shutdown
    t.join()
  File "/home/kinow/anaconda3/lib/python3.7/threading.py", line 1032, in join
    self._wait_for_tstate_lock()
  File "/home/kinow/anaconda3/lib/python3.7/threading.py", line 1048, in _wait_for_tstate_lock
    elif lock.acquire(block, timeout):
KeyboardInterrupt

Pull requests welcome!
This is an Open Source project - please consider contributing a bug fix
yourself (please read CONTRIBUTING.md before starting any work though).

[kinow]

p.s.: looking again at the logs looks like a Cylc UI Server issue with ZMQ perhaps? Might have to transfer the issue over there?

[kinow]

Still happening, tried today with complex and five, after rebasing to latest commit, and re-creating virtual environments.

Sending a cylc kill five|complex resolves it, and the UI then displays the workflow as stopped. But a CTRL+C doesn't stop the workflow that was started with --no-detach.

Environment: Ubuntu LTS, Miniconda, and Python 3.8. The issue doesn't occur in Python 3.7.