cylc monitor rewrite

[oliver-sanders]

Closes #1685

Jet-lag washup - re-implement cylc monitor using urwid.

• A direct replacement for the old cylc monitor.
  • Gets data direct from the suite.
    • Uses GraphQL so we can add the option to go via the UI Server later.
    • One second (non-adaptive) polling.
  • Performs global updating (there is scope for partial updating).
• Follows the new Cylc UI - TreeView interface which is easily ported to a terminal interface.
• Uses unicode to replicate task icons.
  • Note font size can be changed with ctrl +.

What's supported:

• The full Workflow-Cycle-Family-Task-Job tree.
• A predefined set of job information.
• Task and job icons (including 25,50,75% progress indicators).
• Family grouping with consolidated task icons.

What's not supported:

• Multiple workflows (to be supported via GScan-esque panel at a future date).
• Cycle consolidated task icons (computationally expensive).
• Mutations (attempt after/with Suite API - On-The-Fly #2123).
• ctrl c exit, use q or ctrl d.

TODO

• Fix tests which depend on the old cylc monitor.
• Unit test the new code where possible quite difficult.
• Write visual test via HTML translation.
• Add suite state summary to toolbar.
• Add task state filtering.
• Investigate performance with larger suites. attempt after delta updates

Requirements check-list

• I have read CONTRIBUTING.md and added my name as a Code Contributor.
• Contains logically grouped changes (else tidy your branch by rebase).
• Does not contain off-topic changes (use other PRs for other changes).
• Appropriate tests are included (unit and/or functional).
• Appropriate change log entry included.
• (master branch) I have opened a documentation PR at cylc/cylc-doc/pull/XXXX.

[hjoliver]

Need to add urwid to setup.py.

[hjoliver]

Found a couple of issues in my environment already, but it's a WIP; generally: god damn it, that is cool. Well done. 🎉

[oliver-sanders]

The test I added for the new cylc monitor is failing on Travis for some reason, I'll take a look at it and hopefully simplify a bit. The test uses a date-time cycling suite which isn't using UTC mode = True so hopefully that covers the time zone issue.

Have added a TODO for performance, there might be some easy wins. If nothing else, we can lower the refresh rate for heavy suites.

[hjoliver]

@oliver-sanders - I'm keen to get this in, and we can enhance it further (e.g. status summaries at top) in future PRs.

However, (CylcCon 2020) @trwhitcomb wanted to keep the old cylc monitor for "quick overview" use cases, at least until he's satisfied that this really is better for that. To make that easy, we need to rename this new app anyway as it is likely to grow command funcitionality - at which point it will no longer be merely a "monitor".

What about cylc treeview?

[oliver-sanders]

and we can enhance it further (e.g. status summaries at top) in future PRs.

Things I'll try and do in this PR:

• State summary at the top
• Filtering
• Copyable text

Things I'll try and punt to future PRs:

• Async updater (so that refreshing data doesn't lockup the UI)
• Configurable / semi-intelligent update frequency.

I think once filtering/state-totals/copyable text are done the use case for the old cylc monitor is expended?

What about cylc treeview?

With the addition of a gscan column this program becomes a CLI equivalent to the Cylc WUI. Perhaps:

• cylc cui (as opposed to WUI)
• cylc cli (more intuitive version of the above)
• cylc term[inal] (meh)
• cylc console (more fun)
• cylc command-centre (woohoo)

[hjoliver]

Things I'll try and do in this PR:

Great.

Things I'll try and punt to future PRs:

Yep, good.

I think once filtering/state-totals/copyable text are done the use case for the old cylc monitor is expended?

I tend to agree, but we can treat that as a separate issue since the two utils won't have the same name anymore.

With the addition of a gscan column this program becomes a CLI equivalent to the Cylc WUI.

Ah, good point. The name might require some thought. cylc cui makes the most sense, but it ain't pretty.

[kinow]

Two approvals, if Travis passes with no issues I think it should be ready to be merged. Thanks @oliver-sanders !

[kinow]

One failure, but doesn't look related. I don't seem to have access to restart the build (button not displaying, maybe a Travis glitch). Doing some manual testing...

[kinow]

Found another weird case with five, but we can merge and move it to another issue if you prefer @oliver-sanders .

The terminal at the left top corner has the workflow five. The one below has the families2.

I did a cylc hold five, followed by cylc release five after some seconds. Then did the same for families2.

However, five's tui terminal never came back? Actually, I cannot use h, or the arrow keys. Simply nothing is working.

Did a quick strace on both cylc tui instances, not sure if helpful.

Then after hitting CTRL + C, it printed an exception and the process started working again.

[oliver-sanders]

Hard to reproduce, if you get it again could you see if the suite is still responsive, try a graphql request. I'm going to keep Tui open on a suite in the background and see if it freezes at any point.

[kinow]

if you get it again could you see if the suite is still responsive

It was running, at least the logs were showing the tasks/jobs and new cyclepoints as it normally does. After clicking CTRL + C on the cylc tui terminal, that terminal started showing the updated data too (I didn't need to touch the suite).

Will merge & create an issue for that tomorrow morning so we can track the issue 👍 thanks!

[oliver-sanders]

Would rather fix now but I'm finding it very difficult to reproduce and harder to debug. (I'm getting other issues on my platform at the moment which are hard to separate).

[oliver-sanders]

Managed to reproduce, the traceback is coming from zmq.sugar.context.Context.__del__.

I think what's going on is that something has gone wrong with the connection so Tui has attempted to create a new connection. This has caused zmqto try and remove/recreate the context which, it would seem, causes traceback. Perhaps a bug in zmq. If so this problem is bigger than Tui.

Investigating...

[kinow]

This has caused zmqto try and remove/recreate the context which, it would seem, causes traceback. Perhaps a bug in zmq. If so this problem is bigger than Tui.

Not sure if related, but these are the issues in flow/uiserver I created when working on Cylc UI and that have some ZMQ stuff in the tracebacks:

• ZMQServer "Operation cannot be accomplished in current state" exception causes loop to get stalled #3414
• zmq.error.ZMQError: Socket operation on non-socket cylc-uiserver#115

[hjoliver]

Investigating...

If you confirm that as the cause, maybe for the moment we should not attempt to automatically reconnect? (Or is it possible to do it in a way that is more like starting the application again from scratch)?

[dwsutherland]

@oliver-sanders appears to be some consistent failures in Travis chunk3:

./tests/shutdown/18-client-on-dead-suite.t                     (Wstat: 0 Tests: 5 Failed: 2)
  Failed tests:  3, 5
./tests/shutdown/10-no-port-file.t                             (Wstat: 0 Tests: 5 Failed: 1)
  Failed test:  4

[oliver-sanders]

Should be fixed now.

[hjoliver]

Tests pass, two approvals, and this is quite self-contained ... going for the merge 🎉

[kinow]

Synced local branch, then tested again with five and families2, both running with no-detach and with cylc tui attached to both.

Randomly held then released workflows. The first two times, it worked fine for five. Except the " that it displayed. Or is that a symbol for paused @oliver-sanders ?

Then on the third time the workflow exited, but that's a known issue. Started it again with cylc run --no-detach five, and now the workflow is running fine, no errors, and cylc tui five that was already running is now showing the tasks again. All good.

cylc hold five, and now the workflow is still happy, no errors in the logs. But cylc tui is now showing five as running. However, it might be something wrong with Cylc Flow. I odn't see it moving the logs and showing that health check line which I normally see when I hold five. But I can still ping/poll it. 🤔

Better to merge it and move it to a separate issue if others have a similar problem (might be something I am doing).

[hjoliver]

Must admit I didn't test hold/release extensively.

" is definitely the hold indicator, but @kinow seems to be right that it doesn't get applied and removed consistently.

[hjoliver]

Testing with a simpler suite:

[scheduling]
   cycling mode = integer
   initial cycle point = 1
   final cycle point = 10
   [[dependencies]]
        [[[P1]]]
            graph = "foo[-P1] => foo => bar & baz"
[runtime]
   [[root]]
       script = sleep 10

It appears that the hold indicator does not get applied and/or removed to complete cycles of finished tasks. (For which it is kind of meaningless anyway ... but probably easy for @oliver-sanders to fix).

[oliver-sanders]

Yep, confirmed.

The tasks which appear to be out-of-sync should not be visible and are somehow left-over from the previous update.

If you run watch cylc dump <suite> against cylc tui <suite> the out-of-sync tasks should appear in Tui but not in Dump.

[oliver-sanders]

@kinow @hjoliver

After a bit of digging the issue is not here (you can replicate the same in WUI), Tui is simply displaying the data it was provided.

This is an artifact of the two task pools we currently have in Cylc, the SOS TaskPool and the data store. The data store contains tasks which aren't present in the TaskPool, when you hold the suite these tasks aren't held because they aren't present in the data store task pool.

[kinow]

Thanks for investigating it @oliver-sanders . I guess that might be something being fixed. @dwsutherland mentioned he has been finding small issues in the data store while working on the UIS deltas (if I understand that correctly).

[hjoliver]

This is an artifact of the two task pools we currently have in Cylc, the SOS TaskPool and the data store...

Ah yes, makes sense, I should have realized that. I talked to @dwsutherland about this a while ago Not his fault, there's some ambiguity about what exactly to provide to the UI for the default "view" until one or other of the following is settled on and solved:

• SOS: UI shows pure SOS task pool as now
• SOS: UI shows "fake" n-distance window centered on "active tasks" (we noted in the workshop there are several problems with this)
• SOD: UI shows n-distance window centered on the (n=0) SOD scheduler task pool

For work on SOD I really need to see just the scheduler task pool (so was using the old monitor which still feed off of the old scheduler data store).

[oliver-sanders]

This isn't a bug, it's absolutely correct, when you hold a suite you are only holding active tasks, not archived ones.

• In the WUI we can display all n>1 tasks with opacity 0.5 which will make it clear that they are not "live".
• In the TUI that isn't really an option, we will have to do something more clever.

[hjoliver]

What’s not a bug? Showing the hold indicator on “archived” tasks, or displaying those archived tasks at all in the current UI? I would say the former is a bug (as you say, you can’t hold an archived task), and the latter isn’t a bug but is sort of wrong in that it doesn’t conform to our current SOS task pool or any of the future options as I listed above.

[oliver-sanders]

@kinow seems to be right that it doesn't get applied and removed consistently

I guess that might be something being fixed

This isn't a bug, it's absolutely correct

What’s not a bug?

When you hold a suite only the "active" tasks actually get held and so show up that way in the GUI.

Technically this is the correct behaviour, the "archived" tasks are not held, the database backs this up.

[hjoliver]

Technically this is the correct behaviour, the "archived" tasks are not held, the database backs this up.

I agree with that, but I thought (maybe I was wrong?) that the hold indicator was being applied to archived tasks in cylc tui (or at least not removed from them once they left the task pool).

[oliver-sanders]

Looks like tasks keep the status they had when they were in the task pool. Subsequent holds, etc do not effect "archived" tasks.