Improve logging

[matthewrmshin]

This is built on top of #2609, so only the final 3 changesets are applicable, but I think it is now worth exposing for discussion.

The rationale of this comes from the discussions of #2505. As a first step, we need to really simplify how we do logging, i.e. we should go back to mainly rely on the functionality of Python's logging module in the standard library where possible.

We now have:

• Just log/suite/log*.
• Improved calls to logger. E.g.:
  • Exceptions are logged using the logger.exception method.
  • Log with format string + positional arguments instead of pre-formatted string.
• Can, in theory, expose log form cherrypy. (Not done.)
• One log handler class (mainly to handle rotation with time-stamped files) and one log formatter class. (The latter should be unnecessary when we move to Python 3.)
• The --verbose and --debug options are now used to control the level of the logger.
• Log header reprinted on top of continuation log files - useful information when inspecting the logs for a long running suite. E.g.:
  • Suite URL and PID.
  • Restart number and log file number.
  • Initial/final cycle point, etc.

(In the future, we should be able to take this change further to allow users to configure logging in their own way at suite or user/site level. I am not doing this yet so this change can still be used under Python 2.6 on some ancient platforms.)

Close #386.

[hjoliver]

(I'm intending to review this after #2609, which I'm part way through, since this is based on top of that).

[hjoliver]

@matthewrmshin - this now has conflicts. And post merge of #2609 perhaps you could rebase this branch to remove the spurious commits and make the review easier?

[matthewrmshin]

@hjoliver done.

[matthewrmshin]

(Actually, just found a logical conflict with the new cylc review.)

[matthewrmshin]

(Should now have got it.)

[hjoliver]

@kinow - any chance you could review this today (since we're in a hurry to get the release out)? It is actually quite a straightforward change - mostly single liners, and some bulk code relocations.

[kinow]

Sure thing @hjoliver , just finishing some work in another branch, then get some strong coffee and start reviewing this one 👍

[oliver-sanders]

I'm happy here, I only wish we had made this decision before #1958, it would have made life a lot easier!

My only outstanding comment is that we might want to put in a test to test the log file header and to ensure that no "INFO" lines get printed out before the header (which would break the deamonise logic) as this is particularly easy to break.

[matthewrmshin]

#1958 was still worth it though. It was the first attempt to address #386. It also gave us a better foundation for the logging mechanism. The real wake up call was the discussion in #2505, however (about 16 months after #1958).

[matthewrmshin]

(We should merge #2809 before this one. The new logging logic needs to be propagated into the relocated host appointer module.)

[matthewrmshin]

Branch re-based + new commit:

• Site test with latest changes.
• Replace a few missed print and sys.*.write statements with logging statements - revealed during site tests with Auto stop-restart #2809.

[hjoliver]

Nice improvement; LGTM.