Consume Console/Logger singleton/subsystems to prevent interleaving in @console_rules

[kwlzn]

breakout task from the v2 FE HLD here

---

After #6571 is completed, we will be in a situation where the @rules that feed @console_rules are rendered using a curses-like UI, but because we will be writing to stderr in an uncoordinated fashion, usage of the existing Console singleton from a @console_rule will cause interleaved output to stderr (ie, log messages will be-mixed-with/corrupt the ascii codes that position the cursor).

To fix this, we should change the implementation of the Console singleton such that it always send to stdout, and either

1. sends messages on a queue that is consumed and rendered by the rendering loop added to Scheduler::execute in Curses UX renders during @console_rule execution with interleaved console output #6571.
2. buffers messages until they are explicitly requested by a caller.

Additionally, we should either:

1. Add a Logger intrinsic that an @rule may Select that will be customized/prefixed for that @rule
2. Capture all logging (via the python logging module) that happens within the externs that call arbitrary python code
   ... and then render the output of logging the same way we render output from the Console singleton.

[stuhood]

Have updated the description. This is blocked by #6571.

[stuhood]

This is unblocked!

[stuhood]

Starting this one.

[stuhood]

Since we want to (sometimes) pipe all logging through the UX by enqueuing it somewhere, we need the ability to flip a switch that will globally affect logging. While we could in theory have this switch and enqueuing exist on the Python side, it feels more natural to me to move logging to rust before/while doing this. So, my strategy for this ticket looks roughly as follows:

1. Implement a rust log crate Logger that roughly emulates the simplelog CombinedLogger, and delegates to a combination of three other runtime-re-configurable Loggers:
   1. An optional simplelog WriteLogger for a log file (to implement pantsd.log)
   2. An optional simplelog WriteLogger for stderr
   3. A collection of instances of a custom Logger that enqueue items for each running UX instance
2. Expose methods to re-configure the rust global logger for use by src/python/pants/init/logging.py and by the UX when it enables and disables.
3. Install and consume the output of the custom UX Logger in the UX loop using an output queue.
4. Implement a python logging Handler that delegates all logging to the rust log crate, and update src/python/pants/init/logging.py to install this Handler rather than the handlers it currently installs.
5. Modify the Console singleton to produce output on the same output queue.

Using simplelog is a somewhat arbitrary choice: it's mostly just to get a useful WriteLogger implementation without a bunch of other boilerplate involved.

This implements a portion of the description: when this is implemented, we will not yet have a Logger singleton that can be injected into @rules, but Console will write to the appropriate UX instance, and all standard python and rust logging will be enqueued such that it will not collide with instances of the UX.

[Eric-Arellano]

@stuhood that sounds well thought out to me.

Something I'm a little confused on, who is the consumer of this API? You mention several ways we may want to do logging—log file, stderr, enqueuing, etc. Is the intent that Pants developers choose which of these 3 they want to do when implementing a particular goal? Or Pants users pass something like a CLI flag to set behavior? Something in between?

[stuhood]

@Eric-Arellano : The approaches mentioned above are implementation details below the --v2 API: how @rule developers will consume logging will either be via standard python logging (for now), or via an injected Logger singleton.

See the high level design from https://docs.google.com/document/d/1Hn73YlhTPROlULTMa_3A-Fdv7hAPiHFII8rvXri5l7E/edit?usp=sharing for more info on what more explicit logging would look like, with a Logger singleton.

[illicitonion]

Sounds reasonable :) A few more moving parts than I'd like, but I can't point at any that I'd get rid of, so...

[Eric-Arellano]

@stuhood Google doc complains I need permission. Mind giving read access to all with link?

[stuhood]

@stuhood Google doc complains I need permission. Mind giving read access to all with link?

It should be accessible to everyone on pants-devel@. Unfortunately, the doc was created internally, so I can't do more than that with this one. I'll continue to remind folks to create this kind of doc with their personal accounts though.

[stuhood]

@Eric-Arellano : Were you able to see it using your pants-devel@ account?

[Eric-Arellano]

@illicitonion and @blorente had some great discussion V2 logging in #7729 (comment) that I'm copying here.

I just had a bunch of discussion with @blorente about this...

I think this is a reasonable short-term hack, but I have some different ideas for the long term.

I think that there are two use-cases for this logging:

1. Show some watcher (either a user, or travis) that pants isn't hanging.
2. Give enough useful information that if one particular test is hanging, the user knows which test to debug.

There are two "modes" of pants running:

a. A user is running streaming to a live TTY.
b. Pants is running piped to something that isn't a live TTY.

The v2 UI is meant to solve both 1 and 2, for case a.
This is manually re-implementing the heavy-hitters portion of the v2 UI. Ideally, we wouldn't be > manually implementing this for every console_rule (or rule type in general).

I think the solution I'd like to see long term is something along the lines of:
Remove this logging.
When streaming to a live TTY, we just rely on the v2 UI to display this information.
When not streaming to a live TTY (either on travis, or when piping to a file, or whatever), we periodically dump the heavy hitters to stderr. Maybe every (configurable) 30 seconds, or maybe when the heavy hitters we'd be displaying change, or maybe some combination of the two.

Until the v2 UI lands, I'm happy to add this logging, but I think it's an until-v2-ui hack rather than the right long-term solution...

[blorente]

Broken off a small self-contained part of this in #7906

[stuhood]

@gshuflin fixed this a while back! Thanks Greg.