Add option to attach Subsystems to streaming workunits

[gshuflin]

Problem

Now that we have the infrastructure to stream workunits while pants is running, we need a way to be able to tell some component of the code to do something useful with those workunits. We also want plugins to be able to receive streamed workunits.

Solution

A global Subsystem with a specific method handle_workunits can be registered to receive streaming workunits, by using the new global option --streaming-workunit-handlers, which expects a list of Python import paths as strings. At the beginning of a pants run, pants_local_runner will dynamically import all specified Subsystem classes, and every time StreamingWorkunitHandler receives workunits from the engine, it will pass them along to the handle_workunits method of all registered Subsystems.

The string arguments need to be a fully-qualified import path including the class of the Subsystem itself. For instance, ./pants --v2 --no-v1 --streaming-workunits-handlers="['pants.reporting.workunits.Workunits']" binary examples/src/python/example/hello/main/ will register a Subsystem subclass Workunits defined in the module pants.reporting.workunits to receive streaming workunits.

[Eric-Arellano]

Nit: don't use """ unless you are doing a multiline string like this:

"""Line 1
Line 2
Line 3"""

What you are instead using is implicit string concatenation. I think you used """ because you need to use " in the string itself. Instead, what you can use is \".

[blorente]

I don't think it's necessary in this case, but just bringing it up in case it is: Should this option removal follow some kind of deprecation cycle?

[blorente]

Is there some way we can define an interface or mixin or something of the sort to capture all these constraints? (i.e. the component should have a method called handle_workunits, it should be a global subsystem...).

[blorente]

Could we add checking for the signature of handle_workunits?

[blorente]

Also none of my comments are blockers :)