-
-
Notifications
You must be signed in to change notification settings - Fork 636
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add option to attach Subsystems to streaming workunits #8720
Changes from all commits
d63c0d5
194421d
1a94648
37042b2
4d27241
cfd32da
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,15 +1,20 @@ | ||
# Copyright 2015 Pants project contributors (see CONTRIBUTORS.md). | ||
# Licensed under the Apache License, Version 2.0 (see LICENSE). | ||
|
||
import importlib | ||
import inspect | ||
from typing import Dict, Optional, Tuple, Type, TypeVar, Union, cast | ||
import logging | ||
from typing import Callable, Dict, Iterable, List, Optional, Tuple, Type, TypeVar, Union, cast | ||
|
||
from pants.option.optionable import Optionable | ||
from pants.option.options import Options | ||
from pants.option.scope import ScopeInfo | ||
from pants.subsystem.subsystem_client_mixin import SubsystemClientMixin, SubsystemDependency | ||
|
||
|
||
logger = logging.getLogger(__name__) | ||
gshuflin marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
||
|
||
class SubsystemError(Exception): | ||
"""An error in a subsystem.""" | ||
|
||
|
@@ -169,3 +174,55 @@ def get_options(self): | |
:API: public | ||
""" | ||
return self._scoped_options | ||
|
||
@staticmethod | ||
def get_streaming_workunit_callbacks(subsystem_names: Iterable[str]) -> List[Callable]: | ||
""" | ||
This method is used to dynamically generate a list of callables | ||
intended to be passed to StreamingWorkunitHandler. The caller provides a | ||
collection of strings representing a Python import path to a class that | ||
implements the `Subsystem` class. It will then inspect these classes for | ||
the presence of a special method called `handle_workunits`, which expects a | ||
single non-self argument - namely, a tuple of Python dictionaries | ||
representing workunits. | ||
|
||
For instance, you might invoke this method with something like: | ||
|
||
`Subsystem.get_streaming_workunit_callbacks(["pants.reporting.workunits.Workunit"])` | ||
|
||
And this will result in the method attempting to dynamically-import a | ||
module called "pants.reporting.workunits", inspecting it for the presence | ||
of a class called `Workunit`, getting a global instance of this Subsystem, | ||
and returning a list containing a single reference to the | ||
`handle_workunits` method defined on it - and returning an empty list and | ||
emitting warnings if any of these steps fail. | ||
""" | ||
gshuflin marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
||
callables = [] | ||
|
||
for name in subsystem_names: | ||
try: | ||
name_components = name.split(".") | ||
module_name = ".".join(name_components[:-1]) | ||
class_name = name_components[-1] | ||
module = importlib.import_module(module_name) | ||
subsystem_class = getattr(module, class_name) | ||
except (IndexError, AttributeError, ModuleNotFoundError, ValueError) as e: | ||
logger.warning(f"Invalid module name: {name}: {e}") | ||
continue | ||
except ImportError as e: | ||
logger.warning(f"Could not import {module_name}: {e}") | ||
continue | ||
try: | ||
subsystem = subsystem_class.global_instance() | ||
except AttributeError: | ||
logger.warning(f"{subsystem_class} is not a global subsystem.") | ||
continue | ||
|
||
try: | ||
callables.append(subsystem.handle_workunits) | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Is there some way we can define an interface or mixin or something of the sort to capture all these constraints? (i.e. the component should have a method called There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Could we add checking for the signature of |
||
except AttributeError: | ||
logger.warning(f"{subsystem_class} does not have a method named `handle_workunits` defined.") | ||
continue | ||
|
||
return callables |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't think it's necessary in this case, but just bringing it up in case it is: Should this option removal follow some kind of deprecation cycle?