Implement hook_module config option. (#539)

Author: tobiasraabe

docs/source/changes.md

@@ -23,6 +23,8 @@ releases are available on [PyPI](https://pypi.org/project/pytask) and
 - {pull}`528` improves the codecov setup and coverage.
 - {pull}`535` reenables and fixes tests with Jupyter.
 - {pull}`536` allows partialed functions to be task functions.
+- {pull}`539` implements the {confval}`hook_module` configuration value and
+  `--hook-module` commandline option to register hooks.
 - {pull}`540` changes the CLI entry-point and allow `pytask.build(tasks=task_func)` as
   the signatures suggested.
 - {pull}`542` refactors the plugin manager.

docs/source/how_to_guides/how_to_write_a_plugin.md renamed to docs/source/how_to_guides/extending_pytask.md

@@ -1,11 +1,71 @@
-# How to write a plugin
+# Extending pytask
 
-Since pytask is based on pluggy, it is extensible. In this section, you will learn some
-key concepts you need to know to write a plugin. It won't deal with pluggy in detail,
-but if you are interested feel free to read [pluggy](../explanations/pluggy.md). A quick
-look at the first paragraphs might be useful nonetheless.
+pytask can be extended since it is built upon
+[pluggy](https://pluggy.readthedocs.io/en/latest/), a plugin system for Python.
 
-## Preparation
+How does it work? Throughout the execution, pytask arrives at entrypoints, called hook
+functions. When pytask calls a hook function it loops through hook implementations and
+each hook implementation can alter the result of the entrypoint.
+
+The full list of hook functions is specified in {doc}`../reference_guides/hookspecs`.
+
+More general information about pluggy can be found in its
+[documentation](https://pluggy.readthedocs.io/en/latest/).
+
+There are two ways to add new hook implementations.
+
+1. Using the {option}`pytask build --hook-module` commandline option or the
+   {confval}`hook_module` configuration value.
+1. Packaging your plugin as a Python package to publish and share it.
+
+(hook-module)=
+
+## Using `--hook-module` and `hook_module`
+
+The easiest and quickest way to extend pytask is to create a module, for example,
+`hooks.py` and register it temporarily via the commandline option or permanently via the
+configuration.
+
+```console
+pytask --hook-module hooks.py
+```
+
+or
+
+```toml
+[tool.pytask.ini_options]
+hook_module = ["hooks.py"]
+```
+
+The value can be a path. If the path is relative it is assumed to be relative to the
+configuration file or relative to the current working directory as a fallback.
+
+The value can also be a module name. For example, if `hooks.py` lies your projects
+package called `myproject` which is importable, then, you can also use
+
+```toml
+[tool.pytask.ini_options]
+hook_module = ["myproject.hooks"]
+```
+
+In `hooks.py` we can add another commandline option to `pytask build` by providing an
+addition hook implementation for the hook specification
+{func}`~_pytask.hookspecs.pytask_extend_command_line_interface`.
+
+```python
+import click
+from _pytask.pluginmanager import hookimpl
+
+
+@hookimpl
+def pytask_extend_command_line_interface(cli):
+    """Add parameters to the command line interface."""
+    cli.commands["build"].params.append(click.Option(["--hello"]))
+```
+
+## Packaging a plugin
+
+### Preparation
 
 Before you start implementing your plugin, the following notes may help you.
 
@@ -24,11 +84,11 @@ Before you start implementing your plugin, the following notes may help you.
   for your plugin to get feedback from other developers. Your proposal should be concise
   and explain what problem you want to solve and how.
 
-## Writing your plugin
+### Writing your plugin
 
 This section explains some steps which are required for all plugins.
 
-### Set up the setuptools entry-point
+#### Set up the setuptools entry-point
 
 pytask discovers plugins via `setuptools` entry-points. Following the approach advocated
 for by [setuptools_scm](https://github.com/pypa/setuptools_scm), the entry-point is
@@ -65,7 +125,7 @@ For a complete example with `setuptools_scm` and `pyproject.toml` see the
 The entry-point for pytask is called `"pytask"` and points to a module called
 `pytask_plugin.plugin`.
 
-### `plugin.py`
+#### `plugin.py`
 
 `plugin.py` is the entrypoint for pytask to your package. You can put all of your hook
 implementations in this module, but it is recommended to imitate the structure of pytask

docs/source/how_to_guides/index.md

@@ -20,7 +20,7 @@ how_to_influence_build_order
 hashing_inputs_of_tasks
 using_task_returns
 writing_custom_nodes
-how_to_write_a_plugin
+extending_pytask
 the_data_catalog
 ```
 

docs/source/reference_guides/configuration.md

@@ -97,6 +97,21 @@ editor_url_scheme = "no_link"
 
 ````
 
+````{confval} hook_module
+
+Register additional modules containing hook implementations.
+
+```toml
+hook_modules = ["myproject.hooks", "hooks.py"]
+```
+
+You can use module names and paths as values. Relative paths are assumed to be relative
+to the configuration file or the current working directory.
+
+{ref}`This how-to guide <hook-module>` has more information.
+
+````
+
 ````{confval} ignore
 
 pytask can ignore files and directories and exclude some tasks or reduce the duration of

docs/source/tutorials/plugins.md

@@ -19,7 +19,7 @@ You can find plugins in many places.
 - Search on [anaconda.org](https://anaconda.org/search?q=pytask) or
   [prefix.dev](https://prefix.dev) for related packages.
 
-## How to implement your plugin
+## How to extend pytask
 
-Follow the {doc}`guide on writing a plugin <../how_to_guides/how_to_write_a_plugin>` to
-write your plugin.
+Follow the {doc}`guide on extending pytask <../how_to_guides/extending_pytask>` to add
+your own hook implementations or write your plugin.

src/_pytask/click.py

@@ -1,27 +1,36 @@
 """Contains code related to click."""
 from __future__ import annotations
 
-import enum
 import inspect
+from enum import Enum
 from gettext import gettext as _
+from gettext import ngettext
 from typing import Any
 from typing import ClassVar
+from typing import TYPE_CHECKING
 
 import click
 from _pytask import __version__ as version
 from _pytask.console import console
+from click import Choice
+from click import Command
+from click import Context
+from click import Parameter
 from click.parser import split_opt
 from click_default_group import DefaultGroup
 from rich.highlighter import RegexHighlighter
 from rich.panel import Panel
 from rich.table import Table
 from rich.text import Text
 
+if TYPE_CHECKING:
+    from collections.abc import Sequence
+
 
 __all__ = ["ColoredCommand", "ColoredGroup", "EnumChoice"]
 
 
-class EnumChoice(click.Choice):
+class EnumChoice(Choice):
     """An enum-based choice type.
 
     The implementation is copied from https://github.com/pallets/click/pull/2210 and
@@ -35,17 +44,15 @@ class EnumChoice(click.Choice):
 
     """
 
-    def __init__(self, enum_type: type[enum.Enum], case_sensitive: bool = True) -> None:
+    def __init__(self, enum_type: type[Enum], case_sensitive: bool = True) -> None:
         super().__init__(
             choices=[element.value for element in enum_type],
             case_sensitive=case_sensitive,
         )
         self.enum_type = enum_type
 
-    def convert(
-        self, value: Any, param: click.Parameter | None, ctx: click.Context | None
-    ) -> Any:
-        if isinstance(value, enum.Enum):
+    def convert(self, value: Any, param: Parameter | None, ctx: Context | None) -> Any:
+        if isinstance(value, Enum):
             value = value.value
         value = super().convert(value=value, param=param, ctx=ctx)
         if value is None:
@@ -68,7 +75,7 @@ class ColoredGroup(DefaultGroup):
 
     def format_help(
         self: DefaultGroup,
-        ctx: click.Context,
+        ctx: Context,
         formatter: Any,  # noqa: ARG002
     ) -> None:
         """Format the help text."""
@@ -114,12 +121,62 @@ def format_help(
         )
 
 
-class ColoredCommand(click.Command):
+def _iter_params_for_processing(
+    invocation_order: Sequence[Parameter], declaration_order: Sequence[Parameter]
+) -> list[Parameter]:
+    def sort_key(item: Parameter) -> tuple[bool, float]:
+        # Hardcode the order of the config and paths parameters so that they are always
+        # processed first even if other eager parameters are chosen. The rest follows
+        # https://click.palletsprojects.com/en/8.1.x/advanced/#callback-evaluation-order.
+        if item.name == "paths":
+            return False, -3
+
+        if item.name == "config":
+            return False, -2
+
+        if item.name == "hook_module":
+            return False, -1
+
+        try:
+            idx: float = invocation_order.index(item)
+        except ValueError:
+            idx = float("inf")
+
+        return not item.is_eager, idx
+
+    return sorted(declaration_order, key=sort_key)
+
+
+class ColoredCommand(Command):
     """A command with colored help pages."""
 
+    def parse_args(self, ctx: Context, args: list[str]) -> list[str]:
+        if not args and self.no_args_is_help and not ctx.resilient_parsing:
+            click.echo(ctx.get_help(), color=ctx.color)
+            ctx.exit()
+
+        parser = self.make_parser(ctx)
+        opts, args, param_order = parser.parse_args(args=args)
+
+        for param in _iter_params_for_processing(param_order, self.get_params(ctx)):
+            value, args = param.handle_parse_result(ctx, opts, args)
+
+        if args and not ctx.allow_extra_args and not ctx.resilient_parsing:
+            ctx.fail(
+                ngettext(
+                    "Got unexpected extra argument ({args})",
+                    "Got unexpected extra arguments ({args})",
+                    len(args),
+                ).format(args=" ".join(map(str, args)))
+            )
+
+        ctx.args = args
+        ctx._opt_prefixes.update(parser._opt_prefixes)
+        return args
+
     def format_help(
-        self: click.Command,
-        ctx: click.Context,
+        self: Command,
+        ctx: Context,
         formatter: Any,  # noqa: ARG002
     ) -> None:
         """Format the help text."""
@@ -142,9 +199,7 @@ def format_help(
         )
 
 
-def _print_options(
-    group_or_command: click.Command | DefaultGroup, ctx: click.Context
-) -> None:
+def _print_options(group_or_command: Command | DefaultGroup, ctx: Context) -> None:
     """Print options formatted with a table in a panel."""
     highlighter = _OptionHighlighter()
 
@@ -195,7 +250,7 @@ def _print_options(
 
 
 def _format_help_text(  # noqa: C901, PLR0912, PLR0915
-    param: click.Parameter, ctx: click.Context
+    param: Parameter, ctx: Context
 ) -> Text:
     """Format the help of a click parameter.
 
@@ -264,7 +319,7 @@ def _format_help_text(  # noqa: C901, PLR0912, PLR0915
             and not default_value
         ):
             default_string = ""
-        elif isinstance(default_value, enum.Enum):
+        elif isinstance(default_value, Enum):
             default_string = str(default_value.value)
         else:
             default_string = str(default_value)