diff --git a/docs/extensions.md b/docs/extensions.md index b325b2a4..033d90ba 100644 --- a/docs/extensions.md +++ b/docs/extensions.md @@ -1,96 +1,573 @@ # Extensions -You can pass extensions to the loader to augment its capacities: +Extensions allow to enhance or customize the data that Griffe collects. + +## Using extensions + +Extensions can be specified both on the command-line +(in the terminal), and programmatically (in Python). + +### On the command-line + +On the command-line, you can specify extensions to use +with the `-e`, `--extensions` option. This option +accepts a single positional argument which can take +two forms: + +- a comma-separated list of extensions +- a JSON list of extensions + +Extensions can accept options: the comma-separated list +does not allow to specify options, while the JSON list does. +See examples below. + +With both forms, each extension refers to one of these three things: + +- the name of a built-in extension's module, for example `dynamic_docstrings` + (this is just an example, this built-in extension does not exist) +- the Python dotted-path to a module containing one or more extensions, + or to an extension directly, for example `package.module` and `package.module.ThisExtension` +- the file path to a Python script, and an optional extension name, separated by a colon, + for example `scripts/griffe_exts.py` and `scripts/griffe_exts.py:ThisExtension` + +The specified extension modules can contain more than one extension: +Griffe will pick up and load every extension declared or imported within the modules. +If options are specified for a module that contains +multiple extensions, the same options will be passed to all the extensions, +so extension writers must make sure that all extensions within +a single module accept the same options. +If they don't, Griffe will abort with an error. + +To specify options in the JSON form, use a dictionary instead of a string: +the dictionary's only key is the extension identifier (built-in name, Python path, file path) +and its value is a dictionary of options. + +Some examples: + +```bash +griffe dump griffe -e pydantic,scripts/exts.py:DynamicDocstrings,griffe_attrs +``` + +```bash +griffe check --search src griffe -e '[ + {"pydantic": {"schema": true}}, + { + "scripts/exts.py:DynamicDoctrings": { + "paths": ["mypkg.mymod.myobj"] + } + }, + "griffe_attrs" +]' +``` + +In the above two examples, `pydantic` would be a built-in extension, +`scripts/exts.py:DynamicDocstrings` the file path plus name of a local extension, +and `griffe_attrs` the name of a third-party package that exposes +one or more extensions. + +### Programmatically + +Within Python code, extensions can be specified +with the `extensions` parameter of the [`GriffeLoader` class][griffe.loader.GriffeLoader] +or [`load` function][griffe.loader.load]. + +The parameter accepts an instance of the [`Extensions` class][griffe.extensions.Extensions]. +Such an instance is created with the help of the [`load_extensions` function][griffe.extensions.load_extensions], +which itself accepts a list of strings, dictionaries, extension classes and extension instances. + +Strings and dictionaries are used the same way as [on the command-line](#on-the-command-line). +Extension instances are used as such, and extension classes are instantiated +without any options. + +Example: ```python -from griffe.loader import GriffeLoader -from griffe.extensions import VisitorExtension, Extensions, When +import griffe + +from mypackage.extensions import ThisExtension, ThisOtherExtension + +extensions = griffe.load_extensions( + [ + {"pydantic": {"schema": true}}, + {"scripts/exts.py:DynamicDoctrings": {"paths": ["mypkg.mymod.myobj"]}}, + "griffe_attrs", + ThisExtension(option="value"), + ThisOtherExtension, + ] +) + +data = griffe.load("mypackage", extensions=extensions) +``` + +### In MkDocs + +MkDocs and its mkdocstrings plugin can be configured to use Griffe extensions: + +```yaml title="mkdocs.yml" +plugins: +- mkdocstrings: + handlers: + python: + options: + extensions: + - pydantic: {schema: true} + - scripts/exts.py:DynamicDoctrings: + paths: [mypkg.mymod.myobj] + - griffe_attrs +``` + +The `extensions` key accepts a list that is passed to the +[`load_extensions` function][griffe.extensions.load_extensions]. +See [how to use extensions programmatically](#programmatically) to learn more. + +## Writing extensions + +In the next section we give a bit of context on how Griffe works, +to show how extensions can integrate into the data collection process. +Feel free to skip to the [Events and hooks](#events-and-hooks) section +or the [Full example](#full-example) section +if you'd prefer to see concrete examples first. + +### How it works + +To extract information from your Python sources, +Griffe tries to build Abstract Syntax Trees by parsing the sources +with [`ast`][] utilities. + +If the source code is not available +(the modules are built-in or compiled), +Griffe imports the modules and builds object trees instead. -# import extensions -from some.package import TheirExtension +Griffe then follows the [Visitor pattern](https://www.wikiwand.com/en/Visitor_pattern) +to walk the tree and extract information. +For ASTs, Griffe uses its [Visitor agent][griffe.agents.visitor] +and for object trees, it uses its [Inspector agent][griffe.agents.inspector]. +Sometimes during the walk through the tree (depth-first order), +both the visitor and inspector agents will trigger events. +These events can be hooked on by extensions to alter or enhance +Griffe's behavior. Some hooks will be passed just the current +node being visited, others will be passed both the node +and an instance of an [Object][griffe.dataclasses.Object] subclass, +such as a [Module][griffe.dataclasses.Module], +a [Class][griffe.dataclasses.Class], +a [Function][griffe.dataclasses.Function], +or an [Attribute][griffe.dataclasses.Attribute]. +Extensions will therefore be able to modify these instances. -# or define your own -class ClassStartsAtOddLineNumberExtension(VisitorExtension): - when = When.after_all +The following flow chart shows an example of an AST visit. +The tree is simplified: actual trees have a lot more nodes +like `if/elif/else` nodes, `try/except/else/finally` nodes, +[and many more][ast.AST]. - def visit_classdef(self, node) -> None: - if node.lineno % 2 == 1: - self.visitor.current.labels.add("starts at odd line number") +```mermaid +flowchart TB +M(Module definition) --- C(Class definition) & F(Function definition) +C --- m(Function definition) & A(Variable assignment) +``` + +The following flow chart shows an example of an object tree inspection. +The tree is simplified as well: +[many more types of objects are handled][griffe.agents.nodes.ObjectKind]. + +```mermaid +flowchart TB +M(Module) --- C(Class) & F(Function) +C --- m(Method) & A(Attribute) +``` + +For a more concrete example, let say that we visit (or inspect) +an AST (or object tree) for a given module, and that this module +contains a single class, which itself contains a single method: + +- the agent (visitor or inspector) will walk through the tree + by starting with the module node +- it will instantiate a [Module][griffe.dataclasses.Module], + then walk through its members, continuing with the class node +- it will instantiate a [Class][griffe.dataclasses.Class], + then walk through its members, continuing with the function node +- it will instantiate a [Function][griffe.dataclasses.Function] +- then it will go back up and finish walking since there are + no more nodes to walk through + + +Every time the agent enters a node, +creates an object instance, +or finish handling members of an object, +it will trigger an event. + +The flow of events is drawn in the following flowchart: +```mermaid +flowchart TB +visit_mod{{enter module node}} +event_mod_node{{"on_node event
on_module_node event"}} +create_mod{{create module instance}} +event_mod_instance{{"on_instance event
on_module_instance event"}} +visit_mod_members{{visit module members}} +visit_cls{{enter class node}} +event_cls_node{{"on_node event
on_class_node event"}} +create_cls{{create class instance}} +event_cls_instance{{"on_instance event
on_class_instance event"}} +visit_cls_members{{visit class members}} +visit_func{{enter func node}} +event_func_node{{"on_node event
on_function_node event"}} +create_func{{create function instance}} +event_func_instance{{"on_instance event
on_function_instance event"}} +event_cls_members{{"on_members event
on_class_members event"}} +event_mod_members{{"on_members event
on_module_members event"}} -extensions = Extensions(TheirExtension, ClassStartsAtOddLineNumberExtension) -griffe = GriffeLoader(extensions=extensions) -fastapi = griffe.load_module("fastapi") +start{start} --> visit_mod +visit_mod --> event_mod_node +event_mod_node --> create_mod +create_mod --> event_mod_instance +event_mod_instance --> visit_mod_members +visit_mod_members --1--> visit_cls +visit_cls --> event_cls_node +event_cls_node --> create_cls +create_cls --> event_cls_instance +event_cls_instance --> visit_cls_members +visit_cls_members --1--> visit_func +visit_func --> event_func_node +event_func_node --> create_func +create_func --> event_func_instance +event_func_instance --> visit_cls_members +visit_cls_members --2--> event_cls_members +event_cls_members --> visit_mod_members +visit_mod_members --2--> event_mod_members +event_mod_members --> finish{finish} + +class event_mod_node event +class event_mod_instance event +class event_cls_node event +class event_cls_instance event +class event_func_node event +class event_func_instance event +class event_cls_members event +class event_mod_members event +classDef event stroke:#3cc,stroke-width:2 ``` -Extensions are subclasses of a custom version of [`ast.NodeVisitor`][ast.NodeVisitor]. -Griffe uses a node visitor as well, that we will call the *main visitor*. -The extensions are instantiated with a reference to this main visitor, -so they can benefit from its capacities (navigating the nodes, getting the current -class or module, etc.). +Hopefully this flowchart gave you a pretty good idea +of what happens when Griffe collects data from a Python module. +The next setion will explain in more details +the different events that are triggered, +and how to hook onto them in your extensions. + +### Events and hooks + +There are 3 generic **events**: + +- [`on_node`][griffe.extensions.base.Extension.on_node] +- [`on_instance`][griffe.extensions.base.Extension.on_instance] +- [`on_members`][griffe.extensions.base.Extension.on_members] + +There are also specific **events** for each object kind: -Each time a node is visited, the main visitor will make the extensions visit the node as well. -Implement the `visit_` methods to visit nodes of certain types, -and act on their properties. See the [full list of AST nodes](#ast-nodes). +- [`on_module_node`][griffe.extensions.base.Extension.on_module_node] +- [`on_module_instance`][griffe.extensions.base.Extension.on_module_instance] +- [`on_module_members`][griffe.extensions.base.Extension.on_module_members] +- [`on_class_node`][griffe.extensions.base.Extension.on_class_node] +- [`on_class_instance`][griffe.extensions.base.Extension.on_class_instance] +- [`on_class_members`][griffe.extensions.base.Extension.on_class_members] +- [`on_function_node`][griffe.extensions.base.Extension.on_function_node] +- [`on_function_instance`][griffe.extensions.base.Extension.on_function_instance] +- [`on_attribute_node`][griffe.extensions.base.Extension.on_attribute_node] +- [`on_attribute_instance`][griffe.extensions.base.Extension.on_attribute_instance] -!!! warning "Important note" - Because the main visitor recursively walks the tree itself, - calling extensions on each node, - **you must not visit child nodes in your `.visit_*` methods!** - Otherwise, nodes down the tree will be visited twice or more: - once by the main visitor, and as many times more as your extension is called. - Let the main visitor do the walking, and just take care of the current node, - without handling its children. +The "on node" events are triggered when the agent (visitor or inspector) +starts handling a node in the tree (AST or object tree). -You can access the main visitor state and data through the `.visitor` attribute, -and the nodes instances are extended with additional attributes and properties: +The "on instance" events are triggered when the agent +just created an instance of [Module][griffe.dataclasses.Module], +[Class][griffe.dataclasses.Class], +[Function][griffe.dataclasses.Function], +or [Attribute][griffe.dataclasses.Attribute], +and added it as a member of its parent. + +The "on members" events are triggered when the agent +just finished handling all the members of an object. +Functions and attributes do not have members, +so there are no "on members" event for these two kinds. + +**Hooks** are methods that are called when a particular +event is triggered. To target a specific event, +the hook must be named after it. + +**Extensions** are classes that inherit from +[Griffe's Extension base class][griffe.extensions.Extension] +and define some hooks as methods: ```python +import ast +from griffe import Extension, Object, ObjectNode + + class MyExtension(Extension): - def visit_functiondef(self, node) -> None: - node.parent # the parent node - node.children # the list of children nodes - node.siblings # all the siblings nodes, from top to bottom - node.previous_siblings # only the previous siblings, from closest to top - node.next_siblings # only the next siblings, from closest to bottom - node.previous # first previous sibling - node.next # first next sibling - - self.visitor # the main visitor - self.visitor.current # the current data object - self.visitor.current.kind # the kind of object: module, class, function, attribute -``` - -See the data classes ([`Module`][griffe.dataclasses.Module], -[`Class`][griffe.dataclasses.Class], [`Function`][griffe.dataclasses.Function] -and [`Attribute`][griffe.dataclasses.Attribute]) -for a complete description of their methods and attributes. - -Extensions are run at certain moments while walking the Abstract Syntax Tree (AST): - -- before the visit/inspection: `When.before_all`. - The current node has been grafted to its parent. - If this node represents a data object, the object (`self.visitor.current`/`self.inspector.current`) **is not** yet instantiated. -- before the children visit/inspection: `When.before_children`. - If this node represents a data object, the object (`self.visitor.current`/`self.inspector.current`) **is** now instantiated. - Children **have not** yet been visited/inspected. -- after the children visit/inspection: `When.after_children`. - Children **have** now been visited/inspected. -- after the visit/inspection: `When.after_all` - -See [the `When` enumeration][griffe.extensions.When]. - -To tell the main visitor to run your extension at a certain time, -set its `when` attribute: + def on_instance(self, node: ast.AST | ObjectNode, obj: Object) -> None: + """Do something with `node` and/or `obj`.""" +``` + +Hooks are always defined as methods of a class inheriting from +[Extension][griffe.extensions.Extension], never as standalone functions. + +Since hooks are declared in a class, feel free to also declare state variables +(or any other variable) in the `__init__` method: ```python +import ast +from griffe import Extension, Object, ObjectNode + + class MyExtension(Extension): - when = When.after_children + def __init__(self) -> None: + super().__init__(self) + self.state_thingy = "initial stuff" + self.list_of_things = [] + + def on_instance(self, node: ast.AST | ObjectNode, obj: Object) -> None: + """Do something with `node` and/or `obj`.""" ``` -By default, it will run the extension after the visit/inspection of the node: -that's when the full data for this node and its children is loaded. +### Static/dynamic support + +Extensions can support both static and dynamic analysis of modules. +If a module is scanned statically, your extension hooks +will receive AST nodes (from the [ast][] module of the standard library). +If the module is scanned dynamically, +your extension hooks will receive [object nodes][griffe.ObjectNode]. + +To support static analysis, dynamic analysis, or both, +you can therefore check the type of the received node: + +```python +import ast +from griffe import Extension, Object, ObjectNode + + +class MyExtension(Extension): + def on_instance(self, node: ast.AST | ObjectNode, obj: Object) -> None: + """Do something with `node` and/or `obj`.""" + if isinstance(node, ast.AST): + ... # apply logic for static analysis + else: + ... # apply logic for dynamic analysis +``` + +Since hooks also receive instantiated modules, classes, functions and attributes, +most of the time you will not need to use the `node` argument +other than for checking its type and deciding what to do based on the result. +If you do need to, read the next section explaining how to visit trees. + +### Visiting trees + +Extensions provide basic functionality to help you visit trees: + +- [`visit`][griffe.extensions.base.Extension.visit]: call `self.visit(node)` + to start visiting an abstract syntax tree. +- [`generic_visit`][griffe.extensions.base.Extension.generic_visit]: call + `self.generic_visit(node)` to visit each subnode of a given node. +- [`inspect`][griffe.extensions.base.Extension.inspect]: call `self.inspect(node)` + to start visiting an object tree. Nodes contain references to the runtime objects, + see [`ObjectNode`][griffe.agents.nodes.ObjectNode]. +- [`generic_inspect`][griffe.extensions.base.Extension.generic_inspect]: call + `self.generic_inspect(node)` to visit each subnode of a given node. + +Calling `self.visit(node)` or `self.inspect(node)` will do nothing +unless you actually implement methods that handle specific types of nodes: + +- for ASTs, methods must be named `visit_` where `` + is replaced with the lowercase name of the node's class. For example, + to allow visiting [`ClassDef`][ast.ClassDef] nodes, you must + implement the `visit_classdef` method: + + ```python + import ast + from griffe import Extension + + + class MyExtension(Extension): + def visit_classdef(node: ast.ClassDef) -> None: + # do something with the node + ... + # then visit the subnodes + # (it only makes sense if you implement other methods + # such as visit_functiondef or visit_assign for example) + self.generic_visit(node) + ``` + + See the [list of existing AST classes](#ast-nodes) to learn + what method you can implement. + +- for object trees, methods must be named `inspect_`, + where `` is replaced with the string value of the node's kind. + The different kinds are listed in the [`ObjectKind`][griffe.agents.nodes.ObjectKind] enumeration. + For example, to allow inspecting coroutine nodes, you must implement + the `inspect_coroutine` method: + + ```python + from griffe import Extension, ObjectNode + + + class MyExtension(Extension): + def inspect_coroutine(node: ObjectNode) -> None: + # do something with the node + ... + # then visit the subnodes if it makes sense + self.generic_inspect(node) + ``` + +### Extra data + +All Griffe objects (modules, classes, functions, attributes) +can store additional (meta)data in their `extra` attribute. +This attribute is a dictionary of dictionaries. The first +layer is used as namespacing: each extension writes +into its own namespace, or integrates with other projects +by reading/writing in their namespaces, +according to what they support and document. + +```python +import ast +from griffe import Extension, Object, ObjectNode + +self_namespace = "my_extension" + + +class MyExtension(Extension): + def on_instance(self, node: ast.AST | ObjectNode, obj: Object) -> None: + obj.extra[self_namespace]["some_key"] = "some_value" +``` + +For example, [mkdocstrings-python](https://mkdocstrings.github.io/python) +looks into the `mkdocstrings` namespace for a `template` key. +Extensions can therefore provide a custom template value by writing +into `extra["mkdocstrings"]["template"]`: + +```python +import ast +from griffe import Extension, ObjectNode, Class + +self_namespace = "my_extension" +mkdocstrings_namespace = "mkdocstrings" + + +class MyExtension(Extension): + def on_class_instance(self, node: ast.AST | ObjectNode, cls: Class) -> None: + obj.extra[mkdocstrings_namespace]["template"] = "my_custom_template" +``` + +[Read more about mkdocstrings handler extensions.](https://mkdocstrings.github.io/usage/handlers/#handler-extensions) + +### Options + +Extensions can be made to support options. +These options can then be passed from the [command-line](#on-the-command-line) using JSON, +from Python directly, or from other tools like MkDocs, in `mkdocs.yml`. + +```python +import ast +from griffe import Attribute, Extension, ObjectNode + + +class MyExtension(Extension): + def __init__(self, option1: str, option2: bool = False) -> None: + super().__init__(self) + self.option1 = option1 + self.option2 = option2 + + def on_attribute_instance(self, node: ast.AST | ObjectNode, attr: Attribute) -> None: + if self.option2: + ... # do something +``` + +### Logging + +To better integrate with Griffe and other tools in the ecosystem +(notably MkDocs), use Griffe loggers to log messages: + +```python +import ast +from griffe import Extension, ObjectNode, Module, get_logger + +logger = get_logger(__name__) + + +class MyExtension(Extension): + def on_module_members(self, node: ast.AST | ObjectNode, mod: Module) -> None: + logger.info(f"Doing some work on module {mod.path} and its members") +``` + +### Full example + +The following example shows how one could write a "dynamic docstrings" extension +that dynamically import objects that declare their docstrings dynamically, +to improve support for such docstrings. +The extension is configurable to run only on user-selected objects. + +Package structure (or just write your extension in a local script): + +```tree +./ + pyproject.toml + src/ + dynamic_docstrings/ + __init__.py + extension.py +``` + + +```python title="./src/dynamic_docstrings/extension.py" +import ast +import inspect +from griffe import Docstring, Extension, Object, ObjectNode, get_logger, dynamic_import + +logger = get_logger(__name__) + + +class DynamicDocstrings(Extension): + def __init__(self, object_paths: list[str] | None = None) -> None: + self.object_paths = object_paths + + def on_instance(self, node: ast.AST | ObjectNode, obj: Object) -> None: + if isinstance(node, ObjectNode): + return # skip runtime objects, their docstrings are already right + + if self.object_paths and obj.path not in self.object_paths: + return # skip objects that were not selected + + # import object to get its evaluated docstring + try: + runtime_obj = dynamic_import(obj.path) + docstring = runtime_obj.__doc__ + except ImportError: + logger.debug(f"Could not get dynamic docstring for {obj.path}") + return + except AttributeError: + logger.debug(f"Object {obj.path} does not have a __doc__ attribute") + return + + # update the object instance with the evaluated docstring + docstring = inspect.cleandoc(docstring) + if obj.docstring: + obj.docstring.value = docstring + else: + obj.docstring = Docstring(docstring, parent=obj) +``` + +You can then expose this extension in the top-level module of your package: + +```python title="./src/dynamic_docstrings/__init__.py" +from dynamic_docstrings.extension import DynamicDocstrings + +__all__ = ["DynamicDocstrings"] +``` + +This will allow users to load and use this extension +by referring to it as `dynamic_docstrings` (your Python package name). + +See [how to use extensions](#using-extensions) to learn more +about how to load and use your new extension. ## AST nodes diff --git a/mkdocs.yml b/mkdocs.yml index 03f0cb91..84d8d5ff 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -88,7 +88,11 @@ markdown_extensions: - pymdownx.magiclink - pymdownx.snippets: check_paths: true -- pymdownx.superfences +- pymdownx.superfences: + custom_fences: + - name: mermaid + class: mermaid + format: !!python/name:pymdownx.superfences.fence_code_format - pymdownx.tabbed: alternate_style: true slugify: !!python/object/apply:pymdownx.slugs.slugify diff --git a/src/griffe/__init__.py b/src/griffe/__init__.py index fc522877..76e1ffc3 100644 --- a/src/griffe/__init__.py +++ b/src/griffe/__init__.py @@ -7,8 +7,28 @@ from __future__ import annotations +from griffe.agents.nodes import ObjectNode +from griffe.dataclasses import Attribute, Class, Docstring, Function, Module, Object from griffe.diff import find_breaking_changes +from griffe.extensions import Extension, load_extensions from griffe.git import load_git +from griffe.importer import dynamic_import from griffe.loader import load +from griffe.logger import get_logger -__all__: list[str] = ["find_breaking_changes", "load", "load_git"] +__all__: list[str] = [ + "Attribute", + "Class", + "Docstring", + "dynamic_import", + "Extension", + "Function", + "find_breaking_changes", + "get_logger", + "load", + "load_extensions", + "load_git", + "Module", + "Object", + "ObjectNode", +] diff --git a/src/griffe/agents/inspector.py b/src/griffe/agents/inspector.py index f3f3c3fc..1c6deed4 100644 --- a/src/griffe/agents/inspector.py +++ b/src/griffe/agents/inspector.py @@ -215,7 +215,9 @@ def inspect_module(self, node: ObjectNode) -> None: Parameters: node: The node to inspect. """ - self.current = Module( + self.extensions.call("on_node", node=node) + self.extensions.call("on_module_node", node=node) + self.current = module = Module( name=self.module_name, filepath=self.filepath, parent=self.parent, @@ -223,7 +225,11 @@ def inspect_module(self, node: ObjectNode) -> None: lines_collection=self.lines_collection, modules_collection=self.modules_collection, ) + self.extensions.call("on_instance", node=node, obj=module) + self.extensions.call("on_module_instance", node=node, mod=module) self.generic_inspect(node) + self.extensions.call("on_members", node=node, obj=module) + self.extensions.call("on_module_members", node=node, mod=module) def inspect_class(self, node: ObjectNode) -> None: """Inspect a class. @@ -231,6 +237,9 @@ def inspect_class(self, node: ObjectNode) -> None: Parameters: node: The node to inspect. """ + self.extensions.call("on_node", node=node) + self.extensions.call("on_class_node", node=node) + bases = [] for base in node.obj.__bases__: if base is object: @@ -244,7 +253,11 @@ def inspect_class(self, node: ObjectNode) -> None: ) self.current.set_member(node.name, class_) self.current = class_ + self.extensions.call("on_instance", node=node, obj=class_) + self.extensions.call("on_class_instance", node=node, cls=class_) self.generic_inspect(node) + self.extensions.call("on_members", node=node, obj=class_) + self.extensions.call("on_class_members", node=node, cls=class_) self.current = self.current.parent # type: ignore[assignment] def inspect_staticmethod(self, node: ObjectNode) -> None: @@ -335,6 +348,9 @@ def handle_function(self, node: ObjectNode, labels: set | None = None) -> None: node: The node to inspect. labels: Labels to add to the data object. """ + self.extensions.call("on_node", node=node) + self.extensions.call("on_function_node", node=node) + try: signature = getsignature(node.obj) except Exception: # noqa: BLE001 @@ -371,6 +387,11 @@ def handle_function(self, node: ObjectNode, labels: set | None = None) -> None: ) obj.labels |= labels self.current.set_member(node.name, obj) + self.extensions.call("on_instance", node=node, obj=obj) + if obj.is_attribute: + self.extensions.call("on_attribute_instance", node=node, attr=obj) + else: + self.extensions.call("on_function_instance", node=node, func=obj) def inspect_attribute(self, node: ObjectNode) -> None: """Inspect an attribute. @@ -387,6 +408,9 @@ def handle_attribute(self, node: ObjectNode, annotation: str | Name | Expression node: The node to inspect. annotation: A potentiel annotation. """ + self.extensions.call("on_node", node=node) + self.extensions.call("on_attribute_node", node=node) + # TODO: to improve parent = self.current labels: set[str] = set() @@ -421,6 +445,8 @@ def handle_attribute(self, node: ObjectNode, annotation: str | Name | Expression if node.name == "__all__": parent.exports = set(node.obj) + self.extensions.call("on_instance", node=node, obj=attribute) + self.extensions.call("on_attribute_instance", node=node, attr=attribute) _kind_map = { diff --git a/src/griffe/agents/nodes/_runtime.py b/src/griffe/agents/nodes/_runtime.py index 80b0584f..18fec125 100644 --- a/src/griffe/agents/nodes/_runtime.py +++ b/src/griffe/agents/nodes/_runtime.py @@ -24,18 +24,31 @@ class ObjectKind(enum.Enum): """Enumeration for the different kinds of objects.""" MODULE: str = "module" + """Modules.""" CLASS: str = "class" + """Classes.""" STATICMETHOD: str = "staticmethod" + """Static methods.""" CLASSMETHOD: str = "classmethod" + """Class methods.""" METHOD_DESCRIPTOR: str = "method_descriptor" + """Method descriptors.""" METHOD: str = "method" + """Methods.""" BUILTIN_METHOD: str = "builtin_method" + """Built-in ethods.""" COROUTINE: str = "coroutine" + """Coroutines""" FUNCTION: str = "function" + """Functions.""" BUILTIN_FUNCTION: str = "builtin_function" + """Built-in functions.""" CACHED_PROPERTY: str = "cached_property" + """Cached properties.""" PROPERTY: str = "property" + """Properties.""" ATTRIBUTE: str = "attribute" + """Attributes.""" def __str__(self) -> str: return self.value diff --git a/src/griffe/agents/visitor.py b/src/griffe/agents/visitor.py index 4d975d44..55e301e6 100644 --- a/src/griffe/agents/visitor.py +++ b/src/griffe/agents/visitor.py @@ -211,7 +211,9 @@ def visit_module(self, node: ast.Module) -> None: Parameters: node: The node to visit. """ - module = Module( + self.extensions.call("on_node", node=node) + self.extensions.call("on_module_node", node=node) + self.current = module = Module( name=self.module_name, filepath=self.filepath, parent=self.parent, @@ -219,8 +221,11 @@ def visit_module(self, node: ast.Module) -> None: lines_collection=self.lines_collection, modules_collection=self.modules_collection, ) - self.current = module + self.extensions.call("on_instance", node=node, obj=module) + self.extensions.call("on_module_instance", node=node, mod=module) self.generic_visit(node) + self.extensions.call("on_members", node=node, obj=module) + self.extensions.call("on_module_members", node=node, mod=module) def visit_classdef(self, node: ast.ClassDef) -> None: """Visit a class definition node. @@ -228,6 +233,9 @@ def visit_classdef(self, node: ast.ClassDef) -> None: Parameters: node: The node to visit. """ + self.extensions.call("on_node", node=node) + self.extensions.call("on_class_node", node=node) + # handle decorators decorators = [] if node.decorator_list: @@ -261,7 +269,11 @@ def visit_classdef(self, node: ast.ClassDef) -> None: class_.labels |= self.decorators_to_labels(decorators) self.current.set_member(node.name, class_) self.current = class_ + self.extensions.call("on_instance", node=node, obj=class_) + self.extensions.call("on_class_instance", node=node, cls=class_) self.generic_visit(node) + self.extensions.call("on_members", node=node, obj=class_) + self.extensions.call("on_class_members", node=node, cls=class_) self.current = self.current.parent # type: ignore[assignment] def decorators_to_labels(self, decorators: list[Decorator]) -> set[str]: @@ -313,6 +325,9 @@ def handle_function(self, node: ast.AsyncFunctionDef | ast.FunctionDef, labels: node: The node to visit. labels: Labels to add to the data object. """ + self.extensions.call("on_node", node=node) + self.extensions.call("on_function_node", node=node) + labels = labels or set() # handle decorators @@ -348,6 +363,8 @@ def handle_function(self, node: ast.AsyncFunctionDef | ast.FunctionDef, labels: ) attribute.labels |= labels self.current.set_member(node.name, attribute) + self.extensions.call("on_instance", node=node, obj=attribute) + self.extensions.call("on_attribute_instance", node=node, attr=attribute) return # handle parameters @@ -456,6 +473,8 @@ def handle_function(self, node: ast.AsyncFunctionDef | ast.FunctionDef, labels: function.labels |= labels + self.extensions.call("on_instance", node=node, obj=function) + self.extensions.call("on_function_instance", node=node, func=function) if self.current.kind is Kind.CLASS and function.name == "__init__": self.current = function # type: ignore[assignment] # temporary assign a function self.generic_visit(node) @@ -541,6 +560,8 @@ def handle_attribute( node: The node to visit. annotation: A potential annotation. """ + self.extensions.call("on_node", node=node) + self.extensions.call("on_attribute_node", node=node) parent = self.current labels = set() @@ -625,6 +646,8 @@ def handle_attribute( if name == "__all__": with suppress(AttributeError): parent.exports = safe_get__all__(node, self.current) # type: ignore[arg-type] + self.extensions.call("on_instance", node=node, obj=attribute) + self.extensions.call("on_attribute_instance", node=node, attr=attribute) def visit_assign(self, node: ast.Assign) -> None: """Visit an assignment node. diff --git a/src/griffe/cli.py b/src/griffe/cli.py index d71e3ab5..6e290836 100644 --- a/src/griffe/cli.py +++ b/src/griffe/cli.py @@ -35,7 +35,7 @@ from griffe.stats import _format_stats if TYPE_CHECKING: - from griffe.extensions import Extension, Extensions + from griffe.extensions import Extensions, ExtensionType DEFAULT_LOG_LEVEL = os.getenv("GRIFFE_LOG_LEVEL", "INFO").upper() @@ -98,6 +98,13 @@ def _load_packages( _level_choices = ("DEBUG", "INFO", "WARNING", "ERROR", "CRITICAL") +def _extensions_type(value: str) -> Sequence[str | dict[str, Any]]: + try: + return json.loads(value) + except json.JSONDecodeError: + return value.split(",") + + def get_parser() -> argparse.ArgumentParser: """Return the CLI argument parser. @@ -133,7 +140,7 @@ def add_common_options(subparser: argparse.ArgumentParser) -> None: "-e", "--extensions", default={}, - type=json.loads, + type=_extensions_type, help="A list of extensions to use.", ) loading_options.add_argument( @@ -277,7 +284,7 @@ def dump( full: bool = False, docstring_parser: Parser | None = None, docstring_options: dict[str, Any] | None = None, - extensions: Sequence[str | dict[str, Any] | Extension | type[Extension]] | None = None, + extensions: Sequence[str | dict[str, Any] | ExtensionType | type[ExtensionType]] | None = None, resolve_aliases: bool = False, resolve_implicit: bool = False, resolve_external: bool = False, @@ -356,7 +363,7 @@ def check( against_path: str | Path | None = None, *, base_ref: str | None = None, - extensions: Sequence[str | dict[str, Any] | Extension | type[Extension]] | None = None, + extensions: Sequence[str | dict[str, Any] | ExtensionType | type[ExtensionType]] | None = None, search_paths: Sequence[str | Path] | None = None, allow_inspection: bool = True, verbose: bool = False, diff --git a/src/griffe/extensions/__init__.py b/src/griffe/extensions/__init__.py index 7557172c..2138fe5f 100644 --- a/src/griffe/extensions/__init__.py +++ b/src/griffe/extensions/__init__.py @@ -3,6 +3,7 @@ from griffe.extensions.base import ( Extension, Extensions, + ExtensionType, InspectorExtension, VisitorExtension, When, @@ -13,6 +14,7 @@ __all__ = [ "Extensions", "Extension", + "ExtensionType", "InspectorExtension", "VisitorExtension", "When", diff --git a/src/griffe/extensions/base.py b/src/griffe/extensions/base.py index 575e71a1..959514e7 100644 --- a/src/griffe/extensions/base.py +++ b/src/griffe/extensions/base.py @@ -5,6 +5,7 @@ import enum import os import sys +import warnings from collections import defaultdict from importlib.util import module_from_spec, spec_from_file_location from inspect import isclass @@ -15,12 +16,13 @@ from griffe.importer import dynamic_import if TYPE_CHECKING: - from ast import AST + import ast from types import ModuleType from griffe.agents.inspector import Inspector from griffe.agents.nodes import ObjectNode from griffe.agents.visitor import Visitor + from griffe.dataclasses import Attribute, Class, Function, Module, Object class When(enum.Enum): @@ -46,7 +48,12 @@ class VisitorExtension: def __init__(self) -> None: """Initialize the visitor extension.""" - super().__init__() + warnings.warn( + "Visitor extensions are deprecated in favor of the new, more developer-friendly Extension. " + "See https://mkdocstrings.github.io/griffe/extensions/", + DeprecationWarning, + stacklevel=1, + ) self.visitor: Visitor = None # type: ignore[assignment] def attach(self, visitor: Visitor) -> None: @@ -57,7 +64,7 @@ def attach(self, visitor: Visitor) -> None: """ self.visitor = visitor - def visit(self, node: AST) -> None: + def visit(self, node: ast.AST) -> None: """Visit a node. Parameters: @@ -73,7 +80,12 @@ class InspectorExtension: def __init__(self) -> None: """Initialize the inspector extension.""" - super().__init__() + warnings.warn( + "Inspector extensions are deprecated in favor of the new, more developer-friendly Extension. " + "See https://mkdocstrings.github.io/griffe/extensions/", + DeprecationWarning, + stacklevel=1, + ) self.inspector: Inspector = None # type: ignore[assignment] def attach(self, inspector: Inspector) -> None: @@ -93,13 +105,151 @@ def inspect(self, node: ObjectNode) -> None: getattr(self, f"inspect_{node.kind}", lambda _: None)(node) -Extension = Union[VisitorExtension, InspectorExtension] +class Extension: + """Base class for Griffe extensions.""" + + def visit(self, node: ast.AST) -> None: + """Visit a node. + + Parameters: + node: The node to visit. + """ + getattr(self, f"visit_{ast_kind(node)}", lambda _: None)(node) + + def generic_visit(self, node: ast.AST) -> None: + """Visit children nodes. + + Parameters: + node: The node to visit the children of. + """ + for child in ast_children(node): + self.visit(child) + + def inspect(self, node: ObjectNode) -> None: + """Inspect a node. + + Parameters: + node: The node to inspect. + """ + getattr(self, f"inspect_{node.kind}", lambda _: None)(node) + + def generic_inspect(self, node: ObjectNode) -> None: + """Extend the base generic inspection with extensions. + + Parameters: + node: The node to inspect. + """ + for child in node.children: + if not child.alias_target_path: + self.inspect(child) + + def on_node(self, *, node: ast.AST | ObjectNode) -> None: + """Run when visiting a new node during static/dynamic analysis. + + Parameters: + node: The currently visited node. + """ + + def on_instance(self, *, node: ast.AST | ObjectNode, obj: Object) -> None: + """Run when an Object has been created. + + Parameters: + node: The currently visited node. + obj: The object instance. + """ + + def on_members(self, *, node: ast.AST | ObjectNode, obj: Object) -> None: + """Run when members of an Object have been loaded. + + Parameters: + node: The currently visited node. + obj: The object instance. + """ + + def on_module_node(self, *, node: ast.AST | ObjectNode) -> None: + """Run when visiting a new module node during static/dynamic analysis. + + Parameters: + node: The currently visited node. + """ + + def on_module_instance(self, *, node: ast.AST | ObjectNode, mod: Module) -> None: + """Run when a Module has been created. + + Parameters: + node: The currently visited node. + mod: The module instance. + """ + + def on_module_members(self, *, node: ast.AST | ObjectNode, mod: Module) -> None: + """Run when members of a Module have been loaded. + + Parameters: + node: The currently visited node. + mod: The module instance. + """ + + def on_class_node(self, *, node: ast.AST | ObjectNode) -> None: + """Run when visiting a new class node during static/dynamic analysis. + + Parameters: + node: The currently visited node. + """ + + def on_class_instance(self, *, node: ast.AST | ObjectNode, cls: Class) -> None: + """Run when a Class has been created. + + Parameters: + node: The currently visited node. + cls: The class instance. + """ + + def on_class_members(self, *, node: ast.AST | ObjectNode, cls: Class) -> None: + """Run when members of a Class have been loaded. + + Parameters: + node: The currently visited node. + cls: The class instance. + """ + + def on_function_node(self, *, node: ast.AST | ObjectNode) -> None: + """Run when visiting a new function node during static/dynamic analysis. + + Parameters: + node: The currently visited node. + """ + + def on_function_instance(self, *, node: ast.AST | ObjectNode, func: Function) -> None: + """Run when a Function has been created. + + Parameters: + node: The currently visited node. + func: The function instance. + """ + + def on_attribute_node(self, *, node: ast.AST | ObjectNode) -> None: + """Run when visiting a new attribute node during static/dynamic analysis. + + Parameters: + node: The currently visited node. + """ + + def on_attribute_instance(self, *, node: ast.AST | ObjectNode, attr: Attribute) -> None: + """Run when an Attribute has been created. + + Parameters: + node: The currently visited node. + attr: The attribute instance. + """ + + +ExtensionType = Union[VisitorExtension, InspectorExtension, Extension] class Extensions: """This class helps iterating on extensions that should run at different times.""" - def __init__(self, *extensions: Extension) -> None: + def __init__(self, *extensions: ExtensionType) -> None: """Initialize the extensions container. Parameters: @@ -107,9 +257,10 @@ def __init__(self, *extensions: Extension) -> None: """ self._visitors: dict[When, list[VisitorExtension]] = defaultdict(list) self._inspectors: dict[When, list[InspectorExtension]] = defaultdict(list) + self._extensions: list[Extension] = [] self.add(*extensions) - def add(self, *extensions: Extension) -> None: + def add(self, *extensions: ExtensionType) -> None: """Add extensions to this container. Parameters: @@ -118,8 +269,10 @@ def add(self, *extensions: Extension) -> None: for extension in extensions: if isinstance(extension, VisitorExtension): self._visitors[extension.when].append(extension) - else: + elif isinstance(extension, InspectorExtension): self._inspectors[extension.when].append(extension) + else: + self._extensions.append(extension) def attach_visitor(self, parent_visitor: Visitor) -> Extensions: """Attach a parent visitor to the visitor extensions. @@ -221,6 +374,17 @@ def after_inspection(self) -> list[InspectorExtension]: """ return self._inspectors[When.after_all] + def call(self, event: str, *, node: ast.AST | ObjectNode, **kwargs: Any) -> None: + """Call the extension hook for the given event. + + Parameters: + event: The trigerred event. + node: The AST or Object node. + **kwargs: Additional arguments like a Griffe object. + """ + for extension in self._extensions: + getattr(extension, event)(node=node, **kwargs) + builtin_extensions: set[str] = { "hybrid", @@ -238,7 +402,9 @@ def _load_extension_path(path: str) -> ModuleType: return module -def load_extension(extension: str | dict[str, Any] | Extension | type[Extension]) -> Extension: +def load_extension( + extension: str | dict[str, Any] | ExtensionType | type[ExtensionType], +) -> ExtensionType | list[ExtensionType]: """Load a configured extension. Parameters: @@ -254,28 +420,44 @@ def load_extension(extension: str | dict[str, Any] | Extension | type[Extension] An extension instance. """ ext_object = None + ext_classes = (VisitorExtension, InspectorExtension, Extension) - if isinstance(extension, (VisitorExtension, InspectorExtension)): + # If it's already an extension instance, return it. + if isinstance(extension, ext_classes): return extension - if isclass(extension) and issubclass(extension, (VisitorExtension, InspectorExtension)): + # If it's an extension class, instantiate it (without options) and return it. + if isclass(extension) and issubclass(extension, ext_classes): return extension() + # If it's a dictionary, we expect the only key to be an import path + # and the value to be a dictionary of options. if isinstance(extension, dict): import_path, options = next(iter(extension.items())) - else: # we consider it's a string + # Otherwise we consider it's an import path, without options. + else: import_path = str(extension) options = {} + # If the import path contains a colon, we split into path and class name. + if ":" in import_path: + import_path, extension_name = import_path.split(":", 1) + else: + extension_name = None + + # If the import path corresponds to a built-in extension, expand it. if import_path in builtin_extensions: import_path = f"griffe.extensions.{import_path}" + # If the import path is a path to an existing file, load it. elif os.path.exists(import_path): try: ext_object = _load_extension_path(import_path) except ImportError as error: raise ExtensionNotLoadedError(f"Extension module '{import_path}' could not be found") from error + # If the extension wasn't loaded yet, we consider the import path + # to be a Python dotted path like `package.module` or `package.module.Extension`. if not ext_object: try: ext_object = dynamic_import(import_path) @@ -284,16 +466,30 @@ def load_extension(extension: str | dict[str, Any] | Extension | type[Extension] except ImportError as error: raise ExtensionNotLoadedError(f"Error while importing extension '{import_path}': {error}") from error - if isclass(ext_object) and issubclass(ext_object, (VisitorExtension, InspectorExtension)): + # If the loaded object is an extension class, instantiate it with options and return it. + if isclass(ext_object) and issubclass(ext_object, ext_classes): return ext_object(**options) # type: ignore[misc] - try: - return ext_object.Extension(**options) # type: ignore[union-attr] - except AttributeError as error: - raise ExtensionNotLoadedError(f"Extension module '{import_path}' has no 'Extension' attribute") from error - - -def load_extensions(exts: Sequence[str | dict[str, Any] | Extension | type[Extension]]) -> Extensions: + # Otherwise the loaded object is a module, so we get the extension class by name, + # instantiate it with options and return it. + if extension_name: + try: + return getattr(ext_object, extension_name)(**options) + except AttributeError as error: + raise ExtensionNotLoadedError( + f"Extension module '{import_path}' has no '{extension_name}' attribute", + ) from error + + # No class name was specified so we search all extension classes in the module + # instantiate each with the same options, and return them. + extensions = [] + for obj in vars(ext_object).values(): + if isclass(obj) and issubclass(obj, ext_classes) and obj not in ext_classes: + extensions.append(obj) + return [ext(**options) for ext in extensions] + + +def load_extensions(exts: Sequence[str | dict[str, Any] | ExtensionType | type[ExtensionType]]) -> Extensions: """Load configured extensions. Parameters: @@ -304,5 +500,9 @@ def load_extensions(exts: Sequence[str | dict[str, Any] | Extension | type[Exten """ extensions = Extensions() for extension in exts: - extensions.add(load_extension(extension)) + ext = load_extension(extension) + if isinstance(ext, list): + extensions.add(*ext) + else: + extensions.add(ext) return extensions diff --git a/tests/test_extensions.py b/tests/test_extensions.py new file mode 100644 index 00000000..a846ff52 --- /dev/null +++ b/tests/test_extensions.py @@ -0,0 +1,132 @@ +"""Tests for the `extensions` module.""" + +from __future__ import annotations + +from typing import TYPE_CHECKING, Any + +import pytest + +from griffe.extensions import Extension, load_extensions +from griffe.tests import temporary_visited_module + +if TYPE_CHECKING: + import ast + + from griffe.agents.nodes import ObjectNode + from griffe.dataclasses import Attribute, Class, Function, Module, Object + + +class ExtensionTest(Extension): # noqa: D101 + def __init__(self, *args: Any, **kwargs: Any) -> None: # noqa: D107 + super().__init__() + self.records: list[str] = [] + self.args = args + self.kwargs = kwargs + + def on_attribute_instance(self, *, node: ast.AST | ObjectNode, attr: Attribute) -> None: # noqa: D102,ARG002 + self.records.append("on_attribute_instance") + + def on_attribute_node(self, *, node: ast.AST | ObjectNode) -> None: # noqa: D102,ARG002 + self.records.append("on_attribute_node") + + def on_class_instance(self, *, node: ast.AST | ObjectNode, cls: Class) -> None: # noqa: D102,ARG002 + self.records.append("on_class_instance") + + def on_class_members(self, *, node: ast.AST | ObjectNode, cls: Class) -> None: # noqa: D102,ARG002 + self.records.append("on_class_members") + + def on_class_node(self, *, node: ast.AST | ObjectNode) -> None: # noqa: D102,ARG002 + self.records.append("on_class_node") + + def on_function_instance(self, *, node: ast.AST | ObjectNode, func: Function) -> None: # noqa: D102,ARG002 + self.records.append("on_function_instance") + + def on_function_node(self, *, node: ast.AST | ObjectNode) -> None: # noqa: D102,ARG002 + self.records.append("on_function_node") + + def on_instance(self, *, node: ast.AST | ObjectNode, obj: Object) -> None: # noqa: D102,ARG002 + self.records.append("on_instance") + + def on_members(self, *, node: ast.AST | ObjectNode, obj: Object) -> None: # noqa: D102,ARG002 + self.records.append("on_members") + + def on_module_instance(self, *, node: ast.AST | ObjectNode, mod: Module) -> None: # noqa: D102,ARG002 + self.records.append("on_module_instance") + + def on_module_members(self, *, node: ast.AST | ObjectNode, mod: Module) -> None: # noqa: D102,ARG002 + self.records.append("on_module_members") + + def on_module_node(self, *, node: ast.AST | ObjectNode) -> None: # noqa: D102,ARG002 + self.records.append("on_module_node") + + def on_node(self, *, node: ast.AST | ObjectNode) -> None: # noqa: D102,ARG002 + self.records.append("on_node") + + +@pytest.mark.parametrize( + "extension", + [ + # with module path + "tests.test_extensions", + {"tests.test_extensions": {"option": 0}}, + # with extension path + "tests.test_extensions.ExtensionTest", + {"tests.test_extensions.ExtensionTest": {"option": 0}}, + # with filepath + "tests/test_extensions.py", + {"tests/test_extensions.py": {"option": 0}}, + # with filepath and extension name + "tests/test_extensions.py:ExtensionTest", + {"tests/test_extensions.py:ExtensionTest": {"option": 0}}, + # with instance + ExtensionTest(option=0), + # with class + ExtensionTest, + ], +) +def test_loading_extensions(extension: str | dict[str, dict[str, Any]] | Extension | type[Extension]) -> None: + """Test the extensions loading mechanisms. + + Parameters: + extension: Extension specification (parametrized). + """ + extensions = load_extensions([extension]) + loaded: ExtensionTest = extensions._extensions[0] # type: ignore[assignment] + # We cannot use isinstance here, + # because loading from a filepath drops the parent `tests` package, + # resulting in a different object than the present ExtensionTest. + assert loaded.__class__.__name__ == "ExtensionTest" + if isinstance(extension, (dict, ExtensionTest)): + assert loaded.kwargs == {"option": 0} + + +def test_extension_events() -> None: + """Test events triggering.""" + extension = ExtensionTest() + with temporary_visited_module( + """ + attr = 0 + def func(): ... + class Class: + cattr = 1 + def method(self): ... + """, + extensions=load_extensions([extension]), + ): + pass + events = [ + "on_attribute_instance", + "on_attribute_node", + "on_class_instance", + "on_class_members", + "on_class_node", + "on_function_instance", + "on_function_node", + "on_instance", + "on_members", + "on_module_instance", + "on_module_members", + "on_module_node", + "on_node", + ] + assert set(events) == set(extension.records)