Add pants-plugins/stevedore_extensions to add teach pants' dependency inference about our runtime-loaded plugins

[cognifloyd]

Background

This is another part of introducing pants, as discussed in various TSC meetings.

Related PRs can be found in:

• pantsbuild milestone (which includes PRs since Initialize pantsbuild #5713)
• pantsbuild label (which also includes preparatory PRs that came before adding pantsbuild)

Overview of this PR

We use openstack/stevedore for several dynamic (discovered at runtime) StackStorm plugins. We use this to load:

• runners,
• orquesta functions,
• sso backends,
• the metrics driver, and
• the rbac backend.

Since a stevedore extension does not use the python import system, pants does not know how to infer dependencies on any of this code. We need to extend the dependency inference system to teach it how to handle stevedore extensions. This PR adds a pants plugin that strives to accomplish that by allowing us to:

• use stevedore_extension targets to inform pants about what code is required for a given extension (where each extension is identified by a stevedore namespace and a plugin name), and
• use stevedore_namespace fields to record dependencies on extensions in the given namespace.

When a target has the stevedore_namespace field, this plugin will add dependencies on all of the stevedore_extension targets that define plugins in that namespace. For example, that means that pants will include all of our runners whenever code depends on the st2common.runners.runner namespace.

Scope of this PR

This PR only adds the stevedore_extension target and adds the stevedore_namespace field to the python_tests and python_test targets.

After adding the dependency inference (in this PR) there is one more thing required for python_test targets to successfully get access, via stevedore, to the relevant python code. We need to generate an entry_points.txt file. We will include that in the next PR, which you can preview at: pants-plugins-stevedore_extensions...pants-plugins-stevedore_extensions-entry_points_txt

Another follow-up PR will also add the stevedore_namespaces field to python_distribution targets (a wheel is a distribution). If you want a preview of that PR, you can look at: pants-plugins-stevedore_extensions-entry_points_txt...pants-plugins-stevedore_extensions-setup_py

Note, also, that I have not enabled the plugin yet:

st2/pants.toml

Line 30 in 7148058

#"stevedore_extensions",

That keeps this PR focused on the plugin itself including adding tests for it. A third follow-up PR will add the stevedore_extension target and stevedore_namespaces field metadata to our BUILD files. If you want to explore some of that metadata, you can look at: pants-plugins-stevedore_extensions-setup_py...pants-stevedore_extensions

stevedore_extension target

In the plugin, the stevedore_extension target is defined here:

st2/pants-plugins/stevedore_extensions/target_types.py

Lines 124 to 132 in 7148058

	class StevedoreExtension(Target):
	alias = "stevedore_extension"
	core_fields = (
	*COMMON_TARGET_FIELDS,
	StevedoreNamespaceField,
	StevedoreEntryPointsField,
	PythonResolveField,
	)
	help = "Entry points used to generate setuptools metadata for stevedore."

There are several fields on this target. In this example BUILD file, we see three fields: name, namespace, and entry_points. (Yes, there are other fields available on the target. But, those are only needed for other pants features; we won't use them in our BUILD files.)

st2/contrib/runners/noop_runner/BUILD

Lines 1 to 7 in b0d27d8

	stevedore_extension(
	name="runner",
	namespace="st2common.runners.runner",
	entry_points={
	"noop": "noop_runner.noop_runner",
	},
	)

Look in target_types.py to see the definition of the namespace (StevedoreNamespaceField) and entry_points (StevedoreEntryPointsField) fields. In particular, please review the help text for each of these fields.

stevedore_namespaces field (on python_tests/python_test targets)

In the plugin, the stevedore_namespaces field is defined here:

st2/pants-plugins/stevedore_extensions/target_types.py

Lines 136 to 147 in 7148058

	class StevedoreNamespacesField(StringSequenceField):
	alias = "stevedore_namespaces"
	help = softwrap(
	"""
	List the stevedore namespaces required by this target.
	All stevedore_extension targets with these namespaces will be added as
	dependencies so that they are available on PYTHONPATH during tests.
	The stevedore namespace format (my.stevedore.extension) is similar
	to a python namespace.
	"""
	)

When we identify tests that need particular stevedore extensions, we list the required namespaces in BUILD files like this:

st2/st2common/tests/unit/BUILD

Lines 6 to 13 in b0d27d8

	python_tests(
	name="tests",
	dependencies=[
	# several files import tests.unit.base which is ambiguous. Tell pants which one to use.
	"st2common/tests/unit/base.py",
	],
	stevedore_namespaces=["st2common.runners.runner"],
	)

In this example, the unit tests get a dependency on all of the runners.

Dependency inference rules

To make dependency inference add these dependencies, the pants plugin adds several rules.

Dependencies on stevedore_extension targets

Starting with the inferred dependencies, and going backwards in the rule graph, here are the rules to infer dependencies on stevedore_extension targets.

st2/pants-plugins/stevedore_extensions/python_dependency_injection.py

Lines 79 to 86 in 7148058

	@rule(
	desc="Infer stevedore_extension target dependencies for python_tests based on namespace list.",
	level=LogLevel.DEBUG,
	)
	async def infer_stevedore_namespace_dependencies(
	request: InferStevedoreNamespaceDependencies,
	stevedore_extensions: StevedoreExtensions,
	) -> InferredDependencies:

This rule needs StevedoreExtensions which comes from this rule:

st2/pants-plugins/stevedore_extensions/python_dependency_injection.py

Lines 53 to 59 in 7148058

	@rule(
	desc="Creating map of stevedore_extension namespaces to StevedoreExtension targets",
	level=LogLevel.DEBUG,
	)
	async def map_stevedore_extensions(
	stevedore_extensions: AllStevedoreExtensionTargets,
	) -> StevedoreExtensions:

Which needs AllStevedoreExtensions from this rule (which gets AllTargets from pants provided rules):

st2/pants-plugins/stevedore_extensions/target_types_rules.py

Lines 50 to 53 in 7148058

	@rule(desc="Find all StevedoreExtension targets in project", level=LogLevel.DEBUG)
	def find_all_stevedore_extension_targets(
	targets: AllTargets,
	) -> AllStevedoreExtensionTargets:

stevedore_extension targets depend on python code

The stevedore_extension targets would not be very helpful if they didn't pull in dependencies on relevant python code. So, we parse the entry_points field and infer dependencies on all of the python code that provides those entry points. Starting from

st2/pants-plugins/stevedore_extensions/target_types_rules.py

Lines 158 to 165 in 7148058

	@rule(
	desc="Inferring dependency from the stevedore_extension `entry_points` field",
	level=LogLevel.DEBUG,
	)
	async def infer_stevedore_entry_points_dependencies(
	request: InferStevedoreExtensionDependencies,
	python_setup: PythonSetup,
	) -> InferredDependencies:

In that rule, it requests ResolvedStevedoreEntryPoints from the pants engine, which pulls in this rule:

st2/pants-plugins/stevedore_extensions/target_types_rules.py

Lines 59 to 65 in 7148058

	@rule(
	desc="Determining the entry points for a `stevedore_extension` target",
	level=LogLevel.DEBUG,
	)
	async def resolve_stevedore_entry_points(
	request: ResolveStevedoreEntryPointsRequest,
	) -> ResolvedStevedoreEntryPoints:

Thus we have an inferred set of dependencies from python_test through stevedore_extension to python_source.

Developing pants plugins

Pants has extensive documentation on the plugin API, including how to create targets, how to write rules, and there's even a mini tutorial about how to add a formatter (adding formatters is a very common thing for plugins to do).

• Plugins overview
• The Target API
  • Concepts eg: defines targets and fields.
  • Creating new fields
  • Creating new targets
  • Extending existing targets
• The Rules API
• Testing Plugins

[amanda11]

A few queries.

[cognifloyd]

You can see this here: https://github.com/pantsbuild/pants/blob/2.14.x/src/python/pants/backend/python/target_types_rules.py#L146-L199

[cognifloyd]

Rebased on master

[amanda11]

Happy with the changes, just the change on the comment as per suggestion to add.

[cognifloyd]

rebased

[cognifloyd]

I submitted pantsbuild/pants#18132 to try and get this plugin into upstream pants instead of in our repo. In doing that I realized I could vastly simplify it, so I did a major refactor. Even if it's not accepted upstream, the new version of the plugin will be easier to review with fewer and smaller PRs to get it all in.

I'm marking this as draft until I see where the upstreaming effort lands.

[cognifloyd]

Woohoo! This was merged upstream: pantsbuild/pants#18132

I will revisit this once there is a pants release we can use (prob dev or rc) that includes this plugin. Then we'll be able to add all the stevedore entry points metadata without adding the plugin. In the meantime, I plan to add the python_distribution metadata for building wheels.