Make ExternalTool exportable (#21258)

This MR wires `ExternalTool`s into the export machinery.

It exposes them under a separate cli arg `--bin`. Although it uses some
of the same machinery as `--resolve`, there are several differences that
I think support a separate flag (and some separate internal):
1. `ExternalTool`s don't have a resolve. They therefore must not show up
for generating lockfiles. We have to implement this separation
interally, so we should probably surface that.
2. They would be invoked directly (instead of `source
dist/export/.../activate`)
3. We can put them all in a folder "dist/export/bin" so people can get
them all

Closes #21251 

Bikeshedding:
- [x] Do we like `--bin` (yes)
- [x] Is putting all the bins in the same dir what we want (yes, we want
a "bin" folder with all the binaries. But we have to put each binary's
digest in its own folder to prevent collisions of supporting files)
- [-] Do we want to make all TemplatedExternalTools exportable? We could
extend Subsystem.rules and add the UnionRule there (deferred from this
MR)

---------

Co-authored-by: Huon Wilson <wilson.huon@gmail.com>

docs/docs/using-pants/setting-up-an-ide.mdx

@@ -51,6 +51,10 @@ The `--py-resolve-format=symlinked_immutable_virtualenv` option symlinks to an i
 
 `pants export` can also create a virtualenv for each of the Python tools you use via Pants, such as `black`, `isort`, `pytest`, `mypy`, `flake8` and so on. This allows you to configure your editor to use the same version of the tool as Pants does for workflows like formatting on save. To use a custom version of these tools, follow [the instructions for creating a tool lockfile](../python/overview/lockfiles#lockfiles-for-tools).
 
+### Binary tools
+
+`pants export` can export many tools fetched by Pants. For example, `pants export --bin=taplo`.
+
 ## Generated code
 
 If you're using [Protobuf and gRPC](../python/integrations/protobuf-and-grpc.mdx), you may want your editor to be able to index and navigate the generated source code.

docs/docs/writing-plugins/common-plugin-tasks/add-a-formatter.mdx

@@ -178,3 +178,8 @@ Now, when you run `pants fmt ::` or `pants lint ::`, your new formatter should r
 ## 4. Add tests (optional)
 
 Refer to [Testing rules](../the-rules-api/testing-plugins.mdx).
+
+
+## 5. Make the tool exportable (optional)
+
+Refer to [Allowing tool export](allowing-tool-export.mdx) to allow users to export the tool for use in external programs.

docs/docs/writing-plugins/common-plugin-tasks/add-a-linter.mdx

@@ -211,6 +211,7 @@ def rules():
     return [
       	*collect_rules(),
         ShellcheckRequest.rules(partitioner_type=PartitionerType.DEFAULT_SINGLE_PARTITION),
+        UnionRule(ExportableTool, Shellcheck),  # allows exporting the `shellcheck` binary
     ]
 ```
 
@@ -229,3 +230,7 @@ Now, when you run `pants lint ::`, your new linter should run.
 ## 4. Add tests (optional)
 
 Refer to [Testing rules](../the-rules-api/testing-plugins.mdx).
+
+## 5. Make the tool exportable (optional)
+
+Refer to [Allowing tool export](allowing-tool-export.mdx) to allow users to export the tool for use in external programs.

docs/docs/writing-plugins/common-plugin-tasks/add-codegen.mdx

@@ -19,9 +19,7 @@ This guide assumes that you are running a code generator that already exists out
 
 If you are instead writing your own code generation logic inline, you can skip Step 2. In Step 4, rather than running a `Process`, use [`CreateDigest`](../the-rules-api/file-system.mdx).
 
-1. Create a target type for the protocol
-
----
+## 1. Create a target type for the protocol
 
 You will need to define a new target type to allow users to provide metadata for their protocol files, e.g. their `.proto` files. See [Creating new targets](../the-target-api/creating-new-targets.mdx) for a guide on how to do this.
 
@@ -91,9 +89,7 @@ def rules():
 
 This example hardcodes the injected address. You can instead add logic to your rule to make this dynamic. For example, in Pants's Protobuf implementation, Pants looks for a `python_requirement` target with `protobuf`. See [protobuf/python/python_protobuf_subsystem.py](https://github.com/pantsbuild/pants/blob/main/src/python/pants/backend/codegen/protobuf/python/python_protobuf_subsystem.py).
 
-2. Install your code generator
-
----
+## 2. Install your code generator
 
 There are several ways for Pants to install your tool. See [Installing tools](../the-rules-api/installing-tools.mdx). This example will use `ExternalTool` because there is already a pre-compiled binary for Protoc.
 
@@ -130,9 +126,7 @@ class Protoc(ExternalTool):
         return "./bin/protoc"
 ```
 
-3. Create a `GenerateSourcesRequest`
-
----
+## 3. Create a `GenerateSourcesRequest`
 
 `GenerateSourcesRequest` tells Pants the `input` and the `output` of your code generator, such as going from `ProtobufSourceField -> PythonSourceField`. Pants will use this to determine when to use your code generation implementation.
 
@@ -165,9 +159,7 @@ def rules():
     ]
 ```
 
-4. Create a rule for your codegen logic
-
----
+## 4. Create a rule for your codegen logic
 
 Your rule should take as a parameter the `GenerateSourcesRequest` from Step 3 and the `Subsystem` (or `ExternalTool`) from Step 2. It should return `GeneratedSources`.
 
@@ -286,18 +278,18 @@ def rules():
 Run `pants export-codegen path/to/file.ext` to ensure Pants is correctly generating the file. This will write the generated file(s) under the `dist/` directory, using the same path that will be used during Pants runs.
 :::
 
-5. Audit call sites to ensure they've enabled codegen
-
----
+## 5. Audit call sites to ensure they've enabled codegen
 
 Call sites must opt into using codegen, and they must also specify what types of sources they're expecting. See [Rules and the Target API](../the-rules-api/rules-and-the-target-api.mdx) about `SourcesField`.
 
 For example, if you added a code generator that goes from `ProtobufSourceField -> JavaSourceField`, then Pants's Python backend would not use your new implementation because it ignores `JavaSourceField`.
 
 You should check that everywhere you're expecting is using your new codegen implementation by manually testing it out. Create a new protocol target, add it to the `dependencies` field of a target, and then run goals like `pants package` and `pants test` to make sure that the generated file works correctly.
 
-6. Add tests (optional)
-
----
+## 6. Add tests (optional)
 
 Refer to [Testing rules](../the-rules-api/testing-plugins.mdx).
+
+## 7. Make the tool exportable (optional)
+
+Refer to [Allowing tool export](allowing-tool-export.mdx) to allow users to export the tool for use in external programs.

docs/docs/writing-plugins/common-plugin-tasks/allowing-tool-export.mdx

@@ -0,0 +1,61 @@
+---
+    title: Making a tool exportable
+    sidebar_position: 10
+---
+
+How to make a tool exportable with the `export` goal.
+
+---
+
+Backends that implement the `export` goal can indicate binaries that should be exported. These will have their contents exported to a subfolder in the `dist/bins` directory, and the binary itself will be linked in `dist/bin`.
+
+## Downloadable Tools
+
+Subclasses of `ExternalTool` (including `TemplatedExternalTool`) have the logic for exporting implemented. Tools are marked for export as follows:
+
+1. Implement `ExternalTool.generate_exe` if the default is not correct. For instance, a tool downloaded might include a binary, a readme, and a license. This method will point to the binary within the downloaded files.
+
+2. Register a `UnionRule` with `ExportableTool`. For example, `UnionRule(ExportableTool, FortranLint)`
+
+## Implementing for new backends
+
+Backends need to implement:
+
+1. A subclass of `ExportRequest`
+
+```python
+@dataclass(frozen=True)
+class ExportExternalToolRequest(ExportRequest):
+    pass
+```
+
+2. A rule from this subclass to `ExportResults`
+
+```python
+@rule
+async def export_external_tools(
+    request: ExportExternalToolRequest, export: ExportSubsystem
+) -> ExportResults:
+```
+
+3. Inside of that rule, fill the `ExportResult.exported_binaries` field.
+
+```python
+ExportResult(
+    description=f"Export tool {req.resolve}",
+    reldir=dest,
+    digest=downloaded_tool.digest,
+    resolve=req.resolve,
+    exported_binaries=(ExportedBinary(name=Path(exe).name, path_in_export=exe),),
+)
+```
+
+4. For every tool, mark it for export registering a `UnionRule` with `ExportableTool`.
+
+```python
+def rules():
+    return [
+        ...,
+        `UnionRule(ExportableTool, FortranLint)`,
+    ]
+```

docs/notes/2.24.x.md

@@ -34,6 +34,12 @@ If you encounter such discrepancies, and you can't resolve them easily, please [
 
 The "legacy" system will be removed in the 2.25.x series.
 
+### Goals
+
+#### Export
+
+Many tools that Pants downloads can now be exported using [the new `export --bin` option](https://www.pantsbuild.org/2.24/reference/goals/export#bin). For example, `pants export --bin="helm"` will export the `helm` binary to `dist/export/bin/helm`. For each tool, all the files are exported to a subfolder in `dist/export/bins/`, and the main executable is linked to `dist/export/bin/`.
+
 ### Backends
 
 #### JVM

src/python/pants/backend/build_files/fmt/buildifier/rules.py

@@ -4,12 +4,14 @@
 from pants.backend.build_files.fmt.base import FmtBuildFilesRequest
 from pants.backend.build_files.fmt.buildifier.subsystem import Buildifier
 from pants.core.goals.fmt import AbstractFmtRequest, FmtResult
+from pants.core.goals.resolves import ExportableTool
 from pants.core.util_rules.external_tool import download_external_tool
 from pants.engine.internals.native_engine import MergeDigests
 from pants.engine.intrinsics import merge_digests
 from pants.engine.platform import Platform
 from pants.engine.process import Process, fallible_to_exec_result_or_raise
 from pants.engine.rules import collect_rules, implicitly, rule
+from pants.engine.unions import UnionRule
 from pants.util.logging import LogLevel
 from pants.util.strutil import pluralize
 
@@ -51,4 +53,5 @@ def rules():
     return [
         *collect_rules(),
         *BuildifierRequest.rules(),
+        UnionRule(ExportableTool, Buildifier),
     ]

src/python/pants/backend/cc/subsystems/compiler.py

@@ -6,6 +6,7 @@
 import logging
 from typing import Iterable
 
+from pants.core.goals.resolves import ExportableTool
 from pants.core.util_rules.external_tool import TemplatedExternalTool
 from pants.engine.rules import Rule, collect_rules
 from pants.engine.unions import UnionRule
@@ -147,4 +148,5 @@ def rules() -> Iterable[Rule | UnionRule]:
         *collect_rules(),
         *CCSubsystem.rules(),
         *ExternalCCSubsystem.rules(),
+        UnionRule(ExportableTool, ExternalCCSubsystem),
     )

src/python/pants/backend/codegen/protobuf/go/rules.py

@@ -9,6 +9,7 @@
 from collections import defaultdict
 from dataclasses import dataclass
 
+from pants.backend.codegen.protobuf import protoc
 from pants.backend.codegen.protobuf.protoc import Protoc
 from pants.backend.codegen.protobuf.target_types import (
     AllProtobufTargets,
@@ -644,6 +645,7 @@ def rules():
         UnionRule(GoModuleImportPathsMappingsHook, ProtobufGoModuleImportPathsMappingsHook),
         ProtobufSourcesGeneratorTarget.register_plugin_field(GoOwningGoModAddressField),
         ProtobufSourceTarget.register_plugin_field(GoOwningGoModAddressField),
+        *protoc.rules(),
         # Rules needed for this to pass src/python/pants/init/load_backends_integration_test.py:
         *assembly.rules(),
         *build_pkg.rules(),

src/python/pants/backend/codegen/protobuf/java/rules.py

@@ -5,6 +5,7 @@
 from dataclasses import dataclass
 from pathlib import PurePath
 
+from pants.backend.codegen.protobuf import protoc
 from pants.backend.codegen.protobuf.java import dependency_inference, symbol_mapper
 from pants.backend.codegen.protobuf.java.subsystem import JavaProtobufGrpcSubsystem
 from pants.backend.codegen.protobuf.protoc import Protoc
@@ -204,6 +205,7 @@ def rules():
         *collect_rules(),
         *dependency_inference.rules(),
         *symbol_mapper.rules(),
+        *protoc.rules(),
         UnionRule(GenerateSourcesRequest, GenerateJavaFromProtobufRequest),
         UnionRule(ExportableTool, JavaProtobufGrpcSubsystem),
         ProtobufSourceTarget.register_plugin_field(PrefixedJvmJdkField),

src/python/pants/backend/codegen/protobuf/lint/buf/register.py

@@ -4,7 +4,15 @@
 from pants.backend.codegen.protobuf.lint.buf import skip_field
 from pants.backend.codegen.protobuf.lint.buf.format_rules import rules as buf_format_rules
 from pants.backend.codegen.protobuf.lint.buf.lint_rules import rules as buf_lint_rules
+from pants.backend.codegen.protobuf.lint.buf.subsystem import BufSubsystem
+from pants.core.goals.resolves import ExportableTool
+from pants.engine.unions import UnionRule
 
 
 def rules():
-    return (*buf_format_rules(), *buf_lint_rules(), *skip_field.rules())
+    return (
+        *buf_format_rules(),
+        *buf_lint_rules(),
+        *skip_field.rules(),
+        UnionRule(ExportableTool, BufSubsystem),
+    )

src/python/pants/backend/codegen/protobuf/protoc.py

@@ -1,9 +1,9 @@
 # Copyright 2018 Pants project contributors (see CONTRIBUTORS.md).
 # Licensed under the Apache License, Version 2.0 (see LICENSE).
-
-
+from pants.core.goals.resolves import ExportableTool
 from pants.core.util_rules.external_tool import TemplatedExternalTool
 from pants.engine.platform import Platform
+from pants.engine.unions import UnionRule
 from pants.option.option_types import BoolOption
 
 
@@ -51,3 +51,7 @@ class Protoc(TemplatedExternalTool):
 
     def generate_exe(self, plat: Platform) -> str:
         return "./bin/protoc"
+
+
+def rules():
+    return (UnionRule(ExportableTool, Protoc),)

src/python/pants/backend/codegen/protobuf/python/rules.py

@@ -4,6 +4,7 @@
 import os
 from pathlib import PurePath
 
+from pants.backend.codegen.protobuf import protoc
 from pants.backend.codegen.protobuf.protoc import Protoc
 from pants.backend.codegen.protobuf.python.additional_fields import PythonSourceRootField
 from pants.backend.codegen.protobuf.python.grpc_python_plugin import GrpcPythonPlugin
@@ -17,6 +18,7 @@
 from pants.backend.python.util_rules import pex
 from pants.backend.python.util_rules.pex import PexResolveInfo, VenvPex, VenvPexRequest
 from pants.backend.python.util_rules.pex_environment import PexEnvironment
+from pants.core.goals.resolves import ExportableTool
 from pants.core.util_rules.external_tool import DownloadedExternalTool, ExternalToolRequest
 from pants.core.util_rules.source_files import SourceFilesRequest
 from pants.core.util_rules.stripped_source_files import StrippedSourceFiles
@@ -245,4 +247,6 @@ def rules():
         *collect_rules(),
         *pex.rules(),
         UnionRule(GenerateSourcesRequest, GeneratePythonFromProtobufRequest),
+        *protoc.rules(),
+        UnionRule(ExportableTool, GrpcPythonPlugin),
     ]

src/python/pants/backend/codegen/protobuf/scala/rules.py

@@ -5,6 +5,7 @@
 import os
 from dataclasses import dataclass
 
+from pants.backend.codegen.protobuf import protoc
 from pants.backend.codegen.protobuf.protoc import Protoc
 from pants.backend.codegen.protobuf.scala import dependency_inference, symbol_mapper
 from pants.backend.codegen.protobuf.scala.subsystem import PluginArtifactSpec, ScalaPBSubsystem
@@ -314,6 +315,7 @@ def rules():
         *symbol_mapper.rules(),
         UnionRule(GenerateSourcesRequest, GenerateScalaFromProtobufRequest),
         UnionRule(ExportableTool, ScalaPBSubsystem),
+        *protoc.rules(),
         ProtobufSourceTarget.register_plugin_field(PrefixedJvmJdkField),
         ProtobufSourcesGeneratorTarget.register_plugin_field(PrefixedJvmJdkField),
         ProtobufSourceTarget.register_plugin_field(PrefixedJvmResolveField),

src/python/pants/backend/cue/goals/fix.py

@@ -5,6 +5,7 @@
 
 from typing import Iterable
 
+from pants.backend.cue import subsystem
 from pants.backend.cue.rules import _run_cue
 from pants.backend.cue.subsystem import Cue
 from pants.backend.cue.target_types import CueFieldSet
@@ -37,4 +38,5 @@ def rules() -> Iterable[Rule]:
     return (
         *collect_rules(),
         *CueFmtRequest.rules(),
+        *subsystem.rules(),
     )

src/python/pants/backend/cue/goals/lint.py

@@ -5,6 +5,7 @@
 
 from typing import Any, Iterable
 
+from pants.backend.cue import subsystem
 from pants.backend.cue.rules import _run_cue
 from pants.backend.cue.subsystem import Cue
 from pants.backend.cue.target_types import CueFieldSet
@@ -39,4 +40,5 @@ def rules() -> Iterable[Rule]:
     return (
         *collect_rules(),
         *CueLintRequest.rules(),
+        *subsystem.rules(),
     )

src/python/pants/backend/cue/subsystem.py

@@ -3,8 +3,10 @@
 
 from __future__ import annotations
 
+from pants.core.goals.resolves import ExportableTool
 from pants.core.util_rules.external_tool import TemplatedExternalTool
 from pants.engine.platform import Platform
+from pants.engine.unions import UnionRule
 from pants.option.option_types import ArgsListOption, SkipOption
 from pants.util.strutil import help_text
 
@@ -42,3 +44,7 @@ class Cue(TemplatedExternalTool):
 
     def generate_exe(self, plat: Platform) -> str:
         return "cue"
+
+
+def rules():
+    return (UnionRule(ExportableTool, Cue),)

src/python/pants/backend/docker/lint/hadolint/rules.py

@@ -10,6 +10,7 @@
 from pants.backend.docker.subsystems.dockerfile_parser import DockerfileInfo, DockerfileInfoRequest
 from pants.backend.docker.target_types import DockerImageSourceField
 from pants.core.goals.lint import LintResult, LintTargetsRequest
+from pants.core.goals.resolves import ExportableTool
 from pants.core.util_rules.config_files import ConfigFiles, ConfigFilesRequest
 from pants.core.util_rules.external_tool import DownloadedExternalTool, ExternalToolRequest
 from pants.core.util_rules.partitions import PartitionerType
@@ -18,6 +19,7 @@
 from pants.engine.process import FallibleProcessResult, Process
 from pants.engine.rules import Get, MultiGet, collect_rules, rule
 from pants.engine.target import FieldSet, Target
+from pants.engine.unions import UnionRule
 from pants.util.logging import LogLevel
 from pants.util.strutil import pluralize
 
@@ -106,4 +108,5 @@ def rules():
     return [
         *collect_rules(),
         *HadolintRequest.rules(),
+        UnionRule(ExportableTool, Hadolint),
     ]