Add new `visibility` field to targets to limit target dependees. #15803

kaos · 2022-06-10T21:18:10Z

Example

python_sources(
  ...,
  visibility=["<private>"]  # will only be importable from this subtree. Default is `["<public>"]`.
)

resources(
  ...,
  visibility=["src/python/some/lib/", "<private>"]  # will be private, except for `some/lib` which will be granted access.
)

Example error message:

pants.engine.target.VisibilityViolationError: src/python/pants/backend/docker/goals/tailor.py is not visible to src/python/pants/backend/docker/register.py.

Visibility for python_source src/python/pants/backend/docker/goals/tailor.py : <private>.

Also introduces a new `defaults` target which allows to dynamically provide "scoped" default field values. Closes pantsbuild#13393. [ci skip-rust] [ci skip-build-wheels]

thejcannon · 2022-06-10T22:57:18Z

Did you just implement #13767? 😐

kaos · 2022-06-11T04:36:11Z

Did you just implement #13767? 😐

Oh, yeah this would need some more work, but yes pretty much ;)

kaos · 2022-06-11T06:07:54Z

However I feel the defaults may be a more involved topic better suited in a dedicated PR, so am considering limiting this one to just the visibility field and implementation. That leaves it at a sub optimal use, but could serve as a smoother introduction.

Eric-Arellano · 2022-06-13T23:19:57Z

Cool!! I agree it would be better to have this only focus on visibility and leave off defaults

stuhood

Thanks!

stuhood · 2022-06-15T16:57:45Z

src/python/pants/engine/internals/visibility.py

+class ValidateVisibilityRulesRequest(ValidateDependenciesRequest):
+    field_set_type = ValidateVisibilityRulesFieldSet


Technically ValidateDependenciesRequest operates on the consuming target, rather than on the producing target. That means that if someone defined a consuming target which didn't have the visibility field, the producer's visibility wouldn't be validated. I'm not really sure how ValidateDependenciesRequest could be adapted to solve that.

I don't think that that is worth worrying too much about, but maybe worth making a note of here.

We could have a check that enforces all Target types to have a visibility field (and all that include the COMMON_TARGET_FIELDS get it).

I think this would be a potential gotcha for plugin authors, which a note here would be of moot help.

This is addressed by #15876 so that we can register the visibility field as a plugin field directly on Target and it will apply to all targets.

stuhood · 2022-06-15T17:05:14Z

src/python/pants/engine/target.py

+                return True
+            if rule == self.PRIVATE_VISIBILITY:
+                return to_path.startswith(target_path)
+            if to_path.startswith(rule):


I think that this would need to validate that rule ends in a slash as well?

I'd suggest switching both of these to

pants/src/python/pants/util/dirutil.py

Lines 44 to 48 in b98649e

def fast_relpath_optional(path: str, start: str) -> str | None:

"""A prefix-based relpath, with no normalization or support for returning `..`.

Returns None if `start` is not a directory-aware prefix of `path`.

"""

, which avoids the need for the trailing slash.

Switching the entire logic to instead use Specs for determining valid targets.

stuhood · 2022-06-15T17:06:22Z

src/python/pants/engine/internals/visibility.py

+def rules():
+    return (
+        *collect_rules(),
+        UnionRule(ValidateDependenciesRequest, ValidateVisibilityRulesRequest),


Given that this is already using a pluggable mechanism, is it worth putting it in its own backend, to avoid the overhead of checking visibility if a repo isn't using it at all?

Yeah, I like that.

Eric-Arellano · 2022-06-15T19:20:44Z

Awesome! My main open question is what the DSL for users should look like. For example, will there be a need to only allow certain targets to consume another target? Or this should all be filesystem based?

If you're willing, I think it would be helpful to either use something like a Google Doc or GitHub discussion so that we can fully scope out the ideal user experience. What do you think?

kaos · 2022-06-15T20:31:51Z

Yeah, I considered the possibility to use specs rather than just paths, so you can single out specific targets, or just the targets at one level rather than recursive etc.

Seeing the fruitful use of a design doc for the defaults feature, I can see the benefit to use that here as well.

kaos · 2022-06-15T21:43:45Z

Start of fleshing out a design doc here: https://docs.google.com/document/d/1DdVZ4DqX7uwiYuCL_ntb17yPxyez1fbiu8K0whZ78Lg/edit#

cognifloyd · 2022-06-16T15:28:03Z

Could I use visibility to say "this module should be a leaf" (ie it can be imported by anything but must not import any of the other local modules)?

cognifloyd · 2022-06-16T16:20:31Z

Could I use a glob in the path? like visibility = ["contrib/runners/*/tests", "<private>"]

cognifloyd · 2022-06-16T16:32:56Z

Could we rename this PR? I think dependees is better than dependencies when describing how this affects targets.

- Add new `visibility` field to targets to limit target dependencies.
+ Add new `visibility` field to targets to limit target dependees.

Still todo: move into dedicated backend plugin.

kaos · 2022-06-19T20:04:28Z

Marking as draft, as there are still quite a few moving pieces before this one is ready.
Notably: #15836 and #15876.

thejcannon · 2022-06-21T19:18:38Z

src/python/pants/engine/internals/visibility.py

@@ -0,0 +1,85 @@
+# Copyright 2020 Pants project contributors (see CONTRIBUTORS.md).


Whopsie 😂

thejcannon · 2022-06-21T19:20:15Z

Not necessary for this PR, we can file a feature request ticket later. But another interesting visibility use-case is setting visibility to "test". E.g. "only test utilities or tests can depend on this". It helps me make sure no prod code depends on a test utility or a test

kaos · 2022-06-22T13:34:32Z

Since the visibility rules now simply are Specs, it would help if we could shoe-horn in this to be a feasible feature for the Specs to support selecting (or excluding) targets by type as well.

Perhaps something along the lines of:

# exclude `python_test` and `python_testutil` targets from the src/python/pants tree..
!src/python/pants|python_test,python_testutil

this would mean that you could run goals on a select subset of target types, like:

# escaping or quoting of the pipe required from the shell..
./pants fmt lint check ::\|python_sources,shell_sources

…ity rule.

kaos · 2022-07-07T20:08:31Z

I swapped out the bespoke visibility check in the explorer backend for this new feature to try it out. Could drop that switch before landing if desired.

…ckend. # Rust tests and lints will be skipped. Delete if not intended. [ci skip-rust] # Building wheels and fs_util will be skipped. Delete if not intended. [ci skip-build-wheels]

kaos · 2022-07-07T20:43:06Z

Also, I think that we'd maybe want to run some sort of benchmark using this to see if the current approach is feasible from a performance perspective. My time will become rather limited soon to pursue this further for some time (month-ish or more)

Eric-Arellano · 2022-08-03T21:41:48Z

it would help if we could shoe-horn in this to be a feasible feature for the Specs to support selecting (or excluding) targets by type as well.

That would be --filter-target-type=python_test, which now works from anywhere.

--

Sorry I have not reviewed this! Is this still ready for review?

kaos · 2022-08-21T02:18:02Z

Sorry I have not reviewed this!

No worries.

Is this still ready for review?

I believe so. Although I'm not setup for dev work on my new machine just yet.

kaos · 2022-10-28T12:52:42Z

I have a new angle for this feature. I'll also sync some with @benjyw who've expressed an interest in tackling this feature, and update the design doc going forward.

benjyw · 2022-10-28T15:36:23Z

A GitHub Discussion linking to the design doc would be great!

kaos · 2022-10-28T17:55:10Z

#17389

Add new visibility field to targets to limit target dependencies.

096010b

Also introduces a new `defaults` target which allows to dynamically provide "scoped" default field values. Closes pantsbuild#13393. [ci skip-rust] [ci skip-build-wheels]

kaos added the category:new feature label Jun 10, 2022

kaos requested review from stuhood, thejcannon and Eric-Arellano June 10, 2022 21:18

kaos added 5 commits June 14, 2022 09:46

Merge remote-tracking branch 'upstream/main' into visibility_13393

f6ea376

Drop defaults implementation/support. Will be in a dedicated PR.

860cfb8

engine_error decorator did not catch missing exceptions.

f24fc4f

Add tests.

a7ade86

Update field help. +tweaks.

80fadc6

kaos marked this pull request as ready for review June 14, 2022 11:32

stuhood reviewed Jun 15, 2022

View reviewed changes

kaos changed the title ~~Add new visibility field to targets to limit target dependencies.~~ Add new visibility field to targets to limit target dependees. Jun 16, 2022

Merge branch 'main' into visibility_13393

6c4c2c3

kaos mentioned this pull request Jun 19, 2022

"Visibility" support (akin to Bazel's) #13393

Closed

Refactoring.

0ca0fa9

Still todo: move into dedicated backend plugin.

kaos marked this pull request as draft June 19, 2022 19:59

thejcannon reviewed Jun 21, 2022

View reviewed changes

kaos added 4 commits June 25, 2022 19:15

Merge branch 'main' into visibility_13393

176bbd8

Refactor into an experimental backend.

57d24e4

Replace bespoke explorer backend dependency validation with a visibil…

c337338

…ity rule.

Initial docs.

6cbdd02

kaos marked this pull request as ready for review June 25, 2022 19:13

kaos added 2 commits July 7, 2022 21:43

Merge branch 'main' into visibility_13393

35a86fe

Do not include target generators in the visibility validation logic.

8810fc6

Only allow the graphql server libraries to be used by the explorer ba…

33fd100

…ckend. # Rust tests and lints will be skipped. Delete if not intended. [ci skip-rust] # Building wheels and fs_util will be skipped. Delete if not intended. [ci skip-build-wheels]

Eric-Arellano mentioned this pull request Aug 9, 2022

dependency linter plugin #16446

Closed

kaos closed this Oct 28, 2022

kaos deleted the visibility_13393 branch November 24, 2022 00:45

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add new `visibility` field to targets to limit target dependees. #15803

Add new `visibility` field to targets to limit target dependees. #15803

kaos commented Jun 10, 2022 •

edited

Loading

thejcannon commented Jun 10, 2022

kaos commented Jun 11, 2022

kaos commented Jun 11, 2022

Eric-Arellano commented Jun 13, 2022

stuhood left a comment

stuhood Jun 15, 2022

kaos Jun 15, 2022

kaos Jun 19, 2022

stuhood Jun 15, 2022

kaos Jun 19, 2022

stuhood Jun 15, 2022

kaos Jun 15, 2022

Eric-Arellano commented Jun 15, 2022

kaos commented Jun 15, 2022

kaos commented Jun 15, 2022

cognifloyd commented Jun 16, 2022

cognifloyd commented Jun 16, 2022

cognifloyd commented Jun 16, 2022 •

edited

Loading

kaos commented Jun 19, 2022

thejcannon Jun 21, 2022

kaos Jun 21, 2022

thejcannon commented Jun 21, 2022

kaos commented Jun 22, 2022

kaos commented Jul 7, 2022

kaos commented Jul 7, 2022

Eric-Arellano commented Aug 3, 2022

kaos commented Aug 21, 2022 •

edited

Loading

kaos commented Oct 28, 2022

benjyw commented Oct 28, 2022

kaos commented Oct 28, 2022

		class ValidateVisibilityRulesRequest(ValidateDependenciesRequest):
		field_set_type = ValidateVisibilityRulesFieldSet

	def fast_relpath_optional(path: str, start: str) -> str \| None:
	"""A prefix-based relpath, with no normalization or support for returning `..`.

	Returns None if `start` is not a directory-aware prefix of `path`.
	"""

		@@ -0,0 +1,85 @@
		# Copyright 2020 Pants project contributors (see CONTRIBUTORS.md).

Add new visibility field to targets to limit target dependees. #15803

Add new visibility field to targets to limit target dependees. #15803

Conversation

kaos commented Jun 10, 2022 • edited Loading

thejcannon commented Jun 10, 2022

kaos commented Jun 11, 2022

kaos commented Jun 11, 2022

Eric-Arellano commented Jun 13, 2022

stuhood left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Eric-Arellano commented Jun 15, 2022

kaos commented Jun 15, 2022

kaos commented Jun 15, 2022

cognifloyd commented Jun 16, 2022

cognifloyd commented Jun 16, 2022

cognifloyd commented Jun 16, 2022 • edited Loading

kaos commented Jun 19, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

thejcannon commented Jun 21, 2022

kaos commented Jun 22, 2022

kaos commented Jul 7, 2022

kaos commented Jul 7, 2022

Eric-Arellano commented Aug 3, 2022

kaos commented Aug 21, 2022 • edited Loading

kaos commented Oct 28, 2022

benjyw commented Oct 28, 2022

kaos commented Oct 28, 2022

Add new `visibility` field to targets to limit target dependees. #15803

Add new `visibility` field to targets to limit target dependees. #15803

kaos commented Jun 10, 2022 •

edited

Loading

cognifloyd commented Jun 16, 2022 •

edited

Loading

kaos commented Aug 21, 2022 •

edited

Loading