[5.1.0] cherrypick subpackages support (bazelbuild#14780)

* Split ConfiguredAttributeMapper into a new build target.

PiperOrigin-RevId: 411055066

* Part 1 of the Implementation for new 'subpackages()` built-in helper function.

Design proposal: https://docs.google.com/document/d/13UOT0GoQofxDW40ILzH2sWpUOmuYy6QZ7CUmhej9vgk/edit#

This CL modifies the globber infrastructure to support an additional mode of listing sub-directories.

* Add new Globber Operation enum allowing, Globber implementations to
  discriminate between glob, glob w/directories and the future sub-packages
  use-case.

* Modify UnixGlob to replace Predicate and bools with UnixGlobPathDiscriminator interface for:
  a) Determining whether to traverse a sub-directory (previously was lambda)
  b) function for determing what entries to include in the List<Path> produced by UnixGlob.globAsync.

  These allow relatively simple re-use of the same logic for both subpackages and glob

4) Add a few tests for UnixGlob to ensure both cases continue to work as expected.

PiperOrigin-RevId: 421125424

* Part 2 Implementation for new 'subpackages()` built-in helper function.

Design proposal: https://docs.google.com/document/d/13UOT0GoQofxDW40ILzH2sWpUOmuYy6QZ7CUmhej9vgk/edit#

Overview:

Add StarlarkNativeModule 'subpackages' function with parameters that mirror glob()

PiperOrigin-RevId: 422652954

* Fix some typographical errors in the 'subpackages' docs.

PiperOrigin-RevId: 425942284

Co-authored-by: kkress <kkress@google.com>
Co-authored-by: Googler <noreply@google.com>

src/main/java/com/google/devtools/build/docgen/templates/be/functions.vm

@@ -19,6 +19,7 @@ title: Functions
     <li><a href="#exports_files">exports_files</a></li>
     <li><a href="#glob">glob</a></li>
     <li><a href="#select">select</a></li>
+    <li><a href="#subpackages">subpackages</a></li>
   </ul>
 </div>
 #end
@@ -636,48 +637,54 @@ sh_binary(
   <li><code>select</code> works with most, but not all, attributes. Incompatible
   attributes are marked <code>nonconfigurable</code> in their documentation.
 
-  </li>
-</ul>
-
-By default, Bazel produces the following error when no conditions match:
-<pre class="code">
-Configurable attribute "foo" doesn't match this configuration (would a default
-condition help?).
-Conditions checked:
- //pkg:conditionA.
- //pkg:conditionB.
-</pre>
+<!-- =================================================================
+                        subpackages()
+     =================================================================
+  -->
 
-You can signal more precise errors with <code>no_match_error</code>.
+<h2 id="subpackages">subpackages</h2>
 
-<h3 id="select_example">Examples</h3>
+<pre>subpackages(include, exclude=[], allow_empty=True)</pre>
 
-<pre class="code">
-config_setting(
-    name = "windows",
-    values = {
-        "crosstool_top": "//crosstools/windows",
-    },
-)
+<p>
+  <code>subpackages()</code> is a helper function, similar to <code>glob()</code>
+  that lists subpackages instead of files and directories.  It uses the same
+  path patterns as <code>glob()</code> and can match any subpackage that is a
+  direct decendant of the currently loading BUILD file.  See <a
+  href="#glob">glob</a> for a detailed explanation and examples of include and
+  exclude patterns.
+</p>
 
-cc_binary(
-    name = "multiplatform_app",
-    ...
-    linkopts = select({
-        ":windows": [
-            "-Wl,windows_support1.lib",
-            "-Wl,windows_support2.lib",
-        ],
-        "//conditions:default": [],
-    ...
-)
-</pre>
+<p>
+  The resulting list of subpackages returned is in sorted order and contains
+  paths relative to the current loading package that match the given patterns in
+  <code>include</code> and not those in <code>exclude</code>.
 
-<p>In the above example, <code>multiplatform_app</code> links with additional
-  options when invoked with <code>bazel build //pkg:multiplatform_app
-  --crosstool_top=//crosstools/windows </code>.
+<h3 id=subpackages_example">Example</h3>
 
 <p>
+  The following example lists all the direct subpackages for the package <code>foo/BUILD</code>
+
+<pre class="code">
+# The following BUILD files exist:
+# foo/BUILD
+# foo/bar/baz/BUILD
+# foo/sub/BUILD
+# foo/sub/deeper/BUILD
+#
+# In foo/BUILD a call to
+subs = subpackages(include = ["**"])
+
+# results in subs == ["sub", "bar/baz"]
+#
+# 'sub/deeper' is not included because it is a subpackage of 'foo/sub' not of
+# 'foo'
+</pre>
+
+    <p>
+    In general it is preferred that instead of calling this function directly
+    that users use the 'subpackages' module of
+    <a href="https://github.com/bazelbuild/bazel-skylib">skylib</a>.
 
 #if (!$singlePage)
 #parse("com/google/devtools/build/docgen/templates/be/footer.vm")

src/main/java/com/google/devtools/build/lib/analysis/BUILD

@@ -400,6 +400,7 @@ java_library(
         "//src/main/java/com/google/devtools/build/lib/exec:test_xml_output_parser_exception",
         "//src/main/java/com/google/devtools/build/lib/graph",
         "//src/main/java/com/google/devtools/build/lib/packages",
+        "//src/main/java/com/google/devtools/build/lib/packages:configured_attribute_mapper",
         "//src/main/java/com/google/devtools/build/lib/packages:exec_group",
         "//src/main/java/com/google/devtools/build/lib/packages/semantics",
         "//src/main/java/com/google/devtools/build/lib/profiler",

src/main/java/com/google/devtools/build/lib/includescanning/BUILD

@@ -33,6 +33,7 @@ java_library(
         "//src/main/java/com/google/devtools/build/lib/exec:module_action_context_registry",
         "//src/main/java/com/google/devtools/build/lib/exec:spawn_strategy_resolver",
         "//src/main/java/com/google/devtools/build/lib/packages",
+        "//src/main/java/com/google/devtools/build/lib/packages:globber",
         "//src/main/java/com/google/devtools/build/lib/profiler",
         "//src/main/java/com/google/devtools/build/lib/rules/cpp",
         "//src/main/java/com/google/devtools/build/lib/skyframe:containing_package_lookup_value",

src/main/java/com/google/devtools/build/lib/includescanning/IncludeParser.java

@@ -38,6 +38,7 @@
 import com.google.devtools.build.lib.concurrent.BlazeInterners;
 import com.google.devtools.build.lib.events.Event;
 import com.google.devtools.build.lib.includescanning.IncludeParser.Inclusion.Kind;
+import com.google.devtools.build.lib.packages.Globber;
 import com.google.devtools.build.lib.packages.NoSuchPackageException;
 import com.google.devtools.build.lib.profiler.Profiler;
 import com.google.devtools.build.lib.profiler.ProfilerTask;
@@ -347,7 +348,7 @@ ImmutableSet<Artifact> getPathLevelHintedInclusions(
                   containingPackageLookupValue.getContainingPackageName(),
                   containingPackageLookupValue.getContainingPackageRoot(),
                   pattern,
-                  /*excludeDirs=*/ true,
+                  Globber.Operation.FILES,
                   relativePath.relativeTo(packageFragment)));
         } catch (InvalidGlobPatternException e) {
           env.getListener()

src/main/java/com/google/devtools/build/lib/packages/BUILD

@@ -10,21 +10,51 @@ filegroup(
     visibility = ["//src:__subpackages__"],
 )
 
+java_library(
+    name = "configured_attribute_mapper",
+    srcs = ["ConfiguredAttributeMapper.java"],
+    deps = [
+        ":packages",
+        "//src/main/java/com/google/devtools/build/lib/analysis:config/config_matching_provider",
+        "//src/main/java/com/google/devtools/build/lib/cmdline",
+        "//third_party:guava",
+        "//third_party:jsr305",
+    ],
+)
+
+java_library(
+    name = "globber",
+    srcs = ["Globber.java"],
+)
+
+java_library(
+    name = "globber_utils",
+    srcs = ["GlobberUtils.java"],
+    deps = [
+        ":globber",
+        "//third_party:error_prone_annotations",
+    ],
+)
+
 java_library(
     name = "packages",
     srcs = glob(
         ["*.java"],
         exclude = [
             "BuilderFactoryForTesting.java",  # see builder_factory_for_testing
+            "Globber.java",
+            "GlobberUtils.java",
             "ExecGroup.java",
+            "ConfiguredAttributeMapper.java",
         ],
     ),
     deps = [
         ":exec_group",
+        ":globber",
+        ":globber_utils",
         "//src/main/java/com/google/devtools/build/docgen/annot",
         "//src/main/java/com/google/devtools/build/lib/actions:execution_requirements",
         "//src/main/java/com/google/devtools/build/lib/actions:thread_state_receiver",
-        "//src/main/java/com/google/devtools/build/lib/analysis:config/config_matching_provider",
         "//src/main/java/com/google/devtools/build/lib/analysis:config/fragment",
         "//src/main/java/com/google/devtools/build/lib/analysis:config/fragment_class_set",
         "//src/main/java/com/google/devtools/build/lib/analysis:config/transitions/configuration_transition",