Expose rules from language backends with an application to python_dist() creation #5747

cosmicexplorer · 2018-04-25T20:39:19Z

Problem

It's not possible to use rules defined outside of src/python/pants/engine in the rest of Pants.

Solution

Modify load_backend() in extension_loader.py to understand a rules entrypoint. If a backend has a method named rules() defined in its register.py, that method will be invoked to get a list of rules to make the scheduler with.

Result

We can now use rules defined in language backends. I created a register.py in the native backend so that we can test that this works (it's consumed in BuildLocalPythonDistributions). I have a diff that expands the native backend which depends on this functionality.

Note: Extending this to cover plugins as well is trivial (make the exact same change, but to load_plugins() instead).

stuhood

This looks great... thanks!

stuhood · 2018-04-25T20:45:27Z

src/python/pants/option/global_options.py

@@ -71,6 +71,8 @@ def register_bootstrap_options(cls, register):
             default=['pants.backend.graph_info',
                      'pants.backend.python',
                      'pants.backend.jvm',
+                      'pants.backend.native',
+                      'pants.backend.python',


python is already in the list.

Fixed in 241692f! Thanks.

stuhood · 2018-04-25T20:50:33Z

src/python/pants/backend/python/tasks/build_local_python_distributions.py

@@ -103,6 +103,9 @@ def _copy_sources(self, dist_tgt, dist_target_dir):
                                  src_relative_to_target_base)
      shutil.copyfile(abs_src_path, src_rel_to_results_dir)

+  def _request_single(self, product, subject):
+    return self.context._scheduler.product_request(product, [subject])[0]


This technically is not supposed to be exposed yet... should leave a comment here pointing to #5440.

Left a comment pointing to #4769 after offline discussion in 241692f, let me know if that's what you were thinking.

baroquebobcat

This is really cool! I'd like to see some more tests of the loading functionality

baroquebobcat · 2018-04-25T20:56:28Z

src/python/pants/bin/engine_initializer.py

@@ -169,11 +174,14 @@ def setup_legacy_graph(pants_ignore_patterns,

    # Create a Scheduler containing graph and filesystem tasks, with no installed goals. The
    # LegacyBuildGraph will explicitly request the products it needs.
+    if not rules:
+      rules = []


This param defaulting feels like it should live above the comment about creating the scheduler. Maybe after if not build_file_aliases

Moved to after if not build_file_aliases in 4cb2a18!

baroquebobcat · 2018-04-25T20:57:16Z

src/python/pants/build_graph/build_configuration.py

+    """
+
+    if not isinstance(rules, Iterable):
+      raise TypeError('The rules must be an iterable, given {}'.format(rules))


Might be good to explicitly use the repr by changing {} to {!r}.

Done in 4cb2a18!

baroquebobcat · 2018-04-25T21:00:02Z

src/python/pants/backend/native/register.py

+from pants.util.osutil import get_normalized_os_name
+
+
+class Platform(datatype('Platform', ['normed_os_name'])):


maybe use normalized_os_name?

Done in 056c125!

baroquebobcat · 2018-04-25T21:05:31Z

src/python/pants/init/extension_loader.py

@@ -144,4 +144,8 @@ def invoke_entrypoint(name):
  if subsystems:
    build_configuration.register_subsystems(subsystems)

+  rules = invoke_entrypoint('rules')
+  if rules:
+    build_configuration.register_rules(rules)


I'd like to see some unit tests for this in test_extension_loader.py

It's only 3 lines long... seems like overkill?

Fair. My thought was that as this is a plugin-writer facing change, it should have tests. If we're currently thinking of it as being experimental, then it's fine without. It'd be nice if that were noted somehow though.

It's definitely covered by tests, in that the rest of this change relies on it working... it just doesn't seem worth a unit test. It's definitely not code with high cyclomatic complexity, for example.

I totally agree that it's important to test anything that goes out to plugin writers, but as far as I am aware this change only affects backends within the pants codebase. I did however write an extremely small test anyway (see here). The natural extension of this PR is to do the same for plugins, and I absolutely think that change be paired with testprojects using it and integration tests as well, probably, but for this PR I think passing the python dist testing that exists is sufficient to prove it works.

cosmicexplorer · 2018-04-26T00:30:02Z

CI is failing at compiling native code and the error message isn't visible because pex decodes the stdout/stderr to ascii in their Installer class (which is the right thing to do there I think?), but that throws. I'm 99% sure I know what the failure is regardless and will fix that.

cosmicexplorer · 2018-04-26T09:15:55Z

Ok, so setting aside the above, CI is also failing when building the engine for either platform with (on osx):

[=== 05:49 Installing and testing package pantsbuild.pants-1.7.0.dev1+3e5b23a3 ===]
Exception caught: (<type 'exceptions.ValueError'>)
  File "/var/folders/vy/rcv48w3j4w79llzf_x6qnvw40000gn/T/pants.XXXXX.7MLVcUnZ/bin/pants", line 11, in <module>
    sys.exit(main())
  File "/var/folders/vy/rcv48w3j4w79llzf_x6qnvw40000gn/T/pants.XXXXX.7MLVcUnZ/lib/python2.7/site-packages/pants/bin/pants_loader.py", line 69, in main
    PantsLoader.run()
  File "/var/folders/vy/rcv48w3j4w79llzf_x6qnvw40000gn/T/pants.XXXXX.7MLVcUnZ/lib/python2.7/site-packages/pants/bin/pants_loader.py", line 65, in run
    cls.load_and_execute(entrypoint)
  File "/var/folders/vy/rcv48w3j4w79llzf_x6qnvw40000gn/T/pants.XXXXX.7MLVcUnZ/lib/python2.7/site-packages/pants/bin/pants_loader.py", line 58, in load_and_execute
    entrypoint_main()
  File "/var/folders/vy/rcv48w3j4w79llzf_x6qnvw40000gn/T/pants.XXXXX.7MLVcUnZ/lib/python2.7/site-packages/pants/bin/pants_exe.py", line 39, in main
    PantsRunner(exiter, start_time=start_time).run()
  File "/var/folders/vy/rcv48w3j4w79llzf_x6qnvw40000gn/T/pants.XXXXX.7MLVcUnZ/lib/python2.7/site-packages/pants/bin/pants_runner.py", line 53, in run
    return runner.run()
  File "/var/folders/vy/rcv48w3j4w79llzf_x6qnvw40000gn/T/pants.XXXXX.7MLVcUnZ/lib/python2.7/site-packages/pants/bin/local_pants_runner.py", line 49, in run
    self._run()
  File "/var/folders/vy/rcv48w3j4w79llzf_x6qnvw40000gn/T/pants.XXXXX.7MLVcUnZ/lib/python2.7/site-packages/pants/bin/local_pants_runner.py", line 95, in _run
    self._exiter).setup()
  File "/var/folders/vy/rcv48w3j4w79llzf_x6qnvw40000gn/T/pants.XXXXX.7MLVcUnZ/lib/python2.7/site-packages/pants/bin/goal_runner.py", line 201, in setup
    goals, context = self._setup_context()
  File "/var/folders/vy/rcv48w3j4w79llzf_x6qnvw40000gn/T/pants.XXXXX.7MLVcUnZ/lib/python2.7/site-packages/pants/bin/goal_runner.py", line 172, in _setup_context
    self._global_options.subproject_roots
  File "/var/folders/vy/rcv48w3j4w79llzf_x6qnvw40000gn/T/pants.XXXXX.7MLVcUnZ/lib/python2.7/site-packages/pants/bin/goal_runner.py", line 114, in _init_graph
    include_trace_on_error=self._options.for_global_scope().print_exception_stacktrace
  File "/var/folders/vy/rcv48w3j4w79llzf_x6qnvw40000gn/T/pants.XXXXX.7MLVcUnZ/lib/python2.7/site-packages/pants/bin/engine_initializer.py", line 188, in setup_legacy_graph
    scheduler = LocalScheduler(workdir, dict(), tasks, project_tree, native, include_trace_on_error=include_trace_on_error)
  File "/var/folders/vy/rcv48w3j4w79llzf_x6qnvw40000gn/T/pants.XXXXX.7MLVcUnZ/lib/python2.7/site-packages/pants/engine/scheduler.py", line 339, in __init__
    self._scheduler.assert_ruleset_valid()
  File "/var/folders/vy/rcv48w3j4w79llzf_x6qnvw40000gn/T/pants.XXXXX.7MLVcUnZ/lib/python2.7/site-packages/pants/engine/scheduler.py", line 129, in assert_ruleset_valid
    raise ValueError(str(value))
Exception message: Rules with errors: 1
  (SetupPyInvocationEnvironment, (Select(Platform), Select(NativeToolchain)), get_setup_py_env):
    no matches for Select(NativeToolchain) with subject types: PathGlobs, Address, Specs, BuildFileAddress, BuildFileAddresses, ExecuteProcessRequest
    no matches for Select(Platform) with subject types: Specs, BuildFileAddress, ExecuteProcessRequest, BuildFileAddresses, Address, PathGlobs
'pants list src::' failed in venv!
Dry run release failed.
The command "./pants --version && ./build-support/bin/release.sh -n" exited with 1

This rule is not exhibiting this issue locally (on osx) in the python dist integration test, for example (which completes successfully), but when I run that command above locally (on osx) I get the same error (not ./pants list src::, that works fine, but the release script). Why might this ruleset be marked invalid here and not elsewhere? Platform is provided as a SingletonRule and NativeToolchain as a RootRule in src/python/pants/backend/native/register.py.

baroquebobcat

Thanks for the changes. Looks good, modulo passing CI.

baroquebobcat · 2018-04-26T16:28:18Z

src/python/pants/backend/native/register.py

+from pants.util.osutil import get_normalized_os_name
+
+
+class Platform(datatype('Platform', ['normalized_os_name'])):


I think this should have a docstring, if you've got a good idea for one handy.

I do -- this datatype was ripped from my real dev branch here which has a ton more changes (and Platform isn't defined in register.py, for example), so I was a little hasty. I want to leave the docstring off for now because I'm going to move this class out of register.py for sure immediately after this pr gets merged (and keeping the docstring out makes it less legitimate).

Does that make sense? I can totally add the docstring too with great ease, just wanted to get your thoughts on my thoughts.

That works for me.

stuhood approved these changes Apr 25, 2018

View reviewed changes

baroquebobcat reviewed Apr 25, 2018

View reviewed changes

baroquebobcat approved these changes Apr 26, 2018

View reviewed changes

cosmicexplorer mentioned this pull request Apr 27, 2018

Expand engine readme #5759

Merged

cosmicexplorer added 16 commits April 27, 2018 00:01

add some simple examples to demonstrate how to use backend rules

8fd04c2

...actually make the changes to consume backend rules in register.py

258c767

expand a couple docstrings

c3dc339

revert accidental change to a test target

2b7c33d

remove extraneous log statement

5620ce6

fix lint errors

d421e5a

respond to review comments

9ac7e0f

add unit test and change field name

8f5ee6a

fix import order

5d847a4

respond to review comments

efe75b8

add native backend to release.sh

c0b4b36

isolate native toolchain path and hope for the best

6a86987

add backend plugin dependency (intentionally named differently)

a39f21d

explicitly set CC and CXX because distutils

26cc5dd

add LDSHARED to setup.py environment for fun

4ff80a2

try using gcc on travis

1c5218b

cosmicexplorer force-pushed the expose-rules-from-language-backends branch from 1418e08 to 94f68f5 Compare April 27, 2018 07:07

cosmicexplorer added 7 commits April 27, 2018 00:09

fix datatype() declarations

94f68f5

fix old datatype usages

1c7e08d

revert whitespace edit

152e322

unclear why these are necessary

514c1de

replicate the previous behavior

1cfe416

try again to isolate the path, this time using ARCHFLAGS

32b39e3

fix arguments for edited setup py env rule

14eeaff

restore the previous functionality again

7236837

wisechengyi merged commit f9b2ed7 into pantsbuild:master Apr 28, 2018

This was referenced Jun 15, 2018

C/C++ targets which can be compiled/linked and used in python_dist() with ctypes #5815

Merged

Let plugins declare @rules and move the native backend into contrib #5970

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Expose rules from language backends with an application to python_dist() creation #5747

Expose rules from language backends with an application to python_dist() creation #5747

cosmicexplorer commented Apr 25, 2018

stuhood left a comment

stuhood Apr 25, 2018

cosmicexplorer Apr 25, 2018

stuhood Apr 25, 2018

cosmicexplorer Apr 25, 2018

baroquebobcat left a comment

baroquebobcat Apr 25, 2018

cosmicexplorer Apr 26, 2018

baroquebobcat Apr 25, 2018

cosmicexplorer Apr 26, 2018

baroquebobcat Apr 25, 2018

cosmicexplorer Apr 26, 2018

baroquebobcat Apr 25, 2018

stuhood Apr 25, 2018

baroquebobcat Apr 25, 2018

stuhood Apr 25, 2018

cosmicexplorer Apr 26, 2018

baroquebobcat Apr 26, 2018

cosmicexplorer commented Apr 26, 2018 •

edited

Loading

cosmicexplorer commented Apr 26, 2018

baroquebobcat left a comment

baroquebobcat Apr 26, 2018

cosmicexplorer Apr 26, 2018

baroquebobcat Apr 26, 2018

		from pants.util.osutil import get_normalized_os_name


		class Platform(datatype('Platform', ['normed_os_name'])):

Expose rules from language backends with an application to python_dist() creation #5747

Expose rules from language backends with an application to python_dist() creation #5747

Conversation

cosmicexplorer commented Apr 25, 2018

Problem

Solution

Result

stuhood left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

baroquebobcat left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cosmicexplorer commented Apr 26, 2018 • edited Loading

cosmicexplorer commented Apr 26, 2018

baroquebobcat left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cosmicexplorer commented Apr 26, 2018 •

edited

Loading