Introduce a native pants client. #11922

jsirois · 2021-04-14T22:05:06Z

Currently the client only handles talking to a pantsd brought up by
other means. The Pants repo ./pants script is updated to optionally use
the native client and prop up pantsd using the python client as needed.

Work towards #11831

jsirois · 2021-04-14T22:11:57Z

I've posted this PR as one big change. I'm happy to break into smaller changes if reviewers wish, but this may make more sense since the change is broken up naturally by modules, which are all ~200 lines or less and conceptually tightly constrained. To look at things in dependency order from none to some:

build_root.rs # Find the build root.
pantsd.rs # Find pantsd.
client.rs # Talk to pantsd via nails.
options/id.rs # How to identify an option.
options/parse.rs # peg parser for our args / env lists syntax.
options/mod.rs # Knits options sources together in a single options stack that's the public interface to getting options. Also defines OptionsSource which 7-9 below implement.
options/env.rs
options/args.rs
options/config.rs # Config toml parser.
main.rs

Outstanding bits in my mind:

Terminal attribute save and restore wrapped around the nailgun session.
Options system test coverage.

jsirois · 2021-04-14T22:14:01Z

Obligatory timings:

Benchmark #1: ./pants --no-anonymous-telemetry-enabled -V
  Time (mean ± σ):     701.5 ms ±   6.5 ms    [User: 482.0 ms, System: 42.8 ms]
  Range (min … max):   689.5 ms … 710.4 ms    10 runs
 
Benchmark #2: USE_NATIVE_PANTS=1 ./pants --no-anonymous-telemetry-enabled -V
  Time (mean ± σ):     256.6 ms ±   3.4 ms    [User: 63.8 ms, System: 20.3 ms]
  Range (min … max):   252.6 ms … 264.6 ms    11 runs
 
Benchmark #3: ./.pants --no-anonymous-telemetry-enabled -V
  Time (mean ± σ):     191.0 ms ±   8.6 ms    [User: 1.7 ms, System: 2.8 ms]
  Range (min … max):   182.3 ms … 217.8 ms    15 runs
 
Summary
  './.pants --no-anonymous-telemetry-enabled -V' ran
    1.34 ± 0.06 times faster than 'USE_NATIVE_PANTS=1 ./pants --no-anonymous-telemetry-enabled -V'
    3.67 ± 0.17 times faster than './pants --no-anonymous-telemetry-enabled -V'

Note that USE_NATIVE_PANTS adds ~60ms of overhead just activating the pants venv and no-oping the rust build.

The failure no-ops when the native client doesn't have a pantsd to talk to look like this:

$ ./.pants -V
The last pid for the pantsd controlling /home/jsirois/dev/pantsbuild/jsirois-pants was 202856 but it's no longer running: ESRCH: No such process
$ hyperfine './.pants -V' --ignore-failure
Benchmark #1: ./.pants -V
  Time (mean ± σ):       1.5 ms ±   0.3 ms    [User: 1.6 ms, System: 1.4 ms]
  Range (min … max):     1.1 ms …   3.9 ms    593 runs
 
  Warning: Command took less than 5 ms to complete. Results might be inaccurate.
  Warning: Ignoring non-zero exit code.
  Warning: Statistical outliers were detected. Consider re-running this benchmark on a quiet PC without any interferences from other programs. It might help to use the '--warmup' or '--prepare' options.

And that last hyperfine warning is very nice to get.

jsirois · 2021-04-14T22:18:20Z

@benjyw added you in particular for options/parse.rs. I ended up going down the parser road after a bit of fiddling with Regex and reading your py list parsing code.

src/rust/engine/client/src/options/parse.rs

jsirois · 2021-04-14T22:39:53Z

src/rust/engine/client/src/options/parse.rs

+    }
+}
+
+fn format_parse_error(


This goes through some effort to present a useful error message. For example this shows both the native client error message and the python error message since any failure currently in the native client prior to nailgun session establishment has the pants script falling through to the python client:

$ USE_NATIVE_PANTS=1 PANTS_CONFIG_FILES="-['/etc/hosts'], ?(\"/dev/null\") " ./pants -V Problem parsing PANTS_CONFIG_FILES string list value: 1:-['/etc/hosts'], 2: ?("/dev/null") ---------^ 3: Expected an optional list edit action of '+' indicating `add` or '-' indicating `remove` at line 2 column 10 The value cannot be evaluated as a literal expression: IndentationError('unexpected indent', ('<string>', 2, 9, ' ?("/dev/null")\n')) Given raw value: 1: ['/etc/hosts'], 2: ?("/dev/null") Traceback (most recent call last): File "/home/jsirois/dev/pantsbuild/jsirois-pants/src/python/pants/util/eval.py", line 47, in parse_expression parsed_value = eval(val) File "<string>", line 2 ?("/dev/null") ^ IndentationError: unexpected indent ...

Or:

$ USE_NATIVE_PANTS=1 PANTS_CONFIG_FILES="-['/etc/hosts'],+(\"/dev/null\"]" ./pants -V Problem parsing PANTS_CONFIG_FILES string list value: 1:-['/etc/hosts'],+("/dev/null"] -----------------------------^ Expected "," or the end of a tuple indicated by ')' at line 1 column 30 The value cannot be evaluated as a literal expression: SyntaxError('unexpected EOF while parsing', ('<string>', 1, 13, '("/dev/null"]')) Given raw value: 1: ("/dev/null"] Traceback (most recent call last): File "/home/jsirois/dev/pantsbuild/jsirois-pants/src/python/pants/util/eval.py", line 47, in parse_expression parsed_value = eval(val) File "<string>", line 1 ("/dev/null"] ^ SyntaxError: unexpected EOF while parsing

src/rust/engine/client/src/pantsd.rs

src/rust/engine/client/src/build_root.rs

src/rust/engine/client/src/client.rs

src/rust/engine/client/src/options/mod.rs

src/rust/engine/client/src/pantsd.rs

src/rust/engine/client/src/options/mod.rs

benjy · 2021-04-14T23:47:45Z

@benjy added you in particular for options/parse.rs. I ended up going down the parser road after a bit of fiddling with Regex and reading your py list parsing code.

This is not the benjy you're looking for :)

jsirois · 2021-04-15T00:14:02Z

Sorry @benjy. That's not the 1st and it won't be the last. It must be a bit of a curse having a common name handle on a big service.

tdyas · 2021-04-15T00:36:59Z

Sorry the rustfmt config is https://github.com/rust-lang/rustfmt/blob/master/Configurations.md#group_imports

jsirois · 2021-04-15T01:07:40Z

Ok, great @tdyas. That will be excellent, but also reformat the whole codebase. As such I'll defer that to its own disruptive change.

stuhood

Huge. Great work John!

build-support/bin/rust/bootstrap_code.sh

src/rust/engine/client/Cargo.toml

src/rust/engine/client/src/build_root.rs

src/rust/engine/client/src/pantsd.rs

src/rust/engine/client/src/options/parse.rs

src/rust/engine/client/src/options/mod.rs

src/rust/engine/client/src/options/parse.rs

src/rust/engine/client/src/pantsd.rs

benjyw · 2021-04-15T17:02:59Z

@benjyw added you in particular for options/parse.rs. I ended up going down the parser road after a bit of fiddling with Regex and reading your py list parsing code.

This is epic stuff! My rust knowledge is very entry-level, but the options-parsing part seems right to me, or at least it tracks with how the python options system behaves, esp wrt list-valued options. Ultimately the proof will be in the tests.

jsirois · 2021-04-16T16:28:36Z

Thanks for the feedback folks. I've addressed most or all of the missing feature / bug feedback. Still unit tests to fill out and a bit more feedback to address. I'll pick that all up on 4/21-4/22.

jsirois · 2021-04-22T23:26:49Z

Alright, I believe all feedback is addressed and CI is now green. I just need to take one last pass at beefing up options tests. I'll be getting to that 4/26-4/27.

Currently the client only handles talking to a pantsd brought up by other means. The Pants repo ./pants script is updated to optionally use the native client and prop up pantsd using the python client as needed. Work towards pantsbuild#11831 # Building wheels and fs_util will be skipped. Delete if not intended. [ci skip-build-wheels]

# Building wheels and fs_util will be skipped. Delete if not intended. [ci skip-build-wheels]

stuhood · 2021-06-17T17:18:09Z

Does this one have blockers?

jsirois · 2021-06-17T19:01:48Z

No external blockers. I just need time to finish tests to land it.

…/11831 [ci skip-build-wheels]

# Building wheels and fs_util will be skipped. Delete if not intended. [ci skip-build-wheels]

…/11831 [ci skip-build-wheels]

stuhood · 2021-07-02T18:25:47Z

So excited to see this land!

…/11831 [ci skip-build-wheels]

Refactor launch_pantsd into a test utility to support this. # Building wheels and fs_util will be skipped. Delete if not intended. [ci skip-build-wheels]

This found a few bugs, particularly handling escape sequences for implicit adds. # Building wheels and fs_util will be skipped. Delete if not intended. [ci skip-build-wheels]

[ci skip-build-wheels]

# Building wheels and fs_util will be skipped. Delete if not intended. [ci skip-build-wheels]

[ci skip-build-wheels]

# Building wheels and fs_util will be skipped. Delete if not intended. [ci skip-build-wheels]

jsirois · 2021-07-06T05:01:33Z

I'd still like to write tests for the toml option source and the OptionParser sandwich, but I'll be away for a while again; so I'm going to land this and circle back on that remaining test coverage.

Internal changes omitted from the release notes: * [internal] `./pants lock` uses transitive closure as inputs ([pantsbuild#12312](pantsbuild#12312)) * [internal] Add experimental `./pants lock` to generate a lockfile with pip-compile ([pantsbuild#12300](pantsbuild#12300)) * upgrade tokio and futures crates ([pantsbuild#12308](pantsbuild#12308)) * upgrade tonic, tower, and hyper crates ([pantsbuild#12294](pantsbuild#12294)) * Set up pants to use the latest version of humbug ([pantsbuild#12302](pantsbuild#12302)) * Refactor BUILD file parsing. ([pantsbuild#12279](pantsbuild#12279)) * Use get_type_hints() to get typed type hints. ([pantsbuild#12287](pantsbuild#12287)) * [internal] Fix parameters of convert_source_to_sources_test ([pantsbuild#12274](pantsbuild#12274)) * Fix assert arguments in a test ([pantsbuild#12273](pantsbuild#12273)) * Introduce a native pants client. ([pantsbuild#11922](pantsbuild#11922)) * Don't run CI in forks ([pantsbuild#12267](pantsbuild#12267)) * Change how we embed docsite links in help text. ([pantsbuild#12261](pantsbuild#12261))

Internal changes omitted from the release notes: * [internal] `./pants lock` uses transitive closure as inputs ([pantsbuild#12312](pantsbuild#12312)) * [internal] Add experimental `./pants lock` to generate a lockfile with pip-compile ([pantsbuild#12300](pantsbuild#12300)) * upgrade tokio and futures crates ([pantsbuild#12308](pantsbuild#12308)) * upgrade tonic, tower, and hyper crates ([pantsbuild#12294](pantsbuild#12294)) * Set up pants to use the latest version of humbug ([pantsbuild#12302](pantsbuild#12302)) * Refactor BUILD file parsing. ([pantsbuild#12279](pantsbuild#12279)) * Use get_type_hints() to get typed type hints. ([pantsbuild#12287](pantsbuild#12287)) * [internal] Fix parameters of convert_source_to_sources_test ([pantsbuild#12274](pantsbuild#12274)) * Fix assert arguments in a test ([pantsbuild#12273](pantsbuild#12273)) * Introduce a native pants client. ([pantsbuild#11922](pantsbuild#11922)) * Don't run CI in forks ([pantsbuild#12267](pantsbuild#12267)) * Change how we embed docsite links in help text. ([pantsbuild#12261](pantsbuild#12261)) * Add JavacSubsystem and JavacBinary, allowing user-configurable JVM selection ([pantsbuild#12283](pantsbuild#12283)) * Revert "Deprecate unused `--process-execution-local-enable-nailgun` (pantsbuild#11600)" ([pantsbuild#12257](pantsbuild#12257))

Internal changes omitted from the release notes: * [internal] `./pants lock` uses transitive closure as inputs ([#12312](#12312)) * [internal] Add experimental `./pants lock` to generate a lockfile with pip-compile ([#12300](#12300)) * upgrade tokio and futures crates ([#12308](#12308)) * upgrade tonic, tower, and hyper crates ([#12294](#12294)) * Set up pants to use the latest version of humbug ([#12302](#12302)) * Refactor BUILD file parsing. ([#12279](#12279)) * Use get_type_hints() to get typed type hints. ([#12287](#12287)) * [internal] Fix parameters of convert_source_to_sources_test ([#12274](#12274)) * Fix assert arguments in a test ([#12273](#12273)) * Introduce a native pants client. ([#11922](#11922)) * Don't run CI in forks ([#12267](#12267)) * Change how we embed docsite links in help text. ([#12261](#12261)) * Add JavacSubsystem and JavacBinary, allowing user-configurable JVM selection ([#12283](#12283)) * Revert "Deprecate unused `--process-execution-local-enable-nailgun` (#11600)" ([#12257](#12257))

@jsirois

To prepare to make the address syntax more complicated as part of #13882, move to parsing addresses using a generated `peg` parser (using the same parser crate that @jsirois introduced in #11922). Additionally, simplify `Address.spec` and `Address.path_safe_spec` slightly by removing non-trivial early returns and reducing the total number of f-strings. [ci skip-build-wheels]

The Pants native client which was introduced in #11922 has so far only been usable in the Pants repo when the `USE_NATIVE_PANTS` environment variable was set. To make the native client available to end users, this change begins distributing the native-client binary in Pants wheels. A corresponding change in the `scie-pants` repo (pantsbuild/scie-pants#172) uses the native client to run `pants`. To reduce the surface area of `scie-pants` (rather than having it be responsible for handling the special `75` exit code similar to the `pants` script integration), this PR also moves to having the native-client execute its own fallback (via `execv`) to the Pants entrypoint. In future, the `pantsbuild/pants` `pants` script could also use that facility (e.g. by specifying a separate `pants_server` bash script as the entrypoint to use as the `_PANTS_SERVER_EXE`). ---- As originally demonstrated on #11831, the native client is still much faster than the legacy client. Using pantsbuild/scie-pants#172, the timings look like: ``` Benchmark #1: PANTS_NO_NATIVE_CLIENT=true PANTS_SHA=836cceb74e6af042e7c82677f3ceb4927efce20e scie-pants-macos-x86_64 help Time (mean ± σ): 1.161 s ± 0.067 s [User: 830.6 ms, System: 79.2 ms] Range (min … max): 1.054 s … 1.309 s 10 runs Benchmark #2: PANTS_SHA=836cceb74e6af042e7c82677f3ceb4927efce20e scie-pants-macos-x86_64 help Time (mean ± σ): 271.0 ms ± 30.6 ms [User: 8.9 ms, System: 6.9 ms] Range (min … max): 241.5 ms … 360.6 ms 12 runs Summary 'PANTS_SHA=836cceb74e6af042e7c82677f3ceb4927efce20e scie-pants-macos-x86_64 help' ran 4.29 ± 0.54 times faster than 'PANTS_NO_NATIVE_CLIENT=true PANTS_SHA=836cceb74e6af042e7c82677f3ceb4927efce20e scie-pants-macos-x86_64 help' ``` Fixes #11831.

jsirois requested review from stuhood, Eric-Arellano, tdyas and benjyw April 14, 2021 22:14

jsirois commented Apr 14, 2021

View reviewed changes

src/rust/engine/client/src/options/parse.rs Show resolved Hide resolved

jsirois commented Apr 14, 2021

View reviewed changes

src/rust/engine/client/src/pantsd.rs Show resolved Hide resolved

tdyas reviewed Apr 14, 2021

View reviewed changes

src/rust/engine/client/src/options/mod.rs Outdated Show resolved Hide resolved

tdyas reviewed Apr 14, 2021

View reviewed changes

src/rust/engine/client/src/options/mod.rs Outdated Show resolved Hide resolved

stuhood approved these changes Apr 15, 2021

View reviewed changes

jsirois force-pushed the issues/11831 branch from 426eef8 to 6c7f284 Compare April 21, 2021 22:27

jsirois mentioned this pull request Apr 22, 2021

Set up group_imports = "StdExternalCrate" in rustfmt.toml #11961

Open

jsirois force-pushed the issues/11831 branch 2 times, most recently from 8f61ba1 to c994d0b Compare April 22, 2021 23:23

jsirois mentioned this pull request Apr 23, 2021

Fix python process name extraction. #11966

Merged

jsirois added 4 commits May 11, 2021 07:44

Debug macOS test failure.

a8f85da

# Building wheels and fs_util will be skipped. Delete if not intended. [ci skip-build-wheels]

Tom's feedback.

a3b9a7e

# Building wheels and fs_util will be skipped. Delete if not intended. [ci skip-build-wheels]

Fix parser precedence.

9fb47cf

# Building wheels and fs_util will be skipped. Delete if not intended. [ci skip-build-wheels]

jsirois added 2 commits June 22, 2021 12:51

Merge remote-tracking branch 'refs/remotes/upstream/main' into issues…

215e683

…/11831 [ci skip-build-wheels]

Fix shell fmt and lint.

f541ac1

# Building wheels and fs_util will be skipped. Delete if not intended. [ci skip-build-wheels]

jsirois force-pushed the issues/11831 branch from c994d0b to f541ac1 Compare June 22, 2021 20:30

jsirois added 2 commits July 1, 2021 19:42

Merge remote-tracking branch 'refs/remotes/upstream/main' into issues…

680cc46

…/11831 [ci skip-build-wheels]

Merge remote-tracking branch 'refs/remotes/upstream/main' into issues…

7f8047a

…/11831 [ci skip-build-wheels]

jsirois added 7 commits July 5, 2021 10:11

Merge remote-tracking branch 'refs/remotes/upstream/main' into issues…

cdb0807

…/11831 [ci skip-build-wheels]

Add a client test.

61e0654

Refactor launch_pantsd into a test utility to support this. # Building wheels and fs_util will be skipped. Delete if not intended. [ci skip-build-wheels]

Add parse_tests.

711885e

This found a few bugs, particularly handling escape sequences for implicit adds. # Building wheels and fs_util will be skipped. Delete if not intended. [ci skip-build-wheels]

Add Args tests.

45d4cd9

[ci skip-build-wheels]

Fix test string formatting.

4db027a

# Building wheels and fs_util will be skipped. Delete if not intended. [ci skip-build-wheels]

Add Env tests.

c87cdad

[ci skip-build-wheels]

Clean up Args tests.

6f1be0c

# Building wheels and fs_util will be skipped. Delete if not intended. [ci skip-build-wheels]

jsirois merged commit 751fe01 into pantsbuild:main Jul 6, 2021

jsirois deleted the issues/11831 branch July 6, 2021 05:02

benjyw mentioned this pull request Jul 6, 2021

Port pantsd client to rust. #11831

Closed

illicitonion mentioned this pull request Jul 9, 2021

Prep for 2.7.0.dev0 #12315

Merged

stuhood mentioned this pull request Oct 13, 2021

Add [cli.alias] config section for command line alias support. #13228

Merged

stuhood mentioned this pull request Feb 3, 2022

[internal] Parse addresses using a generated parser #14346

Merged

stuhood mentioned this pull request Aug 17, 2022

Switch pants distribution model from "python embeds rust" to "rust embeds python" #7369

Closed

stuhood mentioned this pull request May 9, 2023

Include the Pants native client in released wheels #18957

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Introduce a native pants client. #11922

Introduce a native pants client. #11922

jsirois commented Apr 14, 2021

jsirois commented Apr 14, 2021 •

edited

Loading

jsirois commented Apr 14, 2021

jsirois commented Apr 14, 2021 •

edited

Loading

jsirois Apr 14, 2021

jsirois Apr 14, 2021

benjy commented Apr 14, 2021

jsirois commented Apr 15, 2021

tdyas commented Apr 15, 2021

jsirois commented Apr 15, 2021

stuhood left a comment

benjyw commented Apr 15, 2021

jsirois commented Apr 16, 2021

jsirois commented Apr 22, 2021

stuhood commented Jun 17, 2021

jsirois commented Jun 17, 2021

stuhood commented Jul 2, 2021

jsirois commented Jul 6, 2021

Introduce a native pants client. #11922

Introduce a native pants client. #11922

Conversation

jsirois commented Apr 14, 2021

jsirois commented Apr 14, 2021 • edited Loading

jsirois commented Apr 14, 2021

jsirois commented Apr 14, 2021 • edited Loading

jsirois Apr 14, 2021

Choose a reason for hiding this comment

jsirois Apr 14, 2021

Choose a reason for hiding this comment

benjy commented Apr 14, 2021

jsirois commented Apr 15, 2021

tdyas commented Apr 15, 2021

jsirois commented Apr 15, 2021

stuhood left a comment

Choose a reason for hiding this comment

benjyw commented Apr 15, 2021

jsirois commented Apr 16, 2021

jsirois commented Apr 22, 2021

stuhood commented Jun 17, 2021

jsirois commented Jun 17, 2021

stuhood commented Jul 2, 2021

jsirois commented Jul 6, 2021

jsirois commented Apr 14, 2021 •

edited

Loading

jsirois commented Apr 14, 2021 •

edited

Loading