Remove MultiPlatform Process abstractions #12725

illicitonion · 2021-09-01T09:51:39Z

We don't actually use these anywhere - instead, always operate on a
single platform-specific or platform-independent Process.

stuhood

Thanks a lot!

stuhood · 2021-09-01T16:19:28Z

src/rust/engine/process_execution/src/remote.rs

@@ -843,16 +842,6 @@ impl crate::CommandRunner for CommandRunner {
    )
    .await
  }
-


So, one of the more important places where the MPP constraint actually used to be used... seemingly no longer exists. The speculating command runner used to skip running remotely if the remote platform didn't match the local platform.

I believe that that constraint application is still necessary... but I think that with https://docs.google.com/document/d/1hS0n6vp-hBxy4Xo-q9QA_elofzQtOubjfN4nSeIgp90/edit# , the relevant place to apply it might be in node.rs to explicitly choose between a remote and a local CommandRunner based on which matches the platform. Or perhaps it is still in a CommandRunner implementation, but immediately below a remote cache (if one is configured).

Having said that, there are a few gaps in here, and we're essentially starting over for cross-platform speculation. So fine for a followup.

stuhood · 2021-09-01T16:19:54Z

src/python/pants/backend/python/util_rules/pex.py

@@ -559,10 +553,12 @@ async def build_pex_component(
        ),
    )

+    process = dataclasses.replace(process, platform=platform)
+
    # NB: Building a Pex is platform dependent, so in order to get a PEX that we can use locally


Nit: This comment should probably be on the dataclasses.replace call.

We don't actually use these anywhere - instead, always operate on a single platform-specific or platform-independent Process. [ci skip-build-wheels]

stuhood · 2021-12-04T05:56:43Z

I'm following up to land this one to ease making Process creation async for #13787. Thanks again!

Internal changes: * [internal] Remove superfluous f-string specifiers ([#13821](#13821)) * [internal] scala: extract annotations as consumed types ([#13810](#13810)) * [jvm] Split nailgun digest from input file digests ([#13813](#13813)) * [internal] jvm: add jre_major_version and use stderr to properly extract version ([#13812](#13812)) * [internal] Clean up Go `embed` support's handling of dependencies ([#13801](#13801)) * [internal] scala: handle package object syntax in parser ([#13809](#13809)) * [internal] java: fix junit sentinel rule setup ([#13815](#13815)) * [internal] upgrade to rust v1.57.0 ([#13807](#13807)) * [internal] add failing test for FrozenDict equality issue ([#13389](#13389)) * [internal] Use `PyObject` instead of `Value` in more places ([#13802](#13802)) * Remove MultiPlatform Process abstractions ([#12725](#12725)) * [internal] add `JvmToolBase` for lockfile handling for JVM tools ([#13777](#13777)) * [internal] Port `MergeDigests` to Rust ([#13773](#13773)) * [jvm] Spawn nailgun servers outside the pool lock ([#13796](#13796)) * [internal] DRY loading internal Go binaries ([#13800](#13800)) * [internal] Convert unit tests to use pytest ([#13798](#13798)) * [internal] remove dead code - socket util. ([#13797](#13797)) * [internal] Reorganize Go parser scripts ([#13791](#13791)) * Adds Jackson core/datatype to `JVM_ARTIFACT_MAPPINGS` ([#13792](#13792)) * [internal] go: initial support for embedding resources ([#13743](#13743)) * [internal] Refer to `go.mod` path when downloading packages ([#13786](#13786)) * [internal] More robust Go dependency inference tests ([#13785](#13785)) * [internal] `tailor` doesn't add `go_package` for `testdata` folder ([#13783](#13783)) * [internal] Add Scala backend to dryrun for wheel builds. ([#13772](#13772)) * [internal] Unify JVM thirdparty resolves ([#13771](#13771)) * [internal] scala: infer dependencies from consumed symbols and intermediate scopes ([#13758](#13758)) * [internal] java: infer scala encoded symbols ([#13739](#13739)) * [internal] scala: parse and report package scopes ([#13738](#13738)) * [internal] go: configure included env vars for `GoSdkProcess` ([#13734](#13734)) * Fix some `./cargo audit` vulnerabilities. ([#13728](#13728)) * [internal] fix non-empty __init__.py ([#13730](#13730)) * Compute RepositoryPex directly from addresses. ([#13716](#13716)) * Upgrade to cargo-audit 0.16.0. ([#13729](#13729)) * Simplify `NativeHandler`. ([#13727](#13727)) * [internal] Switch to a maintained fork of the `fuse` crate for `brfs`. ([#13679](#13679)) * [internal] Add infrastructure to support deprecating field names ([#13666](#13666)) * [internal] Introduce OptionalPex/OptionalPexRequest. ([#13712](#13712)) * [internal] tailor adds go_package targets ([#13703](#13703)) * [internal] Remove unused testproject for pants-plugin ([#13704](#13704)) * [internal] Rename ambiguous `subpath` variable for Go code ([#13701](#13701)) * [internal] scala: generate the JVM names seen by Java code for Scala code ([#13696](#13696)) * Use RequirementsPexRequest in run_pex_binary.py. ([#13693](#13693)) * [internal] Refactor finding owning `go_mod` for first-party packages ([#13695](#13695)) * [internal] repl: add append_only_caches / run_in_workspace attributes to ReplRequest ([#13599](#13599)) * [internal] switch back to official `cargo-ensure-prefix` crate ([#13692](#13692)) * [internal] scala: extract type names from all Type.* AST nodes ([#13685](#13685)) * [internal] Convert unit tests to use pytest ([#13652](#13652)) * Unblock codegen support for java export analysis (#13645) ([#13675](#13675)) * [internal] upgrade to Rust 2021 Edition ([#13644](#13644)) * [internal] Don't store derived values on `go_first_party_package` targets ([#13676](#13676)) * Upgrade to py03 0.15.1. ([#13725](#13725)) * Add PyPDF2 to module mapping ([#13717](#13717)) * Upgrade console and indacatif. ([#13726](#13726))

illicitonion requested review from stuhood and tdyas September 1, 2021 09:51

illicitonion force-pushed the rm-mper branch from 46d0551 to acc90ac Compare September 1, 2021 11:13

stuhood approved these changes Sep 1, 2021

View reviewed changes

patricklaw mentioned this pull request Sep 21, 2021

Add Java dependency analysis types and launcher using javaparser library. #12890

Merged

Remove MultiPlatform Process abstractions

a515fbe

We don't actually use these anywhere - instead, always operate on a single platform-specific or platform-independent Process. [ci skip-build-wheels]

stuhood force-pushed the rm-mper branch from acc90ac to a515fbe Compare December 4, 2021 05:55

stuhood enabled auto-merge (squash) December 4, 2021 05:55

stuhood merged commit 315b3b0 into pantsbuild:main Dec 4, 2021

illicitonion mentioned this pull request Dec 6, 2021

Prep for 2.9.0.dev3 #13823

Merged

Eric-Arellano mentioned this pull request Jan 18, 2022

[internal] Add Go Protobuf compiler #13985

Merged

kaos mentioned this pull request Apr 22, 2023

Support docformatter 1.6.0+ #18790

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Remove MultiPlatform Process abstractions #12725

Remove MultiPlatform Process abstractions #12725

illicitonion commented Sep 1, 2021

stuhood left a comment

stuhood Sep 1, 2021 •

edited

Loading

stuhood Sep 1, 2021

stuhood commented Dec 4, 2021

@@ @@ -843,16 +842,6 @@ impl crate::CommandRunner for CommandRunner { @@
                   )
                   .await
                 }

Remove MultiPlatform Process abstractions #12725

Remove MultiPlatform Process abstractions #12725

Conversation

illicitonion commented Sep 1, 2021

stuhood left a comment

Choose a reason for hiding this comment

stuhood Sep 1, 2021 • edited Loading

Choose a reason for hiding this comment

stuhood Sep 1, 2021

Choose a reason for hiding this comment

stuhood commented Dec 4, 2021

stuhood Sep 1, 2021 •

edited

Loading