Port `Field` to Rust #19143

stuhood · 2023-05-24T22:55:00Z

This change ports Field to Rust.

One important takeaway was that pyo3 gives us less of an advantage when subclassing is in use, because most methods of the class need to accept themselves as PyCell or PyRef (in order to gain access to their subclass instance, which isn't a member of the wrapped struct). Those methods can then only be called via pyo3 as well.

Despite that, I'll likely still port Target to Rust for consistency, and because it has some longer methods which I expect to see actual performance benefit from porting.

Eric-Arellano

Thanks! Any benefit to porting the Field subclasses like BoolField?

stuhood · 2023-05-26T16:39:46Z

Thanks! Any benefit to porting the Field subclasses like BoolField?

Possibly, but not in the interest of making porting graph.py easier, I don't think. The Rust code won't ever need to interact with the subclasses.

stuhood · 2023-05-31T18:13:17Z

Friendly ping.

thejcannon · 2023-06-01T15:01:40Z

I'll take a look ASAP, although @huonw might have better review goggles than the Joshs 😄

Any back-of-the-envelope perf numbers to share?

thejcannon

I didn't look too closely at the Rust code. I figure the perf has to be better just from porting. And if the tests pass that's a good indicator of code soundness.

thejcannon · 2023-06-01T15:05:15Z

src/python/pants/engine/internals/native_engine.pyi

+# Type alias to express the intent that the type should be immutable and hashable. There's nothing
+# to actually enforce this, outside of convention.
+ImmutableValue = Any


FYI The Python std library (and therefore core devs) have roughly said that hashability implies immutability. Search https://docs.python.org/3/library/dataclasses.html for "mutability".

So in that regard, this could be an alias for typing.Hashable

That's potentially backwards incompatible... but assuming that mypy doesn't misbehave, it should only catch bugs? Will try it.

That causes a number of type checking errors in our code (and more in user code most likely): will save for another change.

What were the errors? That's concerning...

I didn't look into it, but there were at least 10.

My goal with these port PRs is not to change behavior (unless it makes it easier to accomplish them), since my feeling right now is that getting out of our performance hole is the most important goal.

src/python/pants/engine/internals/native_engine.pyi

thejcannon · 2023-06-01T15:10:55Z

src/python/pants/engine/target_test.py

-    with pytest.raises(FrozenInstanceError):
+    with pytest.raises(AttributeError):


oh bummer...

stuhood · 2023-06-01T17:10:55Z

Any back-of-the-envelope perf numbers to share?

No, haven't done any benchmarking... I'm really not expecting anything here though, given how tiny the various functions were.

thejcannon · 2023-06-01T17:35:23Z

(FYI force-pushing means I can't review just what's changed since my last review. Usually in order to not overwhelm myself I trust past-Josh's review of the code and just review the diff. In this instance I have no clue what changed since the last review 😢 )

stuhood · 2023-06-01T18:21:53Z

(FYI force-pushing means I can't review just what's changed since my last review. Usually in order to not overwhelm myself I trust past-Josh's review of the code and just review the diff. In this instance I have no clue what changed since the last review 😢 )

Nothing has changed since your last review (for the reasons in the threads): if anything had, I would have created a separate commit.

src/rust/engine/Cargo.toml

Eric-Arellano

Nice!

cognifloyd · 2023-06-03T21:13:44Z

src/rust/engine/src/externs/target.rs

+        // TODO: Should be `RequiredFieldMissingException`.
+        Err(PyValueError::new_err(format!(


Is using an error defined in python more difficult than using the built in errors? This is the only conerning thing I see in this PR (from my post-review merge). Very cool stuff!

### Internal * upgrade to Rust v1.70.0 ([#19228](#19228)) * Remove the last mentions of NO_TOOL_LOCKFILE. ([#19229](#19229)) * Upgrade Helm unittest ([#19220](#19220)) * Prepare `2.17.0rc0`. ([#19226](#19226)) * Use a trait for remote action result caching ([#19097](#19097)) * Port `Field` to Rust ([#19143](#19143)) * Upgrade `pyo3` to `0.19`. ([#19223](#19223)) * Prepare `2.16.0rc5`. ([#19221](#19221)) * internal: Create dep inference request type ([#19001](#19001)) * Avoid requiring Python when trying to install Python (using Pyenv) ([#19208](#19208)) * Fix secondary ownership warning semantics ([#19191](#19191)) * Bump os_pipe from 1.1.3 to 1.1.4 in /src/rust/engine ([#19202](#19202)) * Include committer date in local version identifier of unstable builds ([#19179](#19179)) * Ensure we set the AWS region. ([#19178](#19178))

Field construction was ported to Rust in #19143, which incidentally broke the field deprecation check. As a result, a deprecation warning was emitted even if the user was not setting the field explicitly.

stuhood added the category:internal CI, fixes for not-yet-released features, etc. label May 24, 2023

stuhood force-pushed the stuhood/field-to-rust branch 3 times, most recently from bd168c2 to 000f6af Compare May 25, 2023 16:36

stuhood requested review from jriddy, thejcannon and Eric-Arellano May 25, 2023 16:53

stuhood marked this pull request as ready for review May 25, 2023 16:53

Eric-Arellano reviewed May 25, 2023

View reviewed changes

stuhood enabled auto-merge (squash) May 26, 2023 16:39

thejcannon disabled auto-merge May 31, 2023 19:09

thejcannon enabled auto-merge (squash) May 31, 2023 19:10

thejcannon disabled auto-merge June 1, 2023 15:13

thejcannon approved these changes Jun 1, 2023

View reviewed changes

stuhood force-pushed the stuhood/field-to-rust branch from 000f6af to 8b22bb6 Compare June 1, 2023 17:09

stuhood enabled auto-merge (squash) June 1, 2023 17:09

thejcannon disabled auto-merge June 1, 2023 17:34

huonw approved these changes Jun 1, 2023

View reviewed changes

src/rust/engine/Cargo.toml Outdated Show resolved Hide resolved

stuhood added 3 commits June 1, 2023 12:35

Upgrade pyo3 to 0.19.

b353a79

Swap to the native implementation of Field.

06cafe0

Fix test format string.

1c16922

stuhood force-pushed the stuhood/field-to-rust branch from 8b22bb6 to 1c16922 Compare June 1, 2023 19:56

Review feedback.

56ff309

Eric-Arellano approved these changes Jun 1, 2023

View reviewed changes

stuhood enabled auto-merge (squash) June 1, 2023 20:56

stuhood merged commit d97e4ef into pantsbuild:main Jun 1, 2023

stuhood deleted the stuhood/field-to-rust branch June 1, 2023 21:19

wisechengyi mentioned this pull request Jun 3, 2023

Prep 2.18.0.dev1 #19233

Merged

cognifloyd reviewed Jun 3, 2023

View reviewed changes

This was referenced Oct 12, 2023

Deprecate the platforms field on pex_binary. #20012

Merged

Fix broken target field deprecation check. #20023

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Port `Field` to Rust #19143

Port `Field` to Rust #19143

stuhood commented May 24, 2023 •

edited

Loading

Eric-Arellano left a comment

stuhood commented May 26, 2023

stuhood commented May 31, 2023

thejcannon commented Jun 1, 2023

thejcannon left a comment

thejcannon Jun 1, 2023

stuhood Jun 1, 2023 •

edited

Loading

stuhood Jun 1, 2023

thejcannon Jun 1, 2023

stuhood Jun 1, 2023

thejcannon Jun 1, 2023

stuhood commented Jun 1, 2023

thejcannon commented Jun 1, 2023

stuhood commented Jun 1, 2023

Eric-Arellano left a comment

cognifloyd Jun 3, 2023

		with pytest.raises(FrozenInstanceError):
		with pytest.raises(AttributeError):

		// TODO: Should be `RequiredFieldMissingException`.
		Err(PyValueError::new_err(format!(

Port Field to Rust #19143

Port Field to Rust #19143

Conversation

stuhood commented May 24, 2023 • edited Loading

Eric-Arellano left a comment

Choose a reason for hiding this comment

stuhood commented May 26, 2023

stuhood commented May 31, 2023

thejcannon commented Jun 1, 2023

thejcannon left a comment

Choose a reason for hiding this comment

thejcannon Jun 1, 2023

Choose a reason for hiding this comment

stuhood Jun 1, 2023 • edited Loading

Choose a reason for hiding this comment

stuhood Jun 1, 2023

Choose a reason for hiding this comment

thejcannon Jun 1, 2023

Choose a reason for hiding this comment

stuhood Jun 1, 2023

Choose a reason for hiding this comment

thejcannon Jun 1, 2023

Choose a reason for hiding this comment

stuhood commented Jun 1, 2023

thejcannon commented Jun 1, 2023

stuhood commented Jun 1, 2023

Eric-Arellano left a comment

Choose a reason for hiding this comment

cognifloyd Jun 3, 2023

Choose a reason for hiding this comment

Port `Field` to Rust #19143

Port `Field` to Rust #19143

stuhood commented May 24, 2023 •

edited

Loading

stuhood Jun 1, 2023 •

edited

Loading