Adding DigestTrie/Snapshot diffing #14872

thejcannon · 2022-03-22T14:20:29Z

In order to handle the upcoming change to "unify" both fmt and lint processes for formatters, we need to have the ability to find what files changed in a formatter rule run. This PR adds a recursive diff function to the Rust DigestTrie, which is also exposed via the PySnapshot/Snapshot Python byding/type.

The diff method returns a complete set of differences (ours/theirs files/dirs + changed) as this was readily available and could be useful in other contexts. For the purpose of my future changes, I just need the changed files.

Tests have been added Python-side as that is easiest to author/grok.

[ci skip-build-wheels]

# Building wheels and fs_util will be skipped. Delete if not intended. [ci skip-build-wheels]

src/python/pants/engine/fs.py

src/python/pants/engine/fs_test.py

src/rust/engine/fs/src/directory.rs

thejcannon · 2022-03-22T14:29:55Z

src/python/pants/engine/internals/native_engine.pyi

+    # Use pants.engine.fs.SnapshotDiff for semantic names
+    def diff(


Should we just prefix this with an underscore to really ward off callers?

Yea, that might make sense. In practice, users will probably consume this via a @rule that computes a diff string in some format using difflib...? And so this would be an implementation detail of that @rule rather than something consumed directly.

The plan is for this to be called in FmtResult.message to compute the message. Well not "this" but pants.engine.fs.SnapshotDiff.from_snapshots.

# Building wheels and fs_util will be skipped. Delete if not intended. [ci skip-build-wheels]

thejcannon · 2022-03-22T16:24:43Z

The test that fails seems unrelated to my changes (the new tests pass in the output)

=================================== FAILURES ===================================
_________________________ test_workspace_in_goal_rule __________________________

    def test_workspace_in_goal_rule() -> None:
        class WorkspaceGoalSubsystem(GoalSubsystem):
            name = "workspace-goal"
    
        class WorkspaceGoal(Goal):
            subsystem_cls = WorkspaceGoalSubsystem
    
        @dataclass(frozen=True)
        class DigestRequest:
            create_digest: CreateDigest
    
>       @rule
        def digest_request_singleton() -> DigestRequest:

src/python/pants/engine/fs_test.py:996: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
src/python/pants/engine/rules.py:338: in rule
    return inner_rule(*args, **kwargs, rule_type=RuleType.rule, cacheable=True)
src/python/pants/engine/rules.py:328: in inner_rule
    return rule_decorator(*args, **kwargs)
src/python/pants/engine/rules.py:249: in rule_decorator
    type_hints = get_type_hints(func)
/opt/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/typing.py:1013: in get_type_hints
    value = _eval_type(value, globalns, localns)
/opt/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/typing.py:263: in _eval_type
    return t._evaluate(globalns, localns)
/opt/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/typing.py:467: in _evaluate
    eval(self.__forward_code__, globalns, localns),
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

>   ???
E   NameError: name 'DigestRequest' is not defined

<string>:1: NameError
- generated xml file: /tmp/process-executionbfcqNX/src.python.pants.engine.fs_test.py.tests.xml -


=========================== short test summary info ============================
FAILED src/python/pants/engine/fs_test.py::test_workspace_in_goal_rule - Name...
========================= 1 failed, 57 passed in 7.51s =========================

stuhood

Thanks, looks great!

src/python/pants/engine/fs.py

stuhood · 2022-03-22T20:15:13Z

src/python/pants/engine/internals/native_engine.pyi

+    # Use pants.engine.fs.SnapshotDiff for semantic names
+    def diff(


Yea, that might make sense. In practice, users will probably consume this via a @rule that computes a diff string in some format using difflib...? And so this would be an implementation detail of that @rule rather than something consumed directly.

src/rust/engine/fs/src/directory.rs

stuhood · 2022-03-22T21:13:17Z

The test that fails seems unrelated to my changes (the new tests pass in the output)

Related to the addition of from __future__ import annotations, perhaps...?

thejcannon · 2022-03-23T01:04:23Z

The test that fails seems unrelated to my changes (the new tests pass in the output)

Related to the addition of from __future__ import annotations, perhaps...?

Ahh! I think the rule parsing code should be flexible enough to handle this, but we can workaround it here.

# Building wheels and fs_util will be skipped. Delete if not intended. [ci skip-build-wheels]

# Rust tests and lints will be skipped. Delete if not intended. [ci skip-rust] # Building wheels and fs_util will be skipped. Delete if not intended. [ci skip-build-wheels]

stuhood

Thanks!

Adding DigestTrie/Snapshot diffing

1b5718b

# Building wheels and fs_util will be skipped. Delete if not intended. [ci skip-build-wheels]

thejcannon added the category:internal CI, fixes for not-yet-released features, etc. label Mar 22, 2022

thejcannon requested a review from stuhood March 22, 2022 14:20

Removing stale comment

25f6e34

# Building wheels and fs_util will be skipped. Delete if not intended. [ci skip-build-wheels]

thejcannon commented Mar 22, 2022

View reviewed changes

thejcannon added 2 commits March 22, 2022 09:31

Add NB

d3bdd86

# Building wheels and fs_util will be skipped. Delete if not intended. [ci skip-build-wheels]

clippy

7e4c5a1

# Building wheels and fs_util will be skipped. Delete if not intended. [ci skip-build-wheels]

stuhood approved these changes Mar 22, 2022

View reviewed changes

thejcannon added 4 commits March 22, 2022 20:09

PR comments

55458d1

# Building wheels and fs_util will be skipped. Delete if not intended. [ci skip-build-wheels]

helpers

e3c0db9

# Building wheels and fs_util will be skipped. Delete if not intended. [ci skip-build-wheels]

underscore

079a4bb

# Building wheels and fs_util will be skipped. Delete if not intended. [ci skip-build-wheels]

underscore

c7c1083

# Rust tests and lints will be skipped. Delete if not intended. [ci skip-rust] # Building wheels and fs_util will be skipped. Delete if not intended. [ci skip-build-wheels]

stuhood approved these changes Mar 23, 2022

View reviewed changes

thejcannon merged commit 681944c into pantsbuild:main Mar 23, 2022

thejcannon deleted the triediff branch March 23, 2022 16:54

benjyw mentioned this pull request Mar 26, 2022

Prepare 2.12.0.dev0. #14919

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Adding DigestTrie/Snapshot diffing #14872

Adding DigestTrie/Snapshot diffing #14872

thejcannon commented Mar 22, 2022 •

edited

Loading

thejcannon Mar 22, 2022

stuhood Mar 22, 2022

thejcannon Mar 22, 2022

thejcannon commented Mar 22, 2022

stuhood left a comment

stuhood Mar 22, 2022

stuhood commented Mar 22, 2022

thejcannon commented Mar 23, 2022

stuhood left a comment

		# Use pants.engine.fs.SnapshotDiff for semantic names
		def diff(

Adding DigestTrie/Snapshot diffing #14872

Adding DigestTrie/Snapshot diffing #14872

Conversation

thejcannon commented Mar 22, 2022 • edited Loading

thejcannon Mar 22, 2022

Choose a reason for hiding this comment

stuhood Mar 22, 2022

Choose a reason for hiding this comment

thejcannon Mar 22, 2022

Choose a reason for hiding this comment

thejcannon commented Mar 22, 2022

stuhood left a comment

Choose a reason for hiding this comment

stuhood Mar 22, 2022

Choose a reason for hiding this comment

stuhood commented Mar 22, 2022

thejcannon commented Mar 23, 2022

stuhood left a comment

Choose a reason for hiding this comment

thejcannon commented Mar 22, 2022 •

edited

Loading