Add specific and actionable instructions to stale lockfile errors #12699

chrisjrn · 2021-08-30T20:46:19Z

This changes the lockfile staleness messages to include information on how to reconfigure pants to resolve the error. Language is largely similar to that proposed in #12654, with minor changes:

Language changes to explain the need for action on the user's part (you can fix this by rather than please change )

Closes #12654

# Rust tests and lints will be skipped. Delete if not intended. [ci skip-rust] # Building wheels and fs_util will be skipped. Delete if not intended. [ci skip-build-wheels]

…r messages # Rust tests and lints will be skipped. Delete if not intended. [ci skip-rust] # Building wheels and fs_util will be skipped. Delete if not intended. [ci skip-build-wheels]

# Rust tests and lints will be skipped. Delete if not intended. [ci skip-rust] # Building wheels and fs_util will be skipped. Delete if not intended. [ci skip-build-wheels]

[ci skip-rust] [ci skip-build-wheels]

# Rust tests and lints will be skipped. Delete if not intended. [ci skip-rust] # Building wheels and fs_util will be skipped. Delete if not intended. [ci skip-build-wheels]

[ci skip-rust] [ci skip-build-wheels]

chrisjrn · 2021-08-30T21:29:27Z

OK, the tests don't pass due to timing out. Those will need a rewrite; one moment

Eric-Arellano

Thank you Chris

src/python/pants/backend/python/goals/pytest_runner.py

src/python/pants/backend/python/typecheck/mypy/rules.py

src/python/pants/backend/python/util_rules/pex.py

Eric-Arellano · 2021-08-30T22:52:01Z

src/python/pants/backend/python/util_rules/pex.py

+        else:
+            yield (
+                "To regenerate your lockfile based on your current configuration, run "
+                "`./pants generate-lockfiles`. "


Suggested change

"`./pants generate-lockfiles`. "

"`./pants generate-lockfiles --resolve={tool_name}`. "

src/python/pants/backend/python/util_rules/pex.py

src/python/pants/backend/python/util_rules/pex_test.py

# Rust tests and lints will be skipped. Delete if not intended. [ci skip-rust] # Building wheels and fs_util will be skipped. Delete if not intended. [ci skip-build-wheels]

chrisjrn · 2021-08-31T16:31:27Z

I think I've caught everything here. Let me know if you've got any questions.

# Rust tests and lints will be skipped. Delete if not intended. [ci skip-rust] # Building wheels and fs_util will be skipped. Delete if not intended. [ci skip-build-wheels]

chrisjrn · 2021-08-31T18:15:46Z

src/python/pants/backend/python/util_rules/pex.py

+        elif isinstance(requirements, ToolCustomLockfile):
+            yield f"the lockfile at {requirements.file_path} "
+        else:
+            raise Exception("Check the type hinting on `requirements`")


type hints on requirements say that this line should never be hit, but I'm not sure mypy agrees.

Better is raise AssertionError. And yeah this is good to keep. Although note that it will fail much earlier on line 619 due to AttributeError if it isn't a _ToolLockfileMixin.

Maybe you can assert that it's an instance of _ToolLockfileMixin? It is totally valid to not support invalidation for non-tool lockfiles in Pants 2.7.

Yeah, it will definitely fail earlier.

I didn't really want to expose _ToolLockfileMixin, it's mostly there to ensure consistency of field names through DRY more than an implementation detail, we can change that if you think it helps.

I think it's fine to keep it private w/ the underscore, but still import it here. I strongly suspect this code will change in Pants 2.8 when we add support for multiple user lockfiles - so it makes sense for the type to be an implementation detail.

For what it's worth, my expectation is that we'll need a separate error message function to take care of user lockfiles

Eric-Arellano · 2021-08-31T18:20:43Z

src/python/pants/backend/python/subsystems/python_tool_base.py

        extra_requirements: Iterable[str] = (),
+        uses_source_plugins: bool = False,


You can simplify things by chaing extra_requirements to be requirements_from_source_plugins: Iterable[str] | None = None. If requirements_from_source_plugins is None, assume source plugins are not used.

Note that both MyPy and Pylint will pass an empty FrozenOrderedSet, rather than None, if no plugins are used:

pants/src/python/pants/backend/python/typecheck/mypy/subsystem.py

Lines 219 to 221 in cd24eba

@dataclass(frozen=True)

class MyPyFirstPartyPlugins:

requirement_strings: FrozenOrderedSet[str]

pants/src/python/pants/backend/python/typecheck/mypy/subsystem.py

Lines 288 to 291 in cd24eba

return PythonLockfileRequest.from_tool(

mypy, constraints, extra_requirements=first_party_plugins.requirement_strings

)

Firstly, that line is stale, and I'll be deleting it shortly. I'm not sure that relying on this parameter being set to determine overall tool behaviour is a good idea, because the impact to 3rd-party plugin developers of whether they use None or a falsey iterator is pretty obscure at the moment. I've added a field at the class level, which is what we're actually using to implement the behaviour, which keeps things explicit.

Eric-Arellano · 2021-08-31T18:22:16Z

src/python/pants/backend/python/util_rules/pex.py

+@dataclass(frozen=True)
+class ToolProgrammaticLockfile(LockfileContent, _ToolLockfileMixin):
+    pass


I don't think this makes sense to have without having call sites using this.

Instead, I think we simply don't support anything other than ToolCustomLockfile and ToolDefaultLockfile for now, as mentioned below.

src/python/pants/backend/python/util_rules/pex.py

Eric-Arellano · 2021-08-31T18:25:13Z

src/python/pants/backend/python/util_rules/pex.py

+        elif isinstance(requirements, ToolCustomLockfile):
+            yield f"the lockfile at {requirements.file_path} "
+        else:
+            raise Exception("Check the type hinting on `requirements`")


Better is raise AssertionError. And yeah this is good to keep. Although note that it will fail much earlier on line 619 due to AttributeError if it isn't a _ToolLockfileMixin.

Maybe you can assert that it's an instance of _ToolLockfileMixin? It is totally valid to not support invalidation for non-tool lockfiles in Pants 2.7.

Eric-Arellano · 2021-08-31T18:26:23Z

src/python/pants/backend/python/util_rules/pex.py

+        else:
+            yield (
+                "To regenerate your lockfile based on your current configuration, run "
+                "`./pants generate-lockfiles`. "


# Rust tests and lints will be skipped. Delete if not intended. [ci skip-rust] # Building wheels and fs_util will be skipped. Delete if not intended. [ci skip-build-wheels]

Eric-Arellano · 2021-08-31T19:27:08Z

src/python/pants/backend/python/util_rules/pex.py

        elif isinstance(requirements, ToolCustomLockfile):
            yield f"the lockfile at {requirements.file_path} "
        else:
-            raise Exception("Check the type hinting on `requirements`")
+            assert isinstance(requirements, (ToolCustomLockfile, ToolDefaultLockfile))


This needs to be moved higher up in the function. Once you do that, you can simply do if isinstance(requirements, ToolDefaultLockfile) then else (meaning ToolCustomLockfile). MyPy should have "tightened" the types for you appropriately.

Now that I've added an assertion, mypy is complaining that it's unreachable. Much better!

deletes

Eric-Arellano · 2021-08-31T19:27:46Z

src/python/pants/backend/python/util_rules/pex.py

    else:
        message = "TODO: add support for user lockfiles"


This can be removed. The funciton only supports tool lockfiles, meaning that you can't hook up the invalidation check for anything else in Pants 2.7.

I've replaced with a comment and a false assertion

# Rust tests and lints will be skipped. Delete if not intended. [ci skip-rust] # Building wheels and fs_util will be skipped. Delete if not intended. [ci skip-build-wheels]

Eric-Arellano

Thanks Chris! Looks great

Eric-Arellano · 2021-08-31T20:26:12Z

src/python/pants/backend/python/util_rules/pex_test.py

+        if (ir or ic)
+    ],
+)
+def test_validate_metadata(


You're right - this does make a already big file even bigger and more complex. And this will only get worse when we add multiple user lockfiles.

What do you think about this home being in lockfile_metadata.py? Maybe it'd even make sense to rename to lockfile_invalidation.py?

Sounds like a good fix for the next round of changes :)

…ntsbuild#12699) This changes the lockfile staleness messages to include information on how to reconfigure pants to resolve the error. Language is largely similar to that proposed in pantsbuild#12654, with minor changes: * Language changes to explain the need for action on the user's part (_you can fix this by_ rather than _please change_ ) Closes pantsbuild#12654 # Rust tests and lints will be skipped. Delete if not intended. [ci skip-rust]

…errypick of #12699). (#12717) This changes the lockfile staleness messages to include information on how to reconfigure pants to resolve the error. Language is largely similar to that proposed in #12654, with minor changes: * Language changes to explain the need for action on the user's part (_you can fix this by_ rather than _please change_ ) Closes #12654 Cherry-picks `1cfe8dd`. [ci skip-rust]

Christopher Neugebauer added 6 commits August 30, 2021 13:26

Start plumbing through the default lockfile info

6799c70

# Rust tests and lints will be skipped. Delete if not intended. [ci skip-rust] # Building wheels and fs_util will be skipped. Delete if not intended. [ci skip-build-wheels]

Add options_scope_name to PexRequest so we can display it in erro…

e9a186f

…r messages # Rust tests and lints will be skipped. Delete if not intended. [ci skip-rust] # Building wheels and fs_util will be skipped. Delete if not intended. [ci skip-build-wheels]

Add new error messages (untested)

bff09b8

# Rust tests and lints will be skipped. Delete if not intended. [ci skip-rust] # Building wheels and fs_util will be skipped. Delete if not intended. [ci skip-build-wheels]

Plumbs tool names into PexRequest

d83fd1b

[ci skip-rust] [ci skip-build-wheels]

Formatting; fixes minor logic error

7bf8105

# Rust tests and lints will be skipped. Delete if not intended. [ci skip-rust] # Building wheels and fs_util will be skipped. Delete if not intended. [ci skip-build-wheels]

Adds (very slow) test for warning contents for invalid lockfiles

5d4f8d6

[ci skip-rust] [ci skip-build-wheels]

chrisjrn requested a review from Eric-Arellano August 30, 2021 20:46

Eric-Arellano reviewed Aug 30, 2021

View reviewed changes

Christopher Neugebauer added 4 commits August 30, 2021 16:53

Make faster tests for metadata validation errors

500b597

# Rust tests and lints will be skipped. Delete if not intended. [ci skip-rust] # Building wheels and fs_util will be skipped. Delete if not intended. [ci skip-build-wheels]

Refactors error message info into _ToolLockfileMixin dataclass

3584729

# Rust tests and lints will be skipped. Delete if not intended. [ci skip-rust] # Building wheels and fs_util will be skipped. Delete if not intended. [ci skip-build-wheels]

Removes PexRequest.options_scope_name

045461a

# Rust tests and lints will be skipped. Delete if not intended. [ci skip-rust] # Building wheels and fs_util will be skipped. Delete if not intended. [ci skip-build-wheels]

Address code review feedback

289a30f

# Rust tests and lints will be skipped. Delete if not intended. [ci skip-rust] # Building wheels and fs_util will be skipped. Delete if not intended. [ci skip-build-wheels]

adds an f

5117423

# Rust tests and lints will be skipped. Delete if not intended. [ci skip-rust] # Building wheels and fs_util will be skipped. Delete if not intended. [ci skip-build-wheels]

chrisjrn commented Aug 31, 2021

View reviewed changes

Eric-Arellano reviewed Aug 31, 2021

View reviewed changes

Christopher Neugebauer added 2 commits August 31, 2021 11:58

Remove TODO

3a9ca94

# Rust tests and lints will be skipped. Delete if not intended. [ci skip-rust] # Building wheels and fs_util will be skipped. Delete if not intended. [ci skip-build-wheels]

Removes ToolProgrammaticLockfile and uses_source_plugins

ae10f73

# Rust tests and lints will be skipped. Delete if not intended. [ci skip-rust] # Building wheels and fs_util will be skipped. Delete if not intended. [ci skip-build-wheels]

Eric-Arellano reviewed Aug 31, 2021

View reviewed changes

Christopher Neugebauer added 2 commits August 31, 2021 12:31

Replace TODO string with comment and always-failing assertion

cb952ec

# Rust tests and lints will be skipped. Delete if not intended. [ci skip-rust] # Building wheels and fs_util will be skipped. Delete if not intended. [ci skip-build-wheels]

Remove unreachable code

5bb9975

# Rust tests and lints will be skipped. Delete if not intended. [ci skip-rust] # Building wheels and fs_util will be skipped. Delete if not intended. [ci skip-build-wheels]

Eric-Arellano requested a review from stuhood August 31, 2021 20:23

oops

5cd5a3e

# Rust tests and lints will be skipped. Delete if not intended. [ci skip-rust] # Building wheels and fs_util will be skipped. Delete if not intended. [ci skip-build-wheels]

Eric-Arellano approved these changes Aug 31, 2021

View reviewed changes

Eric-Arellano merged commit 1cfe8dd into pantsbuild:main Aug 31, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add specific and actionable instructions to stale lockfile errors #12699

Add specific and actionable instructions to stale lockfile errors #12699

chrisjrn commented Aug 30, 2021 •

edited

Loading

chrisjrn commented Aug 30, 2021

Eric-Arellano left a comment

Eric-Arellano Aug 30, 2021

Eric-Arellano Aug 31, 2021

chrisjrn commented Aug 31, 2021

chrisjrn Aug 31, 2021

Eric-Arellano Aug 31, 2021

chrisjrn Aug 31, 2021

Eric-Arellano Aug 31, 2021

chrisjrn Aug 31, 2021

Eric-Arellano Aug 31, 2021

chrisjrn Aug 31, 2021

Eric-Arellano Aug 31, 2021

chrisjrn Aug 31, 2021

Eric-Arellano Aug 31, 2021

Eric-Arellano Aug 31, 2021

Eric-Arellano Aug 31, 2021

chrisjrn Aug 31, 2021

Eric-Arellano Aug 31, 2021

chrisjrn Aug 31, 2021

Eric-Arellano left a comment

Eric-Arellano Aug 31, 2021

chrisjrn Aug 31, 2021

Eric-Arellano Aug 31, 2021

	"`./pants generate-lockfiles`. "
	"`./pants generate-lockfiles --resolve={tool_name}`. "

		extra_requirements: Iterable[str] = (),
		uses_source_plugins: bool = False,

	@dataclass(frozen=True)
	class MyPyFirstPartyPlugins:
	requirement_strings: FrozenOrderedSet[str]

	return PythonLockfileRequest.from_tool(
	mypy, constraints, extra_requirements=first_party_plugins.requirement_strings
	)

Add specific and actionable instructions to stale lockfile errors #12699

Add specific and actionable instructions to stale lockfile errors #12699

Conversation

chrisjrn commented Aug 30, 2021 • edited Loading

chrisjrn commented Aug 30, 2021

Eric-Arellano left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

chrisjrn commented Aug 31, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Eric-Arellano left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

chrisjrn commented Aug 30, 2021 •

edited

Loading