Reinitialize pantsd for most configuration changes #10035
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…tches how we have always used it in the engine), and break the false dependency from the NailgunServer to the Scheduler: it only needs the Executor. [ci skip-jvm-tests]
…r the server to be long-lived and the services/scheduler to be short lived. [ci skip-rust-tests] [ci skip-jvm-tests]
|
Commits are useful to look at independently. In particular, the last commit could use special attention, as it adjusts which bootstrap options are marked |
…ch can drop and recreate them based on options changes. [ci skip-jvm-tests]
…e expanding the options that affect Scheduler creation to include the entire set of bootstrap options. Please review this commit closely. [ci skip-jvm-tests]
stuhood
force-pushed
the
stuhood/pantsd-reinit-rather-than-restart
branch
from
9698e0d
to
8be37b0
Compare
gshuflin
reviewed
Jun 12, 2020
| @@ -188,11 +189,20 @@ def override_thread_logging_destination_to_just_stderr(self): | |||
| def match_path_globs(self, path_globs: PathGlobs, paths: Iterable[str]) -> bool: | |||
| return cast(bool, self.lib.match_path_globs(path_globs, tuple(paths))) | |||
|
|
|||
| def nailgun_server_await_bound(self, scheduler, nailgun_server) -> int: | |||
| return cast(int, self.lib.nailgun_server_await_bound(scheduler, nailgun_server)) | |||
| def nailgun_server_await_bound(self, nailgun_server) -> int: | |||
There was a problem hiding this comment.
Can you put a type anno on nailgun_server?
There was a problem hiding this comment.
I can't... the type is unnameable right now because it's self.lib.PyNailgunServer rather than something importable.
I think that we can fix that (I'm not sure that we need to use importlib anymore?), and I thought about fixing it here, but it was off topic.
# Delete this line to force CI to run the JVM tests. [ci skip-jvm-tests]
jsirois
approved these changes
Jun 15, 2020
gshuflin
approved these changes
Jun 15, 2020
This was referenced Jun 16, 2020
Closed
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Problem
Currently we restart pantsd for most configuration changes, and exclude a small set of bootstrap options (by marking them
daemon=False) that should not trigger a restart. But in most cases, restarting is heavy-handed. We'd like to be able to keep more and more of our state alive over time as we continue to remove global mutable state (in order to allow us to tackle #7654, among other things).Additionally, the pantsd client currently implements the fingerprinting that decides when the server should restart, which blocks moving the pantsd client to rust. We'd like the client to only need to interact with a small set of options to simplify its implementation.
Solution
Move the nailgun server out of the
PantsServicemodel and directly into thePantsDaemonclass. Share aPantsDaemonCorebetween the daemon andDaemonPantsRunnerthat encapsulates the currentSchedulerand all live services. Finally, have thePantsDaemonCoreimplement fingerprinting to decide when to reinitialize/recreate theScheduler(without restarting) and trim down the options that trigger a restart (daemon=True) to only those that are used to start the daemon itself (rather than to create theScheduler).Result
pantsdwill stay up through the vast majority of options changes (with the exception of a handful of "micro-bootstrap" options), and will instead reinitialize theSchedulerfor bootstrap options changes with some useful output when it does so.Example:
This prepares to port the client to rust, and unblocks a fix for #8200, by having the
PantsDaemonclass tear down the nailgun server cleanly in the foreground if any services exit. Fixes #6114, fixes #7573, and fixes #10041.