The client error when pantsd restarts is not self-explanatory #8200
Assignees
Comments
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
stuhood
changed the title
|
I am seeing this too. In the middle of a test session run. |
|
pantsd log: |
|
@asherf : That isn't quite enough of the pantsd log, but I'm assuming that somewhere shortly above those lines there was a warning about the invalidation globs having been matched? |
|
stuhood
added a commit
that referenced
this issue
Jun 3, 2020
### Problem The `PantsDaemon` class was playing double duty as both the entrypoint for the pantsd server, and as a launcher for the client. We will soon need to make significant changes there in order to support #8200 and #7654. ### Solution Extract a `PantsDaemonProcessManager` base class for process metadata reads/writes and a `PantsDaemonClient` subclass to consume the metadata to decide whether to launch the server. This is 100% code moves... no logic changed. ### Result The `PantsDaemon` class is now exclusively a server, and `PantsDaemonClient` is now exclusively a client. [ci skip-rust-tests] [ci skip-jvm-tests]
stuhood
added a commit
that referenced
this issue
Jun 9, 2020
### Problem `pantsd` does not implement garbage collection of the `Graph` (see #7675), but additionally, there are likely a few Python-level reference cycles beyond those that we have already discovered. ### Solution We will eventually implement #7675, and it will need a config value to control how much it collects. But in the meantime, having a configurable built-in cap on total memory usage is generally useful, and can consume the same flag that our eventual collection will. ### Result `pantsd` will restart itself when it uses more than the configured amount of memory (defaulting to 4GB). As mentioned in the test comment, until #8200 is fixed, the message rendered when we restart will not be particularly friendly, so we should likely not cherry-pick this to 1.29.x, which will not receive #8200. This is not a complete fix for #9999, but I'm going to resolve it in favor of tracking followup in #7675. [ci skip-rust-tests] [ci skip-jvm-tests]
stuhood
added a commit
that referenced
this issue
Jun 15, 2020
### Problem Currently we restart pantsd for most configuration changes, and exclude a small set of bootstrap options (by marking them `daemon=False`) that should not trigger a restart. But in most cases, restarting is heavy-handed. We'd like to be able to keep more and more of our state alive over time as we continue to remove global mutable state (in order to allow us to tackle #7654, among other things). Additionally, the pantsd client currently implements the fingerprinting that decides when the server should restart, which blocks moving the pantsd client to rust. We'd like the client to only need to interact with a small set of options to simplify its implementation. ### Solution Move the nailgun server out of the `PantsService` model and directly into the `PantsDaemon` class. Share a `PantsDaemonCore` between the daemon and `DaemonPantsRunner` that encapsulates the current `Scheduler` and all live services. Finally, have the `PantsDaemonCore` implement fingerprinting to decide when to reinitialize/recreate the `Scheduler` (without restarting) and trim down the options that trigger a restart (`daemon=True`) to only those that are used to start the daemon itself (rather than to create the `Scheduler`). ### Result `pantsd` will stay up through the vast majority of options changes (with the exception of a handful of "micro-bootstrap" options), and will instead reinitialize the `Scheduler` for bootstrap options changes with some useful output when it does so. Example: ``` $ ./pants help 23:26:22 [INFO] initializing pantsd... 23:26:24 [INFO] pantsd initialized. Pants 1.30.0.dev0 https://pypi.org/pypi/pantsbuild.pants/1.30.0.dev0 Usage: <snip> $ ./pants --no-v1 help 23:26:31 [INFO] initialization options changed: reinitializing pantsd... 23:26:32 [INFO] pantsd initialized. Pants 1.30.0.dev0 https://pypi.org/pypi/pantsbuild.pants/1.30.0.dev0 Usage: <snip> ``` This prepares to port the client to rust, and unblocks a fix for #8200, by having the `PantsDaemon` class tear down the nailgun server cleanly in the foreground if any services exit. Fixes #6114, fixes #7573, and fixes #10041.
stuhood
added a commit
that referenced
this issue
Jun 16, 2020
### Problem As described in #8200, the errors when pantsd shuts down during a run are not particularly helpful. ### Solution When pantsd has chosen to exit (due to either of the `PantsService` instances exiting: generally the `SchedulerService` which watches the pythonpath and other invalidation globs for changes), wait for any ongoing connections to complete before exiting. ### Result The server will wait for ongoing runs to complete before exiting. Fixes #8200.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
When
pantsdrestarts (generally because its code or plugin code has been edited), the error is not self explanatory.Under v2 this might look like:
whereas under v1 it might look like:
The text was updated successfully, but these errors were encountered: