Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Pythonpath invalidation of pantsd can be overly broad with large sourceroots #9902

Open
stuhood opened this issue May 29, 2020 · 3 comments
Open
Labels
backend: Python Python backend-related issues

Comments

@stuhood
Copy link
Member

stuhood commented May 29, 2020

pantsd defaults to restarting for changes to any file located on its own pythonpath (since the contents might have been loaded into memory). But the toolchain repo (for example) puts the entire src/python directory on the pythonpath... and that contains significantly more than just plugins. This causes pantsd to restart much more often than it actually needs to.

A few potential options here:

  1. moving files to split sourceroots and narrow the pythonpath for plugins
    • This is potentially painful. While having plugins isolated from other source code is conceptually clean, to be fully effective it would need to be a complete split, which would make it awkward to have shared libraries that are both used as targets in the repo, and as part of plugins.
  2. changing source plugin loading to require that source plugins actually be declared as targets, so that we have precise dependency info
    • This would be pretty interesting, and would resolve a hermeticity leak that exists today with plugins. But I think that it would require a bootstrap scheduler independent of the runtime scheduler to build/load plugins.
  3. attempting to hook into Python's import syntax to dynamically build the list of files/directories to watch
  4. asking users to exclude (!/path) things from invalidation globs that are part of the pythonpath, but which they know aren't relevant.
    • This is frustratingly manual, and can both under and over account for things.
@stuhood stuhood added the pantsd label May 29, 2020
@cosmicexplorer
Copy link
Contributor

Would like to note @illicitonion's comment here about the possibility of super super fine-grained (possibly using content-addressed code) invalidation of @rules potentially being possible: #9779 (comment)

@stuhood
Copy link
Member Author

stuhood commented Jun 16, 2020

The cost of this issue is significantly reduced via #10052 and by the fact that the invalidation glob fingerprinting became lazy in #10035. I'm going to remove it from TODO for the project.

As @cosmicexplorer and @illicitonion mentioned though, it's likely that we can invalidate differently for plugins from pants' own implementation code. It's likely that we could hot-reload plugins/backends, since we don't have any global state that would interfere with using something like https://stackoverflow.com/questions/437589/how-do-i-unload-reload-a-python-module (nonetheless: potentially dragons).

@stuhood
Copy link
Member Author

stuhood commented Nov 6, 2020

Talking with @asherf about this, it seemed like one of the issues here is that we don't have any particular target associated with plugin loading, and so we can't use dependency inference (for example) to apply our much more fine-grained inference to plugins.

This is the bootstrap problem: we need to load the engine to use our target logic, and we need to load (some) plugins to load the engine. This relates to #10360 (comment) : if we had a bootstrap instance of the engine, we could use that to then load loose source plugins with file watching and inference.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backend: Python Python backend-related issues
Projects
None yet
Development

No branches or pull requests

4 participants