Pythonpath invalidation of pantsd can be overly broad with large sourceroots

[stuhood]

pantsd defaults to restarting for changes to any file located on its own pythonpath (since the contents might have been loaded into memory). But the toolchain repo (for example) puts the entire src/python directory on the pythonpath... and that contains significantly more than just plugins. This causes pantsd to restart much more often than it actually needs to.

A few potential options here:

1. moving files to split sourceroots and narrow the pythonpath for plugins
   • This is potentially painful. While having plugins isolated from other source code is conceptually clean, to be fully effective it would need to be a complete split, which would make it awkward to have shared libraries that are both used as targets in the repo, and as part of plugins.
2. changing source plugin loading to require that source plugins actually be declared as targets, so that we have precise dependency info
   • This would be pretty interesting, and would resolve a hermeticity leak that exists today with plugins. But I think that it would require a bootstrap scheduler independent of the runtime scheduler to build/load plugins.
3. attempting to hook into Python's import syntax to dynamically build the list of files/directories to watch
   • This would be magical in an interesting way, but feels fragile: both positive (file existed) and negative (file did not exist) would need to be hooked.
   • Possibly related to https://pyoxidizer.readthedocs.io/en/oxidized_importer-0.1/oxidized_importer.html in some way...
4. asking users to exclude (!/path) things from invalidation globs that are part of the pythonpath, but which they know aren't relevant.
   • This is frustratingly manual, and can both under and over account for things.

[cosmicexplorer]

Would like to note @illicitonion's comment here about the possibility of super super fine-grained (possibly using content-addressed code) invalidation of @rules potentially being possible: #9779 (comment)

[stuhood]

The cost of this issue is significantly reduced via #10052 and by the fact that the invalidation glob fingerprinting became lazy in #10035. I'm going to remove it from TODO for the project.

As @cosmicexplorer and @illicitonion mentioned though, it's likely that we can invalidate differently for plugins from pants' own implementation code. It's likely that we could hot-reload plugins/backends, since we don't have any global state that would interfere with using something like https://stackoverflow.com/questions/437589/how-do-i-unload-reload-a-python-module (nonetheless: potentially dragons).

[stuhood]

Talking with @asherf about this, it seemed like one of the issues here is that we don't have any particular target associated with plugin loading, and so we can't use dependency inference (for example) to apply our much more fine-grained inference to plugins.

This is the bootstrap problem: we need to load the engine to use our target logic, and we need to load (some) plugins to load the engine. This relates to #10360 (comment) : if we had a bootstrap instance of the engine, we could use that to then load loose source plugins with file watching and inference.