Scalable Polonius support on nightly

[nikomatsakis]

Metadata
Owner(s)	@lqd
Team(s)	types
Goal document	2024h2/Polonius

Summary

Improve the Nightly support for Polonius, next generation borrow checking, so that it can scale

Tasks and status

• Design review (@nikomatsakis)
• Factoring out higher-ranked concerns from the main path (@amandasystems)
  • rewrite invalid universe constraints with outlives 'static constraints — PR 123720
  • completely remove placeholders — in progress PR 130227
• Replace parts of the borrow checker with location-insensitive Polonius
• Location-sensitive prototype on nightly (@lqd) — in progress
  • create structures for location-dependent outlives constraints
  • build new constraint graph from typeck constraints and liveness constraints
  • update NLLs for required changes to local & region liveness, loan liveness & loan scopes, (possibly unreachable) kills, bidirectional traversal & active loans
  • limit regressions about diagnostics when using the new constraints on diagnostics tailored to the old constraints
  • land on nightly
• Debugging / dump tool for analysis of location-sensitive analysis (@lqd)
• Tests and validation of location-sensitive Polonius (@lqd)
  • make the full test suite pass — in progress
  • do a crater run for assertions and backwards-compatibility
  • expand test suite with tests about the new capabilities
• Location-sensitive pass on nightly, tested on CI (@lqd)
• Standard reviews (types )

[nikomatsakis]

This issue is intended for status updates only.

For general questions or comments, please contact the owner(s) directly.

[lqd]

Status update:

• we've completed the first task of removing universe errors out of the main path and moving higher-ranked concerns out of the borrowck, in @amandasystems' PR #123720. During SCC construction, we now detect these constraints and rewrite them to regular constraints outliving 'static. We have possible improvements in mind that we can do in the future here, but we've also just started trying to remove placeholders, by replacing them with constraints again.

• we've also made more progress on the location-sensitive analysis, though not as much as we'd like. By now we have a "localized" constraint graph to implement propagation in a flow-sensitive manner, with constraints from typeck and liveness, and have it hooked it up to the existing borrow checking in non-lexical scope and active loans computation. It's still buggy though: there are a couple hundred of UI test failures at the moment, including important cases that don't work yet. Some of these failures are due to the diagnostics subsystem not yet handling flow-sensitive concerns. The next immediate steps are to analyze and fix the test failures, and to help with this we're also expanding the existing NLL logging, MIR dumps, and visualizations.

• and we've also started work on the item to use more of the polonius model in the borrow-checker, though we're still in the preparation phase to have an overview of what we want, need, and can change here. Currently, the gated location-insensitive analysis is mainly implemented in the borrowck dataflow analyses, but we want to expand this to replace code and data existing in both models, avoid duplicate work, and simplify the implementation. For example, we can either change components internals to do so, or have them generic enough to work with both models: this applies at least to region liveness, region values, SCC values, etc. We also want to do this unification while taking the location-sensitive requirements into account, as well as constraints that existing downstream code can't be impacted or changed too much for now (e.g. diagnostics).

[lqd]

Key developments:

• We were able to fix a bunch of test failures due to some off-by-one errors (in both typeck and livenesss constraints) from the old "mid-points" that don't exist anymore.
• We're still debugging the large number of test failures, and continued the design work on the unified debugging visualization to help with that. We're currently manually doing the tracing work, and after a few run-throughs within the team, should be able to automate it.
• We've also focused on some examples that we want to accept but do not yet, in particular all the variations of issue #47680. Some adjustments to the active loans computation may be needed, and/or the locations where effects happen (which also relate to the mid-points item above), which was unexpected. It shouldn't be hard to fix though.
• Amanda has continued cleaning up placeholders in the work-in-progress PR #130227.
• There was no particular progress on using more of the polonius model within the borrowck,. We also had a bit less time than usual with me going to RustConf, and Amanda going to EuroRust and doing a presentation on polonius 🎉. We'll add a link to the recording once they're made available.

[lqd]

Key developments, with good progress:

• I've completed the loan liveness + active loans changes I mentioned last time, with kills stopping traversal in the new localized constraint graph
• issues with missing liveness edges when regions flowed into universal regions were fixed
• issues with liveness optimizations for NLLs, that don't apply anymore in a location-sensitive context, were fixed
• I'm currently double-checking invariant edges on assignments/calls: they should be bidirectional, but some of the edges allowing backwards propagation are missing, causing some test failures
• I identified an issue in diagnostics causing 80 or so differences in tests, adding to the diagnostics differences caused by the NLL optimization described above: some errors and handcrafted diagnostics are tailored to the current liveness NLL behavior and structure.
• a lot of work was done on the NLL+polonius data visualization ("debugger"). There's still quite a lot to do here but it already was crucial in diagnosing and fixing the above issues, even in its early state
• overall around 70 real test failures are left to analyze, and then I'll look at the diagnostics differences to see what can be done. The ICEs there expecting NLL constraints should be fixable. The others may require more creative solutions, e.g. piggybacking on the existing code.
• we're also still working out how to best land all this work
• Amanda has done non-trivial work in [WIP] Remove placeholders completely rust#130227 which now also handles type tests. The PR should be ready to review, and a crater run is currently in-progress.
• Amanda's EuroRust talk on polonius from last month is also now available on YouTube.