Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Abstract the pretty printer's ringbuffer to be infinitely sized #92923

Merged
merged 1 commit into from
Jan 18, 2022

Conversation

dtolnay
Copy link
Member

@dtolnay dtolnay commented Jan 15, 2022

This PR backports dtolnay/prettyplease@8e5e83c from the prettyplease crate into rustc_ast_pretty.

Using a dedicated RingBuffer type with non-wrapping indices, instead of manually %-ing indices into a capped sized buffer, unlocks a number of simplifications to the pretty printing algorithm implementation in followup commits such as dtolnay/prettyplease@fcb5968 and dtolnay/prettyplease@4427ced.

This change also greatly reduces memory overhead of the pretty printer. The old implementation always grows its buffer to 205920 bytes even for files without deeply nested code, because it only wraps its indices when they hit the maximum tolerable size of the ring buffer (the size after which the pretty printer will crash if there are that many tokens buffered). In contrast, the new implementation uses memory proportional to the peak number of simultaneously buffered tokens only, not the number of tokens that have ever been in the buffer.

Speaking of crashing the pretty printer and "maximum tolerable size", the constant used for that in the old implementation is a lie:

// Yes 55, it makes the ring buffers big enough to never fall behind.
let n: usize = 55 * linewidth;

It was raised from 3 to 55 in #33934 because that was empirically the size that avoided crashing on one particular test crate, but according to #33934 (comment) other syntax trees still crash at that size. There is no reason to believe that any particular size is good enough for arbitrary code, and using a large number like 55 adds overhead to inputs that never need close to that much of a buffer. The new implementation eliminates this tradeoff.

@rustbot rustbot added the T-compiler Relevant to the compiler team, which will review and decide on the PR/issue. label Jan 15, 2022
@rust-highfive
Copy link
Collaborator

r? @petrochenkov

(rust-highfive has picked a reviewer for you, use r? to override)

@rust-highfive rust-highfive added the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label Jan 15, 2022
@petrochenkov
Copy link
Contributor

@bors r+

@bors
Copy link
Contributor

bors commented Jan 15, 2022

📌 Commit 7b69ad504592961a5b9e78426d8760f2d3eb475d has been approved by petrochenkov

@bors bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Jan 15, 2022
@bors
Copy link
Contributor

bors commented Jan 17, 2022

☔ The latest upstream changes (presumably #92996) made this pull request unmergeable. Please resolve the merge conflicts.

@bors bors added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. labels Jan 17, 2022
@dtolnay
Copy link
Member Author

dtolnay commented Jan 17, 2022

@dtolnay
Copy link
Member Author

dtolnay commented Jan 17, 2022

@bors r=petrochenkov

@bors
Copy link
Contributor

bors commented Jan 17, 2022

📌 Commit 7b5b3cf has been approved by petrochenkov

@bors bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels Jan 17, 2022
bors added a commit to rust-lang-ci/rust that referenced this pull request Jan 18, 2022
…askrgr

Rollup of 14 pull requests

Successful merges:

 - rust-lang#92629 (Pick themes on settings page, not every page)
 - rust-lang#92640 (Fix ICEs related to `Deref<Target=[T; N]>` on newtypes)
 - rust-lang#92701 (Add some more attribute validation)
 - rust-lang#92803 (Hide mobile sidebar on some clicks)
 - rust-lang#92830 (Rustdoc style cleanups)
 - rust-lang#92866 ("Does exists" typos fix)
 - rust-lang#92870 (add `rustc_diagnostic_item` attribute to `AtomicBool` type)
 - rust-lang#92914 (htmldocck: Add support for `/text()` in ``@snapshot`)`
 - rust-lang#92923 (Abstract the pretty printer's ringbuffer to be infinitely sized)
 - rust-lang#92946 (Exclude llvm-libunwind from the self-contained set on s390x-musl targets)
 - rust-lang#92947 (rustdoc: Use `intersperse` in a `visit_path` function)
 - rust-lang#92997 (Add `~const` bound test for negative impls)
 - rust-lang#93004 (update codegen test for LLVM 14)
 - rust-lang#93016 (Stabilize vec_spare_capacity)

Failed merges:

 - rust-lang#92924 (Delete pretty printer tracing)

r? `@ghost`
`@rustbot` modify labels: rollup
@bors bors merged commit 04b2073 into rust-lang:master Jan 18, 2022
@rustbot rustbot added this to the 1.60.0 milestone Jan 18, 2022
@dtolnay dtolnay deleted the ringbuffer branch January 18, 2022 20:31
matthiaskrgr added a commit to matthiaskrgr/rust that referenced this pull request Jan 19, 2022
Pretty printer algorithm revamp step 2

This PR follows rust-lang#92923 as a second chunk of modernizations backported from https://github.com/dtolnay/prettyplease into rustc_ast_pretty.

I've broken this up into atomic commits that hopefully are sensible in isolation. At every commit, the pretty printer is compilable and has runtime behavior that is identical to before and after the PR. None of the refactoring so far changes behavior.

The general theme of this chunk of commits is: the logic in the old pretty printer is doing some very basic things (pushing and popping tokens on a ring buffer) but expressed in a too-low-level way that I found makes it quite complicated/subtle to reason about. There are a number of obvious invariants that are "almost true" -- things like `self.left == self.buf.offset` and `self.right == self.buf.offset + self.buf.data.len()` and `self.right_total == self.left_total + self.buf.data.sum()`. The reason these things are "almost true" is the implementation tends to put updating one side of the invariant unreasonably far apart from updating the other side, leaving the invariant broken while unrelated stuff happens in between. The following code from master is an example of this:

https://github.com/rust-lang/rust/blob/e5e2b0be26ea177527b60d355bd8f56cd473bd00/compiler/rustc_ast_pretty/src/pp.rs#L314-L317

In this code the `advance_right` is reserving an entry into which to write a next token on the right side of the ring buffer, the `check_stack` is doing something totally unrelated to the right boundary of the ring buffer, and the `scan_push` is actually writing the token we previously reserved space for. Much of what this PR is doing is rearranging code to shrink the amount of stuff in between when an invariant is broken to when it is restored, until the whole thing can be factored out into one indivisible method call on the RingBuffer type.

The end state of the PR is that we can entirely eliminate `self.left` (because it's now just equal to `self.buf.offset` always) and `self.right` (because it's equal to `self.buf.offset + self.buf.data.len()` always) and the whole `Token::Eof` state which used to be the value of tokens that have been reserved space for but not yet written.

I found without these changes the pretty printer implementation to be hard to reason about and I wasn't able to confidently introduce improvements like trailing commas in `prettyplease` until after this refactor. The logic here is 43 years old at this point (Graydon translated it as directly as possible from the 1979 pretty printing paper) and while there are advantages to following the paper as closely as possible, in `prettyplease` I decided if we're going to adapt the algorithm to work better for Rust syntax, it was worthwhile making it easier to follow than the original.
@dtolnay dtolnay added the A-pretty Area: Pretty printing (including `-Z unpretty`) label Dec 28, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-pretty Area: Pretty printing (including `-Z unpretty`) S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. T-compiler Relevant to the compiler team, which will review and decide on the PR/issue.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants