Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix Parser::look_ahead #127636

Merged
merged 3 commits into from
Jul 12, 2024
Merged

Fix Parser::look_ahead #127636

merged 3 commits into from
Jul 12, 2024

Conversation

nnethercote
Copy link
Contributor

Parser::look_ahead has a slow but simple general case, and a fast special case that is hit most of the time. But the special case is buggy and behaves differently to the general case. There are also no unit tests. This PR fixes all of this, resulting in a Parser::look_ahead that is equally fast, slightly simpler, more correct, and better tested.

r? @davidtwco

@nnethercote
Add unit tests for Parser::look_ahead.
dad9557 
It's currently buggy, so some of the test results are surprising, as
described in the `FIXME` comments. The bugs will be fixed in subsequent
commits.
@nnethercote
Remove the bogus special case from Parser::look_ahead.
ebe1305 
The general case at the bottom of `look_ahead` is slow, because it
clones the token cursor. Above it there is a special case for
performance that is hit most of the time and avoids the cloning.
Unfortunately, its behaviour differs from the general case in two ways.

- When within a pair of delimiters, if you look any distance past the
  closing delimiter you get the closing delimiter instead of what comes
  after the closing delimiter.

- It uses `tree_cursor.look_ahead(dist - 1)` which totally confuses
  tokens with token trees. This means that only the first token in a
  token tree will be seen. E.g. in a sequence like `{ a }` the `a` and
  `}` will be skipped over. Bad!

It's likely that these differences weren't noticed before now because
the use of `look_ahead` in the parser is limited to small distances and
relatively few contexts.

Removing the special case causes slowdowns up of to 2% on a range of
benchmarks. The next commit will add a new, correct special case to
regain that lost performance.
@nnethercote
Add a new special case to Parser::look_ahead.
100f3fd 
This new special case is simpler than the old special case because it
only is used when `dist == 1`. But that's still enough to cover ~98% of
cases. This results in equivalent performance to the old special case,
and identical behaviour as the general case.
@rustbot rustbot added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. T-compiler Relevant to the compiler team, which will review and decide on the PR/issue. labels Jul 12, 2024
@nnethercote
Copy link
Contributor Author

Local results showed that the new code has the same performance as the old code. Let's double check that.

@bors try @rust-timer queue

@rust-timer

This comment has been minimized.

Sign in to view
@rustbot rustbot added the S-waiting-on-perf Status: Waiting on a perf run to be completed. label Jul 12, 2024
@bors
Copy link
Contributor

bors commented Jul 12, 2024

⌛ Trying commit 100f3fd with merge ca51f06...

bors added a commit to rust-lang-ci/rust that referenced this pull request Jul 12, 2024
@bors
Auto merge of rust-lang#127636 - nnethercote:fix-Parser-look_ahead, r…
ca51f06 
…=<try>

Fix `Parser::look_ahead`

`Parser::look_ahead` has a slow but simple general case, and a fast special case that is hit most of the time. But the special case is buggy and behaves differently to the general case. There are also no unit tests. This PR fixes all of this, resulting in a `Parser::look_ahead` that is equally fast, slightly simpler, more correct, and better tested.

r? `@davidtwco`
@bors
Copy link
Contributor

bors commented Jul 12, 2024

☀️ Try build successful - checks-actions
Build commit: ca51f06 (ca51f06fb90f2b3589f27de3373c6f83b881b6a5)

@rust-timer

This comment has been minimized.

Sign in to view
@rust-timer
Copy link
Collaborator

Finished benchmarking commit (ca51f06): comparison URL.

Overall result: ✅ improvements - no action needed

Benchmarking this pull request likely means that it is perf-sensitive, so we're automatically marking it as not fit for rolling up. While you can manually mark this PR as fit for rollup, we strongly recommend not doing so since this PR may lead to changes in compiler perf.

@bors rollup=never
@rustbot label: -S-waiting-on-perf -perf-regression

Instruction count

This is a highly reliable metric that was used to determine the overall result at the top of this comment.

mean range count
Regressions ❌
(primary)		 - - 0
Regressions ❌
(secondary)		 - - 0
Improvements ✅
(primary)		 -0.3% [-0.3%, -0.2%] 6
Improvements ✅
(secondary)		 - - 0
All ❌✅ (primary) -0.3% [-0.3%, -0.2%] 6

Max RSS (memory usage)

This benchmark run did not return any relevant results for this metric.

Cycles

Results (primary 2.5%)

This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.

mean range count
Regressions ❌
(primary)		 2.5% [2.5%, 2.5%] 1
Regressions ❌
(secondary)		 - - 0
Improvements ✅
(primary)		 - - 0
Improvements ✅
(secondary)		 - - 0
All ❌✅ (primary) 2.5% [2.5%, 2.5%] 1

Binary size

This benchmark run did not return any relevant results for this metric.

Bootstrap: 704.308s -> 704.516s (0.03%)
Artifact size: 328.70 MiB -> 328.61 MiB (-0.03%)

@rustbot rustbot removed the S-waiting-on-perf Status: Waiting on a perf run to be completed. label Jul 12, 2024
@oli-obk
Copy link
Contributor

oli-obk commented Jul 12, 2024

r? @oli-obk

bors r plus

@bors
Copy link
Contributor

bors commented Jul 12, 2024

📌 Commit 100f3fd has been approved by oli-obk

It is now in the queue for this repository.

@rustbot rustbot assigned oli-obk and unassigned davidtwco Jul 12, 2024
@bors bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Jul 12, 2024
@bors
Copy link
Contributor

bors commented Jul 12, 2024

⌛ Testing commit 100f3fd with merge 62c068f...

@bors
Copy link
Contributor

bors commented Jul 12, 2024

☀️ Test successful - checks-actions
Approved by: oli-obk
Pushing 62c068f to master...

@bors bors added the merged-by-bors This PR was explicitly merged by bors. label Jul 12, 2024
@bors bors merged commit 62c068f into rust-lang:master Jul 12, 2024
7 checks passed
@rustbot rustbot added this to the 1.81.0 milestone Jul 12, 2024
@rust-timer
Copy link
Collaborator

Finished benchmarking commit (62c068f): comparison URL.

Overall result: ✅ improvements - no action needed

@rustbot label: -perf-regression

Instruction count

This is a highly reliable metric that was used to determine the overall result at the top of this comment.

mean range count
Regressions ❌
(primary)		 - - 0
Regressions ❌
(secondary)		 - - 0
Improvements ✅
(primary)		 -0.3% [-0.3%, -0.2%] 3
Improvements ✅
(secondary)		 -2.4% [-2.4%, -2.4%] 1
All ❌✅ (primary) -0.3% [-0.3%, -0.2%] 3

Max RSS (memory usage)

Results (primary -1.6%)

This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.

mean range count
Regressions ❌
(primary)		 - - 0
Regressions ❌
(secondary)		 - - 0
Improvements ✅
(primary)		 -1.6% [-2.5%, -0.7%] 2
Improvements ✅
(secondary)		 - - 0
All ❌✅ (primary) -1.6% [-2.5%, -0.7%] 2

Cycles

This benchmark run did not return any relevant results for this metric.

Binary size

This benchmark run did not return any relevant results for this metric.

Bootstrap: 704.911s -> 703.434s (-0.21%)
Artifact size: 328.85 MiB -> 328.58 MiB (-0.08%)

@nnethercote nnethercote deleted the fix-Parser-look_ahead branch July 13, 2024 01:12
jaisnan pushed a commit to jaisnan/rust-dev that referenced this pull request Jul 29, 2024
@github-actions @celinval
Automatic toolchain upgrade to nightly-2024-07-13 (rust-lang#3339)
3d57f52 
Update Rust toolchain from nightly-2024-07-12 to nightly-2024-07-13
without any other source changes.
This is an automatically generated pull request. If any of the CI checks
fail, manual intervention is required. In such a case, review the
changes at https://github.com/rust-lang/rust from
rust-lang@5315cbe
up to
rust-lang@c6727fc.
The log for this commit range is:
rust-lang@c6727fc9b5 Auto merge of
rust-lang#123351 - beetrees:x86-ret-snan-rust, r=nikic,workingjubilee
rust-lang@62c068feea Auto merge of
rust-lang#127636 - nnethercote:fix-Parser-look_ahead, r=oli-obk
rust-lang@5d76a13bbe Auto merge of
rust-lang#127653 - matthiaskrgr:rollup-72bqgvp, r=matthiaskrgr
rust-lang@f11c2c8e95 Rollup merge of
rust-lang#127648 - Kobzol:ci-lower-timeout, r=pietroalbini
rust-lang@526da2366a Rollup merge of
rust-lang#127627 - lcnr:rustc_search_graph, r=compiler-errors
rust-lang@f5fa6fb602 Rollup merge of
rust-lang#127613 - nikic:riscv-update, r=cuviper
rust-lang@b4f002d2e5 Rollup merge of
rust-lang#127552 - onur-ozkan:unnecessary-git-usage, r=Kobzol
rust-lang@8ceb4e49ff Rollup merge of
rust-lang#127433 - dtolnay:conststrlen, r=workingjubilee
rust-lang@f9b3e8b387 Rollup merge of
rust-lang#126827 - the8472:pidfd-spawn, r=workingjubilee
rust-lang@18152d72a4 Rollup merge of
rust-lang#126639 - sayantn:amx, r=Amanieu
rust-lang@65ea92d4a1 Rollup merge of
rust-lang#124980 - zachs18:rc-allocator, r=Amanieu
rust-lang@05eac57ef6 Auto merge of
rust-lang#127479 - Urgau:rustc-stable-hash, r=michaelwoerister
rust-lang@15f770b143 enable fuzzing of
`SearchGraph`
rust-lang@cae9d480bf Adjust tests for x86
"Rust" ABI changes
rust-lang@3f4b9dd463 Lower timeout of CI
jobs to 4 hours
rust-lang@7f1518bddd Add instability
attribute on private const_strlen function
rust-lang@b286722878 Auto merge of
rust-lang#127635 - matthiaskrgr:rollup-foopajr, r=matthiaskrgr
rust-lang@100f3fd133 Add a new special
case to `Parser::look_ahead`.
rust-lang@ebe1305b1e Remove the bogus
special case from `Parser::look_ahead`.
rust-lang@dad95578b0 Add unit tests for
`Parser::look_ahead`.
rust-lang@ec05c4ea3f Add the feature gate
and target-features
rust-lang@c2b7842555 Rollup merge of
rust-lang#127625 - SkiFire13:revert-comment-deletion, r=workingjubilee
rust-lang@ca576eae4e Rollup merge of
rust-lang#127622 - compiler-errors:builtin-internal, r=lqd
rust-lang@fe564c10ab Rollup merge of
rust-lang#127607 - Zalathar:normalize-hint, r=wesleywiser
rust-lang@83d1a1b252 Rollup merge of
rust-lang#127596 - tesuji:help-unwrap-or, r=compiler-errors
rust-lang@1e7ad4c3ed Rollup merge of
rust-lang#127422 - greaka:master, r=workingjubilee
rust-lang@58fe37f2c3 Rollup merge of
rust-lang#127164 - Nadrieril:clean-lowering-loop, r=matthewjasper
rust-lang@4a31a6c32a Auto merge of
rust-lang#127382 - estebank:const-let, r=compiler-errors
rust-lang@5e311f933d Auto merge of
rust-lang#127614 - matthiaskrgr:rollup-8geziwi, r=matthiaskrgr
rust-lang@a776e5f922 Add doc for
deconstruct_option_or_result
rust-lang@872d7b82e1 Add suggestion for
`Option<&Vec<T>> -> Option<&[T]`
rust-lang@d9170dc666 Add regression test
for issue 127545
rust-lang@4df75140dd Fix aarch64 test
rust-lang@cbe75486f7 Account for `let foo
= expr`; to suggest `const foo: Ty = expr;`
rust-lang@b56dc8ee90 Use verbose style
when suggesting changing `const` with `let`
rust-lang@d9021791eb Revert accidental
comment deletion
rust-lang@b77d3ef7c4 Mark builtin syntax
as internal
rust-lang@fa3ce50f0b Rollup merge of
rust-lang#127605 - nikic:remove-extern-wasm, r=oli-obk
rust-lang@d433f176ef Rollup merge of
rust-lang#127601 - trevyn:issue-127600, r=compiler-errors
rust-lang@47ab86653e Rollup merge of
rust-lang#127599 - tgross35:lazy_cell_consume-rename, r=workingjubilee
rust-lang@a10b4d1463 Rollup merge of
rust-lang#127598 - weiznich:diagnostic_do_not_recommend_also_skips_help,
r=compiler-errors
rust-lang@73c500b3a7 Rollup merge of
rust-lang#127591 - compiler-errors:label-after-primary, r=lcnr
rust-lang@380c78741e Rollup merge of
rust-lang#127588 - uweigand:s390x-f16-doctests, r=tgross35
rust-lang@6fd955549a Rollup merge of
rust-lang#127572 - tbu-:pr_debug_event_nonpacked, r=jhpratt
rust-lang@8de487fdbd Rollup merge of
rust-lang#124599 - estebank:issue-41708, r=wesleywiser
rust-lang@55256c5a18 Update
dist-riscv64-linux to binutils 2.40
rust-lang@977439d9b8 Use uplifted
`rustc-stable-hash` crate in `rustc_data_structures`
rust-lang@f56b2074c6 solve -> solve/mod
rust-lang@08a2992d6b compiletest: Better
error message for bad `normalize-*` headers
rust-lang@8a50bcbdce Remove extern "wasm"
ABI
rust-lang@a01f49e7f3 check is_ident
before parse_ident
rust-lang@ab56fe2053 Rename
`lazy_cell_consume` to `lazy_cell_into_inner`
rust-lang@27d5db166e Allows
`#[diagnostic::do_not_recommend]` to supress trait impls in suggestions
as well
rust-lang@12ae282987 Fix diagnostic and
add a test for it
rust-lang@df72e478b0 Make sure that
labels are defined after the primary span in diagnostics
rust-lang@0065763950 core: Limit
remaining f16 doctests to x86_64 linux
rust-lang@45ad522e87 Don't mark
`DEBUG_EVENT` struct as `repr(packed)`
rust-lang@0134bd2e67 remove unnecessary
`git` usages
rust-lang@42772e98e0 Address review
comments
rust-lang@3e030b38ef Return the
`otherwise_block` instead of passing it as argument
rust-lang@fc40247c6b Factor out the
"process remaining candidates" cases
rust-lang@8a222ffd6b Don't try to save an
extra block
rust-lang@c5062f7318 Move or-pattern
expansion inside the main part of the algorithm
rust-lang@bff4d213fa Factor out the
special handling of or-patterns
rust-lang@5bf50e66f9 Move a function
rust-lang@53d3e6217b Stabilize
const_cstr_from_ptr (CStr::from_ptr, CStr::count_bytes)
rust-lang@585ca16e0b as_simd: fix comment
to be in line with 507583a (rust-lang#121201)
rust-lang@0f643c449a Ensure tests don't
fail on i586 in CI
rust-lang@ec0c755704 Check that we get
somewhat sane PIDs when spawning with pidfds
rust-lang@3e4e31b7bf more fine-grained
feature-detection for pidfd spawning
rust-lang@0ce361938e document safety
properties of the internal Process::new constructor
rust-lang@6687a3f7da use pidfd_spawn for
faster process creation when pidfds are requested
rust-lang@5c46acac04 document the cvt
methods
rust-lang@0e1c832dbd Update
`platform-support.md` to reflect improvements in returning floats on
32-bit x86
rust-lang@952becc0bd Ensure floats are
returned losslessly by the Rust ABI on 32-bit x86
rust-lang@a1ad6346d6 Add fn allocator
method to rc/sync::Weak. Relax Rc<T>/Arc<T>::allocator to allow unsized
T.
rust-lang@2df4f7dd8c Suggest borrowing on
fn argument that is `impl AsRef`

Co-authored-by: celinval <35149715+celinval@users.noreply.github.com>
@compiler-errors compiler-errors mentioned this pull request Aug 9, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@fmease fmease fmease left review comments

@oli-obk oli-obk oli-obk left review comments

Assignees

@oli-obk oli-obk

Labels
merged-by-bors This PR was explicitly merged by bors. S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. T-compiler Relevant to the compiler team, which will review and decide on the PR/issue.
Projects
None yet
Milestone
1.81.0
Development

Successfully merging this pull request may close these issues.
7 participants
@nnethercote @rust-timer @bors @oli-obk @fmease @davidtwco @rustbot