macros: stackless expansion

[jseyfried]

After this PR, macro expansion cannot overflow the stack unless the expanded crate is too deep to fold.
Everything but the stackless placeholder expansion commit is also groundwork for macro modularization.

cc #35896
r? @nrc or @eddyb

[durka]

Hmm, this doesn't seem to be working. I tried it on my most-recursing macro, brainmunch. Having built master and this branch:

[root@li1424-173 brainmunch]# time ../rust/stackless/build/x86_64-unknown-linux-gnu/stage1/bin/rustc -O -Z time-passes src/main.rs | grep expansion
time: 8.082; rss: 68MB  expansion

real    0m9.118s
user    0m8.993s
sys     0m0.100s
[root@li1424-173 brainmunch]# time ../rust/build/x86_64-unknown-linux-gnu/stage1/bin/rustc -o master -O -Z time-passes src/main.rs | grep expansion
time: 8.501; rss: 67MB  expansion

real    0m9.575s
user    0m9.450s
sys     0m0.103s

This branch is first, then master. I would have expected a more impressive time difference and a smaller memory footprint if the expansion is really stackless. Furthermore, I removed #![recursion_limit="5000"] from main.rs and both builds immediately bailed having reached the recursion limit.

[eddyb]

@durka If you don't get a stack overflow with a high recursion limit before this PR, then there's not much to see.
You couldn't notice a large memory or time difference, all that's happening AFAICT is that very deep macro recursion would not cause stack overflows anymore because the real stack isn't used.

[jseyfried]

@durka This PR doesn't change recursion_limit checking -- if macro expansion is more recursive than the recursion limit, it is still an error.

It would probably be a good idea to raise the default recursion limit or even remove it altogether, but that's for a separate PR. Note that if we remove the recursion limit altogether, I think we should limit the total memory usage or total number of expansions so that if there really is unbounded recursion, the user will still get a nice error message (not a hang or an out of memory error). I'd be happy to help if you want to do this.

To test this branch, you can set the recursion limit to something that can't be reached in a reasonable amount of time and space, like 2^32.

This PR doesn't improve performance (I actually measured ~5% regression, but that will be made up by a later PR) nor does it improve memory usage -- after this PR, the heap will have all the information that used to be on the stack and more.

[durka]

Ah I see, I got too excited.

[durka]

Further investigation: I lengthened the test case to stack overflow on master after 1m27s, and it finished compiling on this branch after 5 minutes. Intentionally lowering the recursion limit produces comparable time-to-error for both branches.

[bors]

☔ The latest upstream changes (presumably #35957) made this pull request unmergeable. Please resolve the merge conflicts.

[nrc]

This doesn't seem right - in the dummy span case you're popping an empty PathBuf

[jseyfried]

Fixed.

[bors]

☔ The latest upstream changes (presumably #36240) made this pull request unmergeable. Please resolve the merge conflicts.

[nrc]

r=me with that boilerplate factored away

[jseyfried]

@bors r=nrc

[bors]

📌 Commit 400a4f2 has been approved by nrc

[bors]

⌛ Testing commit 400a4f2 with merge d75c64d...

[bors]

💔 Test failed - auto-win-msvc-64-cargotest

[jseyfried]

@bors r=nrc

[bors]

📌 Commit dc01d96 has been approved by nrc

[jseyfried]

@bors r=nrc

[bors]

📌 Commit 9ac91fa has been approved by nrc

[bors]

⌛ Testing commit 9ac91fa with merge 4a26286...

[bors]

☀️ Test successful - auto-linux-32-nopt-t, auto-linux-32-opt, auto-linux-64-cargotest, auto-linux-64-cross-freebsd, auto-linux-64-debug-opt, auto-linux-64-nopt-t, auto-linux-64-opt, auto-linux-64-opt-rustbuild, auto-linux-64-x-android-t, auto-linux-cross-opt, auto-linux-musl-64-opt, auto-mac-32-opt, auto-mac-64-opt, auto-mac-64-opt-rustbuild, auto-mac-cross-ios-opt, auto-win-gnu-32-opt, auto-win-gnu-32-opt-rustbuild, auto-win-gnu-64-opt, auto-win-msvc-32-opt, auto-win-msvc-64-cargotest, auto-win-msvc-64-opt, auto-win-msvc-64-opt-rustbuild
Approved by: nrc
Pushing 4a26286 to master...

[jseyfried]

@DanielJCampbell This refactoring might help with expansion visualization / stepwise expansion -- feel free to ping me on IRC if you'd like to discuss.

[DanielJCampbell]

Yeah, I will do that - I have a working solution but it's based on Rust from 6 weeks ago, looks like plenty has changed since then.