Stream the dep-graph to a file instead of storing it in-memory.

[cjgillot]

This is a reimplementation of #60035.

Instead of storing the dep-graph in-memory, the nodes are encoded as they come
into the a temporary file as they come. At the end of a successful the compilation,
this file is renamed to be the persistent dep-graph, to be decoded during the next
compilation session.

This two-files scheme avoids overwriting the dep-graph on unsuccessful or crashing compilations.

The structure of the file is modified to be the sequence of (DepNode, Fingerprint, EdgesVec).
The deserialization is responsible for going to the more compressed representation.
The node_count and edge_count are stored in the last 16 bytes of the file,
in order to accurately reserve capacity for the vectors.

At the end of the compilation, the encoder is flushed and dropped.
The graph is not usable after this point: any creation of a node will ICE.

I had to retrofit the debugging options, which is not really pretty.

[rust-highfive]

r? @matthewjasper

(rust-highfive has picked a reviewer for you, use r? to override)

[cjgillot]

@bors try @rust-timer queue

[rust-timer]

Awaiting bors try build completion.

@rustbot label: +S-waiting-on-perf

[bors]

⌛ Trying commit 987aaf8c8ecff5fc3f2d6d4cbc04de6d6803886a with merge 1dcaa7f88c16bc44af2b3dc563a4dd2d5b391823...

[bors]

☀️ Try build successful - checks-actions
Build commit: 1dcaa7f88c16bc44af2b3dc563a4dd2d5b391823 (1dcaa7f88c16bc44af2b3dc563a4dd2d5b391823)

[rust-timer]

Queued 1dcaa7f88c16bc44af2b3dc563a4dd2d5b391823 with parent 4099208, future comparison URL.

[rust-timer]

Finished benchmarking try commit (1dcaa7f88c16bc44af2b3dc563a4dd2d5b391823): comparison url.

Benchmarking this pull request likely means that it is perf-sensitive, so we're automatically marking it as not fit for rolling up. Please note that if the perf results are neutral, you should likely undo the rollup=never given below by specifying rollup- to bors.

Importantly, though, if the results of this run are non-neutral do not roll this PR up -- it will mask other regressions or improvements in the roll up.

@bors rollup=never
@rustbot label: +S-waiting-on-review -S-waiting-on-perf

[cjgillot]

Perf: large reduction in max-rss for incr-comp, but really slow.

[cjgillot]

@bors try @rust-timer queue

[rust-timer]

Awaiting bors try build completion.

@rustbot label: +S-waiting-on-perf

[bors]

⌛ Trying commit 9a2749d6b7dd0d34a20be6a168b77e6cc7dc841f with merge 1ca30158cd250a1d4cd67eea2a8784709fcea9a6...

[bors]

💔 Test failed - checks-actions

[cjgillot]

@bors try @rust-timer queue

[rust-timer]

Awaiting bors try build completion.

@rustbot label: +S-waiting-on-perf

[bors]

⌛ Trying commit 9a2749d6b7dd0d34a20be6a168b77e6cc7dc841f with merge 18ab57771eb73a559bbd9496c7ac99ad573ef776...

[michaelwoerister]

Looks like something went wrong when updating the error checking code.

[michaelwoerister]

OK, I assume that won't really hurt performance.

@bors r+

[bors]

📌 Commit f3dde45 has been approved by michaelwoerister

[bors]

⌛ Testing commit f3dde45 with merge 87ae22ab35bdf856af25cbc0c93da922a7c0708c...

[rust-log-analyzer]

A job failed! Check out the build log: (web) (plain)

Click to see the possible cause of the failure (guessed by this bot)

[bors]

💔 Test failed - checks-actions

[cjgillot]

Seems spurious.
@bors retry

[bors]

⌛ Testing commit f3dde45 with merge d474075...

[bors]

☀️ Test successful - checks-actions
Approved by: michaelwoerister
Pushing d474075 to master...

[Mark-Simulacrum]

https://perf.rust-lang.org/compare.html?start=803ddb83598838fb9de308d283b759ba463e5e80&end=d474075a8f28ae9a410e95d849d009006db4b176&stat=instructions:u performance results for the final collection.

[michaelwoerister]

I'm curious where the syn-opt regression comes from. Everything else looks good to me.

[michaelwoerister]

True, but that's to be expected since we take a measurement per dep-node now instead of measuring a single bulk-serialization at the end. Other benchmarks show the same change but don't seem to regress.

[mati865]

Sorry, I've deleted my comment when I realized that.
monomorphization_collector_graph_walk regressed slightly but not that much to explain the difference.

[michaelwoerister]

Sorry, I've deleted my comment when I realized that.

No worries :)