Miri: GC the dead_alloc_map too

[saethlin]

dead_alloc_map is the last piece of state in the interpreter I can find that leaks. With this PR, all of the long-term memory growth I can find in Miri with programs that do things like run a big loop { or run property tests is attributable to some data structure properties in borrow tracking, and is extremely slow.

My only gripe with the commit in this PR is that I don't have a new test for it. I'd like to have a regression test for this, but it would have to be statistical I think because the peak memory of a process that Linux reports is not exactly the same run-to-run. Which means it would have to not be very sensitive to slow leaks (some guesswork suggests for acceptable CI time we would be checking for like 10% memory growth over a minute or two, which is still pretty fast IMO).

Unless someone has a better idea for how to detect a regression, I think on balance I'm fine with manually keeping an eye on the memory use situation.

r? RalfJung

[bors]

☔ The latest upstream changes (presumably #118134) made this pull request unmergeable. Please resolve the merge conflicts.

[rustbot]

Some changes occurred to the CTFE / Miri engine

cc @rust-lang/miri

The Miri subtree was changed

cc @rust-lang/miri

[RalfJung]

This does make me rather nervous since we're generally relying on the dead_alloc_map having information about all past allocations. But I guess if an AllocId cannot be found anywhere in the machine state then we cannot possibly care about it in the dead_alloc_map, either...

[RalfJung]

Thanks. :)
@bors r+

[bors]

📌 Commit f5dae8e has been approved by RalfJung

It is now in the queue for this repository.

[RalfJung]

is attributable to some data structure properties in borrow tracking

Do you know which data structures?

[bors]

⌛ Testing commit f5dae8e with merge 38eecca...

[saethlin]

Do you know which data structures?

Stack and Tree if I recall. It's not a particularly challenging profiling to do again.

[RalfJung]

Well yeah that is not very specific. But when we're just creating and removing allocations, I don't see why any stacks or trees would keep growing.

[bors]

☀️ Test successful - checks-actions
Approved by: RalfJung
Pushing 38eecca to master...

[rust-timer]

Finished benchmarking commit (38eecca): comparison URL.

Overall result: no relevant changes - no action needed

@rustbot label: -perf-regression

Instruction count

This benchmark run did not return any relevant results for this metric.

Max RSS (memory usage)

Results

This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	3.3%	[3.3%, 3.3%]	1
Regressions ❌ (secondary)	2.2%	[1.9%, 2.4%]	2
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-	-	0
All ❌✅ (primary)	3.3%	[3.3%, 3.3%]	1

Cycles

This benchmark run did not return any relevant results for this metric.

Binary size

This benchmark run did not return any relevant results for this metric.

Bootstrap: 676.494s -> 677.592s (0.16%)
Artifact size: 313.75 MiB -> 313.72 MiB (-0.01%)

[saethlin]

Well yeah that is not very specific.

Yeah, I think I'll need to cook up some special profiling to understand what's going on. All the existing tools just tell you "this call stack allocated" which is not enlightening.

But when we're just creating and removing allocations, I don't see why any stacks or trees would keep growing.

Ah! I'm observing such behavior in property tests or other long running test suites, which surely do more than just create and remove allocations. I suspect there is some global data structure that is accessed or aliased constantly over the tests, and it accumulates dead state that the GC doesn't know how to remove. Or perhaps all the state is live. In either case, I'll need to do some custom tooling to understand it.

But I don't think the remaining memory growth is a big concern. With the rate it's at now, I think I'm running out of patience long before I run out of memory. So I might just post an issue about what's going on here then not work on it unless I can find a test suite where the memory growth is concerning.