Memory usage quadrupled after salsa migration

[alexkirsz]

rust-analyzer version: 2025-03-17

rustc version: nightly/2025-01-04

editor or extension: Zed

relevant settings: N/A

After we updated to the latest release of RA, we've seen its memory usage more than quadruple when running on our codebase. Before the update, RA was routinely taking between 5 to 6GB of memory. After the update, it balloons up to 22GB a few seconds after starting, and can increase up to 30GB during use.

I've bisected the regression to 74620e6, which is the port to the new salsa.

I'm currently trying to capture allocations profiles using Instruments on macOS, but the instrumentation makes RA extremely slow, so it takes quite a while to run. I'll try on a subset of our codebase next, which hopefully should reproduce the issue while being faster to run, and update this issue with the results.

Particularities of our setup:

1. We have about 700 crates, 2800 counting external dependencies.
2. We use Bazel instead of cargo, which means we use rust-project.json (or more specifically, the ongoing discover config integration).

[alexkirsz]

Here are two Instruments.app Allocations traces. I had to run rust-analyzer on a small subset of our workspace (~80 crates), as otherwise running the instrumentation would have taken hours on the full workspace. Even on ~10% of the workspace, Instruments is no longer able to save/reopen the .trace files it creates.

On this subset, I'm only seeing a 2x increase in memory usage, but hopefully it is representative of the underlying issue(s).

The first trace is running on 394374e, the second is running on 74620e6 (after #18964). The traces are very large (10GiB), so I've gzipped them.

1. 705,48MiB
2. 1,60GiB

Please let me know if there are better ways to run memory/heap allocations profiling for RA!

[davidbarsky]

Oh boy, 30 gigabytes is suboptimal. As least temporarily, can you try disabling cache priming in your editor? Here's the configuration you'll need in Zed:

 "lsp": {
    "rust-analyzer": {
      "initialization_options": {
        "cachePriming": {
          "enable": false
        }
      }
    }
  },

Slightly longer term, I think I have a sense as to what's going on: I think cache priming ("Indexing") doesn't increment the revision counter, so we never actually garbage collect anything. cc: @Veykril—how wrong am I about this theory?

[alexkirsz]

@davidbarsky After disabling cache priming, rust-analyzer now only takes 5.5GiB of memory. It's also much faster to start, so I think I'm never going back 😄

Thanks for the quick reply. I had built an older version of RA for the team, but this completely unblocks us for now.

EDIT: Okay, the "much faster to start" part might have been placebo, I'll need to run more comparisons, but it does feel noticeably faster than before the salsa migration.

[davidbarsky]

@davidbarsky After disabling cache priming, rust-analyzer now only takes 5.5GiB of memory. It's also much faster to start, so I think I'm never going back 😄

I'm guessing you already discovered this, but disabling cache priming means that startup is faster, but that faster startup time is paid for by certain operations like symbol search or go-to-def being slightly slower because, well—the indexes they rely on haven't been built yet. We might need to fix stick a Database::trigger_lru_eviction

EDIT: Okay, the "much faster to start" part might have been placebo, I'll need to run more comparisons, but it does feel noticeably faster than before the salsa migration.

See the disclaimer above, but I've noticed that new Salsa is slightly faster on some projects even with cache priming. My suggestion to disable cache priming was intended to buy us time in order to run garbage collection during cache priming.

[Rudxain]

IDK if this helps narrow down the problem, but I haven't observed RA using excessive mem on the UADNG repo

[cormacrelf]

I had to run rust-analyzer on a small subset of our workspace (~80 crates), as otherwise running the instrumentation would have taken hours on the full workspace.

Please let me know if there are better ways to run memory/heap allocations profiling for RA!

In situations like these, https://docs.rs/allocative can be useful. It does something like a GC trace on all active allocations given a root object (eg your god object, whatever owns all the data), and produces flame graph of memory use, nesting indicating ownership. It's a bit of work to derive it everywhere but it seems worthwhile in the circumstances.

Because it has no overhead aside from binary size, you can compile it into release builds and take samples from the wild where and when memory issues are occurring. Just have a button to save a flame graph of current memory use. You're supposed to be able to open the traces in eg Firefox Profiler and do analysis usually reserved for stack tracing profilers, like "invert call tree".