Garbage collection of dirty keys

[pepeiborra]

This runs a garbage collection process when a file is closed, releasing all the keys that remain dirty after a build of all the open files. Essential keys are not GC'ed to avoid information loss and inefficient rebuilds.

For example, if we have A, B and C open in the IDE and close A, the garbage collection will release all the keys that were specific to A, and preserve all the keys that are still needed by B and C.

I have been able to validate heap usage reductions after closing all the files in the editor and triggering a keys GC followed by a heap GC. So it does seem to do what we want.

TODO

• Test
• Measure improvement
• Restore heap tracing code
• Fix flaky tests
• Think a bit more about GC strategy
• GC seems to break diagnostics
• Check plugins with custom rules
• Break down in smaller PRs

EDIT: moved EKG to another PR since it introduces new deps that break the GHC 9 build

[pepeiborra]

This is now ready for review and dogfooding, since I'm not sure whether the problem with GC breaking diagnostics still exists.

[jneira]

great, this was really needed

[jneira]

Maybe we should wait to merge this after the next release?

[jneira]

Maybe related: #713, which is asking for garbage collect diagnostics from files not in or not linked to any actual file of interes

[pepeiborra]

Thanks for linking it. That ticket is about cleaning up diagnostics, not related to this. But there's a comment by @mpickering on performing garbage collection only when the FOI set is modified that is very relevant.

[pepeiborra]

So beautiful:

[wz1000]

Looks good. However, I wonder if it would be neater to use an LRU datastructure to store the hls-graph database itself, instead of manually triggering garbage collection on idle.

[pepeiborra]

Looks good. However, I wonder if it would be neater to use an LRU datastructure to store the hls-graph database itself, instead of manually triggering garbage collection on idle.

Yesterday I changed the collection to trigger on file close, rather than on idle. We rely on dirtiness to inform on which keys are not relevant and can be deleted, so I'm not sure that LRU would be helpful. I have updated the PR description to reflect this.

[mpickering]

I am dogfooding this MR @pepeiborra, thanks for working on it.

[pepeiborra]

Maybe we should wait to merge this after the next release?

Is this still the plan or should we go ahead and merge ?

[github-actions]

We are gonna [skip circleci]

[jneira]

Maybe we should wait to merge this after the next release?

Is this still the plan or should we go ahead and merge ?

i think we have to postpone the release so i would go ahead and merge