Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[fix] remove space leaks in (,) by using strict state #234

Merged
merged 1 commit into from
Aug 16, 2024

Conversation

MangoIV
Copy link
Contributor

@MangoIV MangoIV commented Aug 16, 2024

  • benchmarks on maintainer-list.nix give a baseline of 88M max residency, nice triangle of death and a bit over 4 seconds duration
  • heap profile shows a lot of memory allocated to (,)
  • removing usagge of (,) in own code didn't affect the profile
  • changing to strict state improves memory from 88 to 40 max residency and runtime goes down by 1 second

- benchmarks on maintainer-list.nix give a baseline of 88M max
  residency, nice triangle of death and a bit over 4 seconds duration
- heap profile shows a lot of memory allocated to (,)
- removing usagge of (,) in own code didn't affect the profile
- changing to strict state improves memory from 88 to 40 max residency
  and runtime goes down by 1 second
Copy link

github-actions bot commented Aug 16, 2024

Nixpkgs diff

Copy link
Member

@infinisil infinisil left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Very nice, thanks a lot!

@infinisil infinisil merged commit 14be7e6 into NixOS:master Aug 16, 2024
2 checks passed
@infinisil
Copy link
Member

Can I ask how you're doing the benchmarking? I don't have a lot of experience doing that with Haskell

@MangoIV
Copy link
Contributor Author

MangoIV commented Aug 16, 2024

Can I ask how you're doing the benchmarking? I don't have a lot of experience doing that with Haskell

Heap Profiling: compile the program with profiling (e.g. by using the --enable-profiling with cabal), then run the program with heap profiling enabled in the RTS, e.g. to break down by cost centre and output to an eventlog, +RTS -hc -l-au, then use haskellPackages.eventlog2html to visualize the heap profile.

Time profiling: almost the same, except you run with +RTS -p -l-au , then you can convert the eventlog to a json with haskellPackaes.hs-speedscope and load it into e.g. https://speedscope.app.

The GHC manual also has a section about profiling haskell programs:
https://downloads.haskell.org/ghc/latest/docs/users_guide/profiling.html

If you happen to have space leaks, you often can bring down max residency and run time at once, if you are lucky enough to find the space leak

@MangoIV
Copy link
Contributor Author

MangoIV commented Aug 16, 2024

lazy state is a huge red flag though, so that wasn't too hard, thankfully.

@infinisil
Copy link
Member

Very useful, thanks!

@MangoIV
Copy link
Contributor Author

MangoIV commented Aug 17, 2024

image

I have gotten nixfmt fast enough to feasibly run on hacke-packges.nix and even though the pretty printer is much faster than the parser, it still got a pretty bad space leak

@MangoIV
Copy link
Contributor Author

MangoIV commented Aug 17, 2024

The big brown triangle is labeled with Int, which means that it's probably some arithmetic that is not forced.

@MangoIV
Copy link
Contributor Author

MangoIV commented Aug 17, 2024

image

and this is how it looks when it's gone.

@MangoIV
Copy link
Contributor Author

MangoIV commented Aug 17, 2024

(yeah, the space leak of the parser is still pretty bad, I just now fixed set to force its arguments...)

@MangoIV
Copy link
Contributor Author

MangoIV commented Aug 17, 2024

image

DocE uses lists, which is a really bad datatype for these sorts of things, so there's another space leak (orange triangle in the bottom right)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Status: Done
Development

Successfully merging this pull request may close these issues.

2 participants