Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support sharing ModuleInfoCache between pipeline runs #2753

Merged
merged 6 commits into from
Apr 25, 2024

Conversation

paulcadman
Copy link
Collaborator

@paulcadman paulcadman commented Apr 24, 2024

This PR delays running of the pipeline MCache (renamed to ModuleInfoCache) to allow this cache to be shared between pipeline runs in processRecursiveUpToTyped. processRecursiveUpToTyped is used by the juvix html command to recursively build HTML for modules in a project.

Performance

The docs build is now much faster and takes much less memory:

Before:

$ /usr/bin/time -lh juvix html docs/index.juvix.md	
1m17.41s real		35.39s user		11.42s sys
          5918703616  maximum resident set size
                   0  average shared memory size
                   0  average unshared data size
                   0  average unshared stack size
             3665213  page reclaims
                 697  page faults
                   0  swaps
                   0  block input operations
                   0  block output operations
                   0  messages sent
                   0  messages received
                   0  signals received
              114533  voluntary context switches
               81450  involuntary context switches
        595152097097  instructions retired
        143688878963  cycles elapsed
         19323983744  peak memory footprint

After:

$ /usr/bin/time -lh juvix html docs/index.juvix.md
	8.35s real		5.76s user		0.62s sys
          2992160768  maximum resident set size
                   0  average shared memory size
                   0  average unshared data size
                   0  average unshared stack size
              221870  page reclaims
                 719  page faults
                   0  swaps
                   0  block input operations
                   0  block output operations
                   0  messages sent
                   0  messages received
                   0  signals received
                1965  voluntary context switches
                1962  involuntary context switches
         93909891240  instructions retired
         19317129226  cycles elapsed
          2963053632  peak memory footprint

Notes

  • MCache is renamed to ModuleInfoCache

  • ModuleInfoCache must be defined in a separate module instead of being defined in Compiler.Pipeline.Driver to avoid a cyclic dependency. ModuleInfoCache is now used used in Juvix.Compiler.Pipeline (in PipelineEff) and this module is imported by Juvix.Compiler.Pipeline.Driver).

@paulcadman paulcadman added this to the 0.6.2 milestone Apr 24, 2024
@paulcadman paulcadman self-assigned this Apr 24, 2024
@paulcadman paulcadman marked this pull request as ready for review April 24, 2024 13:18
@paulcadman paulcadman requested a review from janmasrovira April 24, 2024 13:18
@paulcadman paulcadman force-pushed the share-pipeline-moduleinfocache branch from 23e383f to 007f0d8 Compare April 25, 2024 15:25
@paulcadman paulcadman force-pushed the share-pipeline-moduleinfocache branch from 34375f7 to c7739b8 Compare April 25, 2024 15:36
MCache must be defined in a separate module instead of being defined in
Compiler.Pipeline.Driver to avoid a cyclic dependency.

(MCache is now used used in Juvix.Compiler.Pipeline (in PipelineEff) and this module is imported by Juvix.Compiler.Pipeline.Driver).
@janmasrovira janmasrovira force-pushed the share-pipeline-moduleinfocache branch from c7739b8 to 6423d5d Compare April 25, 2024 15:40
@paulcadman paulcadman merged commit 844f302 into main Apr 25, 2024
4 checks passed
@paulcadman paulcadman deleted the share-pipeline-moduleinfocache branch April 25, 2024 16:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

juvix html has high memory usage
2 participants