Replace Marshal by Granular_marshal in ocaml-index #1889
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
voodoos
added a commit
to voodoos/merlin
that referenced
this pull request
Jan 21, 2025
voodoos
added a commit
to voodoos/merlin
that referenced
this pull request
Jan 22, 2025
voodoos
force-pushed
the
granular-marshal
branch
from
bc7eb27 to
e6b4bd3
Compare
voodoos
added a commit
to voodoos/merlin
that referenced
this pull request
Jan 22, 2025
voodoos
force-pushed
the
granular-marshal
branch
from
4c6e689 to
2f18e9b
Compare
voodoos
added a commit
to voodoos/merlin
that referenced
this pull request
Jan 22, 2025
voodoos
force-pushed
the
granular-marshal
branch
from
2f18e9b to
a1495ad
Compare
voodoos
added a commit
to voodoos/merlin
that referenced
this pull request
Jan 22, 2025
voodoos
force-pushed
the
granular-marshal
branch
from
a1495ad to
250b46e
Compare
Optimize use of reachable_words Fix fd leak: always close granlar_marshal in_channel Compressed index_format by de-duplicating filenames/longidents Fix fd leak: keep latest in_channel open for granular_marshal
in the union find structures
voodoos
force-pushed
the
granular-marshal
branch
from
250b46e to
ca0edb2
Compare
|
On my own testing the indexing time is a slight (5%) increase of the indexing time and no increase in the file size. Fetches are now close to instantaneous and Merlin's memory usage vastly reduced since the index are not fully loaded into memory anymore. Indexing one large library in Dune: Impact on the larger Dune parallel build appear to be much lower: 1.6% increase. @liam923 I am going to merge these improvements that make sense for most users, but we should also evaluate the impact of this PR on "larger" codebases. |
voodoos
added a commit
to voodoos/merlin-jst
that referenced
this pull request
Apr 9, 2025
voodoos
added a commit
to voodoos/merlin-jst
that referenced
this pull request
Apr 10, 2025
voodoos
added a commit
to voodoos/merlin-jst
that referenced
this pull request
Apr 23, 2025
voodoos
added a commit
to voodoos/opam-repository
that referenced
this pull request
Jun 24, 2025
CHANGES:
Tue Jun 24 16:10:42 CEST 2025
+ merlin library
- Expose utilities to manipulate typed-holes in `Merlin_analysis.Typed_hole`
(ocaml/merlin#1888)
- `locate` can now disambiguate between files with identical names and contents
(ocaml/merlin#1882)
- `occurrences` now reports stale files (ocaml/merlin#1885)
- `inlay-hints` fix inlay hints on function parameters (ocaml/merlin#1923)
- Fix issues with ident validation and Lid comparison for occurrences (ocaml/merlin#1924)
- Handle class type in outline (ocaml/merlin#1932)
- Handle locally defined value in outline (ocaml/merlin#1936)
- Fix a typer issue triggering assertions in the short-paths graph (ocaml/merlin#1935,
fixes ocaml/merlin#1913)
- Downstreamed a typer fix from 5.3.X that would trigger assertions linked
to scopes bit masks when backtracking the typer cache (ocaml/merlin#1935)
- Add a new selection field to outline results that contains the location of
the symbol itself. (ocaml/merlin#1942)
- Fix destruct hanging when printing patterns with (::). (ocaml/merlin#1944, fixes
ocaml/ocaml-lsp#1489)
- Reproduce and fix a handful of jump-to-definition (locate) issues (ocaml/merlin#1930,
fixes ocaml/merlin#1580 and ocaml/merlin#1588, workaround for ocaml/merlin#1934)
+ ocaml-index
- Improve the granularity of index reading by segmenting the marshalization
of the involved data-structures. (ocaml/merlin#1889)
+ test suite
- Add a test case illustrating wrong open order proposed in issue ocaml/merlin#1900. (ocaml/merlin#1901)
voodoos
added a commit
to voodoos/merlin-jst
that referenced
this pull request
Jun 26, 2025
voodoos
added a commit
to voodoos/merlin-jst
that referenced
this pull request
Jun 27, 2025
voodoos
added a commit
to voodoos/merlin-jst
that referenced
this pull request
Jun 30, 2025
liam923
pushed a commit
to oxcaml/merlin
that referenced
this pull request
Jul 10, 2025
liam923
added a commit
to oxcaml/merlin
that referenced
this pull request
Jul 16, 2025
* Downstream: Fix occurrences when the definition's source is hidden (ocaml/merlin#1865) * Downstream: Use new uid info to fix jumps and provide occurrences in both the interface and the implementation (ocaml/merlin #1857) * Downstream: perform less merges when indexing (ocaml/merlin#1881) Immediately grow the final index instead of building and merging. * Merge project-wide renaming changes from (ocaml/merlin#1877) * Merge changes adding granular marshal (ocaml/merlin#1889) * Disable new upstream test relying on Dune * Downstream: Fixes for renaming(ocaml/merlin#1924) * Reformat files * Undo -> change * Add fold to Granular_set * Resolve conflicts in occurrences.ml * Promote good test changes * Fix json serialization for renaming scope * Promote more good test changes * Review changes in locate.ml * Review 1857 * Handle store_shapes in index.ml * Review occurrences.ml * Make functor renaming test run * Review r-modules-and-types.t * Resolve cr * Add missing .merlin file --------- Co-authored-by: Ulysse Gérard <thevoodoos@gmail.com> Co-authored-by: Ulysse <5031221+voodoos@users.noreply.github.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Initial PR presentation by @Lucccyo:
The current implementation of ocaml-index uses Marshal to store on the disk the data.
Searching for occurrences on massive projects is time-consuming because the search loads all the data structures from the disk to perform the search.
This Pull Request aims to replace Marshal with a granular version to make the ocaml-index more efficient in reading.
It comes with two granular implementations of the data structures set and map, based on the Stdlib implementation.
During a search operation, the program lazily loads only the required part of the ocaml-index.
It works because the heavy nodes of the granular_map and granular_set have link indirections,
introducing serialization boundaries, which allows Marshal to delay the deserialization of their children.