Use new hidden deps Merlin support for handling implicit-transitive-deps false #10535

voodoos · 2024-05-15T15:13:30Z

With release 5.2 the compiler introduces a new flag -H to introduce dependencies that are required for linking but should be hidden from the currently built module. Merlin for 5.2 follows with two new configuration directives: BH and SH that allows to provide hidden build and source paths.

This PR effectively takes advantage of these new directives to correctly communicate hidden dependencies to Merlin.

All versions of Merlin since 4.6 will accept that flag, but only the preview versions for 5.2 can actually take advantage of this additional information.

nojb · 2024-05-17T13:59:12Z

Hello @voodoos, thanks for this PR. Note that #9333 is being worked on at the moment (cc @MA0010). Once that is done, you will be able to simplify the patch because the lists of requires given by Compilation_context.requires will be annotated with a flag indicating whether each dependency is direct or transitive (= hidden), and you won't need to do this computation "just" for Merlin. Moreover, I think it would be great if we could merge this PR at the same time that Dune itself is able to support -H.

Accordingly, I suggest to suspend this PR until #9333 is done; does that sound OK? I hope that there will be a PR for #9333 in the next week or so.

nojb · 2024-06-28T08:08:44Z

@voodoos #10644 has been merged; so it should now be possible to finalize this PR by using Compilation_context.requires_hidden. Thanks for waiting!

Leonidas-from-XIV · 2024-10-04T14:55:10Z

Hi @voodoos could you rebase the PR? We were talking about this in the last dune-dev meeting and it would be nice to have it as part of the 3.17 release.

test/blackbox-tests/test-cases/merlin/implicit-transitive-deps.t/run.t

voodoos · 2024-10-07T12:16:36Z

@Leonidas-from-XIV rebased and signed-off

Leonidas-from-XIV

Could you please add a changelog entry? This is one of the highlights of the release, would be good for people to be aware of it :)

src/dune_rules/merlin/merlin.ml

Use new BH / SH directives Make specific tests for OCaml 5.2 Signed-off-by: Ulysse Gérard <thevoodoos@gmail.com>

Signed-off-by: Ulysse Gérard <thevoodoos@gmail.com>

test/blackbox-tests/test-cases/overlapping-deps.t

Leonidas-from-XIV · 2024-10-10T14:27:45Z

test/blackbox-tests/test-cases/overlapping-deps.t

+  -> required by _build/default/proj2/.merlin-conf/lib-bar
+  -> required by _build/default/proj2/bar.cma
+  -> required by %{cma:proj2/bar} at command line:1
+  [1]


Interesting that this triggers the error now, this could be a regression for the release but it seems like the error is what should've happened here anyway.

Leonidas-from-XIV · 2024-10-10T14:29:45Z

test/blackbox-tests/test-cases/deprecated-library-name/features.t

@@ -96,8 +96,7 @@ that wasn't found:
  1 | (executable (name prog) (libraries a))
                                         ^
  Error: Library "a" not found.
-  -> required by _build/default/c/.prog.eobjs/byte/dune__exe__Prog.cmi
-  -> required by _build/default/c/.prog.eobjs/native/dune__exe__Prog.cmx
+  -> required by _build/default/c/.merlin-conf/exe-prog


My main concern is that this changes so many tests, but given the dependency that is getting reported is an internal one and stays an internal one I believe this is not an issue.

It looks like the PR changes some evaluation order that makes Dune errors after following a different branch. Not sure what causes this however...

@rgrinberg Can you please weight in whether this is an issue? It doesn't seem to be wrong, but maybe the order is important in some way I am not aware of?

I found the difference: before we only "peeked" (with Resolve.peek) at the dependencies and now we "read" them.

I will revert that change as is seems wrong to introduce dependencies here.

Co-authored-by: Marek Kubica <marek@xivilization.net> Signed-off-by: Ulysse <5031221+voodoos@users.noreply.github.com>

Signed-off-by: Ulysse Gérard <thevoodoos@gmail.com>

rgrinberg · 2024-10-10T19:48:39Z

@nojb I think you've assigned yourself to this PR. Do you mind taking a final look, and if everything is OK, merging?

rgrinberg · 2024-10-10T19:49:51Z

doc/changes/10535.md

@@ -0,0 +1,2 @@
+- Use Merlin new configuration directives `BH` and `SH` for hidden dependencies.


I would mention that hidden dependencies correspond to the -h flag instead of internal merlin directives which most people are unaware of.

Indeed, it is always better to write Changes entries from the point of view of the user, not the developer. In this case, something like

- Make Merlin/OCaml-LSP aware of "hidden" dependencies used by `(implicit_transitive_deps false)` via the `-H` compiler flag.

would be more useful for the occasional reader.

I took the liberty of tweaking the Changes entry in this manner. Planning to merge once CI passes. Thanks!

nojb

LGTM

nojb · 2024-10-11T04:30:06Z

doc/changes/10535.md

@@ -0,0 +1,2 @@
+- Use Merlin new configuration directives `BH` and `SH` for hidden dependencies.


Indeed, it is always better to write Changes entries from the point of view of the user, not the developer. In this case, something like

- Make Merlin/OCaml-LSP aware of "hidden" dependencies used by `(implicit_transitive_deps false)` via the `-H` compiler flag.

would be more useful for the occasional reader.

Signed-off-by: Nicolás Ojeda Bär <n.oje.bar@gmail.com>

voodoos · 2024-10-11T08:15:45Z

Thank you @nojb !

…eps false (ocaml#10535) Signed-off-by: Ulysse Gérard <thevoodoos@gmail.com>

@jchavarri

CHANGES: ### Fixed - Show the context name for errors happening in non-default contexts. (ocaml/dune#10414, fixes ocaml/dune#10378, @jchavarri) - Correctly declare dependencies of indexes so that they are rebuilt when needed. (ocaml/dune#10623, @voodoos) - Don't depend on coq-stdlib being installed when expanding variables of the `coq.version` family (ocaml/dune#10631, fixes ocaml/dune#10629, @gares) - Error out if no files are found when using `copy_files`. (ocaml/dune#10649, @jchavarri) - Re_export dune-section private library in the dune-site library stanza, in order to avoid failure when generating and building sites modules with implicit_transitive_deps = false. (ocaml/dune#10650, fixes ocaml/dune#9661, @MA0100) - Expect test fixes: support multiple modes and fix dependencies when there is a custom runner (ocaml/dune#10671, @vouillon) - In a `(library)` stanza with `(extra_objects)` and `(foreign_stubs)`, avoid double linking the extra object files in the final executable. (ocaml/dune#10783, fixes ocaml/dune#10785, @nojb) - Map `(re_export)` library dependencies to the `exports` field in `META` files, and vice-versa. This field was proposed in to https://discuss.ocaml.org/t/proposal-a-new-exports-field-in-findlib-meta-files/13947. The field is included in Dune-generated `META` files only when the Dune lang version is >= 3.17. (ocaml/dune#10831, fixes ocaml/dune#10830, @nojb) - Fix staged pps preprocessors on Windows (which were not working at all previously) (ocaml/dune#10869, fixes ocaml/dune#10867, @nojb) - Fix `dune describe` when an executable is disabled with `enabled_if`. (ocaml/dune#10881, fixes ocaml/dune#10779, @moyodiallo) - Fix an issue where C stubs would be rebuilt whenever the stderr of Dune was redirected. (ocaml/dune#10883, fixes ocaml/dune#10882, @nojb) - Format long lists in s-expressions to fill the line instead of formatting them in a vertical way (ocaml/dune#10892, fixes ocaml/dune#10860, @nojb) - Fix the URL opened by the command `dune ocaml doc`. (ocaml/dune#10897, @gridbugs) - Fix the file referred to in the error/warning message displayed due to the dune configuration version not supporting a particular configuration stanza in use. (ocaml/dune#10923, @H-ANSEN) - Fix `enabled_if` when it uses `env` variable. (ocaml/dune#10936, fixes ocaml/dune#10905, @moyodiallo) - Fix exec -w for relative paths with --root argument (ocaml/dune#10982, @gridbugs) - Do not ignore the `(locks ..)` field in the `test` and `tests` stanza (ocaml/dune#11081, @rgrinberg) - Tolerate files without extension when generating merlin rules. (ocaml/dune#11128, @anmonteiro) ### Added - Make Merlin/OCaml-LSP aware of "hidden" dependencies used by `(implicit_transitive_deps false)` via the `-H` compiler flag. (ocaml/dune#10535, @voodoos) - Add support for the -H flag (introduced in OCaml compiler 5.2) in dune (requires lang versions 3.17). This adaptation gives the correct semantics for `(implicit_transitive_deps false)`. (ocaml/dune#10644, fixes ocaml/dune#9333, ocsigen/tyxml#274, ocaml/dune#2733, ocaml/dune#4963, @MA0100) - Add support for specifying Gitlab organization repositories in `source` stanzas (ocaml/dune#10766, fixes ocaml/dune#6723, @H-ANSEN) - New option to control jsoo sourcemap generation in env and executable stanza (ocaml/dune#10777, fixes ocaml/dune#10673, @hhugo) - One can now control jsoo compilation_mode inside an executable stanza (ocaml/dune#10777, fixes ocaml/dune#10673, @hhugo) - Add support for specifying default values of the `authors`, `maintainers`, and `license` stanzas of the `dune-project` file via the dune config file. Default values are set using the `(project_defaults)` stanza (ocaml/dune#10835, @H-ANSEN) - Add names to source tree events in performance traces (ocaml/dune#10884, @jchavarri) - Add `codeberg` as an option for defining project sources in dune-project files. For example, `(source (codeberg user/repo))`. (ocaml/dune#10904, @nlordell) - `dune runtest` can now run individual tests with `dune runtest mytest.t` (ocaml/dune#11041, @Alizter). - Wasm_of_ocaml support (ocaml/dune#11093, @vouillon) - Add a `coqdep_flags` field to the `coq` field of the `env` stanza, and to the `coq.theory` stanza, allowing to configure `coqdep` flags. (ocaml/dune#11094, @rlepigre) ### Changed - Remove all remnants of the experimental `patch-back-source-tree`. (ocaml/dune#10771, @rgrinberg) - Change the preset value for author and maintainer fields in the `dune-project` file to encourage including emails. (ocaml/dune#10848, @punchagan) - Tweak the preset value for tags in the `dune-project` file to hint at topics not having a special meaning. (ocaml/dune#10849, @punchagan) - Change some colors to improve readability in light-mode terminals (ocaml/dune#10890, @gridbugs) - Forward the linkall flag to jsoo in whole program compilation as well (ocaml/dune#10935, @hhugo) - Configurator uses `pkgconf` as pkg-config implementation when available and forwards it the `target` of `ocamlc -config`. (ocaml/dune#10937, @pirbo) - Enable Dune cache by default. Add a new Dune cache setting `enabled-except-user-rules`, which enables the Dune cache, but excludes user-written rules from it. This is a conservative choice that can avoid breaking rules whose dependencies are not correctly specified. This is the current default. (ocaml/dune#10944, ocaml/dune#10710, @nojb, @ElectreAAS) - Do not add `dune` dependency in `dune-project` when creating projects with `dune init proj`. The Dune dependency is implicitely added when generating opam files (ocaml/dune#11129, @Leonidas-from-XIV)

@jchavarri

CHANGES: ### Fixed - Show the context name for errors happening in non-default contexts. (ocaml/dune#10414, fixes ocaml/dune#10378, @jchavarri) - Correctly declare dependencies of indexes so that they are rebuilt when needed. (ocaml/dune#10623, @voodoos) - Don't depend on coq-stdlib being installed when expanding variables of the `coq.version` family (ocaml/dune#10631, fixes ocaml/dune#10629, @gares) - Error out if no files are found when using `copy_files`. (ocaml/dune#10649, @jchavarri) - Re_export dune-section private library in the dune-site library stanza, in order to avoid failure when generating and building sites modules with implicit_transitive_deps = false. (ocaml/dune#10650, fixes ocaml/dune#9661, @MA0100) - Expect test fixes: support multiple modes and fix dependencies when there is a custom runner (ocaml/dune#10671, @vouillon) - In a `(library)` stanza with `(extra_objects)` and `(foreign_stubs)`, avoid double linking the extra object files in the final executable. (ocaml/dune#10783, fixes ocaml/dune#10785, @nojb) - Map `(re_export)` library dependencies to the `exports` field in `META` files, and vice-versa. This field was proposed in to https://discuss.ocaml.org/t/proposal-a-new-exports-field-in-findlib-meta-files/13947. The field is included in Dune-generated `META` files only when the Dune lang version is >= 3.17. (ocaml/dune#10831, fixes ocaml/dune#10830, @nojb) - Fix staged pps preprocessors on Windows (which were not working at all previously) (ocaml/dune#10869, fixes ocaml/dune#10867, @nojb) - Fix `dune describe` when an executable is disabled with `enabled_if`. (ocaml/dune#10881, fixes ocaml/dune#10779, @moyodiallo) - Fix an issue where C stubs would be rebuilt whenever the stderr of Dune was redirected. (ocaml/dune#10883, fixes ocaml/dune#10882, @nojb) - Fix the URL opened by the command `dune ocaml doc`. (ocaml/dune#10897, @gridbugs) - Fix the file referred to in the error/warning message displayed due to the dune configuration version not supporting a particular configuration stanza in use. (ocaml/dune#10923, @H-ANSEN) - Fix `enabled_if` when it uses `env` variable. (ocaml/dune#10936, fixes ocaml/dune#10905, @moyodiallo) - Fix exec -w for relative paths with --root argument (ocaml/dune#10982, @gridbugs) - Do not ignore the `(locks ..)` field in the `test` and `tests` stanza (ocaml/dune#11081, @rgrinberg) - Tolerate files without extension when generating merlin rules. (ocaml/dune#11128, @anmonteiro) ### Added - Make Merlin/OCaml-LSP aware of "hidden" dependencies used by `(implicit_transitive_deps false)` via the `-H` compiler flag. (ocaml/dune#10535, @voodoos) - Add support for the -H flag (introduced in OCaml compiler 5.2) in dune (requires lang versions 3.17). This adaptation gives the correct semantics for `(implicit_transitive_deps false)`. (ocaml/dune#10644, fixes ocaml/dune#9333, ocsigen/tyxml#274, ocaml/dune#2733, ocaml/dune#4963, @MA0100) - Add support for specifying Gitlab organization repositories in `source` stanzas (ocaml/dune#10766, fixes ocaml/dune#6723, @H-ANSEN) - New option to control jsoo sourcemap generation in env and executable stanza (ocaml/dune#10777, fixes ocaml/dune#10673, @hhugo) - One can now control jsoo compilation_mode inside an executable stanza (ocaml/dune#10777, fixes ocaml/dune#10673, @hhugo) - Add support for specifying default values of the `authors`, `maintainers`, and `license` stanzas of the `dune-project` file via the dune config file. Default values are set using the `(project_defaults)` stanza (ocaml/dune#10835, @H-ANSEN) - Add names to source tree events in performance traces (ocaml/dune#10884, @jchavarri) - Add `codeberg` as an option for defining project sources in dune-project files. For example, `(source (codeberg user/repo))`. (ocaml/dune#10904, @nlordell) - `dune runtest` can now run individual tests with `dune runtest mytest.t` (ocaml/dune#11041, @Alizter). - Wasm_of_ocaml support (ocaml/dune#11093, @vouillon) - Add a `coqdep_flags` field to the `coq` field of the `env` stanza, and to the `coq.theory` stanza, allowing to configure `coqdep` flags. (ocaml/dune#11094, @rlepigre) ### Changed - Remove all remnants of the experimental `patch-back-source-tree`. (ocaml/dune#10771, @rgrinberg) - Change the preset value for author and maintainer fields in the `dune-project` file to encourage including emails. (ocaml/dune#10848, @punchagan) - Tweak the preset value for tags in the `dune-project` file to hint at topics not having a special meaning. (ocaml/dune#10849, @punchagan) - Change some colors to improve readability in light-mode terminals (ocaml/dune#10890, @gridbugs) - Forward the linkall flag to jsoo in whole program compilation as well (ocaml/dune#10935, @hhugo) - Configurator uses `pkgconf` as pkg-config implementation when available and forwards it the `target` of `ocamlc -config`. (ocaml/dune#10937, @pirbo) - Enable Dune cache by default. Add a new Dune cache setting `enabled-except-user-rules`, which enables the Dune cache, but excludes user-written rules from it. This is a conservative choice that can avoid breaking rules whose dependencies are not correctly specified. This is the current default. (ocaml/dune#10944, ocaml/dune#10710, @nojb, @ElectreAAS) - Do not add `dune` dependency in `dune-project` when creating projects with `dune init proj`. The Dune dependency is implicitely added when generating opam files (ocaml/dune#11129, @Leonidas-from-XIV)

voodoos mentioned this pull request May 15, 2024

[5.2] Build occurrences index for Merlin #10422

Merged

nojb self-assigned this May 17, 2024

nojb mentioned this pull request Jun 11, 2024

Add support for -H compiler flag #10644

Merged

voodoos force-pushed the merlin-hidden-deps branch from e2531ee to bcccf15 Compare July 1, 2024 13:35

rgrinberg added the merlin label Jul 15, 2024

voodoos force-pushed the merlin-hidden-deps branch from bcccf15 to 257a373 Compare October 4, 2024 16:11

voodoos commented Oct 4, 2024

View reviewed changes

test/blackbox-tests/test-cases/merlin/implicit-transitive-deps.t/run.t Show resolved Hide resolved

voodoos force-pushed the merlin-hidden-deps branch from a5b1ea6 to e26d0ce Compare October 7, 2024 12:13

voodoos force-pushed the merlin-hidden-deps branch from 7465100 to b3cc73e Compare October 7, 2024 14:55

Leonidas-from-XIV reviewed Oct 7, 2024

View reviewed changes

src/dune_rules/merlin/merlin.ml Outdated Show resolved Hide resolved

voodoos force-pushed the merlin-hidden-deps branch from 3531597 to b4641d9 Compare October 9, 2024 16:24

voodoos added 8 commits October 10, 2024 16:21

Add all dependency source dirs to Merlin's configuration.

75979a2

Use new BH / SH directives Make specific tests for OCaml 5.2 Signed-off-by: Ulysse Gérard <thevoodoos@gmail.com>

Promote test results for OCaml 5.1

415c35c

Signed-off-by: Ulysse Gérard <thevoodoos@gmail.com>

fmt

b1a5400

Signed-off-by: Ulysse Gérard <thevoodoos@gmail.com>

Promote changed tests backtraces.

fe4819f

Signed-off-by: Ulysse Gérard <thevoodoos@gmail.com>

Bump Merlin conf version

75ade1b

Signed-off-by: Ulysse Gérard <thevoodoos@gmail.com>

Promote more consistent behaviour of overlap checks

2f436ef

Signed-off-by: Ulysse Gérard <thevoodoos@gmail.com>

Merlin: Remove unused source_dirs argument.

cd4550b

Signed-off-by: Ulysse Gérard <thevoodoos@gmail.com>

Add changelog entry for ocaml#10535

775915d

Signed-off-by: Ulysse Gérard <thevoodoos@gmail.com>

voodoos force-pushed the merlin-hidden-deps branch from b4641d9 to 775915d Compare October 10, 2024 14:21

Leonidas-from-XIV reviewed Oct 10, 2024

View reviewed changes

voodoos and others added 2 commits October 10, 2024 16:36

Fixect typo

2350eab

Co-authored-by: Marek Kubica <marek@xivilization.net> Signed-off-by: Ulysse <5031221+voodoos@users.noreply.github.com>

Only peek at dependencies

6d113f5

Signed-off-by: Ulysse Gérard <thevoodoos@gmail.com>

Leonidas-from-XIV approved these changes Oct 10, 2024

View reviewed changes

rgrinberg approved these changes Oct 10, 2024

View reviewed changes

nojb approved these changes Oct 11, 2024

View reviewed changes

nojb added 2 commits October 11, 2024 06:30

Merge branch 'main' into merlin-hidden-deps

22ad48e

Changes

feb1cd5

Signed-off-by: Nicolás Ojeda Bär <n.oje.bar@gmail.com>

nojb merged commit b39ad6d into ocaml:main Oct 11, 2024
26 of 27 checks passed

anmonteiro pushed a commit to anmonteiro/dune that referenced this pull request Nov 17, 2024

Use new hidden deps Merlin support for handling implicit-transitive-d…

a143c54

…eps false (ocaml#10535) Signed-off-by: Ulysse Gérard <thevoodoos@gmail.com>

maiste mentioned this pull request Nov 25, 2024

[new release] dune (15 packages) (3.17.0~alpha0) ocaml/opam-repository#26951

Closed

maiste mentioned this pull request Nov 27, 2024

[new release] dune (15 packages) (3.17.0) ocaml/opam-repository#26975

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Use new hidden deps Merlin support for handling implicit-transitive-deps false #10535

Use new hidden deps Merlin support for handling implicit-transitive-deps false #10535

voodoos commented May 15, 2024

nojb commented May 17, 2024

nojb commented Jun 28, 2024

Leonidas-from-XIV commented Oct 4, 2024

voodoos commented Oct 7, 2024

Leonidas-from-XIV left a comment

Leonidas-from-XIV Oct 10, 2024

Leonidas-from-XIV Oct 10, 2024

voodoos Oct 10, 2024

Leonidas-from-XIV Oct 10, 2024

voodoos Oct 10, 2024

rgrinberg commented Oct 10, 2024

rgrinberg Oct 10, 2024

nojb Oct 11, 2024

nojb Oct 11, 2024

nojb left a comment

nojb Oct 11, 2024

voodoos commented Oct 11, 2024

		@@ -0,0 +1,2 @@
		- Use Merlin new configuration directives `BH` and `SH` for hidden dependencies.

Use new hidden deps Merlin support for handling implicit-transitive-deps false #10535

Use new hidden deps Merlin support for handling implicit-transitive-deps false #10535

Conversation

voodoos commented May 15, 2024

nojb commented May 17, 2024

nojb commented Jun 28, 2024

Leonidas-from-XIV commented Oct 4, 2024

voodoos commented Oct 7, 2024

Leonidas-from-XIV left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rgrinberg commented Oct 10, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nojb left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

voodoos commented Oct 11, 2024