Include more compiler constraints for dev tools #12834

Sudha247 · 2025-12-01T10:45:42Z

This adds as dependencies more compiler constraints to dev tools if it needs to use the same compiler.

I was discussing with @Alizter last week that when you try to install ocamllsp in a project built with OxCaml, it doesn't automatically pickup the OxCaml version of ocaml-lsp-server and that one has to explicitly specify this in dune-workspace. Digging further, I found that the package ocaml is only considered as a compiler package, whereas OxCaml and other compiler branches exist as ocaml-variants. To this end, it seemed like it would make sense to add them as compiler dependencies.

This PR is not in the best shape, but I'm opening it early to ask if this is something we should be doing while locking dev tools. If this is worth adding, I'll work on refactoring this a bit, add a cram test, and make it ready for review.

I've run some basic tests with this branch and it does pickup the right dev tools on OxCaml projects.

Leonidas-from-XIV · 2025-12-01T13:55:12Z

bin/lock_dev_tool.ml


+let extra_compiler_names = [ Package_name.of_string "ocaml-base-compiler"
+      ; Package_name.of_string "ocaml-variants"
+      ; Package_name.of_string "ocaml-compiler"


Maybe it would make sense to share the hardcoded list of compiler names with Pkg_toolchain which (for similar reasons) has to hardcode the same list.

Leonidas-from-XIV · 2025-12-01T15:58:28Z

to ask if this is something we should be doing while locking dev tools. If this is worth adding

I would say: yes. It seems sensible to me that the dev-tools would try to pick up the right constraint for the compiler. The reason they don't at the moment seems more like an oversight than an conscious decision.

rgrinberg · 2025-12-02T18:32:13Z

Instead of relying on constraints to resolve to the same compiler version, how about just fixing it using a pin?

Leonidas-from-XIV · 2025-12-03T10:53:34Z

@rgrinberg The issue is that for some dev-tools (most notably the LSP server) one needs to build them with the same version as the compiler that is used in the project. Thus we can:

Either force the user to configure the project and dev-tool to use the same, fixed version, of the ocaml compiler
Inject the compiler constraint when finding a solution for the dev-tool

At the moment we do the latter, but the only constraint we add is on ocaml. This works, but only if you use the mainline compiler. However if your project uses an ocaml-variants compiler, then the dev-tool can't match the same version. Thus for oxcaml you have to do the former manually.

rgrinberg · 2025-12-03T10:57:29Z

However if your project uses an ocaml-variants compiler, then the dev-tool can't match the same version

Why is that?

Sudha247 · 2025-12-03T11:00:16Z

Why is that?

Because it's not explicitly required by the solver currently to match it, so it ends up picking up the base compiler. For instance, if your project uses 5.2.0+ox the solver ends up making 5.2.0 the compiler for dev tools that need matching compilers. (e.g. ocamllsp). This patch adds constraints so the solver also matches other compiler packages besides ocaml.

rgrinberg · 2025-12-03T11:05:40Z

For instance, if your project uses 5.2.0+ox the solver ends up making 5.2.0 the compiler for dev tools that need matching compilers

This is exactly why I'm saying that we shouldn't rely on constraints. Running the solver for the dev tools should pin the version of the compiler so that we'll get exactly the version that we need.

Sudha247 · 2025-12-03T12:03:09Z

Running the solver for the dev tools should pin the version of the compiler

Could you please clarify what you mean by pinning here? Do you mean adding it as a pin here:

dune/bin/lock_dev_tool.ml

Line 62 in 79509aa

; pins = Package_name.Map.empty

rgrinberg · 2025-12-03T12:36:44Z

Yes, that's what I mean. It is a way of directing the solver to use a particular package that does not rely on version constraints.

Leonidas-from-XIV · 2025-12-03T14:55:31Z

But doesn't that require additional configuration (that the user must be aware of to do)? With the dev-tool automatically adding constraints on the ocaml compiler packages that we know about it would just work out of the box.

And we already do that in some cases anyway. Should we stop doing that and ask people to instead pin the compiler?

rgrinberg · 2025-12-03T23:04:53Z

Sorry, I should probably just explain the problem clearly first. Pop quiz:

What happens when a user writes this in their workspace file:

(lang dune 3.21)
(pin
 (url "git+http://github.com/oxcaml/oxcaml#my-cool-branch")
 (package (name ocaml)))

And we try to solve for dev tools?

Answer: there are no constraints that are powerful enough to make sure that the dev tools use this version of OCaml.

Constraints on version numbers are simply not powerful enough in such cases. It's OK that we've chosen an approach that doesn't support the compiler being pinned, but ideally, I would prefer an approach that supports both pinned and unpinned compilers.

Pinning on the other hand is powerful enough to support fixing the version in both cases. I'm not sure if our pinning API is powerful enough to express what we want, but fundamentally, pinning is the thing to use when we want to limit the solver to only see the "right" version of a particular package.

For now, feel free to proceed with the constraints approach if this alternative proves to be much more difficult.

Leonidas-from-XIV · 2025-12-04T09:32:38Z

Thanks for elaborating the problem, I see now what you mean.

We can make the lock dir stanza for dev-tools also implicitly add pins on ocaml, ocaml-variants, etc if they're defined in the workspace. But I think we will still need to add constraints instead of pinning for the cases where the user is using a variant compiler from opam-repository in their project lock and does not have a pin for the compiler itself.

e.g. a case like this:

(package
  ..
  (depends
    (ocaml (>= "5.3"))
    ocaml-option-fp))

Sudha247 · 2025-12-04T10:26:10Z

Yes, thanks for explaining @rgrinberg — I see your point. I tend to agree with @Leonidas-from-XIV that in generic cases it could be a constraint, and that for cases where the user has pinned a compiler, we need to pin the compiler. I can look into the pinning API to see if this is possible.

However, this makes me think: should packages that lie in the intersection between user-pinned packages and the dev tool’s dependencies (which the user is trying to install) also be pinned?

Sudha247 · 2025-12-04T18:09:39Z

There are a few things happening here:

Base Compiler: It is assumed that every project has a dependency to the ocaml package for determining the compiler version. I think this is not true, at least not for all compiler versions. But it is a reasonable compromise to assume that this true, and let's continue with this assumption.
Other Compilers There are more compiler packages people might be using and this is hardcoded as certain compiler packages. This is now shared with Pkg_toolchain, which also needs to hardcode these for some reason. The extra compiler packages are added as dependency constraints. We might be able to pin them, but to echo what @Leonidas-from-XIV said above, I think we should add them as constraints, as the user hasn't explicitly pinned them. This PR addresses it.
Pinned Compilers: Special compiler branches that need to be pinned, as @rgrinberg cited above, are not supported. These need to be added as pins as it has been pinned by the user. This PR isn't addressing this part, but I can try to address it in a different PR later.

Signed-off-by: Sudha Parimala <sudharg247@gmail.com>

Sudha247 · 2025-12-05T10:58:22Z

I've added a cram test; this should be ready for review. I'll open an issue about the pinning part.

Leonidas-from-XIV

I think it is shaping up nicely but needs a bit more work. Also to make the code more understandable without reading this issue first.

Leonidas-from-XIV · 2025-12-05T13:18:45Z

src/dune_rules/pkg_toolchain.mli

    versions. Creates the directory if it doesn't already exist. *)
 val base_dir : unit -> Path.Outside_build_dir.t

+val compiler_package_names : Dune_lang.Package_name.t list


A documentation comment would be nice here. But tbh, I am not sure this is the right place for this to live. Maybe it can be in src/dune_pkg from which both the toolchains as well as the dev-tools can use it?

Leonidas-from-XIV · 2025-12-05T13:20:26Z

test/blackbox-tests/test-cases/pkg/ocamllsp/dev-tool-ocamllsp-special-compiler-version.t

+  > (version 5.2.0)
+  > EOF
+
+  $ dune tools exec ocamllsp


Could you ask narration here that we expect ocamllsp to pick up ocaml-variants?

Leonidas-from-XIV · 2025-12-05T13:21:49Z

test/blackbox-tests/test-cases/pkg/ocamllsp/dev-tool-ocamllsp-special-compiler-version.t

+  >  (allow_empty))
+  > EOF
+
+  $ make_lockdir


I think it would be nicer if both the project and the dev tool used the solver. That way one can also show that the result matches.

Leonidas-from-XIV · 2025-12-05T13:22:37Z

test/blackbox-tests/test-cases/pkg/ocamllsp/dev-tool-ocamllsp-special-compiler-version.t

+  > 
+  > (package
+  >  (name foo)
+  >  (allow_empty))


Missing a dependency on any compiler. If one were to run dune pkg lock it wouldn't pick up any compiler. That's probably not what we want.

Leonidas-from-XIV · 2025-12-05T13:23:31Z

src/dune_rules/dune_rules.ml

 module Stanzas = Stanzas
 module Lock_dir = Lock_dir
 module Pkg_dev_tool = Pkg_dev_tool
+module Pkg_toolchain = Pkg_toolchain


If Pkg_toolchain.compiler_package_names could go somewhere else this doesn't need to be exposed.

Leonidas-from-XIV · 2025-12-05T13:25:51Z

bin/lock_dev_tool.ml

+                "Add a dependency on %S to one of the packages in dune-project and then \
+                 run"
+                (Package_name.to_string pkg_name)
+            ; User_message.command "dune pkg lock"


I would drop the recommendation to run dune pkg lock here. We probably don't want to suggest people that use autolocking to suddenly opt out of it.

Leonidas-from-XIV · 2025-12-05T13:27:06Z

bin/lock_dev_tool.ml

+    let* more_compiler_packages = extra_compiler_constraints () in
    let+ constraint_ = locked_ocaml_compiler_constraint () in
-    [ constraint_ ]
+    constraint_ :: more_compiler_packages


Why do we have two mechanisms to add constraints? I would think that adding the constraints via extra_compiler_constraints should be sufficient. If it is not - why?

Leonidas-from-XIV · 2025-12-05T13:28:02Z

test/blackbox-tests/test-cases/pkg/ocamllsp/dev-tool-ocamllsp-special-compiler-version.t

@@ -0,0 +1,36 @@
+Test the special compiler version is picked up by ocamllsp.


I think it would be easier to call them ocaml variants, since that's what they're called upstream (instead of introducing a new term "special").

Leonidas-from-XIV · 2025-12-05T13:29:32Z

bin/lock_dev_tool.ml

-   "ocaml" package used to compile the project in the default build
-   context.
+   TODO: This only makes sure to match the default compiler packages (defined in
+   Pkg_toolchain). This won't work for custom compilers added as pins. Ideally


This doesn't seem to make sense to me. It is not defined in Pkg_toolchain, it is defined here?

Leonidas-from-XIV · 2025-12-05T13:30:38Z

bin/lock_dev_tool.ml

+
+let locked_ocaml_compiler_version () =
+  let open Memo.O in
+  let* packages = load_packages () in


You can make that let+ and avoid the Memo.return below, but YMMV :)

Leonidas-from-XIV reviewed Dec 1, 2025

View reviewed changes

Alizter self-requested a review December 1, 2025 16:45

rgrinberg added the package management label Dec 2, 2025

Sudha247 added 3 commits December 5, 2025 11:57

Include more compiler constraints for dev tools

44bc925

Signed-off-by: Sudha Parimala <sudharg247@gmail.com>

Code cleanup

b301321

Signed-off-by: Sudha Parimala <sudharg247@gmail.com>

Add cram test

44a0df7

Signed-off-by: Sudha Parimala <sudharg247@gmail.com>

Sudha247 force-pushed the dependencies-compiler-dev-tools branch from cd46c2f to 44a0df7 Compare December 5, 2025 10:57

Sudha247 marked this pull request as ready for review December 5, 2025 10:58

Sudha247 requested a review from Leonidas-from-XIV December 5, 2025 10:58

Sudha247 mentioned this pull request Dec 5, 2025

Dev tools and special compiler branches #12868

Open

Leonidas-from-XIV reviewed Dec 5, 2025

View reviewed changes

		@@ -0,0 +1,36 @@
		Test the special compiler version is picked up by ocamllsp.

Include more compiler constraints for dev tools #12834

Are you sure you want to change the base?

Include more compiler constraints for dev tools #12834

Conversation

Sudha247 commented Dec 1, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Leonidas-from-XIV commented Dec 1, 2025

Uh oh!

rgrinberg commented Dec 2, 2025

Uh oh!

Leonidas-from-XIV commented Dec 3, 2025

Uh oh!

rgrinberg commented Dec 3, 2025

Uh oh!

Sudha247 commented Dec 3, 2025

Uh oh!

rgrinberg commented Dec 3, 2025

Uh oh!

Sudha247 commented Dec 3, 2025

Uh oh!

rgrinberg commented Dec 3, 2025

Uh oh!

Leonidas-from-XIV commented Dec 3, 2025

Uh oh!

rgrinberg commented Dec 3, 2025

Uh oh!

Leonidas-from-XIV commented Dec 4, 2025

Uh oh!

Sudha247 commented Dec 4, 2025

Uh oh!

Sudha247 commented Dec 4, 2025

Uh oh!

Sudha247 commented Dec 5, 2025

Uh oh!

Leonidas-from-XIV left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants