Parallel build issue in 1.9.1

[kit-ty-kate]

When building cppo.1.6.0 on opam on a machine with 72 cores with OCaml 4.02.3, the build has a 1/24 chance to fail with:

#=== ERROR while compiling cppo.1.6.0 =========================================#
# context              2.0.3 | linux/x86_64 | ocaml-base-compiler.4.02.3 | file:///home/opam/opam-repository
# path                 ~/.opam/4.02.3/.opam-switch/build/cppo.1.6.0
# command              ~/.opam/4.02.3/bin/jbuilder build -p cppo -j 71
# exit-code            1
# env-file             ~/.opam/log/cppo-1-fd6dea.env
# output-file          ~/.opam/log/cppo-1-fd6dea.out
### output ###
# [...]
# Warning: Aliases must not have targets, this target will be ignored.
# This will become an error in the future.
# File "_build/.dune/default/src/jbuild", line 5, characters 0-25:
# 5 | (ocamlyacc (cppo_parser))
#     ^^^^^^^^^^^^^^^^^^^^^^^^^
# Warning: File cppo_parser.mli is both generated by a rule and present in the source tree.
# As a result, the rule is currently ignored, however this will become an error in the future.
# To keep the current behavior and get rid of this warning, add a field (fallback) to the rule.
#       ocamlc src/.cppo_main.eobjs/byte/cppo_parser.{cmi,cmti} (exit 2)
# (cd _build/default && /home/opam/.opam/4.02.3/bin/ocamlc.opt -w -40 -g -bin-annot -I src/.cppo_main.eobjs/byte -no-alias-deps -o src/.cppo_main.eobjs/byte/cppo_parser.cmi -c -intf src/cppo_parser.mli)
# File "src/cppo_parser.mli", line 2, characters 13-27:
# Error: Unbound module Cppo_types

- Note: You can use "dune upgrade" to convert your project to dune.
- File "test/jbuild", line 96, characters 26-39:
- 96 |   (action (with-stdout-to test.cppo.out (run ${bin:cppo} ${<})))))
-                                ^^^^^^^^^^^^^
- Warning: Aliases must not have targets, this target will be ignored.
- This will become an error in the future.
- File "_build/.dune/default/src/jbuild", line 5, characters 0-25:
- 5 | (ocamlyacc (cppo_parser))
-     ^^^^^^^^^^^^^^^^^^^^^^^^^
- Warning: File cppo_parser.mli is both generated by a rule and present in the source tree.
- As a result, the rule is currently ignored, however this will become an error in the future.
- To keep the current behavior and get rid of this warning, add a field (fallback) to the rule.
-       ocamlc src/.cppo_main.eobjs/byte/cppo_parser.{cmi,cmti} (exit 2)
- (cd _build/default && /home/opam/.opam/4.02.3/bin/ocamlc.opt -w -40 -g -bin-annot -I src/.cppo_main.eobjs/byte -no-alias-deps -o src/.cppo_main.eobjs/byte/cppo_parser.cmi -c -intf src/cppo_parser.mli)
- File "src/cppo_parser.mli", line 2, characters 13-27:
- Error: Unbound module Cppo_types

Here is the content of the jbuild file in src:

(* -*- tuareg -*- *)
#require "unix"

let version =
  let ic = open_in "../VERSION" in
  let version = input_line ic in
  close_in ic;
  version

let () = Printf.ksprintf Jbuild_plugin.V1.send {|
(jbuild_version 1)

(ocamllex (cppo_lexer))
(ocamlyacc (cppo_parser))

(rule
 ((targets (cppo_version.ml))
  (action
   (with-stdout-to ${@}
    (echo "let cppo_version = \"%s\"")))))

(executable
 ((name cppo_main)
  (package cppo)
  (public_name cppo)
  (libraries (unix str))))
|} version

and here is the content of src:

-rw-r--r-- cppo_command.ml
-rw-r--r-- cppo_eval.ml
-rw-r--r-- cppo_lexer.mll
-rw-r--r-- cppo_main.ml
-rw-r--r-- cppo_parser.mli
-rw-r--r-- cppo_parser.mly
-rw-r--r-- cppo_types.ml
-rw-r--r-- jbuild

I probably don't have enough knowledge in dune but it all seems fine to me, there shouldn't be any problem.
I'm also starting to suspect ocaml/opam-repository#13954 and ocaml/opam-repository#13980 might also be related to this problem.
I'm not sure those problems were there before 1.9.1 but maybe I'm wrong. I think I at least saw the problem on Camomile before but I'm not sure.

[ghost]

Would it be possible to see the _build/log in this case?

[ghost]

I believe the problem is the following:

• (ocamlyacc (cppo_parser)) produces both cppo_parser.mli and cppo_parser.ml
• cppo_parser.mli is present in the source tree

This is no longer allowed in dune files, however it is in jbuild files to be compatible with the original behaviour of jbuilder. I believe the following happens:

• the rule to copy cppo_parser.mli from the source tree to _build kicks in
• ocamldep is called on cppo_parser.mli
• at the same time, the ocamlyacc rule kicks in
• ocalmdep sees an invalid file, silently produces nothing and exit with code 0
• dune thinks cppo_parser.mli has no dependency and as a result doesn't try to build cppo_types.mli before cppo_parser.mli

I'll see if we can improve dune's behavior in this case, but in the meantime a quick fix is to run: rm -f src/cppo_parser.mli just before calling jbuilder

[kit-ty-kate]

menhir seems to also have the same behaviour and I wasn't even aware that they would overwrite the .mli file if it's already present. I did the mistake in one of my project too apparently.

[kit-ty-kate]

Thanks a lot for looking into this!

[ghost]

No problem. The handling of this case was just dodgy. It's good that we finally clean it up.