Add (files) stanza

[nojb]

This PR is an attempt to define a (files) stanza following discussion in #11819. The syntax (and semantics) are rather close to the existing (dirs) stanza. See the doc and the example for more.

Only the dune-file-level stanza is introduced here. Later, we could allow specifying it at the level of dune-project and/or dune-workspace to provide project- or workspace-wide defaults.

I am not very familiar with this part of the code (the source tree scanning logic), so a review by someone knowledgeable would be appreciated.

@gasche: could you try this branch with your use case and report back if it works as expected? You will need to add something like (files :standard \ *.cm* *.o *.a) to the Dune files of the compiler tree. You will also need to bump the (lang dune ...) version in dune-project to 3.21 (I tried to do it quickly, but it needs some other adaptations and I ran out of time to investigate that!).

[gasche]

Thanks a lot!

I tried to do what you suggested, and then I realized that you were hoping that I would know how to update the dune build system for the compiler from dune-lang 1.10 to 3.20, before I could test this. Nope, nope! It takes way more expertise with dune than I have to know how to do this. (It requires using foreign_stubs.)

But then I just took your PR and changed the line Dune_lang.Syntax.since Stanza.syntax (3, 21) to use (1, 10) instead. And it just works! I had to apply the following diff to get the OCaml compiler to build from an already-built tree (after make -j has run).

diff --git i/asmcomp/dune w/asmcomp/dune
index fb3da1060d6..1a869af39f3 100644
--- i/asmcomp/dune
+++ w/asmcomp/dune
@@ -12,6 +12,8 @@
 ;*                                                                        *
 ;**************************************************************************
 
+(files :standard \ emit.ml)
+
 (rule
  (targets arch.ml CSE.ml proc.ml reload.ml scheduling.ml selection.ml
           stackframe.ml)
diff --git i/otherlibs/str/dune w/otherlibs/str/dune
index 0c96b6385a1..94db27e32ee 100644
--- i/otherlibs/str/dune
+++ w/otherlibs/str/dune
@@ -12,6 +12,8 @@
 ;*                                                                        *
 ;**************************************************************************
 
+(files :standard \ *.cm* *.o *.a)
+
 (library
  (name str)
  (modes byte)
diff --git i/otherlibs/unix/dune w/otherlibs/unix/dune
index bc2c4709616..22fdb95f0ba 100644
--- i/otherlibs/unix/dune
+++ w/otherlibs/unix/dune
@@ -12,6 +12,8 @@
 ;*                                                                        *
 ;**************************************************************************
 
+(files :standard \ *.cm* *.o *.a)
+
 (library
  (name unix)
  (wrapped false)
diff --git i/stdlib/dune w/stdlib/dune
index 1e530bc83eb..667821f1614 100644
--- i/stdlib/dune
+++ w/stdlib/dune
@@ -12,6 +12,8 @@
 ;*                                                                        *
 ;**************************************************************************
 
+(files :standard \ *.cm* *.o *.a)
+
 (library
  (name stdlib)
  (libraries dune_support)

Pretty simple! In particular, it is interesting that I did not have to change most of the compiler source directories (like parsing/, typing/ etc.). I am not sure why, but my best guess is that this comes from the copy_files# in the toplevel dune repository, that would somehow have the effect of making dune ignore the rest of those directories and in particular the build artifacts they contain. (This also suggests a way to possibly do without this feature at all to build the compiler with build artifacts lying around, by systematically using these copy_files# directives for all source directories.)

In summary: I had to use a hack, and then it works very well, but maybe a Dune-and-compiler expert could do even better.

(It is a hack to claim that a new feature is available with old dune languages, but it is a harmless hack, right? The guarantees that people want to have is that old dune files keep working with newer Dune versions, and silently adding new stanzas to old dune-language versions preserves this. Removing stanzas/features or changing the build semantics of existing features would be evil, but this is not.)

[Alizter]

@gasche I have a patch updating the compiler to 3.19. Feel free to take over: ocaml/ocaml#14408.

[nojb]

@Alizter I see you put yourself as a reviewer. I will wait for your approval before merging. Thanks!

[gasche]

Thanks everyone (in particular @nojb for writing the PR an @Alizter for fixing the OCaml dune build system) for your help on this issue. I think that this will be notably helpful to compiler developers.