Add promote actions and include stanzas

[ghost]

This PR implements something similar to what was discussed in #371 except for toplevel (promote ...) stanza which will be part of another PR.

Motivations

This PR makes it easier to have generated files that are committed in the source code repository. This method has a few use cases:

• to handle bootstrapping issues
• for single use code generators, when it's easier to review the generated code than the code generator itself

In particular, it provides a nice way to have a jbuild files that are generated and committed in the source tree. This is used in jbuilder itself, in doc/jbuild.

Additionally, this PR provide the necessary support for tools that generate corrected files, such as expectation tests, some ppx rewriters, etc...

Details

This PR adds two new forms in actions:

• (promote (<a> as <b>) (<c> as <d>) ...)
• (promote-if (<a> as <b>) (<c> as <d>) ...)

Both these forms takes as argument a list of S-expression of the form (<file1> as <file2>). Every form (<file1> as <file2>) means that <file1> is a generated file and we want to copy it as <file2> directly in the source tree. The only difference between the two forms is that promote-if will ignore entries where the generated file doesn't exist. This is for a special case detailed below.

In any case, when the generated files are promoted, all the entries in the list are processed at once as this sometimes matters for consistency. For instance, if we were using this in the OCaml compiler itself, we would write this in boot/jbuild:

(alias
 ((name bootstrap)
  (action (promote (../ocamlc as ocamlc) (../ocamldep as ocamldep) ...))))

Jbuilder currently doesn't support dynamic generation of jbuild files, however, using this feature, one can have a generated jbuild file that is committed in the source tree and kept up-to-date. Since in such cases we often want both static stanzas and generated ones, this PR also add an (include <file>) stanza, used to include the contents of another source file in the current jbuild file. By combining include and promotion, one can do the following:

(include jbuild.inc)

(rule (with-stdout-to jbuild.inc.gen (run ./gen-jbuild)))

(alias
 ((name update-jbuild)
  (action (promote (jbuild.inc.gen as jbuild.inc)))))

--promote command line argument

The interpretation of promote actions is controlled via the command line option --promote <mode> which supports the 3 following mode:

• ignore: completely ignore promote actions
• check: check that the two files are equal and print a diff if not
• copy: same as check, but additionally copy over the generated file to the source tree. This is the default

The actual copy is always done after the build has completed, to avoid race conditions.

The -p/--for-release-of-packages command line option implies --promote ignore. This option is meant for release builds and ignoring promotion will often cut down the number of dependencies and speed up the build.

Usage for .corrected files

Several systems such as expectation tests, [@@deriving_inline ...] attribute, cinaps, etc... work by checking that the source file is a quine. I.e. the interpretation of some specific elements in the source files will produce some output that will immediately these elements in the source tree. For instance with expectation tests:

print_string "hello";
[%expect "hello"]

Such systems work by interpreting such elements, replacing the expectation by the actual output and checking the resulting file is the same as the source file. When there are not, a diff must be displayed to the user and optionally the corrected file must be copied over to the source tree.

promote-if is intended to handle such cases, and in particular unify the management of such corrected files. For instance, when running expectation tests, the test runtime might generate some file.ml.corrected files. What we can do is add (promote-if (file.ml.corrected file.ml) ...) after the execution of the test to let jbuilder handle the diffing and promotion.

Currently, these systems all handle the diffing and sometimes promotion themselves. However the promotion part, when done is not compatible with jbuilder as it performs the build in a separate directory. They will need to be slightly modified in order to work with this mechanism.

[Chris00]

comit → commit

[Chris00]

Shouldn't promote-if also be mentioned here?

[ghost]

Indeed

[ghost]

Thinking about it again, I changed the default to copy. "promote" seems to imply that it should copy the files

[ghost]

I forgot to push the commit to change the default, I just pushed it to master.