[RFC] Dune configuration files

[ghost]

This is a proposal for the new configuration files of Dune. The most notable changes are the addition of project files and the generalization of jbuild-ignore files.

dune-project files

This is a new kind of file. It is not needed for experiments but distributed projects should all have one dune-project file at the root.

It contains the following fields:

• (lang <language> <number>) sets the language used for all dune configuration files in the project. <language> is always dune, though maybe one day one might want to provide a different language
• (name <string>) sets the name of the project. It is not mandatory and defaults to something like <anynonous-N>, so that one can use a dune-project file before chosing the project name or for quick experiments. It will be the string %%NAME%% is expanded to by dune subst
• (version <string>) sets the version of the project. This will set the version of all packages defined in the project. When the special %%VERSION%% string is used, the output of git describe ... will be used instead, since it seems to be what many users are already doing by hand
• (plugin <name> <version> <plugin-parameters>...) enable the specified plugin, see the plugin section for details
• (packages <strings>) list the packages defined by the project. This field is automatically inferred when using the opam plugin by looking at the <package>.opam files present in the same directory as the dune-project file. However, I expect that in the future we will do the opposite, i.e. list package names here and Dune will generate the opam files
• (requires <projects>) lists the projects that this projects depends on. The aim is that we can add a dune get command for fetching the transitive closure of a project into a self-contained workspace. <projects> is a list of S-expressions of the form:
  • (git <url>)
  • (git <url> <hash>)
  • <user>/<name> as a shorthand for (git https://github.com/<user>/<name>.git)
  • <user>/<name>#<hash> as a shorthand for (git https://github.com/<user>/<name>.git <hash>)

plugins

One might write one or more plugin stanzas in order to enable the given plugins in a project. The exact syntax is a follow:

(plugin <name> <version> <plugin-parameters>)

for instance:

(plugin ocaml 1.0
  (flags (-w +42)) ; specify the default flags for compiling .ml/.mli files
)
(plugin menhir 1.0)

Enabling a plugin might affect the behavior of Dune in various but will generally extend the language that is available in dune files. The plugin parameters can be guarded by a build profile, in order to use different parameters based on the selected build profile. Build profiles can be selected from the command line via --profile <profile> or in the dune-workspace file, either globally or on a per build context basis. To set specific parameters for a given profile, one must write (profile <profile> <parameters>). For instance:

(plugin ocaml 1.0
  ;; Pass -w +a only when the "dev" profile is selected
  (profile dev
   (flags (-w +a))
  ;; Always set inlining to 20
  (ocamlopt_flags (-inline 20))
)

For now, the following plugins are available:

• ocaml for OCaml specific stuff (library, executables, ocamlyacc, ...)
• menhir for the menhir stanza
• opam for the opam specific behaviors such as acknowledging <package>.opam files

Stanzas such as install and rule are builtin and are controled by (lang dune <version>).

An example of plugins that wouldn't extend the language in dune files would be a travis plugin, that would simply cause the generation of travis files that would be promoted to the source tree.

dune files

These replace jbuild files. Changes compared to current jbuild files:

• the stanza jbuild_version is no longer needed (we use the dune_version from the project file instead)
• what stanzas are available is controled by what was written in the plugins field of the dune project
• files_recursively_in is renamed source_files_recursively_in
• the various elements that are currently deprecated in jbuild files are no longer accepted

dune-fs files

These replace jbuild-ignore files. A dune-fs file contain a list of S-expression of the form (<kind> <glob> <state>), where <kind> is file, dir or _ and where <state> describe how files or directories matched by <glob> are treated. The following states are available:

• standard, this is the default
• raw, this means that the file or sub-tree is considered as a raw data. In particular any dune* file it contains will be treated as a regular source file and won't be interpreted by dune
• ignore, this means that the file is completely ignored. dune will consider that the file or directory doesn't exist and it won't be possible to depend on anything in it. It is the same as running rm -rf on matching files before running dune

When a file or directory is matched by several globs, the last one is used.

Existing jbuild-ignore files will be converted to:

(dir {<dir1>,<dir2>,...} raw)
...

The following configuration is always assumed in any directory:

(file .*    raw)
(dir  [._]* ignore)
(_    .#*   ignore)

This is the current default in jbuilder. To override the default behavior, one can add the following:

(file .* standard)

dune-workspace files

They replace jbuild-workspace files. Contrary to jbuilder, dune-workspace* files won't be used to find the root.

The only notable change is the addition of a (profile <profile>) stanza to set the default build profile and a (profile <profile>) field in context stanzas in order to select a build profile for a given build context.

[ghost]

Replying to #362 (comment) (/cc @lpw25)

I also just prefer the idea of specifying the (versioned) name of the language of the file rather than the version of the tool that interprets it.

Here, I'm proposing to list the plugins and version of Dune required in a single project file. I think it's more natural than listing it in every file. I'm also assuming that there is a builtin language in dune files for things that are independent of the language, since I assume that no one will want to opt it out.

[lpw25]

I think it's more natural than listing it in every file

I agree. Putting these stanzas in the project file is definitely better.

since I assume that no one will want to opt it out

This is an assumption I'd like to avoid baking in. By simply using (lang dune 1.1) rather than (dune_version 1.1) we leave open the possibility for people to implement alternative languages for dune files. Maybe no one will want to implement such a language from scratch and they will always share common elements with the builtin dune language -- but I don't think we should make that assumption if we don't have to.

[jberdine]

Would it make sense to allow dune-project files to specify common values for some stanzas of all contained dune files? I'm thinking something like defining what :standard in flags means in the dune-project file.

[ghost]

This is an assumption I'd like to avoid baking in. By simply using (lang dune 1.1) rather than (dune_version 1.1) we leave open the possibility for people to implement alternative languages for dune files. Maybe no one will want to implement such a language from scratch and they will always share common elements with the builtin dune language -- but I don't think we should make that assumption if we don't have to.

I see. Ok, that makes sense. BTW, just so that it's clear what is covered by this stanza, this includes:

• the set of files read by Dune
• the languages inside these files

I'm wondering whether we should still split lang and plugins or merge them into:

(lang (dune 1.0) (ocaml 1.0) ...)

Would it make sense to allow dune-project files to specify common values for some stanzas of all contained dune files? I'm thinking something like defining what :standard in flags means in the dune-project file.

I started working on this in #419. It still at the experiment/design stage, but so far I was more going in the direction of putting such parameters in dune files directly. This would allow to override the default in any sub-tree of a project. I'm also trying to think about how this will interact with other features we want to add.

[lpw25]

just so that it's clear what is covered by this stanza, this includes:

• the set of files read by Dune
• the languages inside these files

I was just thinking of the language inside these files. If people wanted to use a different name for their build files that could always be added as another stanza.

I'm wondering whether we should still split lang and plugins or merge them into

If they are in a merged stanza I would probably still make the top-level one special like:

(lang dune 1.0 ((ocaml 1.0) ...))

although I think it is also fine to just keep the plugins in a separate stanza.

[lpw25]

I was more going in the direction of putting such parameters in dune files directly

I wonder if they might be better in just the project files. That way, to understand how something in directory project/foo/bar/baz is built you only need to read project/dune-project and project/foo/bar/baz/dune rather than also needing to read project/dune, project/foo/dune and project/foo/bar/dune.

Are there particular use cases for more local environments that you have in mind?

[lpw25]

I wonder if they might be better in just the project files

Having said that, I quite like the idea of being able to use plugins that affect the environment, and I don't think we'll be able to use plugins to interpret dune-project files. So I'm not sure either way.

[lpw25]

Having said that, I quite like the idea of being able to use plugins that affect the environment, and I don't think we'll be able to use plugins to interpret dune-project files. So I'm not sure either way.

I suppose, since the plugins are specified in the project file they could provide a function of type Environment.t -> Environment.t that was run before reading any of the dune files. That might provide enough to have things like a bisect plugin that changes the environment in some profiles.

That approach would lead me to prefer having environments specified in project files -- to keep the build easier to understand, and using a separate plugin stanza since the plugins would now be affecting more that just how dune files were interpreted.

[ghost]

I was just thinking of the language inside these files. If people wanted to use a different name for their build files that could always be added as another stanza.

I was more thinking about Dune itself: in the future we might want to add, remove or rename some configuration files.

Regarding the environment. One thing I'm wondering is whether we should use the environment only for plugins parameters or generalize it to arbitrary variables that could be used for simple meta-programming purposes. For instance that would cover #397 as well.

I think where we put the environment doesn't matter regarding what can evaluate it: we can always delay the interpretation of the environment until we have loaded the plugins. Keeping the environment only in project files to keep things simple is a good point, but if later we also add some simple meta-programming facility and the two features look similar but are two independent systems, it's going to be a bit weird I think.

BTW, in #419 I added a command printenv, which helps a bit in case the environment is per-directory:

$ jbuilder printenv src/
(
 (flags (-w -40 -plop -truc))
 (ocamlc_flags (-g))
 (ocamlopt_flags (-g))
)

[bikallem]

(requires ) lists the projects that this projects depends on. The aim is that we can add a dune get command for fetching the transitive closure of a project into a self-contained workspace. is a list of S-expressions of the form:

(git <url>)
<user>/<name> as a shorthand for (git https://github.com/<user>/<name>.git)

Doesn't this overlap with packages? Are you aiming for use-cases where opam is not there and/or its usage is not desired?

[lpw25]

I think where we put the environment doesn't matter regarding what can evaluate it: we can always delay the interpretation of the environment until we have loaded the plugins.

Good point. Whilst we probably can't use plugins everywhere in a project file we could definitely allow them in some parts.

Keeping the environment only in project files to keep things simple is a good point, but if later we also add some simple meta-programming facility and the two features look similar but are two independent systems, it's going to be a bit weird I think.

I think my objection is to having all the directories in the path affect the environment. I think it would be fine for the environment of project/foo/bar/baz to be defined by the combination of project/dune-project and project/foo/bar/baz/dune. So as long as your syntax for changing the environment is allowed in both dune-project and dune files then you should be able to just have a single system for meta-programming/environment management.

[ghost]

I think my objection is to having all the directories in the path affect the environment. I think it would be fine for the environment of project/foo/bar/baz to be defined by the combination of project/dune-project and project/foo/bar/baz/dune. So as long as your syntax for changing the environment is allowed in both dune-project and dune files then you should be able to just have a single system for meta-programming/environment management.

Indeed. I was also thinking that the metaprogramming idea is likely to take more time to design and implement, while allowing to set the default flags etc is more urgent. So I think we can start by allowing to set some parameters in dune-project files and we'll make sure that whatever we do later is compatible with this.

To avoid confusion about how things are interpreted, I suggest to replace the plugins stanza by a list of plugin stanzas of the form:

(plugin <name> <version> <parameters>)

for instance:

(plugin ocaml 1.0
  (flags          (:standard -w +a))
  (ocamlopt_flags (:standard -inline 100)))

and to select according to the profile:

(plugin ocaml 1.0
  (profile dev
   (flags          (:standard -w +a))
   (ocamlopt_flags (:standard -inline 100))
  (profile release
    (flags ...))))

[ghost]

Doesn't this overlap with packages?

Not really, packages is to describe what packages the project you are defining will produce. While requires is to list the projects you depend on.

The idea of requires is that if you clone all the listed projects and their dependencies recursively, then you get a big workspace where you are guaranteed that your project will build without external dependencies (apart from a C compiler, etc...).

This is very convenient for development.

Are you aiming for use-cases where opam is not there and/or its usage is not desired?

Currently you can already do multi project development with Dune by creating a workspace composed of several projects. This is just to make it easier.

[bikallem]

Thanks for the clarification. Yes, that makes sense now. I was confusing packages/requires with libraries stanza in library/executable rules.

[avsm]

We could probably generate the requires stanza from the opam database, via the dev-repo information. It would be good if it supported a specific git hash so that we can use it as a lock file for dependencies as well. It's likely that simply cloning the master branch of all the dependencies is likely to fail in practise.

[ghost]

@avsm I was wondering about that. Adding a hash/tag/branch name seems fine, but I think it's not enough for all cases. For instance, let's say you create a PR for a multi-repo feature, and you want to provide an easy way for developers to test your work. You might put the hash/branch name in the dune project file directly, but then you have to remember to change them before merging the PR.

Really, what you want is to do is:

$ dune get https://github.com/foo/bar/pull/42

which suggests that the specific branch names should be attached to the PR.

[ghost]

I've updated the description of this PR to integrate the changes discussed here

[andreypopp]

Would multiple dune-fs be supported by dune?

We have a use case with esy where all packages are installed into ./node_modules directory — dune tries to traverse it but we don't need that as packages inside ./node_modules are handled by esy itself.

Right now we are asking users to have node_modules record inside jbuild-ignore file but it would be good if instead of that we could place node_modules/dune-fs file automatically which would instruct dune to ignore everything there.

[ghost]

Yes, you'll be able to put these files anywhere in the tree. BTW, you can already do that with jbuild-ignore files, i.e. you can have node_modules/jbuild-ignore. However, you can't use globs in them, so you'd need one line per sub-directory of node_modules.

[ghost]

Some people mentioned to me that plugin is a bit too technical and doesn't add any value for the user. So instead we can use using:

(using ocaml 1.0)

BTW, this is how I see the versioning of the various plugins such as ocaml: the minor version will be increased whenever we add a new feature but stay backward compatible and the major version will be increased in one of the following cases:

• either when we want to do a breaking change
• either when we want to change the default behavior to keep up with the evolution of the language

For instance with ocaml 2.0 -safe-string should be used by default on older compilers. New major versions of the OCaml plugins should not be too frequent, for instance once or twice a year to follow new releases of the compiler.

[bluddy]

It's a little bit crazy to me that we're mainstreaming jbuilder without consider transitioning to a more common file format. The historical choice of s-expression seems mostly due to Jane Street's preferences, but s-exp has obvious issues:

1. It scares away people who are new to the language.
2. It's not very readable, particular when parentheses start to build up.
3. It's not a well accepted convention in the industry with regard to tooling, knowledge carrying over etc.

JSON makes a lot more sense to me than keeping s-expressions.

[ghost]

how would you write this is JSON?

(chdir foo (run bar x y z))

[bluddy]

One of the problems with s-exp is not knowing how to parse them. It's not clear if something is a list, a dictionary, or a string. I don't actually know how to translate the example you gave because I don't know what it means. Instead, I pulled something out of the dune unit tests and attempted to translate it:

(alias
 ((name runtest)
  (deps (tests.mlt
         (glob_files ${SCOPE_ROOT}/src/.dune.objs/*.cmi)
         (files_recursively_in findlib-db)))
  (action (chdir ${SCOPE_ROOT}
           (progn
            (run ${exe:expect_test.exe} ${<})
            (diff? ${<} ${<}.corrected))))))

vs

{"alias":
  {"name" : "runtest",
   "deps": [
       "test.mlt",
       {"glob_files": "${SCOPE_ROOT}/src/.dune.objs/*.cmi"},
       {"files_recursively_in": "findlib-db"}
    ],
   "action": [
       {"chdir": "$(SCOPE_ROOT)",
        "progn": [ {"run": ["${exe:expect_test.exe}", "${<}"] },
                   {"diff?": ["${<}", "${<}.corrected"]}
                 ]
       }
   ]
  }
}
           

Flattening it a little, we get (using json5):

{alias: true,
 name : "runtest",
 deps: [
       "test.mlt",
       {glob_files: "${SCOPE_ROOT}/src/.dune.objs/*.cmi"},
       {files_recursively_in: "findlib-db"}
    ],
 action:
       {chdir: "$(SCOPE_ROOT)",
        progn: [ {run: ["${exe:expect_test.exe}", "${<}"] },
                 {diff?: ["${<}", "${<}.corrected"]}
               ]
       },
}

[lpw25]

JSON is a rubbish configuration format. It's just not designed for it. It's overly verbose, has no comment syntax, not easily extensible, etc.

Quoting http://json.org:

JSON (JavaScript Object Notation) is a lightweight data-interchange format.

Data-interchange formats make crappy configuration languages, it was true for XML and it's true for JSON.

[mjambon]

@lpw25 json5 has comments, no quotes around fields names, and line breaks in string literals. Those major annoyances of json being gone, I think we have a good format for configuration files.

Now, as pointed out by @diml earlier, there's no good way of writing (chdir foo (run bar x y z)) in json or json5. This is however a program, not configuration data. I think it would help to clarify the scope of jbuild files. Will every user have to put custom logic into their build system? If only a few power users on occasion need to write dune plugins and they have to do so in a "crappy" language, then so be it. "Keep simple things easy and complex things possible" is a good design guideline.

[bluddy]

@mjambon @diml would you kindly explain the meaning of (chdir foo (run bar x y z))?

[yminsky]

My view is that the argument for switching to json or json5 is relatively weak, especially given the fact that there is functionality that we want that it does not support. Also, while s-expressions aren't terribly common outside of the OCaml world, they're quite common inside of it, being used for serialization and display in many OCaml libraries, not just Jane Street's. (Indeed, related to that, we have some work that's in progress to generate some syntax-aware auto-completion support for s-expressions, which we hope to use both inside Jane Street and make available for external use as well.)

Besides, switching now would take precious engineering cycles away from other more important tasks. Beware of Wadler's Law.

[bluddy]

Besides, switching now would take precious engineering cycles away from other more important tasks.

Except that we're aiming to build the main and possibly only build tool for OCaml, and the rebranding gives us a chance to reconsider some design decisions. We want to avoid the possibility of us having to come back later with the feedback of "newbies are being scared away by this and hugely prefer something like bsb-native" leading to yet another tool being formed, or "complex projects just prefer the more natural syntax of other solutions".

I share @agarwal's opinion (seen in his own build tool) that if we're adding complexity at the level of an embedded language, rather than slowly re-implementing lisp, I'd rather we used OCaml. The difference here would be that we're just building a config data structure rather than driving the process, as has been done earlier.

Roughly translating the above example, I'd much rather have

let config env = [
  Alias {
    name="runtest";
    deps=[
      Dep "test.mlt";
      Glob_files (append env.scope_root "/src/.dune.objs/*.cmi");
      Files_recurse "findlib-db"
    ]
  };
  Action {
    default_action with
    chdir=env.scope_root;
    prog= [
        Run (fun state -> cmd ["expect_test.exe"; state.filename]);
        Diff (fun state -> (state.filename, state.filename ^ ".corrected")
    ]
  };
]

Some of these constructors could use functions with optional arguments, but you get the idea.

I find this much clearer than s-exp, and it gives us the ability to completely get rid of shell-isms, or at least deprecate them over time.

[Chris00]

explain the meaning of (chdir foo (run bar x y z))

Change the directory to foo and execute then execute the command — see the actions DSL.

[bluddy]

Change the directory to foo and execute then execute the command — see the actions DSL.

Ah ok. Thanks!

Well, I still prefer the OCaml version of that to a s-exp based language.

[agarwal]

Some people like json. Some like s-expressions. We won't get anywhere trying to convince the other side that json or s-expressions are superior. IMO the real issue is how much do we care to be friendly to newcomers. Every decision like this perpetuates the myth that OCaml is somehow weird. Is that the direction we want to keep going in?

Imagine yourself as a non-experienced OCaml programmer. In that case, JSON is the obvious choice and s-expressions are the non-standard choice. When is it justified to use a non-standard technology? Only if it is substantially superior to the standard one. That's why I use OCaml. It isn't the standard, but is deeply superior to the standard languages. Are s-expressions deeply superior to JSON (for the purpose under consideration)? We should try hard to justify the choice, and if we can't, maybe we should be more normal and do what everyone else is.

how would you write ... (chdir foo (run bar x y z))

If you imagine yourself as a newcomer to OCaml, you would have chosen JSON because it is the standard, and then I think you would have found a solution. You would have made it work somehow. (In fact, I don't see the problem. How about {"chdir": {"dir": "foo", "action": "bar x y z"}}.)

switching now would take precious engineering cycles away from other more important tasks

This is only true if you already believe that the switch isn't valuable, but that's the point being debated. Also, from the perspective of most programmers, we already made a switch. It's us, the OCaml community, that switched to s-expressions. Was it worth the cycles to make that switch? And the cycles aren't complete. You just mentioned more work needed to support auto-completion for s-expressions, but editor support for json is already extensive.

I usually avoid such debates because I'm never sure myself. Maybe we'll make s-expressions the new standard. I'd rather make OCaml the new standard.

[lpw25]

Firstly, can I make a request for people to make new issues if they want to start discussions like this, rather than hijack an existing discussion. Otherwise it makes it harder to manage the discussions.

It's very clear dune isn't going to switch to JSON. I don't think any of the maintainers want to switch, and there have been no convincing arguments (and no technical arguments at all) in favour of the change. At this point the issue should be considered "won't fix" and the discussion concluded. Any further discussion on that point is just going to be wasting people's time.

I suppose the suggestion to use OCaml could be considered a separate issue and usefully discussed further. Perhaps in another issue though.

[bluddy]

Done

[ghost]

This ticket is almost finished. The only two things left are:

1. implement (using <extension> <version>)
2. setting the version using git describe when appropriate

For 1., I have a work in progress. For now the only available extension will be menhir, which is developed inside of Dune. Presenting it as an extension will allow to de-synchronise its versioning from the Dune versioning.

For 2., I'm basically waiting to see what happens with ocaml/opam#2932

[ghost]

This is now done, I opened #880 to discuss the question of versions.

[hcarty]

Is requires intended to be part of 1.0.0?

[ghost]

Not yet, no. It requires a non-negligible amount of work (and design). For now we are focusing on features that require breaking changes, since the switch to new configuration files is a good opportunity for it. requires can be added later without breaking compatibility.

[hcarty]

Thanks @diml - that makes perfect sense. I just wanted to check since the issue was closed.