[new release] dune (1.7.0)

[rgrinberg]

CHANGES:

• Second step of the deprecation of jbuilder: the jbuilder binary
  now emits a warning on every startup and both jbuilder and dune
  emit warnings when encountering jbuild files (Second step of jbuilder migration dune#1752, @diml)

• Change the layout of build artifacts inside _build. The new layout enables
  optimizations that depend on the presence of .cmx files of private modules
  (Change compilation layout dune#1676, @bobot)

• Fix merlin handling of private module visibility (Fix merlin handling of private module dune#1653 @bobot)

• unstable-fmt: use boxes to wrap some lists (dune unstable-fmt: use boxes dune#1608, fix Use box-like algorithm in dune fmt dune#1153, @emillon,
  thanks to @rgrinberg)

• skip directories when looking up programs in the PATH (Skip directories when looking up programs in the PATH dune#1628, fixes
  Dune attempts to run non-executable in $PATH, but only under certain circumstances dune#1616, @diml)

• Use lsof on macOS to implement --stats (Tentative fix for issue #1634 dune#1636, fixes --stats doesn't work on OSX dune#1634, @xclerc)

• Generate dune-package files for every package. These files are installed and
  read instead of META files whenever they are available (Dune Package dune#1329, @rgrinberg)

• Fix preprocessing for libraries with (include_subdirs ..) ((preprocess (action ...)) cannot be used with (include_subdirs unqualified) dune#1624, fix Add test for using (include_subdirs unqualified) + (preprocess (action ...)) dune#1626,
  @nojb, @rgrinberg)

• Do not generate targets for archive that don't match the modes field.
  (Do not generate extra targets for bytecode only libs dune#1632, fix building @all includes .cmx/.o targets dune#1617, @rgrinberg)

• When executing actions, open files lazily and close them as soon as
  possible in order to reduce the maximum number of file descriptors
  opened by Dune (Delay opening redirected files until execing cmd dune#1635, Revert #1635 + fix for #1633 dune#1643, fixes Build failure on macOS with Dune master dune#1633, @jonludlam, @rgrinberg,
  @diml)

• Reimplement the core of Dune using a new generic memoization system
  (Memoization function dune#1489, @rudihorn, @diml)

• Replace the broken cycle detection algorithm by a state of the art
  one from this paper (Memoization function dune#1489,
  @rudihorn)

• Get the correct environment node for multi project workspaces (Fix getting node for directories dune#1648,
  @rgrinberg)

• Add dune compute to call internal memoized functions (Compute Console Command dune#1528,
  @rudihorn, @diml)

• Add --trace-file option to trace dune internals (Add --trace-file option to output catapult traces dune#1639, fix [IDEA] Use Chrome's tracing view to profile dune builds dune#1180, @emillon)

• Add --no-print-directory (borrowed from GNU make) to suppress
  Entering directory messages. (Add --no-print-directory command line argument dune#1668, @dra27)

• Remove --stats and track fd usage in --trace-file (Catapult improvements dune#1667, @emillon)

• Add virtual libraries feature and enable it by default (Enable virtual libraries in dune 1.7 dune#1430 fixes Virtual libraries dune#921,
  @rgrinberg)

• Fix handling of Control+C in watch mode (Fix handling of Control+C in watch mode dune#1678, fixes Ctrl+C doesn't work well in watch mode dune#1671, @diml)

• Look for jsoo runtime in the same dir as the js_of_ocaml binary
  when the ocamlfind package is not available (js_of_ocaml: look for runtime in the same dir as executable if ocamlfind pkg unavailable dune#1467, @nojb)

• Make the seq package available for OCaml >= 4.07 (Add support for the seq package for 4.07+ dune#1714, @rgrinberg)

• Add locations to error messages where a rule fails to generate targets and
  rules that require files outside the build/source directory. (Improve location for get_file_spec errors dune#1708, fixes
  "No rule found for" errors are bad dune#848, @rgrinberg)

• Let Configurator handle sizeof (in addition to negative numbers).
  (configurator: allow sizeof in C symbols import dune#1726, fixes configurator: add an unsigned int type for C symbol imports dune#1723, @Chris00)

• Fix an issue causing menhir generated parsers to fail to build in
  some cases. The fix is to systematically use -short-paths when
  calling ocamlc -i (Always pass -short-paths when calling ocamlc -i dune#1743, fix Dependency error with "dune -p" and "infer true" for menhir dune#1504, @diml)

• Never raise when printing located errors. The code that would print the
  location excerpts was prone to raising. (Never raise when printing errors dune#1744, fix ocaml 4.02.3: internal error raised from Exn.Code_error dune#1736, @rgrinberg)

• Add a dune upgrade command for upgrading jbuilder projects to Dune
  (Automatic jbuild --> dune upgrader dune#1749, @diml)

• When automatically creating a dune-project file, insert the
  detected name in it (Automatic jbuild --> dune upgrader dune#1749, @diml)

• Add (implicit_transitive_deps <bool>) mode to dune projects. When this mode
  is turned off, transitive dependencies are not accessible. Only listed
  dependencies are directly accessible. (Restore obj dir for utop toplevels dune#1734, RFC: add (strict_includes) in (library) and (executable) dune#430, @rgrinberg, @hnrgrgr)

• Add toplevel stanza. This stanza is used to define toplevels with libraries
  already preloaded. (Add a toplevel stanza dune#1713, @rgrinberg)

• Generate .merlin files that account for normal preprocessors defined using a
  subset of the action language. (Use merlin's -pp support dune#1768, @rgrinberg)

• Emit (orig_src_dir <path>) metadata in dune-package for dune packages
  built with --store-orig-source-dir command line flag (also controlled by
  DUNE_STORE_ORIG_SOURCE_DIR env variable). This is later used to generate
  .merlin with S-directives pointed to original source locations and thus
  allowing merlin to see those. (Track original source location in lib_info/dune_package dune#1750, @andreypopp)

• Improve the behavior of dune promote when the files to be promoted have been
  deleted. (Improve the behaviour of promotion when files are missings dune#1775, fixes dune promote internal error dune#1772, @diml)

• unstable-fmt: preserve comments (unstable-fmt: preserve comments dune#1766, @emillon)

• Pass flags correctly when using staged_pps (Pass flags when using staged_pps dune#1779, fixes staged_pps doesn't cope well with bisect_ppx -conditional dune#1774, @diml)

• Fix an issue with the use of (mode promote) in the menhir
  stanza. It was previously causing intermediate mock files to be
  promoted (Do not promote menhir mock files dune#1783, fixes Menhir mode promote will generate .mock file in source directory dune#1781, @diml)

• unstable-fmt: ignore files using OCaml syntax (unstable-fmt: ignore files in OCaml syntax dune#1784, @emillon)

• Configurator: Add which function to replace the which command line utility
  in a cross platform way. (Configurator: Export additional existing functions  dune#1773, fixes Consider adding a cross platform which to configurator dune#1705, @Chris00)

• Make configurator append paths to $PKG_CONFIG_PATH on macOS. Previously it
  was prepending paths and thus $PKG_CONFIG_PATH set by users could have been
  overridden by homebrew installed libraries (Allow $PKG_CONFIG_PATH to have effect on macOS dune#1785, @andreypopp)

• Disallow c/cxx sources that share an object file in the same stubs archive.
  This means that foo.c and foo.cpp can no longer exist in the same library.
  (OSL handling for {c,cxx}_names dune#1788, @rgrinberg)

• Forbid use of %{targets} (or ${@} in jbuild files) inside
  preprocessing actions
  (Forbid %{targets} in pp action dune#1812, fixes Dune 1.6.3 turn ocp-browser into a noop dune#1811, @diml)

• Add DUNE_PROFILE environment variable to easily set the profile. (Add DUNE_PROFILE environment variable dune#1806,
  @rgrinberg)

• Deprecate the undocumented (no_keep_locs) field. It was only
  necessary until virtual libraries were supported (Deprecate no_keep_locs dune#1822, fix deprecate no-keep-locs in favour of virtual libraries dune#1816,
  @diml)

• Rename unstable-fmt to format-dune-file and remove its --inplace option.
  (Add format rules for dune files dune#1821, @emillon).

• Autoformatting: (using fmt 1.1) will also format dune files (Add format rules for dune files dune#1821, @emillon).

• Autoformatting: record dependencies on .ocamlformat-ignore files (Format rules: register .ocamlformat-ignore dep dune#1824,
  fixes allow some files to not be reformatted with @fmt dune#1793, @emillon)

[rgrinberg]

I've looked at the failures and nothing seems relevant.

[kit-ty-kate]

#=== ERROR while compiling mdx.1.1.0 ==========================================#
# context              2.0.3 | linux/x86_64 | ocaml-base-compiler.4.07.1 | git+file:///home/opam/opam-repository
# path                 ~/.opam/4.07/.opam-switch/build/mdx.1.1.0
# command              ~/.opam/4.07/bin/dune build -p mdx -j 71
# exit-code            1
# env-file             ~/.opam/log/mdx-19-d42b2b.env
# output-file          ~/.opam/log/mdx-19-d42b2b.out
### output ###
# File "bin/test/dune", line 7, characters 18-22:
# 7 |  (libraries   cli cppo mdx.top))
#                       ^^^^
# Error: Library "cppo" not found.
# Hint: try: dune external-lib-deps --missing -j 71 -p mdx @@default
#     ocamlopt lib/top/.mdx_top.objs/native/mdx_top.{cmx,o}
# File "_none_", line 1:
# Warning 58: no cmx file was found in path for module Topdirs, and its interface was not compiled with -opaque
# File "_none_", line 1:
# Warning 58: no cmx file was found in path for module Toploop, and its interface was not compiled with -opaque

seems related. It didn't fail earlier according to check.ocamllabs.io and cppo is installed. I'm not sure what's the problem

[rgrinberg]

A couple of points to address that:

• cppo is indeed not a library, it's a binary. I have my doubts that ever worked before.
• I believe that library should set its mode to bytecode only.

[kit-ty-kate]

* cppo is indeed not a library, it's a binary. I have my doubts that ever worked before.

You can see here that it works: http://check.ocamllabs.io/4.07.1/good/mdx.1.1.0
I'm not sure why or how but it's still a change compared to the last release

[rgrinberg]

Okay, I think I understand the difference now. Before 1.7, libraries that did not exist were simply "empty". This was a consequence of the findlib emulation. In 1.7, we have our own internal format for external libraries, so things are a bit more strict. It should not block the release as its still a bug in mdx basically

[samoht]

It should not block the release as its still a bug in mdx basically

I agree and this has been fixed in 1.2.0. 1.1.0 should probably be removed, though.

[avsm]

I agree; I fixed that in a few places including mdx, so older versions can be constrained to dune<1.7 without too many problems (since the fix is so easy and makes the package strictly better)

[avsm]

the spice must flow! congratulations on getting such a big feature-packed release out of the door.

[camelus]

☀️ All lint checks passed 674b1ed

• These packages passed lint tests: dune.1.7.0

---

☀️ Installability check (10441 → 10442)

• new installable packages (1): dune.1.7.0