Skip to content

Commit

Permalink
[dune] Switch to a Dune-based build system.
Browse files Browse the repository at this point in the history
Over the last months we have added support to build Coq with Dune, I
feel we have reached a point where we can start to discuss if we
should replace the make-based system with a Dune-based one.

The *main rationale* for the switch is that maintaining two build
systems is hard, and Dune seems superior to make in almost every
aspect.

Indeed, I think it is going to be hard to find technical arguments to
defend the make-base system. Dune outperforms it on almost every
possible metric.

Additionally, the make-based system has grew organically over the
years, and these days we are spending significant developer resources
on it. The number of bugs that would be fixed by Dune is large [see coq#8052].
It is not a coincidence that few large projects do use `make` anymore,
but most have moved to `CMake`, `Ninja`, or some other alternative.

On this side, Dune provides a well-defined model that seems to git
Coq's present and future necessities well.

*Blockers*: the single blocker for a merge as of today, apart from the
depending PRs, is the lack of native-compute support. This is due to a
technical limitation wrt targets and can be solved in several
different ways. Also, it is likely that a smooth developer profile is
not possible until ocaml/dune#1155 lands in
Dune itself. ocaml/dune#1377 may be
convenient for the reference manual but we can have a small workaround
to generate the install file for now.

*Risk analysis*: it is important to understand the risks that such a
move would entail. After Dune support is merged we could always go
back to a make-based system, however we are very likely to depend on
Dune-specific features that would be hard to replicate. The main risks are:

- lock-in: indeed lock-in risk is significant and Coq's source code
  may depend on some Dune features. On the other hand, Dune is
  strongly poised to be the standard build tool for the OCaml platform
  and ecosystem, and more than 50% of OPAM packages use it.

  It is safe to say that if Dune would become unsupported in the
  future, there would be more important problems than Coq itself.

- lack of in-house knowledge: indeed, Dune requires some specific
  training in order to understand its build rules, which may become a
  problem. This risk is mitigated in 2 different ways and attenuated
  by an additional consideration: first, Dune rules are fairly simple
  and declarative and it is reasonable to expect developers get to
  know them without too much effort. This is one of the reasons for
  the high adoption numbers in the ecosystem. Second, Dune developers
  are very reactive and care about Coq, thus it is safe to assume that
  we would get external help if needed. The additional consideration
  that may make this less of a problem is that our in-house knowledge
  of make may be even worse than Dune's. A non-negligible number of
  developers have expressed discomfort with make and the amount of
  required knowledge to master make seems way higher than what you
  need to use Dune.

- lack of flexibility: indeed, Dune is way less flexible than
  make. Dune only knows how to do well two things: building OCaml
  executables and libraries. However, that is basically 99% of what
  building Coq entails; building other parts [documentation or Coq
  libraries] is fairly straightforward and seem to pose not a problem.

- lack of maturity: indeed, Dune develops fast, it is not free of
  bugs, and some amount of adaptation is expected from us. This risk
  can only be mitigated if there is continued developer interest.  In
  the worst case, Coq could become stuck in a particular Dune version.

- bootstrapping: this is not a problem as witnessed by coq#8615, Dune can
  be bootstrapped very easily as it depends only on OCaml and it is
  designed to do so.
  • Loading branch information
ejgallego committed Oct 18, 2018
1 parent 1692c4b commit 79534f2
Show file tree
Hide file tree
Showing 14 changed files with 490 additions and 491 deletions.
120 changes: 38 additions & 82 deletions .gitlab-ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -54,6 +54,20 @@ after_script:

# TODO figure out how to build doc for installed Coq
.build-template: &build-template
stage: build
artifacts:
name: "$CI_JOB_NAME"
paths:
- _build
expire_in: 1 week
script:
- set -e
- echo 'start:coq.build'
- make world
- echo 'end:coq:build'
- set +e

.build-make-template: &build-make-template
stage: build
artifacts:
name: "$CI_JOB_NAME"
Expand All @@ -67,84 +81,52 @@ after_script:
- set -e

- echo 'start:coq.clean'
- make clean # ensure that `make clean` works on a fresh clone
- make -f Makefile.make clean # ensure that `make clean` works on a fresh clone
- echo 'end:coq.clean'

- echo 'start:coq.config'
- ./configure -warn-error yes -prefix "$(pwd)/_install_ci" ${COQ_EXTRA_CONF}"$COQ_EXTRA_CONF_QUOTE"
- ./configure -warn-error yes -prefix "$(pwd)/_install_ci"
- echo 'end:coq.config'

- echo 'start:coq.build'
- make -j "$NJOBS" byte
- make -j "$NJOBS" world $EXTRA_TARGET
- make test-suite/misc/universes/all_stdlib.v
- make -f Makefile.make -j "$NJOBS" byte
- make -f Makefile.make -j "$NJOBS" world
- make -f Makefile.make test-suite/misc/universes/all_stdlib.v
- echo 'end:coq:build'

- echo 'start:coq.install'
- make install
- make install-byte
- make -f Makefile.make install
- make -f Makefile.make install-byte
- cp bin/fake_ide _install_ci/bin/
- echo 'end:coq.install'

- set +e

.dune-template: &dune-template
dependencies: []
stage: build
artifacts:
name: "$CI_JOB_NAME"
paths:
- _build/
expire_in: 1 week
script:
- set -e
- make -f Makefile.dune "$DUNE_TARGET"
- set +e

# every non build job must set dependencies otherwise all build
# artifacts are used together and we may get some random Coq. To that
# purpose, we add a spurious dependency `not-a-real-job` that must be
# overridden otherwise the CI will fail.

.doc-template: &doc-template
stage: test
dependencies:
- not-a-real-job
script:
- SPHINXENV='COQBIN="'"$PWD"'/_install_ci/bin/" COQBOOT=no'
- make -j "$NJOBS" SPHINXENV="$SPHINXENV" SPHINX_DEPS= refman
- make install-doc-sphinx
artifacts:
name: "$CI_JOB_NAME"
paths:
- _install_ci/share/doc/coq/

# set dependencies when using
.test-suite-template: &test-suite-template
stage: test
dependencies:
- not-a-real-job
script:
- cd test-suite
- make clean
# careful with the ending /
- BIN=$(readlink -f ../_install_ci/bin)/
- LIB=$(readlink -f ../_install_ci/lib/coq)/
- export OCAMLPATH=$(readlink -f ../_install_ci/lib/):"$OCAMLPATH"
- make -j "$NJOBS" BIN="$BIN" LIB="$LIB" all
- make test-suite
artifacts:
name: "$CI_JOB_NAME.logs"
when: on_failure
paths:
- test-suite/logs
- _build/default/test-suite/logs

# set dependencies when using
.validate-template: &validate-template
stage: test
dependencies:
- not-a-real-job
script:
- cd _install_ci
- cd _build/install/default
- find lib/coq/ -name '*.vo' -print0 > vofiles
- for regexp in 's/.vo//' 's:lib/coq/plugins:Coq:' 's:lib/coq/theories:Coq:' 's:/:.:g'; do sed -z -i "$regexp" vofiles; done
- xargs -0 --arg-file=vofiles bin/coqchk -boot -silent -o -m -coqlib lib/coq/
Expand Down Expand Up @@ -189,39 +171,32 @@ after_script:
variables:
- $WINDOWS =~ /enabled/

build:make:base:
<<: *build-make-template

build:base:
<<: *build-template
variables:
COQ_EXTRA_CONF: "-native-compiler yes -coqide opt"
# coqdoc for stdlib, until we know how to build it from installed Coq
EXTRA_TARGET: "stdlib"

# no coqide for 32bit: libgtk installation problems
build:base+32bit:
<<: *build-template
script:
- make voboot
- dune build coq.install
- dune build coqide-server.install
variables:
OPAM_VARIANT: "+32bit"
COQ_EXTRA_CONF: "-native-compiler yes"

build:edge:
<<: *build-template
variables:
OPAM_SWITCH: edge
COQ_EXTRA_CONF: "-native-compiler yes -coqide opt"

build:edge+flambda:
<<: *build-template
variables:
OPAM_SWITCH: edge
OPAM_VARIANT: "+flambda"
COQ_EXTRA_CONF: "-native-compiler yes -coqide opt -flambda-opts "
COQ_EXTRA_CONF_QUOTE: "-O3 -unbox-closures"

build:egde:dune:dev:
<<: *dune-template
variables:
OPAM_SWITCH: edge
DUNE_TARGET: world

windows64:
<<: *windows-template
Expand Down Expand Up @@ -278,30 +253,24 @@ pkg:nix:
- nix-build-coq.drv-0/*/test-suite/logs

doc:refman:
<<: *doc-template
dependencies:
- build:base

doc:refman:dune:
<<: *dune-template
dependencies:
- build:base
stage: test
dependencies:
- build:egde:dune:dev
- build:edge
script:
- make refman
variables:
OPAM_SWITCH: edge
DUNE_TARGET: refman
artifacts:
name: "$CI_JOB_NAME"
paths:
- _build/default/doc/sphinx/_build/html

doc:ml-api:odoc:
doc:ml-api:
stage: test
dependencies:
- build:egde:dune:dev
script: make -f Makefile.dune apidoc
- build:edge
script:
- make apidoc
variables:
OPAM_SWITCH: edge
artifacts:
Expand Down Expand Up @@ -336,19 +305,6 @@ test-suite:edge+flambda:
OPAM_SWITCH: edge
OPAM_VARIANT: "+flambda"

test-suite:egde:dune:dev:
stage: test
dependencies:
- build:egde:dune:dev
script: make -f Makefile.dune test-suite
variables:
OPAM_SWITCH: edge
artifacts:
name: "$CI_JOB_NAME.logs"
when: on_failure
paths:
- _build/default/test-suite/logs

validate:base:
<<: *validate-template
dependencies:
Expand Down
4 changes: 2 additions & 2 deletions .travis.yml
Original file line number Diff line number Diff line change
Expand Up @@ -110,11 +110,11 @@ script:
- echo -en 'travis_fold:end:coq.config\\r'

- echo 'Building Coq...' && echo -en 'travis_fold:start:coq.build\\r'
- make -j $NJOBS $MAIN_TARGET
- make -f Makefile.make -j $NJOBS $MAIN_TARGET
- echo -en 'travis_fold:end:coq.build\\r'

- echo 'Running tests...' && echo -en 'travis_fold:start:coq.test\\r'
- if [ -n "$TEST_TARGET" ]; then $TW make -j $NJOBS $TEST_TARGET; fi
- if [ -n "$TEST_TARGET" ]; then $TW make -f Makefile.make -j $NJOBS $TEST_TARGET; fi
- echo -en 'travis_fold:end:coq.test\\r'
- set +e

Expand Down
Loading

0 comments on commit 79534f2

Please sign in to comment.