Directly call the C compiler #3565

voodoos · 2020-06-18T15:33:21Z

This fixes #3346
Before that PR the C compiler was called via ocamlc. Now, like the C++ compiler the C compiler is called directly by dune.

~~However, ocamlc added a number of flags that were lost in this process: -fPIC, -D_FILE_OFFSET_BITS=64 and -D_REENTRANT.~~

As a first mitigation I decided to hard-code these in the call of the C compiler. It is probably not the best solution, and I will be happy to update this PR with a more programmatic and flexible way to do that. In fact, I am wondering why these flags are not recovered by Fdo.c_flags ctx ?

src/dune/foreign_rules.ml

voodoos · 2020-06-19T12:05:52Z

The current status of this PR is that dune now calls directly the configured C compiler.
That means that config flags in ocamlc_cflags and ocaml_cppflags must be forwarded to that compiler by dune itself.

These flags are set as the default_context_flags, that are used to populate the :standard placeholder of the foreign_stubs's flags field. Users can customize this set of flag as usual:

 (foreign_stubs (language c) (names src1) (flags ((:standard \ -O2) -O3)))

If I understand well this is what was always expected from users. However because dune was using ocamlc to call the C compiler, flags in ocamlc_cflags + ocaml_cppflags were always added to the call (and thus often duplicated).

Therefore user omitting to include the :standard set of flag where still able to build their programs.

This is not the case any-more, and some projects like cpuid or lwt fail to build due to a missing -fPIC flag (and that breaks the CI).

So we need a mechanism to support those broken projects. My first idea would be to always prepend the config flags to the compiler arguments when dune lang is < 2.7. This should reproduce the exact same behaviour as before. Does that seems a reasonable option to you ?

ghost · 2020-06-22T09:57:40Z

So, it seems clear that we have to preserve the current behaviour for existing released projects. Regarding using the new better behaviour past 2.7, technically that seems reasonable, however it does mean that the interpretation of a dune file will change past a minor revision of the language, which could make things difficult to follow for users.

Usually, the way we go about such change is to:

add a flag in the dune-project file to enable the new behaviour immediately
when bumping the major version of the language, flip the default

There is also an open question: if we don't pass the C flags given by ocamlc -config, then when someone writes (cflags -O3) it's probably wrong. For instance because we won't pass the -fPIC as you mentioned. We should discuss this a bit more.

BTW, the minimum we can do right now that is totally uncontroversial is systematically add the flags given by ocamlc -config and make :standard be empty. @voodoos if you do this, we can merge the PR right now and continue discussing the other points.

nojb

Looks good to me. I left a few comments but they are all trivial.

test/blackbox-tests/test-cases/check-alias/run.t

src/dune/foreign_rules.ml

nojb · 2020-06-23T11:56:00Z

src/dune/super_context.ml

-let default_context_flags (ctx : Context.t) =
-  let c = Ocaml_config.ocamlc_cflags ctx.ocaml_config in
+let default_context_flags (_ctx : Context.t) =
+  let c = [] in


Just to make sure I understand, is it the case that the previous value ocamlc_cflags made some flags appear twice in the command-line?

Yes, flags were duplicated when not redefined by the user.

src/dune/super_context.ml

src/dune/foreign_rules.ml

ghost

Looks good. I'd just remove the double rev and add the extra space as @nojb suggested.

src/dune/foreign_rules.ml

rgrinberg · 2020-07-20T23:29:09Z

@voodoos are you still working on this RP or is it good to merge?

voodoos · 2020-07-21T08:02:58Z

@voodoos are you still working on this RP or is it good to merge?

It is good to merge, I just rebased it.

Signed-off-by: Ulysse Gérard <thevoodoos@gmail.com>

nojb · 2020-08-06T18:48:46Z

I haven't looked in detail, but it looks like this change is not fully backwards compatible: the variable %{cc} used to contain the flags ocamlc_cflags but now it doesn't. This may/will cause rules that use %{cc} explicitly to fail.

@NathanReb

…lugin, dune-private-libs and dune-glob (2.7.0) CHANGES: - Write intermediate files in a `.mdx` folder for each `mdx` stanza to prevent the corresponding actions to be executed as part of the `@all` alias (ocaml/dune#3659, @NathanReb) - Read Coq flags from `env` (ocaml/dune#3547 , fixes ocaml/dune#3486, @gares) - Allow bisect_ppx to be enabled/disabled via dune-workspace. (ocaml/dune#3404, @stephanieyou) - Formatting of dune files is now done in the executing dune process instead of in a separate process. (ocaml/dune#3536, @nojb) - Add a `--debug-artifact-substution` flag to help debug problem with version not being captured by `dune-build-info` (ocaml/dune#3589, @jeremiedimino) - Allow the use of the `context_name` variable in the `enabled_if` fields of executable(s) and install stanzas. (ocaml/dune#3568, fixes ocaml/dune#3566, @voodoos) - Fix compatibility with OCaml 4.12.0 when compiling empty archives; no .a file is generated. (ocaml/dune#3576, @dra27) - `$ dune utop` no longer tries to load optional libraries that are unavailable (ocaml/dune#3612, fixes ocaml/dune#3188, @anuragsoni) - Fix dune-build-info on 4.10.0+flambda (ocaml/dune#3599, @emillon, @jeremiedimino). - Allow multiple libraries with `inline_tests` to be defined in the same directory (ocaml/dune#3621, @rgrinberg) - Run exit hooks in jsoo separate compilation mode (ocaml/dune#3626, fixes ocaml/dune#3622, @rgrinberg) - Add (alias ...), (mode ...) fields to (copy_fields ...) stanza (ocaml/dune#3631, @nojb) - (copy_files ...) now supports copying files from outside the workspace using absolute file names (ocaml/dune#3639, @nojb) - Dune does not use `ocamlc` as an intermediary to call C compiler anymore. Configuration flags `ocamlc_cflags` and `ocamlc_cppflags` are always prepended to the compiler arguments. (ocaml/dune#3565, fixes ocaml/dune#3346, @voodoos) - Revert the build optimization in ocaml/dune#2268. This optimization slows down building individual executables when they're part of an `executables` stanza group (ocaml/dune#3644, @rgrinberg) - Use `{dev}` rather than `{pinned}` in the generated `.opam` file. (ocaml/dune#3647, @kit-ty-kate) - Insert correct extension name when editing `dune-project` files. Previously, dune would just insert the stanza name. (ocaml/dune#3649, fixes ocaml/dune#3624, @rgrinberg) - Fix crash when evaluating an `mdx` stanza that depends on unavailable packages. (ocaml/dune#3650, @craigfe) - Fix typo in `cache-check-probablity` field in dune config files. This field now requires 2.7 as it wasn't usable before this version. (ocaml/dune#3652, @edwintorok) - Add `"odoc" {with-doc}` to the dependencies in the generated `.opam` files. (ocaml/dune#3667, @kit-ty-kate) - Do not allow user actions to capture dune's stdin (ocaml/dune#3677, fixes ocaml/dune#3672, @rgrinberg) - `(subdir ...)` stanzas can now appear in dune files used via `(include ...)`. (ocaml/dune#3676, @nojb)

@NathanReb

…lugin, dune-private-libs and dune-glob (2.7.0) CHANGES: - Write intermediate files in a `.mdx` folder for each `mdx` stanza to prevent the corresponding actions to be executed as part of the `@all` alias (ocaml/dune#3659, @NathanReb) - Read Coq flags from `env` (ocaml/dune#3547 , fixes ocaml/dune#3486, @gares) - Add instrumentation framework to toggle instrumentation by `bisect_ppx`, `landmarks`, etc, via dune-workspace and/or the command-line. (ocaml/dune#3404, ocaml/dune#3526 @stephanieyou, @nojb) - Formatting of dune files is now done in the executing dune process instead of in a separate process. (ocaml/dune#3536, @nojb) - Add a `--debug-artifact-substution` flag to help debug problem with version not being captured by `dune-build-info` (ocaml/dune#3589, @jeremiedimino) - Allow the use of the `context_name` variable in the `enabled_if` fields of executable(s) and install stanzas. (ocaml/dune#3568, fixes ocaml/dune#3566, @voodoos) - Fix compatibility with OCaml 4.12.0 when compiling empty archives; no .a file is generated. (ocaml/dune#3576, @dra27) - `$ dune utop` no longer tries to load optional libraries that are unavailable (ocaml/dune#3612, fixes ocaml/dune#3188, @anuragsoni) - Fix dune-build-info on 4.10.0+flambda (ocaml/dune#3599, @emillon, @jeremiedimino). - Allow multiple libraries with `inline_tests` to be defined in the same directory (ocaml/dune#3621, @rgrinberg) - Run exit hooks in jsoo separate compilation mode (ocaml/dune#3626, fixes ocaml/dune#3622, @rgrinberg) - Add (alias ...), (mode ...) fields to (copy_fields ...) stanza (ocaml/dune#3631, @nojb) - (copy_files ...) now supports copying files from outside the workspace using absolute file names (ocaml/dune#3639, @nojb) - Dune does not use `ocamlc` as an intermediary to call C compiler anymore. Configuration flags `ocamlc_cflags` and `ocamlc_cppflags` are always prepended to the compiler arguments. (ocaml/dune#3565, fixes ocaml/dune#3346, @voodoos) - Revert the build optimization in ocaml/dune#2268. This optimization slows down building individual executables when they're part of an `executables` stanza group (ocaml/dune#3644, @rgrinberg) - Use `{dev}` rather than `{pinned}` in the generated `.opam` file. (ocaml/dune#3647, @kit-ty-kate) - Insert correct extension name when editing `dune-project` files. Previously, dune would just insert the stanza name. (ocaml/dune#3649, fixes ocaml/dune#3624, @rgrinberg) - Fix crash when evaluating an `mdx` stanza that depends on unavailable packages. (ocaml/dune#3650, @craigfe) - Fix typo in `cache-check-probablity` field in dune config files. This field now requires 2.7 as it wasn't usable before this version. (ocaml/dune#3652, @edwintorok) - Add `"odoc" {with-doc}` to the dependencies in the generated `.opam` files. (ocaml/dune#3667, @kit-ty-kate) - Do not allow user actions to capture dune's stdin (ocaml/dune#3677, fixes ocaml/dune#3672, @rgrinberg) - `(subdir ...)` stanzas can now appear in dune files used via `(include ...)`. (ocaml/dune#3676, @nojb)

voodoos changed the title ~~Cc call~~ Directly call the C compiler Jun 18, 2020

nojb reviewed Jun 18, 2020

View reviewed changes

src/dune/foreign_rules.ml Outdated Show resolved Hide resolved

voodoos requested review from dra27 and rgrinberg as code owners June 19, 2020 09:37

voodoos force-pushed the cc-call branch 4 times, most recently from 2f25db7 to 194f671 Compare June 22, 2020 12:25

voodoos requested a review from nojb June 22, 2020 13:57

nojb approved these changes Jun 23, 2020

View reviewed changes

ghost suggested changes Jun 23, 2020

View reviewed changes

src/dune/super_context.ml Outdated Show resolved Hide resolved

src/dune/foreign_rules.ml Show resolved Hide resolved

voodoos force-pushed the cc-call branch from 194f671 to 4ff3741 Compare June 30, 2020 11:22

ghost approved these changes Jul 1, 2020

View reviewed changes

voodoos force-pushed the cc-call branch 2 times, most recently from 7aac42b to 97d1c10 Compare July 1, 2020 14:35

rgrinberg reviewed Jul 1, 2020

View reviewed changes

src/dune/foreign_rules.ml Outdated Show resolved Hide resolved

voodoos force-pushed the cc-call branch from c437b06 to beae6d0 Compare July 8, 2020 14:47

voodoos force-pushed the cc-call branch from beae6d0 to 6273553 Compare July 21, 2020 08:00

voodoos added 6 commits July 23, 2020 16:12

Call the C compiler like the CXX one

61a850f

Signed-off-by: Ulysse Gérard <thevoodoos@gmail.com>

Refactor

b9205d5

Signed-off-by: Ulysse Gérard <thevoodoos@gmail.com>

Add cpp_flags to Ocaml_config

1398324

Signed-off-by: Ulysse Gérard <thevoodoos@gmail.com>

No config flags in :standard. Always prepend to cc.

4a6e2da

Signed-off-by: Ulysse Gérard <thevoodoos@gmail.com>

Add CHANGES entry

c6dec37

Signed-off-by: Ulysse Gérard <thevoodoos@gmail.com>

Add sanitization to JSOO test

4990959

Signed-off-by: Ulysse Gérard <thevoodoos@gmail.com>

voodoos force-pushed the cc-call branch from 4853d6d to 4990959 Compare July 23, 2020 14:14

voodoos merged commit fff4be1 into ocaml:master Jul 23, 2020

This was referenced Aug 7, 2020

[Fix] Restore variable %{cc} content (changed by #3565 ) #3686

Closed

Fix variable CC content + add test #3695

Merged

rgrinberg mentioned this pull request Aug 14, 2020

[new release] dune-build-info, dune, dune-configurator, dune-action-plugin, dune-private-libs and dune-glob (2.7.0) ocaml/opam-repository#17005

Merged

hannesm mentioned this pull request Oct 13, 2020

New Xen core platform stack mirage/mirage-xen#23

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Directly call the C compiler #3565

Directly call the C compiler #3565

voodoos commented Jun 18, 2020 •

edited

Loading

voodoos commented Jun 19, 2020 •

edited

Loading

ghost commented Jun 22, 2020

nojb left a comment

nojb Jun 23, 2020

voodoos Jun 30, 2020

ghost left a comment

rgrinberg commented Jul 20, 2020

voodoos commented Jul 21, 2020

nojb commented Aug 6, 2020

Directly call the C compiler #3565

Directly call the C compiler #3565

Conversation

voodoos commented Jun 18, 2020 • edited Loading

voodoos commented Jun 19, 2020 • edited Loading

ghost commented Jun 22, 2020

nojb left a comment

Choose a reason for hiding this comment

nojb Jun 23, 2020

Choose a reason for hiding this comment

voodoos Jun 30, 2020

Choose a reason for hiding this comment

ghost left a comment

Choose a reason for hiding this comment

rgrinberg commented Jul 20, 2020

voodoos commented Jul 21, 2020

nojb commented Aug 6, 2020

voodoos commented Jun 18, 2020 •

edited

Loading

voodoos commented Jun 19, 2020 •

edited

Loading