Mkdir_p.exec: attempted to create unmanaged dir in Dune 1.9

[seantalts]

This used to work fine in Dune 1.8.2 at least. I'm asking to run some programs from outside directories, but that shouldn't be a problem right? I'm passing in that outside directory here as the cmdstan env variable (I should probably capitalize it). Here's the dune file: https://github.com/stan-dev/stanc3/blob/e1f2c7359eca43c478cfba07bcdb7dab64c622e4/test/integration/good/code-gen/dune#L45

halair ~/scm/stanc3 (jenkins/end-to-end) $ cmdstan=~/scm/clean/cmdstan dune runtest test/integration/good/code-gen/
Internal error, please report upstream including the contents of _build/log.
Description:
("Mkdir_p.exec: attempted to create unmanaged dir"
 (p (External /Users/xitrium/scm/clean/cmdstan)))
Backtrace:
Raised at file "src/stdune/exn.ml", line 40, characters 5-10
Called from file "set.ml", line 343, characters 35-38
Called from file "set.ml", line 343, characters 25-33
Called from file "src/stdune/set.ml" (inlined), line 36, characters 18-27
Called from file "src/build_system.ml", line 1451, characters 8-60
Called from file "src/fiber/fiber.ml", line 176, characters 6-11
Re-raised at file "src/stdune/exn.ml" (inlined), line 65, characters 38-65
Called from file "src/stdune/exn_with_backtrace.ml", line 17, characters 2-40
Called from file "src/fiber/fiber.ml", line 82, characters 8-15
Re-raised at file "src/stdune/exn.ml" (inlined), line 65, characters 38-65
Called from file "src/stdune/exn_with_backtrace.ml", line 17, characters 2-40
Called from file "src/fiber/fiber.ml", line 82, characters 8-15
Re-raised at file "src/stdune/exn.ml" (inlined), line 65, characters 38-65
Called from file "src/stdune/exn_with_backtrace.ml", line 17, characters 2-40
Called from file "src/fiber/fiber.ml", line 82, characters 8-15
Re-raised at file "src/stdune/exn.ml" (inlined), line 65, characters 38-65
Called from file "src/stdune/exn_with_backtrace.ml", line 17, characters 2-40
Called from file "src/fiber/fiber.ml", line 82, characters 8-15
Re-raised at file "src/stdune/exn.ml" (inlined), line 65, characters 38-65
Called from file "src/stdune/exn_with_backtrace.ml", line 17, characters 2-40
Called from file "src/fiber/fiber.ml", line 82, characters 8-15
                       
I must not segfault.  Uncertainty is the mind-killer.  Exceptions are
the little-death that brings total obliteration.  I will fully express
my cases.  Execution will pass over me and through me.  And when it
has gone past, I will unwind the stack along its path.  Where the
cases are handled there will be nothing.  Only I will remain.

[rgrinberg]

Does the directory in question already exists? If so, then this is indeed a regression.

[seantalts]

Yep, it exists.

[rgrinberg]

@diml @aalekseyev Actually, I'm wondering if it perhaps it makes sense to forbid this use case. This might be a regression, but it seems a bit off to allow users to run things outside the build dir using the build rules.

[aalekseyev]

If the action has no targets in that directory, that seems harmless to me, but I don't really understand what's going on here. (we chdir to an external directory, run "make" and that somehow creates targets for us? that seems weird)

[rgrinberg]

In the example, we run the command and collect its stdout to create a target, so yes it's a bit weird:

(rule
 (targets eight_schools_ncp_summary.expected)
 (action
   (progn
    (run %{dep:stan2_eight_schools_ncp} random seed=1 sample data file=%{dep:eight_schools.data.R} output file=stan2_eight_schools_ncp.csv)
    (with-stdout-to %{targets}
      (run %{env:cmdstan=cmdstan}/bin/stansummary stan2_eight_schools_ncp.csv))
    (with-stdout-to %{targets}
      (run python %{dep:chop_summary.py} %{targets}))
    )))

[aalekseyev]

Oh, running a command in external directory for its stdout seems totally fine to me.
I thought the chdirs in the other rules were the problem. Is that not so?

[rgrinberg]

Okay, I think I understand the issue better. I think the issue is caused when we sandbox an external path:

let sandbox_managed_paths ~sandbox_dir t =
  match t with
  | External _ -> t
  | In_source_tree p -> append_local sandbox_dir (Local.append local_src   p)
  | In_build_dir   p -> append_local sandbox_dir (Local.append local_build p)

We're clearly not sandboxing it correctly and that creates a Chdir to an external dir, which in turn makes Mkdir_p.exec throw up.

[rgrinberg]

@diml why don't we sandbox external paths? Seems like we can just append the external path after the sandbox dir.

[aalekseyev]

@rgrinberg, I'm not totally familiar with dune sandboxing, but absolute paths are fundamentally harder to change: you must actually rewrite all their occurrences, unlike relative paths where it's enough to "chdir".

Rewriting paths can work for paths that appear on the command-line, but not for the ones that are hardcoded in the script or binary we're running (or get picked up by reading files).

Why are you saying sandboxing is a problem here? I'm still not following.

[ghost]

Currently, the only thing we sandbox is the building of the alias module with OCaml 4.02. Nothing else is sandboxed

[ghost]

BTW, I believe this is the culprit:

(rule
  (targets eight_schools_ncp)
  (deps eight_schools_ncp.hpp)
  (action
  (chdir %{env:cmdstan=cmdstan}
    (run make -j4 %{targets}))))

[rgrinberg]

Why are you saying sandboxing is a problem here? I'm still not following.

It is not the problem. I was looking at the wrong rule and did not notice the explicit chdir.

So in this case I agree with Jeremie that we should just not allow to chdir outside the build directory.

[ghost]

@seantalts, I believe this stanza will produce a command like this:

$ make -j4 <absolute-path-to-you-dune-project>/test/integration/good/code-gen/eight_schools_ncp

You can verify this by running:

$ dune rule eight_schools_ncp

Or, to see it in makefile syntax:

$ dune rule -m eight_schools_ncp

Was that the intent?

[seantalts]

Yep that is the intent. Is this intended to be allowed? I guess the alternative is to write a separate portable ocaml binary for that bit and call it from dune to generate the output to diff?

…

On Wed, May 15, 2019 at 06:01 Jérémie Dimino ***@***.***> wrote: @seantalts <https://github.com/seantalts>, I believe this stanza will produce a command like this: $ make -j4 <absolute-path-to-you-dune-project>/test/integration/good/code-gen/eight_schools_ncp You can verify this by running: $ dune rule eight_schools_ncp Or, to see it in makefile syntax: $ dune rule -m eight_schools_ncp Was that the intent? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#2158?email_source=notifications&email_token=AAGET3EKXNBY5EXLOZT7O5TPVPNPRA5CNFSM4HM4Z6Z2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODVOFLAQ#issuecomment-492590466>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAGET3ED35D6INTTPMAQQJDPVPNPRANCNFSM4HM4Z6ZQ> .

[ghost]

I guess we never considered such a use case. Normally dune only operates inside the workspace. If you build things outside of the workspace, then dune won't track everything that's happening and that might lead to surprising behaviours. Although, unless you run dune instances in parallel, it shouldn't be too bad.

A more idiomatic way to do that with dune would be to copy (or even just symlink) the cmdstan directory into the workspace and use the foreign build sandboxing method.

[seantalts]

Ah, that looks potentially really useful. We're doing a super weird thing here in that this stanc3 compiler emits C++ code that we then want to build and link using cmdstan makefiles into something we then continue testing with dune. Though obviously we could leave the dune ecosystem at any point during this end to end test; I just liked the output diffing part and having it all in one system.

[rgrinberg]

Hmm, I think I closed this issue previously by accident. In fact, it was fixed in master just now. Thanks for the report @seantalts