Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

dune build freezes with cache enabled #3233

Closed
artempyanykh opened this issue Mar 3, 2020 · 5 comments · Fixed by #3249
Closed

dune build freezes with cache enabled #3233

artempyanykh opened this issue Mar 3, 2020 · 5 comments · Fixed by #3249

Comments

@artempyanykh
Copy link
Contributor

Expected Behavior

dune build finishes

Actual Behavior

dune build freezes.

Reproduction

I've added (cache enabled) to ~/.config/dune/config.

Then setup infer Infer with ./configure && make.

Then under infer/src I ran dune cache start --foreground in one terminal and dune build in the other.

dune cache output is

# accept client: 127.0.0.1:49327
# 127.0.0.1:49327: received command: (set-build-root /Users/arr/infer/infer/src/_build)
# 127.0.0.1:49327: ended

At the same time dtruss on dune build process shows a bunch of:

select(0x0, 0x0, 0x0, 0x0, 0x70000CA20F90)               = 0 0

which is very weird -- basically, dune build does polling on 0 file descriptors.

Specifications

  • Version of dune (output of dune --version): 2.3.1
  • Version of ocaml (output of ocamlc --version): 4.08.1
  • Operating system (distribution and version): MacOS 10.15.3
@artempyanykh
Copy link
Contributor Author

I also tried the same steps on CentOS 7.6 and everything worked just fine. Seems like it might be MacOS specific problem.

@ghost
Copy link

ghost commented Mar 4, 2020

/cc @mefyl

The select is probably from the OCaml ticker thread.

@mefyl
Copy link
Collaborator

mefyl commented Mar 7, 2020

This might be the same as #2973. I haven't been able to look into this because I must get my hands on a Mac.

Normally, once the client closes the connection (as we can see in the daemon logs), the daemon will shutdown (2) the socket, and the client read should terminate gracefully with an EOF - unless that's a wrong non-POSIX Linux centric assumptions, but I don't think so. Here it seems the clients stays stuck forever on the read, despite the TCP shutdown.

A workaround would be to send an end message to force the client to quit, but it pains me to deteriorate and complexify the code like that. If that's a glitch in the Ocaml TCP bindings on OSX, it would be great to catch it too.

I'll try my best to find a Mac, or if someone with OSX wants to look into it I'm happy to help in any way.

@artempyanykh
Copy link
Contributor Author

@mefyl thanks for you reply!

I've found a change that fixes the freeze on MacOS, and also makes cache-trimming tests pass. I need to check that it doesn't break anything on Linux. Will do it sometime on Monday.

artempyanykh added a commit to artempyanykh/dune that referenced this issue Mar 9, 2020
@artempyanykh
[cache] Fix dune freezing on MacOS with cache enabled
f1fa7b3 
Summary:
On MacOS shutdown on a socket raises `ENOTCONN` even when only one
side of the duplex communication is closed. This doesn't make much
sense to me (also, man pages are rather fuzzy around this, and on
Linux it behaves differently), but it is what it is.

Since cache/client calls `Unix.shutdown client.fd Unix.SHUTDOWN_SEND`
during teardown, `cache_daemon.ml` gets an exception on its shutdown,
which it swallows and therefore never closes `client.fd`.

Test plan:
1. Cache trimming tests work (and now enabled for all platforms),
2. Freezing described in ocaml#3233 is now gone.
3. Still works fine on Linux.

Fixes ocaml#3233 ocaml#2973
artempyanykh added a commit to artempyanykh/dune that referenced this issue Mar 9, 2020
@artempyanykh
[cache] Fix dune freezing on MacOS with cache enabled
35905cc 
Summary:
On MacOS shutdown on a socket raises `ENOTCONN` even when only one
side of the duplex communication is closed. This doesn't make much
sense to me (also, man pages are rather fuzzy around this, and on
Linux it behaves differently), but it is what it is.

Since cache/client calls `Unix.shutdown client.fd Unix.SHUTDOWN_SEND`
during teardown, `cache_daemon.ml` gets an exception on its shutdown,
which it swallows and therefore never closes `client.fd`.

Test plan:
1. Cache trimming tests work (and now enabled for all platforms),
2. Freezing described in ocaml#3233 is now gone.
3. Still works fine on Linux.

Fixes ocaml#3233 ocaml#2973

Signed-off-by: Artem Pianykh <artem.pyanykh@gmail.com>
@artempyanykh artempyanykh mentioned this issue Mar 9, 2020
Merged
artempyanykh added a commit to artempyanykh/dune that referenced this issue Mar 9, 2020
@artempyanykh
[cache] Fix dune freezing on MacOS with cache enabled
b28043c 
Summary:
On MacOS shutdown on a socket raises `ENOTCONN` even when only one
side of the duplex communication is closed. This doesn't make much
sense to me (also, man pages are rather fuzzy around this, and on
Linux it behaves differently), but it is what it is.

Since cache/client calls `Unix.shutdown client.fd Unix.SHUTDOWN_SEND`
during teardown, `cache_daemon.ml` gets an exception on its shutdown,
which it swallows and therefore never closes `client.fd`.

Test plan:
1. Cache trimming tests work (and now enabled for all platforms),
2. Freezing described in ocaml#3233 is now gone.
3. Still works fine on Linux.

Fixes ocaml#3233 ocaml#2973

Signed-off-by: Artem Pyanykh <artem.pyanykh@gmail.com>
@artempyanykh
Copy link
Contributor Author

@mefyl submitted a PR #3249 . The change is like a couple lines, so should be a quick review.

facebook-github-bot pushed a commit to facebook/infer that referenced this issue Mar 11, 2020
@artempyanykh @facebook-github-bot
[RFC][build] Use dune environments and profiles instead of contexts
fcce3c0 
Summary:
With profiles and `(env ...)` stanza it's possible to consolidate
various ocamlc/ocamlopt/etc setups in a single place.

Where previously we needed to append `dune.common` to every dune file
and specify `flags` and `ocamlopt_flags` now the flags are specified
in `env` and applied accross the board.

This allows to
1. simplify build definitions,
2. avoid the need to generate dune files,
3. use plain sexps instead of OCaml and JBuilder plugin in build
files.

(I'll try to address 2 and 3 in the followup patches).

Existing `make` targets should continue working as before. Also, we
can use dune CLI like so:
```
infer/src$ dune printenv --profile opt # <- very useful for introspection
infer/src$ dune build check
infer/src$ dune build check --profile test
infer/src$ dune build infer.exe --profile dev
infer/src$ dune build infer.exe --profile opt
```

Also, with just 1 context something like `dune runtest` will run unit
tests only once instead of N times, where N is the number of contexts.

Now, there's one difference compared to the previous setup with
contexts:
- Previously, each context had its own build folder, and building infer
in opt context didn't invalidate any of the build artifacts in default
context. Therefore, alternating between `make` and `make opt` had low
overhead at the expense of having N copies of all build artifacts (1
for every context).
- Now, there's just 1 build folder and switching between profiles does
invalidate some artifacts (but not all) and rebuild takes a bit more
time.

So, if you're alternating like crazy between profiles your experience
may get worse (but not necessarily, more on that below). If you want
to trigger an opt build occasionally, you shouldn't notice much
difference.

For those who are concerned about slower build times when alternating
between different build profiles, there's a solution: [dune
cache](https://dune.readthedocs.io/en/stable/caching.html).

You can enable it by creating a file `~/.config/dune/config` with the
following contents:
```
(lang dune 2.0)
(cache enabled)
```

With cache enabled switching between different build profiles (but
also branches and commits) has ~0 overhead.

Dune cache works fine on Linux, but unfortunately there are [certain
problems with
MacOS](ocaml/dune#3233) (hopefully, those
will be fixed soon and then there will be no downsides to using
profiles compared to contexts for our case).

Reviewed By: jvillard

Differential Revision: D20247864

fbshipit-source-id: 5f8afa0db
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

[cache] Fix dune freezing on MacOS with cache enabled artempyanykh/dune
2 participants
@mefyl @artempyanykh