Stalls for 3.8.2 in dune build on Windows GitLab CI

[jonahbeckford]

Behavior

I do a dune build (exact verbose logs are at https://gist.github.com/jonahbeckford/c33118fd45812e2bf73d5f4c4266710a).

I get a stall (deadlock?) under certain conditions:

• Only Dune 3.8.2. (Versions prior to 3.8.2 work! Dune 3.6.2, 3.7.1, 3.8.0 and 3.8.1)
• Only GitLab CI machines. (Not my desktop, and apparently not on GitHub Actions)

Because of the above conditions, I strongly suspect the root cause is #7906 ("Use threaded console, adapt for Windows"). EDIT: Tested that this wasn't the commit.

Reproduction

A dune build of many packages (quite similar to a opam monorepo build).

These packages:

• lwt.5.6.1
• stdint.0.7.2
• memtrace.v0.2.3
• fiber.3.7.0

and the following:

  astring.0.8.5+dune ;                               d950ab785805c6fb51bc48dff6380118
  base-bytes.base+dune ;                             607827e71b1a816b33e16104d966f39c
  base.v0.15.1 ;                                     f3d7f4e72096610b2f14017c351df010
  camlp-streams.5.0.1 ;                              179a0662333b2ac5c7a3cc5009e1020f
  capnp.3.5.0 ;                                      c79f2454d7d9d956f0f45f25e3c25fb2
  cppo.1.6.9 ;                                       64b7f30febf9b6c4977f537ea936992d
  csexp.1.5.2 ;                                      69b9a4107a979db5724ccdb0b1f3fbf8
  cstruct.6.2.0 ;                                    1c750a2ed4098f82b65153917b9ad475
  dum.1.0.3 ;                                        21bd91e73b7e9bccb298854f1bf47e58
  dyn.3.8.0 ;                                        e7bdb1e6bcf1cdc2c004f72faa1ecf52
  easy-format.1.3.4 ;                                e77d2a3c8294190d59006af468ef9f09
  ezjsonm.1.3.0 ;                                    1b532e39182b85e4a787f270fccf2fb4
  ; fiber.3.7.0 ;                                    excluded
  fmt.0.9.0+dune ;                                   c6bc7ec23eb7c17757f5bbf5921cbe7a
  hex.1.5.0 ;                                        c09195f94932559683a73fdd1d9b34c4
  jsonm.1.0.1+dune ;                                 2eb446505d1c6606ed1386e060b10fb5
  ; lwt.5.6.1 ;                                      excluded
  merlin-lib.4.9-414 ;                               de781e33ca6733f2299317216209099d
  ocaml-lsp-server.1.16.1 ;                          85d7dfb00197484dc541be706ad2d512
  ocamlformat-rpc-lib.0.25.1 ;                       058d4868136786112700d4ab4d934749
  ocplib-endian.1.2 ;                                272a0bc808996570167fb33ada28bbd7
  odoc-parser.2.0.0 ;                                2f9b8dafa83ecb0946cc919012d55748
  pp.1.1.2 ;                                         1374b90bfdaffef32447990ac6a29f2f
  ppx_yojson_conv_lib.v0.16.0 ;                      090dde9cceea381878a30d29de2ddc4f
  re.1.10.4 ;                                        56fd03da1fdba0759f9ce7c24ae8e4a5
  res.5.0.1 ;                                        149dbcb0511e2bbc6e09040617c28330
  result.1.5 ;                                       be8c7653e74b05864f855ff6008bc067
  seq.base+dune ;                                    4419c6d032b2ba303d6393c1b229eafd
  sexplib0.v0.15.1 ;                                 d8968ee97fd639e136b4805c8d10fe26
  spawn.v0.15.1 ;                                    34d7a3ab2bb70495057498b88549d111
  ; stdint.0.7.2 ;                                   excluded
  stdio.v0.15.0 ;                                    9d0a6023715b6a014214f07c7b0fdc75
  tezt.3.1.0 ;                                       e392df3a5c51f21dacf031de3c9c8767
  uuidm.0.9.8+dune ;                                 6b19ffff77048ebc701476cf5f629a22
  uutf.1.0.3+dune ;                                  58b861b89d85b6ff2a91d41d172b6080
  yojson.2.1.0 ;                                     6e18f18435a049543e0ab30a5884f7c2

Specifications

• Version of dune (output of dune --version): 3.8.2
• Version of ocaml (output of ocamlc --version): 4.14.0
• Operating system (distribution and version):
  • Windows Container
  • GitLab windows-1809 tag (aka. compatible with Windows 10 1809)
  • https://gitlab.com/gitlab-org/ci-cd/shared-runners/images/gcp/windows-containers/-/blob/main/cookbooks/preinstalled-software/README.md

Additional information

• Link to gist with verbose output (run dune with the --verbose flag):
  https://gist.github.com/jonahbeckford/c33118fd45812e2bf73d5f4c4266710a

[nojb]

Hello; could you try reverting the one-line diff https://github.com/ocaml/dune/pull/7906/files#diff-51b18b303e6585311a301fb7eb4761c4ae1b226d8a29c2c30cf20c80102bc39cR47 and testing if the stall goes away?

[jonahbeckford]

It looks like my guess was wrong. Since there are only six commits between 3.8.1 and 3.8.2, I'll bisect and update the issue when I find the problematic commit.

[jonahbeckford]

Summary:

• 886ff3b is the first commit where stalls appear
• That is "fix(rpc): remove threads": (fix(rpc): remove threads #7418) ([3.8] backport #7418 #7955)

Bisect data (scroll to the right):

FetchGit(dune
# GIT_REPOSITORY https://github.com/ocaml/dune.git GIT_TAG a340b806a6ec2da97c481c64011bfab7b71213a8 # 3.8.1 Jun 5. WORKS.
# GIT_REPOSITORY https://github.com/ocaml/dune.git GIT_TAG 8b7cc718a813fa385ce69c2f4918036d280efb99 # 3.8.1(+3) Jun 15. WORKS.
GIT_REPOSITORY https://github.com/ocaml/dune.git GIT_TAG 886ff3b14adfb876c1b25c5b88c9637b59bd2c37 # 3.8.1(+4) Jun 16. STALLS
# GIT_REPOSITORY https://github.com/ocaml/dune.git GIT_TAG 1405390f8827fcc6616bf1dc1e406a9949d7b6b1 # 3.8.1(+5) Jun 16. STALLS
# GIT_REPOSITORY https://github.com/ocaml/dune.git GIT_TAG e6a7ca69d3b27aa21b59887b5549daafdc931899 # 3.8.2 Jun 16. STALLS.

# Not part of bisection:
# GIT_REPOSITORY https://github.com/jonahbeckford/dune.git GIT_TAG 74a2b680872596ca76d2913913e5f8b7c8b7ba5a # 3.8.2-revert-PR7906. STALLS.
)

cc: @rgrinberg @emillon . Any ideas?

[nojb]

cc: @rgrinberg @emillon . Any ideas?

Not sure it is the same thing, but can you try #8044?

[jonahbeckford]

Wow. That fixed it!

Seems like this critical fix should be backported to 3.8.x.

Raw details:

• I used your PR (which was a branch off main) and added a one-line commit to force its version as 3.8.2. That git tree is https://github.com/jonahbeckford/dune/commits/57f7cfca55ecbc0b4599c70741de86d97dbb6369

FetchGit(dune
GIT_REPOSITORY https://github.com/jonahbeckford/dune.git GIT_TAG 57f7cfca55ecbc0b4599c70741de86d97dbb6369 # 3.8.2.2023-06-25.2a3ca31 with (+1) async_io_self_pipe_win32 and (+1) fix version=3.8.2
)

[nojb]

Wow. That fixed it!

Thanks for the confirmation!

• (which was a branch off main)

Sorry about that, the patch applies cleanly to the 3.8.2 tag. I made a branch based on 3.8.2: https://github.com/nojb/dune/tree/async_io_self_pipe_win32-3.8.2. Could you give it a shot? Just to be 100% sure that we identified the culprit.

[hhugo]

@nojb, I'm testing your branch in ocsigen/js_of_ocaml#1478 to check if i'm witnessing the same issue.

[nojb]

@nojb, I'm testing your branch in ocsigen/js_of_ocaml#1478 to check if i'm witnessing the same issue.

Looks like it is.

[nojb]

I merged #8044 as it looks like it fixes both of these issues. Thanks!

[jonahbeckford]

I verified your change on 3.8.2. Thanks!

Final test notes:

FetchGit(
GIT_REPOSITORY https://github.com/jonahbeckford/dune.git GIT_TAG dd9522145ccaa43d3958bca96b9711ae3b8a0cc8 # 3.8.2(+1 nojb) Jun 26 and (+1 jonahbeckford) force version=3.8.2. WORKS
)

[emillon]

@jonahbeckford you can take the 3.8.3 tag which includes just this if you'd like

[jonahbeckford]

@jonahbeckford you can take the 3.8.3 tag which includes just this if you'd like

Excellent. Thanks!