Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

install.sh hangs during superuser creation under Docker Compose v2 #1133

Closed
mowangjuanzi opened this issue Oct 31, 2021 · 24 comments
Closed

Comments

@mowangjuanzi
Copy link

mowangjuanzi commented Oct 31, 2021

Workaround

With Docker Compose < 2.2.1(?):

./install.sh --no-user-prompt
docker compose run --rm web createuser

Docker Compose >= v2.2.1(?) is known to partially work.

Update

Fixed in docker/compose#9035! 💃 Released in 2.2.3. 👍

Original Report

Docker version 20.10.10, build b485636

Docker Compose version v2.0.1

Version

21.10.0

Steps to Reproduce

sudo ./install.sh

Expected Result

Install Success

Actual Result

Running hooks in /etc/ca-certificates/update.d...
done.
02:45:10 [WARNING] sentry.utils.geo: Error opening GeoIP database: /geoip/GeoLite2-City.mmdb
02:45:10 [WARNING] sentry.utils.geo: Error opening GeoIP database in Rust: /geoip/GeoLite2-City.mmdb
02:45:12 [INFO] sentry.plugins.github: apps-not-configured
Running migrations for default

Project Options: N/A% | | ETA: --:--:--

Project Options: 100% |#########################################| Time: 0:00:00
Operations to perform:
Apply all migrations: admin, auth, contenttypes, nodestore, sentry, sessions, sites, social_auth
Running migrations:
Applying sentry.0240_grouphistory_index... OK
Applying sentry.0241_grouphistory_null_actor... OK
Applying sentry.0242_delete_removed_plugin_data... OK
Applying sentry.0243_delete_visualstudio_repo_data... OK

Would you like to create a user account now? [Y/n]:

At this time, the terminal does not respond. There is no choice but to close the terminal. After closing, the log output is as follows

read /dev/stdin: input/output error

@chadwhitacre
Copy link
Member

D'oh! I'm able to repro. Will work on debugging ...

@chadwhitacre
Copy link
Member

Hrm, something funky with in setup-and-migrate-database.sh. Workaround is:

SKIP_USER_PROMPT=1 ./install.sh

@chadwhitacre
Copy link
Member

Note this though with the workaround:

Did not prompt for user creation due to non-interactive shell.
Run the following command to create one yourself (recommended):

docker-compose run --rm web createuser

@chadwhitacre
Copy link
Member

This seems to only occur when using docker compose instead of docker-compose, but only when called under install.sh. Both variants work fine when called directly.

docker-compose --ansi never run --rm web upgrade
docker compose --ansi never run --rm web upgrade
$ docker-compose --version
docker-compose version 1.29.2, build 5becea4c
$ docker compose version
Docker Compose version v2.0.0-rc.1
$

@chadwhitacre
Copy link
Member

Anything in here look suspicious? What is different when invoked under install.sh than when invoked directly?

--- env.working	2021-11-02 17:23:44.000000000 -0400
+++ env.borked	2021-11-02 17:24:18.000000000 -0400
@@ -1,24 +1,31 @@
 BLOCKSIZE=K
+COMPOSE_PROJECT_NAME=sentry_onpremise
 EDITOR=vim
 HOME=/Users/chadwhitacre
 LANG=en_US.UTF-8
 LESS=--RAW-CONTROL-CHARS --chop-long-lines --LONG-PROMPT
 LOGNAME=chadwhitacre
 LSCOLORS=gxfxcxdxbxegedabagacad
-OLDPWD=/Users/chadwhitacre
+OLDPWD=/Users/chadwhitacre/workbench/getsentry/onpremise
 PAGER=less
 PATH=/Users/chadwhitacre/.pyenv/shims:/Users/chadwhitacre/.volta/bin:/Users/chadwhitacre/bin:~/go/bin:/usr/local/opt/bin:/Users/chadwhitacre/.pyenv/shims:/Users/chadwhitacre/.volta/bin:/Users/chadwhitacre/bin:~/go/bin:/usr/local/opt/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Users/chadwhitacre/.cargo/bin:/usr/local/opt/fzf/bin
-PWD=/Users/chadwhitacre/workbench/getsentry/onpremise
+PWD=/Users/chadwhitacre/workbench/getsentry/onpremise/install
 PYENV_SHELL=zsh
 PYTEST_ADDOPTS=--tb=native --capture=no
 PYTHONDONTWRITEBYTECODE=true
+RELAY_IMAGE=getsentry/relay:nightly
+SENTRY_BIND=9000
+SENTRY_EVENT_RETENTION_DAYS=90
+SENTRY_IMAGE=getsentry/sentry:nightly
 SENTRY_INSTRUMENTATION=1
 SENTRY_POST_MERGE_AUTO_UPDATE=1
 SENTRY_SPA_DSN=https://863de587a34a48c4a4ef1a9238fdb0b1@o19635.ingest.sentry.io/5270453
 SHELL=/bin/zsh
-SHLVL=2
+SHLVL=3
+SNUBA_IMAGE=getsentry/snuba:nightly
 SSH_AUTH_SOCK=/private/tmp/com.apple.launchd.d4lsA35hGO/Listeners
 STY=97460.ttys000.Chads-MBP
+SYMBOLICATOR_IMAGE=getsentry/symbolicator:nightly
 TERM=screen
 TERMCAP=SC|screen|VT 100/ANSI X3.64 virtual terminal:DO=\E[%dB:LE=\E[%dD:RI=\E[%dC:UP=\E[%dA:bs:bt=\E[Z:cd=\E[J:ce=\E[K:cl=\E[H\E[J:cm=\E[%i%d;%dH:ct=\E[3g:do=^J:nd=\E[C:pt:rc=\E8:rs=\Ec:sc=\E7:st=\EH:up=\EM:le=^H:bl=^G:cr=^M:it#8:ho=\E[H:nw=\EE:ta=^I:is=\E)0:li#45:co#130:am:xn:xv:LP:sr=\EM:al=\E[L:AL=\E[%dL:cs=\E[%i%d;%dr:dl=\E[M:DL=\E[%dM:dc=\E[P:DC=\E[%dP:im=\E[4h:ei=\E[4l:mi:IC=\E[%d@:ks=\E[?1h\E=:ke=\E[?1l\E>:vi=\E[?25l:ve=\E[34h\E[?25h:vs=\E[34l:ti=\E[?1049h:te=\E[?1049l:us=\E[4m:ue=\E[24m:so=\E[3m:se=\E[23m:mb=\E[5m:md=\E[1m:mr=\E[7m:me=\E[m:ms:Co#8:pa#64:AF=\E[3%dm:AB=\E[4%dm:op=\E[39;49m:AX:vb=\Eg:G0:as=\E(0:ae=\E(B:ac=\140\140aaffggjjkkllmmnnooppqqrrssttuuvvwwxxyyzz{{||}}~~..--++,,hhII00:po=\E[5i:pf=\E[4i:Km=\E[M:k0=\E[10~:k1=\EOP:k2=\EOQ:k3=\EOR:k4=\EOS:k5=\E[15~:k6=\E[17~:k7=\E[18~:k8=\E[19~:k9=\E[20~:k;=\E[21~:F1=\E[23~:F2=\E[24~:kB=\E[Z:kh=\E[1~:@1=\E[1~:kH=\E[4~:@7=\E[4~:kN=\E[6~:kP=\E[5~:kI=\E[2~:kD=\E[3~:ku=\EOA:kd=\EOB:kr=\EOC:kl=\EOD:km:
 TERM_PROGRAM=Apple_Terminal
@@ -27,6 +34,7 @@
 TMPDIR=/var/folders/ky/wbrpvh1s5f12wpt377s3pn3m0000gn/T/
 USER=chadwhitacre
 VOLTA_HOME=/Users/chadwhitacre/.volta
+WAL2JSON_VERSION=latest
 WINDOW=1
 XPC_FLAGS=0x0
 XPC_SERVICE_NAME=0

@chadwhitacre
Copy link
Member

I exported the added envvars into a separate shell and was unable to trigger the bug.

@chadwhitacre
Copy link
Member

Whittling down a test case ...

Steps

  1. Get the environment into some particular state, probably using reset.sh but commenting out the final install.sh.
  2. Apply the diff below.
  3. Run ./install.sh.

Expected

Get a prompt to create a user account. The prompt accepts input.

$ ./install.sh 
▶ Parsing command line ...

▶ Setting up error handling ...

▶ Setting up / migrating database ...
docker-compose --ansi never run --rm web upgrade
Starting sentry_onpremise_postgres_1 ... 
Starting sentry_onpremise_redis_1    ... 
Starting sentry_onpremise_redis_1    ... done
Starting sentry_onpremise_postgres_1 ... done
Creating sentry_onpremise_web_run    ... 
Creating sentry_onpremise_web_run    ... done
Updating certificates in /etc/ssl/certs...
1 added, 0 removed; done.
Running hooks in /etc/ca-certificates/update.d...
done.
21:48:25 [INFO] sentry.plugins.github: apps-not-configured
Running migrations for default
Operations to perform:
  Apply all migrations: admin, auth, contenttypes, nodestore, sentry, sessions, sites, social_auth
Running migrations:
  No migrations to apply.

Would you like to create a user account now? [Y/n]: n

Run `sentry createuser` to do this later.

Creating missing DSNs
Correcting Group.num_comments counter

$

Actual

Terminal gets boogered up, layout goes wonky and is unresponsive, has to be kill -9ed.

But! If docker-compose is used instead of docker compose under set-up-and-migrate-database.sh in the diff below, then ./install.sh behaves as expected.

$ ./install.sh
▶ Parsing command line ...

▶ Setting up error handling ...

▶ Setting up / migrating database ...
docker compose --ansi never run --rm web upgrade
Container sentry_onpremise_redis_1  Recreate
Container sentry_onpremise_postgres_1  Recreate
Container sentry_onpremise_postgres_1  Recreated
Container sentry_onpremise_redis_1  Recreated
Container sentry_onpremise-postgres-1  Starting
Container sentry_onpremise-redis-1  Starting
Container sentry_onpremise-redis-1  Started
Container sentry_onpremise-postgres-1  Started
Updating certificates in /etc/ssl/certs...
                                          1 added, 0 removed; done.
                                                                   Running hooks in /etc/ca-certificates/update.d...
     done.
          21:49:55 [INFO] sentry.plugins.github: apps-not-configured
                                                                    Running migrations for default
                                                                                                  Operations to perform:
           Apply all migrations: admin, auth, contenttypes, nodestore, sentry, sessions, sites, social_auth
                                                                                                           Running migrations:
                 No migrations to apply.

                                        Would you like to create a user account now? [Y/n]:
$ ps
  PID TTY           TIME CMD
90101 ttys000    0:05.59 screen -D -RR -U
99819 ttys000    0:00.14 -zsh
29156 ttys001    0:12.09 vim
58887 ttys001    0:00.51 /bin/zsh
52669 ttys002    0:00.97 /bin/zsh
54701 ttys002    0:00.01 bash ./install.sh
54705 ttys002    0:00.00 bash ./install.sh
54707 ttys002    0:00.00 tee -a sentry_install_log-2021-11-02_17-49-19.txt
54723 ttys002    0:00.05 docker compose --ansi never run --rm web upgrade
54726 ttys002    0:00.04 /usr/local/bin/com.docker.cli compose --ansi never run --rm web upgrade
54728 ttys002    0:00.59 /usr/local/lib/docker/cli-plugins/docker-compose compose --ansi never run --rm web up
54895 ttys003    0:00.11 /bin/zsh
$ kill -9 54701
$ ps           
  PID TTY           TIME CMD
90101 ttys000    0:05.59 screen -D -RR -U
99819 ttys000    0:00.14 -zsh
29156 ttys001    0:12.09 vim
58887 ttys001    0:00.51 /bin/zsh
52669 ttys002    0:00.98 /bin/zsh
54895 ttys003    0:00.13 /bin/zsh
$

Environment

$ docker-compose --version
docker-compose version 1.29.2, build 5becea4c
$ docker compose version
Docker Compose version v2.0.0
$

Diff

Against c16ae02:

diff --git a/docker-compose.yml b/docker-compose.yml
index 29342e4..2d21540 100644
--- a/docker-compose.yml
+++ b/docker-compose.yml
@@ -20,32 +20,8 @@ x-sentry-defaults: &sentry_defaults
   depends_on:
     redis:
       <<: *depends_on-healthy
-    kafka:
-      <<: *depends_on-healthy
     postgres:
       <<: *depends_on-healthy
-    memcached:
-      <<: *depends_on-default
-    smtp:
-      <<: *depends_on-default
-    snuba-api:
-      <<: *depends_on-default
-    snuba-consumer:
-      <<: *depends_on-default
-    snuba-outcomes-consumer:
-      <<: *depends_on-default
-    snuba-sessions-consumer:
-      <<: *depends_on-default
-    snuba-transactions-consumer:
-      <<: *depends_on-default
-    snuba-subscription-consumer-events:
-      <<: *depends_on-default
-    snuba-subscription-consumer-transactions:
-      <<: *depends_on-default
-    snuba-replacer:
-      <<: *depends_on-default
-    symbolicator:
-      <<: *depends_on-default
   entrypoint: "/etc/sentry/entrypoint.sh"
   command: ["run", "web"]
   environment:
diff --git a/install.sh b/install.sh
index 3886463..c1ac6ee 100755
--- a/install.sh
+++ b/install.sh
@@ -9,21 +9,4 @@ source "$(dirname $0)/install/_lib.sh"  # does a `cd .../install/`, among other
 
 source parse-cli.sh
 source error-handling.sh
-source check-minimum-requirements.sh
-source create-docker-volumes.sh
-source ensure-files-from-examples.sh
-source generate-secret-key.sh
-source replace-tsdb.sh
-source update-docker-images.sh
-source build-docker-images.sh
-source turn-things-off.sh
-source set-up-zookeeper.sh
-source install-wal2json.sh
-source bootstrap-snuba.sh
-source create-kafka-topics.sh
-source upgrade-postgres.sh
 source set-up-and-migrate-database.sh
-source migrate-file-storage.sh
-source relay-credentials.sh
-source geoip.sh
-source wrap-up.sh
diff --git a/install/set-up-and-migrate-database.sh b/install/set-up-and-migrate-database.sh
index 38d4093..3be89ae 100644
--- a/install/set-up-and-migrate-database.sh
+++ b/install/set-up-and-migrate-database.sh
@@ -9,7 +9,9 @@ if [[ -n "${CI:-}" || "${SKIP_USER_PROMPT:-0}" == 1 ]]; then
   echo "  docker-compose run --rm web createuser"
   echo ""
 else
-  $dcr web upgrade
+  cmd="docker compose --ansi never run --rm web upgrade"
+  echo "$cmd"
+  $cmd
 fi
 
 echo "${_endgroup}"

@chadwhitacre
Copy link
Member

A clue! It's triggered by sourcing install/_lib.sh ... 🤨

@chadwhitacre
Copy link
Member

@chadwhitacre
Copy link
Member

Yup. 👍

The following works with docker-compose and fails with docker compose:

diff --git a/install.sh b/install.sh
index c1ac6ee..92c9eeb 100755
--- a/install.sh
+++ b/install.sh
@@ -1,12 +1,4 @@
 #!/usr/bin/env bash
 set -e
-if [[ -n "$MSYSTEM" ]]; then
-  echo "Seems like you are using an MSYS2-based system (such as Git Bash) which is not supported. Please use WSL instead.";
-  exit 1
-fi
-
-source "$(dirname $0)/install/_lib.sh"  # does a `cd .../install/`, among other things
-
-source parse-cli.sh
-source error-handling.sh
-source set-up-and-migrate-database.sh
+exec &> >(tee -a "foo.log")
+docker compose --ansi never run --rm web upgrade
$ cat install.sh 
#!/usr/bin/env bash
set -e
exec &> >(tee -a "foo.log")
docker compose --ansi never run --rm web upgrade
$

Okay! Now what? 🤔

@BYK
Copy link
Member

BYK commented Nov 5, 2021

I think it might be due to this: docker/compose#7306 (comment)

@BYK
Copy link
Member

BYK commented Nov 5, 2021

@chadwhitacre can you try with adding tty: true under the web service definition and see if it helps?

@chadwhitacre
Copy link
Member

Yeah I tried that based on docker/compose#3106. Didn't help. :-/

@BYK
Copy link
Member

BYK commented Nov 5, 2021

@chadwhitacre that one suggests using both tty: true and stdin_open: true. Did you try both, or only tty?

@chadwhitacre
Copy link
Member

I tried both ways (both keys, and only tty).

chadwhitacre added a commit that referenced this issue Nov 5, 2021
@chadwhitacre
Copy link
Member

4086fa3

@chadwhitacre
Copy link
Member

I whittled down a test case.

@chadwhitacre
Copy link
Member

Filed docker/compose#8908. 🤞 ¯\_(ツ)_/¯

@chadwhitacre chadwhitacre changed the title install error install.sh hangs during superuser creation under Docker Compose v2 Nov 9, 2021
@Martmists-GH
Copy link

Likely related: docker-compose up -d hangs indefinitely, never starting the relay or nginx. (might not be indefinitely, but I ended up cancelling it after waiting for 8 hours)

@chadwhitacre
Copy link
Member

@Martmists-GH Did you look at docker/compose#8908 and the test case? It seems related to stdout/err redirection in the shell script that invokes docker compose.

@chadwhitacre
Copy link
Member

Good news! This was fixed in docker/compose#9035. 💃

Once there is a new release of Docker Compose we can close this ticket. 👍

@GingerAdonis
Copy link

The fix has landed in docker compose v2.2.3. https://github.com/docker/compose/releases/tag/v2.2.3

@chadwhitacre
Copy link
Member

Nice! Thanks for reporting back @GingerAdonis. :)

Closing! Will pick up on #1251 ...

@github-actions github-actions bot locked and limited conversation to collaborators Jan 27, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

No branches or pull requests

5 participants