Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make architect dev more robust #661

Merged
merged 8 commits into from
Aug 15, 2022
Merged

Make architect dev more robust #661

merged 8 commits into from
Aug 15, 2022

Conversation

TylerAldrich
Copy link
Contributor

@TylerAldrich TylerAldrich commented Aug 11, 2022

The changes in this PR include:

  • Fixing handling of ctrl+c, ctrl+break on Windows by not passing signals to the underlying child process but just spawning a docker compose stop
  • Fix handling of signals on mac/linux by using detached: true so the above docker compose stop happens and the chlid process doesn't automatically get the sigint.
  • Fix multiple places were interrupting architect dev could leave you with still-running-containers
    • Between when docker compose up is attaching to the logs and actually starting the containers, if you ^C, docker compose leaves your containers hanging. This is a problem with the underlying docker compose command, so to fix this for Architect, ^C doesn't do anything until logs start getting output. This is generally a very short amount of time after the "Attaching..." starts and without artificial delays I was only able to get this to reliably happen accidentally.
    • If a container was restarting right as you ^C, the container will restart after the stop is run and leaves you hanging indefinitely, because not all containers will stop. This was because we checked a condition at the top of a loop, but within the loop await, so the result can change (Unless nodejs is using multiple threads which makes things more complicated, but still a race condition regardless 🤷‍♂️)

note: the require("readline").createInterface isn't necessary anymore. This was tested in both Powershell and CMD - node now supports automatically mapping ctrl+c -> SIGINT and ctrl+break -> SIGBREAK: nodejs/node-v0.x-archive#5054 (comment)

SIGBREAK is a windows-only, listen-only signal that just signifies ctrl+break was used on Windows (see: https://nodejs.org/api/all.html#all_process_signal-events) - it has no affect on other platforms so we don't need to guard it behind process.platform === 'win32'

Catching SIGBREAK just helps us cleanly kill containers with both supported windows "interrupt"-like commands.

Testing Checklist:

  • Windows - cmd
  • Windows - powershell
  • Windows - Linux via wsl2
  • Mac

…ing containers running, run docker compose stop behind the scenes to clean up.
src/commands/dev.ts Outdated Show resolved Hide resolved
@TylerAldrich
Copy link
Contributor Author

It looks like there's a bit of a race in docker compose up where you can input a SIGINT between "Attaching to ..." and when docker compose stop is actually able to stop the containers:

tyler@Tyler-The-King:~/architect-cli$ docker compose -f /home/tyler/.config/architect/docker-compose/architect.yml -p architect up --remove-orphans --renew-anon-volumes --timeout 0
[+] Running 2/0
 ⠿ Container architect-hello-world--api-1  Created                                                                                                                     0.0s
 ⠿ Container architect-gateway-1           Created                                                                                                                     0.0s
Attaching to architect-gateway-1, architect-hello-world--api-1
^CGracefully stopping... (press Ctrl+C again to force)
canceled

In this example the containers weren't stopped at all. As we're kind of just wrapping the docker compose up functionality in our dev command, it's possible for a user to have this issue as well.

We can however wait until the containers are definitely started and have the SIGINT do nothing until we can successfully stop.

TylerAldrich and others added 3 commits August 11, 2022 15:30
…ally passing down the SIGINT from parent->child process for docker compose cmd. Add some special logic to handle fixing a case where a user can SIGINT docker compose in just the right place and leave their containers running unintentionally.
@TylerAldrich TylerAldrich changed the title WIP: Handle architect dev ctrl+c and ctrl+break properly on Windows Powershell/cmd Make architect dev more robust Aug 11, 2022
@ryan-cahill
Copy link
Member

It looks like there's a bit of a race in docker compose up where you can input a SIGINT between "Attaching to ..." and when docker compose stop is actually able to stop the containers:

tyler@Tyler-The-King:~/architect-cli$ docker compose -f /home/tyler/.config/architect/docker-compose/architect.yml -p architect up --remove-orphans --renew-anon-volumes --timeout 0
[+] Running 2/0
 ⠿ Container architect-hello-world--api-1  Created                                                                                                                     0.0s
 ⠿ Container architect-gateway-1           Created                                                                                                                     0.0s
Attaching to architect-gateway-1, architect-hello-world--api-1
^CGracefully stopping... (press Ctrl+C again to force)
canceled

In this example the containers weren't stopped at all. As we're kind of just wrapping the docker compose up functionality in our dev command, it's possible for a user to have this issue as well.

We can however wait until the containers are definitely started and have the SIGINT do nothing until we can successfully stop.

Similarly I get an issue when I start then shortly stop our own local stack while a container is in the middle of being restarted. Note that I had just removed all containers

Once the containers are running they will be accessible via the following urls:
http://registry.arc.localhost:1024/ => cloud--registry-proxy
http://api.arc.localhost:1024/ => cloud--api
http://cloud.arc.localhost:1024/ => cloud--app

http://localhost:1024/ => gateway-1024:1024
http://localhost:8080/ => gateway-1024:8080
http://localhost:50000/ => cloud--registry:5000
http://localhost:50001/ => cloud--registry-proxy:8080
http://localhost:50002/ => cloud--api:8080
http://localhost:50003/ => cloud--api-redis:6379
http://localhost:50004/ => cloud--app:8080

Starting containers...

ERROR: cloud.services.api has encountered an error and is being restarted.
a53ad8b21d2382c1c7205e905b572dfa7488e11a06048417ef120f7a27e5ac33
ERROR: cloud.services.api-redis has encountered an error and is being restarted.
architect-cloud--api-redis-1            |  redis 14:04:01.96 
architect-cloud--api-redis-1            |  redis 14:04:01.96 Welcome to the Bitnami redis container
architect-cloud--api-redis-1            |  redis 14:04:01.96 Subscribe to project updates by watching https://github.com/bitnami/bitnami-docker-redis
architect-cloud--api-redis-1            |  redis 14:04:01.97 Submit issues and feature requests at https://github.com/bitnami/bitnami-docker-redis/issues
architect-cloud--api-redis-1            |  redis 14:04:01.97 
architect-cloud--api-redis-1            |  redis 14:04:01.97 INFO  ==> ** Starting Redis setup **
architect-cloud--api-redis-1            |  redis 14:04:02.01 INFO  ==> Initializing Redis
architect-cloud--api-redis-1            |  redis 14:04:02.03 INFO  ==> Setting Redis config file
architect-cloud--api-redis-1            |  redis 14:04:02.07 INFO  ==> ** Redis setup finished! **
architect-cloud--api-redis-1            |  
architect-cloud--api-redis-1            |  redis 14:04:02.12 INFO  ==> ** Starting Redis **
architect-cloud--api-redis-1            |  1:C 12 Aug 2022 14:04:02.135 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
architect-cloud--api-redis-1            |  1:C 12 Aug 2022 14:04:02.135 # Redis version=5.0.14, bits=64, commit=00000000, modified=0, pid=1, just started
architect-cloud--api-redis-1            |  1:C 12 Aug 2022 14:04:02.135 # Configuration loaded
architect-cloud--api-redis-1            |                  _._                                                  
architect-cloud--api-redis-1            |             _.-``__ ''-._                                             
architect-cloud--api-redis-1            |        _.-``    `.  `_.  ''-._           Redis 5.0.14 (00000000/0) 64 bit
architect-cloud--api-redis-1            |    .-`` .-```.  ```\/    _.,_ ''-._                                   
architect-cloud--api-redis-1            |   (    '      ,       .-`  | `,    )     Running in standalone mode
architect-cloud--api-redis-1            |   |`-._`-...-` __...-.``-._|'` _.-'|     Port: 6379
architect-cloud--api-redis-1            |   |    `-._   `._    /     _.-'    |     PID: 1
architect-cloud--api-redis-1            |    `-._    `-._  `-./  _.-'    _.-'                                   
architect-cloud--api-redis-1            |   |`-._`-._    `-.__.-'    _.-'_.-'|                                  
architect-cloud--api-redis-1            |   |    `-._`-._        _.-'_.-'    |           http://redis.io        
architect-cloud--api-redis-1            |    `-._    `-._`-.__.-'_.-'    _.-'                                   
architect-cloud--api-redis-1            |   |`-._`-._    `-.__.-'    _.-'_.-'|                                  
architect-cloud--api-redis-1            |   |    `-._`-._        _.-'_.-'    |                                  
architect-cloud--api-redis-1            |    `-._    `-._`-.__.-'_.-'    _.-'                                   
architect-cloud--api-redis-1            |        `-._    `-.__.-'    _.-'                                       
architect-cloud--api-redis-1            |            `-._        _.-'                                           
architect-cloud--api-redis-1            |                `-.__.-'                                               
architect-cloud--api-redis-1            |  
architect-cloud--api-redis-1            |  1:M 12 Aug 2022 14:04:02.137 # Server initialized
architect-cloud--api-redis-1            |  1:M 12 Aug 2022 14:04:02.137 # WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled.
architect-cloud--api-redis-1            |  1:M 12 Aug 2022 14:04:02.138 * Ready to accept connections
architect-cloud--registry-1             |  time="2022-08-12T14:04:02.180116848Z" level=warning msg="No HTTP secret provided - generated random secret. This may cause problems with uploads if multiple registries are behind a load-balancer. To provide a shared secret, fill in http.secret in the configuration file or set the REGISTRY_HTTP_SECRET environment variable." environment=local go.version=go1.16.15 instance.id=b1599918-91bf-4fb3-8d2a-ec1352acb7f8 service=registry version="v2.8.1+unknown" 
architect-cloud--registry-1             |  time="2022-08-12T14:04:02.180198786Z" level=info msg="Starting upload purge in 44m0s" environment=local go.version=go1.16.15 instance.id=b1599918-91bf-4fb3-8d2a-ec1352acb7f8 service=registry version="v2.8.1+unknown" 
architect-cloud--registry-1             |  time="2022-08-12T14:04:02.180538464Z" level=info msg="redis not configured" environment=local go.version=go1.16.15 instance.id=b1599918-91bf-4fb3-8d2a-ec1352acb7f8 service=registry version="v2.8.1+unknown" 
architect-cloud--registry-1             |  time="2022-08-12T14:04:02.191193076Z" level=info msg="backend redirection disabled" environment=local go.version=go1.16.15 instance.id=b1599918-91bf-4fb3-8d2a-ec1352acb7f8 service=registry version="v2.8.1+unknown" 
architect-cloud--registry-1             |  time="2022-08-12T14:04:02.196112926Z" level=info msg="using inmemory blob descriptor cache" environment=local go.version=go1.16.15 instance.id=b1599918-91bf-4fb3-8d2a-ec1352acb7f8 service=registry version="v2.8.1+unknown" 
architect-cloud--registry-1             |  time="2022-08-12T14:04:02.196377204Z" level=info msg="listening on [::]:5000" environment=local go.version=go1.16.15 instance.id=b1599918-91bf-4fb3-8d2a-ec1352acb7f8 service=registry version="v2.8.1+unknown" 
architect-cloud--api-1                  |  
architect-cloud--api-1                  |  > architect-cloud-api@0.1.0 start:dev
architect-cloud--api-1                  |  > nest build --webpack --webpackPath webpack-hmr.config.js --watch
architect-cloud--api-1                  |  
architect-gateway-1024-1                |  time="2022-08-12T14:04:02Z" level=info msg="Configuration loaded from flags."
architect-cloud--app-1                  |  
architect-cloud--app-1                  |  > architect-cloud@1.0.0 dev /usr/src/app
architect-cloud--app-1                  |  > nuxt
architect-cloud--app-1                  |  
architect-cloud--registry-proxy-1       |  
architect-cloud--registry-proxy-1       |  > docker-registry-proxy@1.0.0 start:dev
architect-cloud--registry-proxy-1       |  > nodemon src/index.js
architect-cloud--registry-proxy-1       |  
architect-cloud--registry-proxy-1       |  [nodemon] 2.0.19
architect-cloud--registry-proxy-1       |  [nodemon] to restart at any time, enter `rs`
architect-cloud--registry-proxy-1       |  [nodemon] watching path(s): *.*
architect-cloud--registry-proxy-1       |  [nodemon] watching extensions: js,mjs,json
architect-cloud--registry-proxy-1       |  [nodemon] starting `node src/index.js`
^C
Gracefully stopping..... Please Wait.....
ERROR: cloud.services.api-redis failed to restart.

Our process ends, but the container remains running

ryan@ryan-ThinkPad-P53s:~/Code/architect-cli$ docker ps
CONTAINER ID   IMAGE               COMMAND                  CREATED         STATUS         PORTS                     NAMES
b77ab9c37071   bitnami/redis:5.0   "/opt/bitnami/script…"   4 minutes ago   Up 4 minutes   0.0.0.0:50003->6379/tcp   architect-cloud--api-redis-1

@TylerAldrich
Copy link
Contributor Author

@ryan-cahill That issue I ran into as well and should be fixed by this commit here: 4e58ab1 (Ignore the debugging file I committed 😛)

@tjhiggins tjhiggins merged commit eda1cff into rc Aug 15, 2022
@tjhiggins tjhiggins deleted the windows-dev-interrupt branch August 15, 2022 16:23
github-actions bot pushed a commit that referenced this pull request Aug 15, 2022
## [1.22.1-rc.6](v1.22.1-rc.5...v1.22.1-rc.6) (2022-08-15)

### Bug Fixes

* **dev:** Make `architect dev` more robust ([#661](#661)) ([eda1cff](eda1cff))
github-actions bot pushed a commit that referenced this pull request Aug 15, 2022
# [1.23.0](v1.22.1...v1.23.0) (2022-08-15)

### Bug Fixes

* **cli:** Switch from postinstall to prepare ([ffac972](ffac972))
* **dev:** Don't show ping access logs unless they fail ([33dc9b4](33dc9b4))
* **dev:** Make `architect dev` more robust ([#661](#661)) ([eda1cff](eda1cff))
* **exec:** Fix issue with commands run in no-tty but with stdin available exiting prematurely due to stdin closing ([#656](#656)) ([06ef0df](06ef0df))
* **exec:** no local selection when docker is unavailable ([#662](#662)) ([e3bf33b](e3bf33b))
* **register:** Allow register without build ([#660](#660)) ([033f261](033f261))

### Features

* **link:** 473 list linked components ([#658](#658)) ([b40c347](b40c347))
@github-actions
Copy link

🎉 This PR is included in version 1.23.0 🎉

The release is available on:

Your semantic-release bot 📦🚀

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants