Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Failing auto deploy #1601

Open
aequitas opened this issue Dec 20, 2024 · 2 comments · May be fixed by #1602
Open

Failing auto deploy #1601

aequitas opened this issue Dec 20, 2024 · 2 comments · May be fixed by #1602
Labels
infrastructure-docker

Comments

@aequitas
Copy link
Collaborator

Autodeploy was failing for days. Output was:

run-parts: /etc/periodic-docker/15min/auto_update: exit status 125
a465d6468f2d
e90d2c1daca3
crond: USER root pid 7884 cmd run-parts /etc/periodic-docker/15min
main: Pulling from internetstandards/util
Digest: sha256:cb72c5dd365960b9d67608d5b9630fa2f9ec9d6bd8372e5e8e188cde42f3ff32
Status: Image is up to date for ghcr.io/internetstandards/util:main
ghcr.io/internetstandards/util:main
Updating to: 1.9.0.dev176-g2b50952
docker: invalid reference format.
See 'docker run --help'.
run-parts: /etc/periodic-docker/15min/auto_update: exit status 125
a465d6468f2d
e90d2c1daca3
crond: USER root pid 7923 cmd run-parts /etc/periodic-docker/15min
crond: USER root pid 7924 cmd run-parts /etc/periodic-docker/hourly
52710664ffe6
5b5e207c7bb4
main: Pulling from internetstandards/util
Digest: sha256:cb72c5dd365960b9d67608d5b9630fa2f9ec9d6bd8372e5e8e188cde42f3ff32
Status: Image is up to date for ghcr.io/internetstandards/util:main
ghcr.io/internetstandards/util:main
Updating to: 1.9.0.dev176-g2b50952
docker: invalid reference format.
See 'docker run --help'.
run-parts: /etc/periodic-docker/15min/auto_update: exit status 125
a465d6468f2d
e90d2c1daca3
crond: USER root pid 7981 cmd run-parts /etc/periodic-docker/15min
main: Pulling from internetstandards/util
Digest: sha256:cb72c5dd365960b9d67608d5b9630fa2f9ec9d6bd8372e5e8e188cde42f3ff32
Status: Image is up to date for ghcr.io/internetstandards/util:main
ghcr.io/internetstandards/util:main
Updating to: 1.9.0.dev176-g2b50952
docker: invalid reference format.
See 'docker run --help'.

Debug output when running manually:

root@int-dev-docker:/opt/Internet.nl# docker exec -ti internetnl-prod-cron-docker-1 sh -vex /etc/periodic-docker/15min/auto_update
#!/bin/sh

set -e
+ set -e

if [ -z "$AUTO_UPDATE_TO" ];then
  # auto update not configured
  exit 0
fi
+ '[' -z main ]

cd /opt/Internet.nl/
+ cd /opt/Internet.nl/

CURRENT_RELEASE="$RELEASE"
+ CURRENT_RELEASE=1.9.0.dev174-gca80957

IMAGE="${DOCKER_REGISTRY:-ghcr.io/internetstandards}/util:${AUTO_UPDATE_TO}"
+ IMAGE=ghcr.io/internetstandards/util:main

# pull latest image
docker pull "$IMAGE"
+ docker pull ghcr.io/internetstandards/util:main
main: Pulling from internetstandards/util
Digest: sha256:cb72c5dd365960b9d67608d5b9630fa2f9ec9d6bd8372e5e8e188cde42f3ff32
Status: Image is up to date for ghcr.io/internetstandards/util:main
ghcr.io/internetstandards/util:main

# get release info from image
UPSTREAM_RELEASE="$(docker image inspect "$IMAGE" | jq -r '.[0].Config.Labels.release')"
+ docker image inspect ghcr.io/internetstandards/util:main
+ jq -r '.[0].Config.Labels.release'
+ UPSTREAM_RELEASE=1.9.0.dev176-g2b50952

# check if update is required
if [ "$CURRENT_RELEASE" = "$UPSTREAM_RELEASE" ];then
  echo "No update available"
  exit 0
fi
+ '[' 1.9.0.dev174-gca80957 '=' 1.9.0.dev176-g2b50952 ]

echo "Updating to: $UPSTREAM_RELEASE"
+ echo 'Updating to: 1.9.0.dev176-g2b50952'
Updating to: 1.9.0.dev176-g2b50952

# initiate update, don't remove container when done to preserve logging
# as the initiator (cron-docker) will be killed during the deploy process
docker run --pull=always --network none \
  --volume /var/run/docker.sock:/var/run/docker.sock \
  --volume $HOME/.docker:/root/.docker \
  --volume /opt/Internet.nl:/opt/Internet.nl \
  --env "DOCKER_REGISTRY=$DOCKER_REGISTRY" \
  "$DOCKER_REGISTRY/util:$UPSTREAM_RELEASE" \
  /deploy.sh
+ docker run '--pull=always' --network none --volume /var/run/docker.sock:/var/run/docker.sock --volume /root/.docker:/root/.docker --volume /opt/Internet.nl:/opt/Internet.nl --env 'DOCKER_REGISTRY=' /util:1.9.0.dev176-g2b50952 /deploy.sh
docker: invalid reference format.
See 'docker run --help'.

Solved by manually updating to a newer release:

root@int-dev-docker:/opt/Internet.nl# docker run -ti --rm --pull=always \
  --volume /var/run/docker.sock:/var/run/docker.sock \
  --volume $HOME/.docker:/root/.docker \
  --volume /opt/Internet.nl:/opt/Internet.nl \
  --network none \
  --env DOCKER_REGISTRY=ghcr.io/internetstandards \
  ghcr.io/internetstandards/util:1.9.0.dev175-g0e5a6d1 \
  /deploy.sh

After which automatic update worked again:

root@int-dev-docker:/opt/Internet.nl# docker exec -ti internetnl-prod-cron-docker-1 sh -vex /etc/periodic-docker/15min/auto_update
#!/bin/sh

set -e
+ set -e

if [ -z "$AUTO_UPDATE_TO" ];then
  # auto update not configured
  exit 0
fi
+ '[' -z main ]

cd /opt/Internet.nl/
+ cd /opt/Internet.nl/

CURRENT_RELEASE="$RELEASE"
+ CURRENT_RELEASE=1.9.0.dev175-g0e5a6d1

IMAGE="${DOCKER_REGISTRY:-ghcr.io/internetstandards}/util:${AUTO_UPDATE_TO}"
+ IMAGE=ghcr.io/internetstandards/util:main

# pull latest image
docker pull "$IMAGE"
+ docker pull ghcr.io/internetstandards/util:main
main: Pulling from internetstandards/util
Digest: sha256:cb72c5dd365960b9d67608d5b9630fa2f9ec9d6bd8372e5e8e188cde42f3ff32
Status: Image is up to date for ghcr.io/internetstandards/util:main
ghcr.io/internetstandards/util:main

# get release info from image
UPSTREAM_RELEASE="$(docker image inspect "$IMAGE" | jq -r '.[0].Config.Labels.release')"
+ docker image inspect ghcr.io/internetstandards/util:main
+ jq -r '.[0].Config.Labels.release'
+ UPSTREAM_RELEASE=1.9.0.dev176-g2b50952

# check if update is required
if [ "$CURRENT_RELEASE" = "$UPSTREAM_RELEASE" ];then
  echo "No update available"
  exit 0
fi
+ '[' 1.9.0.dev175-g0e5a6d1 '=' 1.9.0.dev176-g2b50952 ]

echo "Updating to: $UPSTREAM_RELEASE"
+ echo 'Updating to: 1.9.0.dev176-g2b50952'
Updating to: 1.9.0.dev176-g2b50952

# initiate update, don't remove container when done to preserve logging
# as the initiator (cron-docker) will be killed during the deploy process
docker run --pull=always --network none \
  --volume /var/run/docker.sock:/var/run/docker.sock \
  --volume $HOME/.docker:/root/.docker \
  --volume /opt/Internet.nl:/opt/Internet.nl \
  --env "DOCKER_REGISTRY=$DOCKER_REGISTRY" \
  "$DOCKER_REGISTRY/util:$UPSTREAM_RELEASE" \
  /deploy.sh
+ docker run '--pull=always' --network none --volume /var/run/docker.sock:/var/run/docker.sock --volume /root/.docker:/root/.docker --volume /opt/Internet.nl:/opt/Internet.nl --env 'DOCKER_REGISTRY=ghcr.io/internetstandards' ghcr.io/internetstandards/util:1.9.0.dev176-g2b50952 /deploy.sh
1.9.0.dev176-g2b50952: Pulling from internetstandards/util
Digest: sha256:cb72c5dd365960b9d67608d5b9630fa2f9ec9d6bd8372e5e8e188cde42f3ff32
Status: Downloaded newer image for ghcr.io/internetstandards/util:1.9.0.dev176-g2b50952
Deploying release: 1.9.0.dev176-g2b50952
'/dist/docker/compose.yaml' -> 'docker/compose.yaml'
'/dist/docker/defaults.env' -> 'docker/defaults.env'
'/dist/docker/host-dist.env' -> 'docker/host-dist.env'
'/dist/docker/user_manage.sh' -> 'docker/user_manage.sh'
 worker-slow Skipped - Image is already being pulled by worker-nassl
 app Skipped - Image is already being pulled by worker-nassl
 webserver Pulling
 redis Pulling
 grafana Pulling
 db-migrate Skipped - Image is already being pulled by worker-nassl
 resolver-permissive Skipped - Image is already being pulled by unbound
 beat Skipped - Image is already being pulled by worker-nassl
 resolver-validating Skipped - Image is already being pulled by unbound
 cron Skipped - Image is already being pulled by cron-docker
 worker Skipped - Image is already being pulled by worker-nassl
 redis-exporter Pulling
 statsd-exporter Pulling
 nginx_logs_exporter Pulling
 rabbitmq Pulling
 prometheus Pulling
 postgres Pulling
 postgresql-exporter Pulling
 celery-exporter Pulling
 routinator Pulling
 worker-nassl Pulling
 node-exporter Pulling
 cron-docker Pulling
 unbound Pulling
 alertmanager Pulling
 docker_stats_exporter Pulling
 nginx_logs_exporter Pulled
 cron-docker Pulled
 grafana Pulled

<<<<snip>>>>>

 700b144b6 Pull complete
 b94a1579bd38 Extracting [=========>                                         ]  32.77kB/177.3kB
 b94a1579bd38 Extracting [==================================================>]  177.3kB/177.3kB
 b94a1579bd38 Extracting [==================================================>]  177.3kB/177.3kB
 b94a1579bd38 Pull complete
 worker-nassl Pulled
 Container internetnl-prod-statsd-exporter-1  Running
 Container internetnl-prod-postgres-1  Running
 Container internetnl-prod-db-migrate-1  Recreate
 Container internetnl-prod-rabbitmq-1  Running
 Container internetnl-prod-celery-exporter-1  Running
 Container internetnl-prod-docker_stats_exporter-1  Running
 Container internetnl-prod-cron-docker-1  Recreate
 Container internetnl-prod-nginx_logs_exporter-1  Running
 Container internetnl-prod-node-exporter-1  Running
 Container internetnl-prod-resolver-validating-1  Recreate
 Container internetnl-prod-alertmanager-1  Running
 Container internetnl-prod-redis-exporter-1  Running
 Container internetnl-prod-cron-1  Recreate
 Container internetnl-prod-routinator-1  Running
 Container internetnl-prod-postgresql-exporter-1  Running
 Container internetnl-prod-redis-1  Running
 Container internetnl-prod-unbound-1  Recreate
 Container internetnl-prod-webserver-1  Recreate
 Container internetnl-prod-prometheus-1  Running
 Container internetnl-prod-grafana-1  Recreate
 Container internetnl-prod-resolver-permissive-1  Recreate
 Container internetnl-prod-db-migrate-1  Recreated
 Container internetnl-prod-beat-1  Recreate
 Container internetnl-prod-grafana-1  Recreated
 Container internetnl-prod-webserver-1  Recreated
 Container internetnl-prod-beat-1  Recreated
@aequitas
Copy link
Collaborator Author

Direct issue is now resolved, recorded logging for future reference.

@bwbroersma
Copy link
Collaborator

bwbroersma commented Dec 20, 2024 •
edited
Loading

Great debugging.
So the issue was docker was called with /util:1.9.0.dev176-g2b50952 instead of ghcr.io/internetstandards/util:1.9.0.dev176-g2b50952 because DOCKER_REGISTRY was empty, which is needed since

I wonder however, since there are quite some ${DOCKER_REGISTRY:-ghcr.io/internetstandards} in the compose files, and also here:

Internet.nl/docker/cron-docker/periodic/15min/auto_update

Line 14 in 2b50952

IMAGE="${DOCKER_REGISTRY:-ghcr.io/internetstandards}/util:${AUTO_UPDATE_TO}"

but it is missing in the two cases here:

Internet.nl/docker/cron-docker/periodic/15min/auto_update

Lines 36 to 37 in 2b50952

--env "DOCKER_REGISTRY=$DOCKER_REGISTRY" \
"$DOCKER_REGISTRY/util:$UPSTREAM_RELEASE" \

  1. I think the last line should be fixed.
  2. Would be great to 'centralize' the ghcr.io/internetstandards, but maybe that's hard since it's a 'top most' variable used in compose?

@bwbroersma bwbroersma reopened this Dec 20, 2024
bwbroersma added a commit to bwbroersma/Internet.nl that referenced this issue Dec 20, 2024
@bwbroersma
Fix auto_update DOCKER_REGISTRY default
3826a26 
Fixes internetstandards#1601
Use ${DOCKER_REGISTRY:-ghcr.io/internetstandards} instead of $DOCKER_REGISTRY.
@bwbroersma bwbroersma linked a pull request Dec 20, 2024 that will close this issue
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
infrastructure-docker
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fix auto_update DOCKER_REGISTRY default bwbroersma/Internet.nl
2 participants
@aequitas @bwbroersma