website build broken during Docker image preparation #3382

richardlau · 2023-06-13T14:41:49Z

@RafaelGSS has pointed out from the webhook log that the build did not succeed:

stderr: Previous HEAD position was 58704a9... chore: updated issue templates descriptions
stderr: HEAD is now at 7a4e29c... blog: add june 2023 security releases (#5434)
stdout: lts: Pulling from library/node
stdout: c98dda1b0e97: Pulling fs layer
stdout: 9a463ac54ed3: Pulling fs layer
stdout: b70766638915: Pulling fs layer
stdout: 4bcc83cd0070: Pulling fs layer
stdout: 217770c726a2: Pulling fs layer
stdout: cec0e089046a: Pulling fs layer
stdout: 6fedcf8a3041: Pulling fs layer
stdout: 471e0a0d1303: Pulling fs layer
stdout: 4bcc83cd0070: Waiting
stdout: 217770c726a2: Waiting
stdout: cec0e089046a: Waiting
stdout: 471e0a0d1303: Waiting
stdout: 9a463ac54ed3: Download complete
stdout: c98dda1b0e97: Verifying Checksum
stdout: c98dda1b0e97: Download complete
stdout: 217770c726a2: Verifying Checksum
stdout: 217770c726a2: Download complete
stdout: b70766638915: Verifying Checksum
stdout: b70766638915: Download complete
stdout: 6fedcf8a3041: Verifying Checksum
stdout: 6fedcf8a3041: Download complete
stdout: 471e0a0d1303: Verifying Checksum
stdout: 471e0a0d1303: Download complete
stdout: cec0e089046a: Verifying Checksum
stdout: cec0e089046a: Download complete
stdout: 4bcc83cd0070: Verifying Checksum
stdout: 4bcc83cd0070: Download complete
stdout: c98dda1b0e97: Pull complete
stdout: 9a463ac54ed3: Pull complete
stdout: b70766638915: Pull complete
stdout: 4bcc83cd0070: Pull complete
stdout: 217770c726a2: Pull complete
stdout: cec0e089046a: Pull complete
stdout: 6fedcf8a3041: Pull complete
stdout: 471e0a0d1303: Pull complete
stdout: Digest: sha256:d244ade7d0b2ad97002102880142c8b9a4b016ea2308af87e9b150c43987b301
stdout: Status: Downloaded newer image for node:lts
stdout: docker.io/library/node:lts
stdout: Get:1 http://deb.debian.org/debian bookworm InRelease [147 kB]
stdout: Get:2 http://deb.debian.org/debian bookworm-updates InRelease [52.1 kB]
stdout: Get:3 http://deb.debian.org/debian-security bookworm-security InRelease [48.0 kB]
stdout: Get:4 http://deb.debian.org/debian bookworm/main amd64 Packages [8904 kB]
stdout: Get:5 http://deb.debian.org/debian-security bookworm-security/main amd64 Packages [17.6 kB]
stdout: Fetched 9169 kB in 2s (3673 kB/s)
stdout: Reading package lists...
stderr: E: Problem executing scripts APT::Update::Post-Invoke 'rm -f /var/cache/apt/archives/*.deb /var/cache/apt/archives/partial/*.deb /var/cache/apt/*.bin || true'
stderr: E: Sub-process returned an error code
event="push", match="ref == "refs/heads/main" && repository.full_name == "nodejs/nodejs.org"", exec="/home/nodejs/build-site.sh nodejs"
Tue Jun 13 2023 14:15:19 GMT+0000 (Coordinated Universal Time)
Took 46090 ms

The error looks to be from

build/ansible/www-standalone/resources/scripts/build-site.sh

Lines 40 to 57 in e96c917

    
           docker pull node:lts 
        
           docker run \ 
        
             --rm \ 
        
             -v ${clonedir}:/website/ \ 
        
             -v /home/nodejs/.npm:/npm/ \ 
        
             node:lts \ 
        
             bash -c " \ 
        
               apt-get update && apt-get install -y rsync && \ 
        
               addgroup nodejs --gid ${nodeuid} && \ 
        
               adduser nodejs --uid ${nodeuid} --gid ${nodegid} --gecos nodejs --disabled-password && \ 
        
               su nodejs -c ' \ 
        
                 npm config set loglevel http && \ 
        
                 npm config set cache /npm/ && \ 
        
                 cd /website/ && \ 
        
                 npm ci && \ 
        
                 $build_cmd \ 
        
               ' \ 
        
             "

richardlau · 2023-06-13T15:14:34Z

The failing command is

build/ansible/www-standalone/resources/scripts/build-site.sh

Line 47 in e96c917

apt-get update && apt-get install -y rsync && \

which is failing on the infra machine:

root@infra-digitalocean-ubuntu1604-x64-1:~# docker run -it --rm node:lts bash
root@8f4af573f6ce:/# apt-get update && apt-get install -y rsync
Get:1 http://deb.debian.org/debian bookworm InRelease [147 kB]
Get:2 http://deb.debian.org/debian bookworm-updates InRelease [52.1 kB]
Get:3 http://deb.debian.org/debian-security bookworm-security InRelease [48.0 kB]
Get:4 http://deb.debian.org/debian bookworm/main amd64 Packages [8904 kB]
Get:5 http://deb.debian.org/debian-security bookworm-security/main amd64 Packages [17.6 kB]
Fetched 9169 kB in 2s (4534 kB/s)
Reading package lists... Done
E: Problem executing scripts APT::Update::Post-Invoke 'rm -f /var/cache/apt/archives/*.deb /var/cache/apt/archives/partial/*.deb /var/cache/apt/*.bin || true'
E: Sub-process returned an error code
root@8f4af573f6ce:/#

node:lts is the currently the same as node:lts-bookworm and fails:

root@infra-digitalocean-ubuntu1604-x64-1:~# docker run -it --rm node:lts-bookworm bash
root@843ac074029e:/# apt-get update && apt-get install -y rsync
Get:1 http://deb.debian.org/debian bookworm InRelease [147 kB]
Get:2 http://deb.debian.org/debian bookworm-updates InRelease [52.1 kB]
Get:3 http://deb.debian.org/debian-security bookworm-security InRelease [48.0 kB]
Get:4 http://deb.debian.org/debian bookworm/main amd64 Packages [8904 kB]
Get:5 http://deb.debian.org/debian-security bookworm-security/main amd64 Packages [17.6 kB]
Fetched 9169 kB in 2s (4786 kB/s)
Reading package lists... Done
E: Problem executing scripts APT::Update::Post-Invoke 'rm -f /var/cache/apt/archives/*.deb /var/cache/apt/archives/partial/*.deb /var/cache/apt/*.bin || true'
E: Sub-process returned an error code
root@843ac074029e:/#

but node:lts-bullseye succeeds:

root@infra-digitalocean-ubuntu1604-x64-1:~# docker run -it --rm node:lts-bullseye bash
root@8526f963090e:/# apt-get update && apt-get install -y rsync
Get:1 http://deb.debian.org/debian bullseye InRelease [116 kB]
Get:2 http://deb.debian.org/debian-security bullseye-security InRelease [48.4 kB]
Get:3 http://deb.debian.org/debian bullseye-updates InRelease [44.1 kB]
Get:4 http://deb.debian.org/debian bullseye/main amd64 Packages [8183 kB]
Get:5 http://deb.debian.org/debian-security bullseye-security/main amd64 Packages [245 kB]
Get:6 http://deb.debian.org/debian bullseye-updates/main amd64 Packages [14.8 kB]
Fetched 8651 kB in 2s (4689 kB/s)
Reading package lists... Done
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following additional packages will be installed:
  libpopt0
Suggested packages:
  openssh-server
The following NEW packages will be installed:
  libpopt0 rsync
0 upgraded, 2 newly installed, 0 to remove and 3 not upgraded.
Need to get 446 kB of archives.
After this operation, 987 kB of additional disk space will be used.
Get:1 http://deb.debian.org/debian bullseye/main amd64 libpopt0 amd64 1.18-2 [49.6 kB]
Get:2 http://deb.debian.org/debian bullseye/main amd64 rsync amd64 3.2.3-4+deb11u1 [396 kB]
Fetched 446 kB in 0s (10.6 MB/s)
debconf: delaying package configuration, since apt-utils is not installed
Selecting previously unselected package libpopt0:amd64.
(Reading database ... 22794 files and directories currently installed.)
Preparing to unpack .../libpopt0_1.18-2_amd64.deb ...
Unpacking libpopt0:amd64 (1.18-2) ...
Selecting previously unselected package rsync.
Preparing to unpack .../rsync_3.2.3-4+deb11u1_amd64.deb ...
Unpacking rsync (3.2.3-4+deb11u1) ...
Setting up libpopt0:amd64 (1.18-2) ...
Setting up rsync (3.2.3-4+deb11u1) ...
invoke-rc.d: could not determine current runlevel
invoke-rc.d: policy-rc.d denied execution of start.
Processing triggers for libc-bin (2.31-13+deb11u6) ...
root@8526f963090e:/#

root@infra-digitalocean-ubuntu1604-x64-1:~# docker images
REPOSITORY   TAG            IMAGE ID       CREATED        SIZE
node         lts            aa392f56f466   12 hours ago   1.09GB
node         lts-bookworm   aa392f56f466   12 hours ago   1.09GB
node         lts-bullseye   78b037dbb659   3 weeks ago    996MB

Oddly I can't reproduce the failures with node:lts/node:lts-bookworm on a separate machine.

mhdawson · 2023-06-13T15:15:47Z

What are the versions of ubuntu on the passing/failing based machine? Maybe that is related.

mhdawson · 2023-06-13T15:16:22Z

Just wondering since I see ubuntu1604 in the name of the website machine

richardlau · 2023-06-13T15:18:37Z

That's because it is still running Ubuntu 16.04.
The other machine I was testing on is running RHEL 8.

targos · 2023-06-13T15:31:24Z

Maybe this can be fixed by upgrading Docker, but I don't know if recent versions of Docker work on Ubuntu 16.04

Temporarily switch to `node:lts-bullseye` while we investigate why the website build is broken with `node:lts`/`node:lts-bookworm`. Refs: #3382

richardlau · 2023-06-13T17:15:48Z

As another datapoint, I tried node:lts-bookworm on test-digitalocean-ubuntu1804-docker-x64-1 and the failing apt update command passes there. That machine is running

Docker version 24.0.1, build 6802122

The RHEL 8 machine I am on is running

Docker version 20.10.12, build e91ed57

while the www server is on

Docker version 20.10.7, build f0df350

richardlau · 2023-06-15T17:10:28Z

FWIW using --security-opt=seccomp:unconfined to start the container allows the bookworm-based containers to run apt-get update && apt-get install -y rsync successfully. I'd rather keep running the production build with the working bullseye-based container than use that option.

root@infra-digitalocean-ubuntu1604-x64-1:~# docker run -it --rm --security-opt=seccomp:unconfined node:lts bash
root@c946f84a1e5c:/# apt-get update && apt-get install -y rsync
Get:1 http://deb.debian.org/debian bookworm InRelease [147 kB]
Get:2 http://deb.debian.org/debian bookworm-updates InRelease [52.1 kB]
Get:3 http://deb.debian.org/debian-security bookworm-security InRelease [48.0 kB]
Get:4 http://deb.debian.org/debian bookworm/main amd64 Packages [8904 kB]
Get:5 http://deb.debian.org/debian-security bookworm-security/main amd64 Packages [24.2 kB]
Fetched 9176 kB in 2s (4859 kB/s)
Reading package lists... Done
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following additional packages will be installed:
  libpopt0
Suggested packages:
  openssh-server python3-braceexpand
The following NEW packages will be installed:
  libpopt0 rsync
0 upgraded, 2 newly installed, 0 to remove and 46 not upgraded.
Need to get 461 kB of archives.
After this operation, 1046 kB of additional disk space will be used.
Get:1 http://deb.debian.org/debian bookworm/main amd64 libpopt0 amd64 1.19+dfsg-1 [43.3 kB]
Get:2 http://deb.debian.org/debian bookworm/main amd64 rsync amd64 3.2.7-1 [417 kB]
Fetched 461 kB in 0s (10.7 MB/s)
debconf: delaying package configuration, since apt-utils is not installed
Selecting previously unselected package libpopt0:amd64.
(Reading database ... 23269 files and directories currently installed.)
Preparing to unpack .../libpopt0_1.19+dfsg-1_amd64.deb ...
Unpacking libpopt0:amd64 (1.19+dfsg-1) ...
Selecting previously unselected package rsync.
Preparing to unpack .../rsync_3.2.7-1_amd64.deb ...
Unpacking rsync (3.2.7-1) ...
Setting up libpopt0:amd64 (1.19+dfsg-1) ...
Setting up rsync (3.2.7-1) ...
invoke-rc.d: could not determine current runlevel
invoke-rc.d: policy-rc.d denied execution of start.
Processing triggers for libc-bin (2.36-9) ...
root@c946f84a1e5c:/#

Daniel15 · 2023-06-22T16:30:39Z

I don't know if recent versions of Docker work on Ubuntu 16.04

Ubuntu 16.04 was released in 2016 and reached its end-of-life in April 2021 (unless you have a paid subscription with Canonical) so anything still using it should really be updated.

Temporarily switch to `node:lts-bullseye` while we investigate why the website build is broken with `node:lts`/`node:lts-bookworm`. Refs: #3383 Refs: #3382

targos · 2024-02-24T08:55:42Z

I think this can be closed. The website is not being built on this server anymore.

MoLow · 2024-02-25T11:11:19Z

+1 closing. can reopen if needed

richardlau added infra incident labels Jun 13, 2023

richardlau mentioned this issue Jun 13, 2023

www: use node:lts-bullseye in build-site.sh #3383

Merged

richardlau added a commit that referenced this issue Jun 13, 2023

www: use node:lts-bullseye in build-site.sh (#3383)

55a899f

Temporarily switch to `node:lts-bullseye` while we investigate why the website build is broken with `node:lts`/`node:lts-bookworm`. Refs: #3382

richardlau added a commit that referenced this issue Jun 13, 2023

www: use node:lts-bullseye in build-site.sh

9db07d3

Temporarily switch to `node:lts-bullseye` while we investigate why the website build is broken with `node:lts`/`node:lts-bookworm`. Refs: #3382

richardlau mentioned this issue Jun 13, 2023

www: use node:lts-bullseye in build-site.sh #3384

Merged

targos mentioned this issue Jun 14, 2023

Avoid upgrading base OS image during the life of a major version of Node.js nodejs/docker-node#1918

Open

richardlau mentioned this issue Jun 15, 2023

Node 20.3 Crashes all the time when executed inside docker nodejs/node#48444

Closed

richardlau added a commit that referenced this issue Jun 29, 2023

www: use node:lts-bullseye in build-site.sh (#3384)

73f466e

Temporarily switch to `node:lts-bullseye` while we investigate why the website build is broken with `node:lts`/`node:lts-bookworm`. Refs: #3383 Refs: #3382

mayeulk mentioned this issue Jul 25, 2023

Use Debian Bookworm fititnt/uwazi-docker#48

Open

MoLow closed this as completed Feb 25, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

website build broken during Docker image preparation #3382

website build broken during Docker image preparation #3382

richardlau commented Jun 13, 2023 •

edited

Loading

richardlau commented Jun 13, 2023

mhdawson commented Jun 13, 2023

mhdawson commented Jun 13, 2023

richardlau commented Jun 13, 2023

targos commented Jun 13, 2023

richardlau commented Jun 13, 2023

richardlau commented Jun 15, 2023 •

edited

Loading

Daniel15 commented Jun 22, 2023

targos commented Feb 24, 2024

MoLow commented Feb 25, 2024

website build broken during Docker image preparation #3382

website build broken during Docker image preparation #3382

Comments

richardlau commented Jun 13, 2023 • edited Loading

richardlau commented Jun 13, 2023

mhdawson commented Jun 13, 2023

mhdawson commented Jun 13, 2023

richardlau commented Jun 13, 2023

targos commented Jun 13, 2023

richardlau commented Jun 13, 2023

richardlau commented Jun 15, 2023 • edited Loading

Daniel15 commented Jun 22, 2023

targos commented Feb 24, 2024

MoLow commented Feb 25, 2024

richardlau commented Jun 13, 2023 •

edited

Loading

richardlau commented Jun 15, 2023 •

edited

Loading