java.io.IOException: Cannot run program "git" on test-equinix-ubuntu2204-x64-1 / test-equinix-ubuntu2204-x64-2

[richardlau]

Recent builds on test-equinix-ubuntu2204-x64-1 are failing with
e.g. https://ci.nodejs.org/job/node-test-commit/68784/console

08:00:01 Caused by: java.io.IOException: Cannot run program "git" (in directory "/home/iojs/build/workspace/node-test-commit"): error=0, Failed to exec spawn helper: pid: 3572706, signal: 11

I'll try restarting the agent.

[richardlau]

Looks like test-equinix-ubuntu2204-x64-2 is also affected, so I'll restart the agent there as well.
test-ibm-ubuntu1804-x64-1 seems to be okay, but has not run as many builds in the affected timeframe.

[richardlau]

Agents on test-equinix-ubuntu2204-x64-1 and test-equinix-ubuntu2204-x64-2 have been updated to the current version and restarted.

https://ci.nodejs.org/job/node-test-commit/68791/console looks like it was able to run git successfully and has started the child jobs.

[targos]

Was it caused by the Jenkins upgrade? Is there a way to know what's the oldest agent version that can properly work with the running Jenkins instance?

[richardlau]

I don't know.

I've just seen test-digitalocean-ubuntu2204-x64-2 has also failed for a similar reason.

I'll check other machines after lunch.

[richardlau]

FWIW I updated the agent version because it seemed a reasonable thing to do since the machines were not working anyway. I had nothing that indicated that it was the agent version at fault.

[richardlau]

Updated agents and restarted on:

• test-digitalocean-ubuntu2204-x64-1
• test-digitalocean-ubuntu2204-x64-2

[richardlau]

ci-release was also affected:

• https://ci-release.nodejs.org/job/iojs+release/10003/console

I'm wondering if this was caused by a Java update. From /var/log/dpkg.log on ci-release:

2024-02-28 06:23:55 startup archives unpack
2024-02-28 06:23:56 upgrade libxml2:amd64 2.9.10+dfsg-5ubuntu0.20.04.6 2.9.10+dfsg-5ubuntu0.20.04.7
2024-02-28 06:23:56 status triggers-pending libc-bin:amd64 2.31-0ubuntu9.14
2024-02-28 06:23:56 status half-configured libxml2:amd64 2.9.10+dfsg-5ubuntu0.20.04.6
2024-02-28 06:23:56 status unpacked libxml2:amd64 2.9.10+dfsg-5ubuntu0.20.04.6
2024-02-28 06:23:56 status half-installed libxml2:amd64 2.9.10+dfsg-5ubuntu0.20.04.6
2024-02-28 06:23:56 status unpacked libxml2:amd64 2.9.10+dfsg-5ubuntu0.20.04.7
2024-02-28 06:23:56 startup packages configure
2024-02-28 06:23:56 configure libxml2:amd64 2.9.10+dfsg-5ubuntu0.20.04.7 <none>
2024-02-28 06:23:56 status unpacked libxml2:amd64 2.9.10+dfsg-5ubuntu0.20.04.7
2024-02-28 06:23:56 status half-configured libxml2:amd64 2.9.10+dfsg-5ubuntu0.20.04.7
2024-02-28 06:23:56 status installed libxml2:amd64 2.9.10+dfsg-5ubuntu0.20.04.7
2024-02-28 06:23:56 trigproc libc-bin:amd64 2.31-0ubuntu9.14 <none>
2024-02-28 06:23:56 status half-configured libc-bin:amd64 2.31-0ubuntu9.14
2024-02-28 06:23:56 status installed libc-bin:amd64 2.31-0ubuntu9.14
2024-02-28 06:23:59 startup archives unpack
2024-02-28 06:23:59 upgrade openjdk-17-jre-headless:amd64 17.0.9+9-1~20.04 17.0.10+7-1~20.04.1
2024-02-28 06:23:59 status half-configured openjdk-17-jre-headless:amd64 17.0.9+9-1~20.04
2024-02-28 06:23:59 status unpacked openjdk-17-jre-headless:amd64 17.0.9+9-1~20.04
2024-02-28 06:23:59 status half-installed openjdk-17-jre-headless:amd64 17.0.9+9-1~20.04
2024-02-28 06:24:04 status unpacked openjdk-17-jre-headless:amd64 17.0.10+7-1~20.04.1
2024-02-28 06:24:04 startup packages configure
2024-02-28 06:24:04 configure openjdk-17-jre-headless:amd64 17.0.10+7-1~20.04.1 <none>
2024-02-28 06:24:04 status unpacked openjdk-17-jre-headless:amd64 17.0.10+7-1~20.04.1
2024-02-28 06:24:04 status half-configured openjdk-17-jre-headless:amd64 17.0.10+7-1~20.04.1
2024-02-28 06:24:04 status installed openjdk-17-jre-headless:amd64 17.0.10+7-1~20.04.1

The earlier build at 6:00am went through successfully.

[richardlau]

So far it looks like it's limited to Ubuntu 22.04. The machines I checked that had the issue did update the jre package just after 6am today.