Errors running Ansible across current cluster #2531

rvagg · 2021-01-28T02:47:25Z

I decided to just run our ansible worker create script across our cluster today in light of the sudo security flaw that was announced. In the process I've encountered a bunch of errors that we should probably look in to. Recording here as a list so we can tick them off. I'd appreciate help in dealing with these.

I didn't run any updates on:

Windows - shouldn't need it, I think they all do auto updates and regular restarts anyway
IBM platforms (including ibmi, aix, rhel-s390x, zos) I'll let @nodejs/build IBM folks deal with that
macOS - @AshCripps would you mind doing these? I'm a little afraid to just run the scripts against the existing infra.

Errors:

Failed 'not secret' (Jenkins secret not in secrets/inventory.yml)

Unreachable

Failed on 'baselayout : run ccache installer'

~~test-joyent-smartos15-x64-3~~ (gone: Replacing machines from Joyent data center #2552)
~~test-joyent-smartos15-x64-4~~ (gone: Replacing machines from Joyent data center #2552)
~~test-joyent-smartos16-x64-3~~ (gone: Replacing machines from Joyent data center #2552)
~~test-joyent-smartos16-x64-4~~ (gone: Replacing machines from Joyent data center #2552)
test-joyent-smartos17-x64-3 "baselayout : run ccache installer" fails running ansible against LinuxONE/smartos #2533
test-joyent-smartos17-x64-4 "baselayout : run ccache installer" fails running ansible against LinuxONE/smartos #2533
~~release-joyent-smartos15-x64-2~~ (gone: Replacing machines from Joyent data center #2552)
release-joyent-smartos17-x64-2 "baselayout : run ccache installer" fails running ansible against LinuxONE/smartos #2533

Failed update

test-digitalocean-debian8-x64-1 (jessie-backports 404)
test-softlayer-centos7-x64-1 (?) ansible: fix centos7 x64 ansible errors #2651
test-digitalocean-freebsd11-x64-2 ("pkg: repository meta /var/db/pkg/FreeBSD.meta has wrong version 2") Errors running Ansible across current cluster #2531 (comment)
test-nearform_intel-ubuntu1604-x64-1 Errors running Ansible across current cluster #2531 (comment)

Failed 'bootstrap : install libselinux-python bindings'

test-digitalocean-fedora30-x64-1 ansible: fix multiple register vars on Fedora #2654
test-digitalocean-fedora30-x64-2 ansible: fix multiple register vars on Fedora #2654
~~release-softlayer-centos6-x64-1~~ removed ansible: add RHEL 8 x64 instances #2886
~~test-softlayer-centos6-x64-1~~ removed ansible: add RHEL 8 x64 instances #2886
~~test-softlayer-centos6-x64-2~~ removed ansible: add RHEL 8 x64 instances #2886

Failed 'set hostname'

(We should remove these entirely)

test-softlayer-ubuntu1404-x64-1 (remove?)
test-softlayer-ubuntu1404-x86-1 (remove?)

Failed 'baselayout : centos7 | install ius'

"Name or service not known", is IUS still a thing?

test-rackspace-centos7-x64-1 ansible: fix centos7 x64 ansible errors #2651
release-digitalocean-centos7-x64-1 ansible: fix centos7 x64 ansible errors #2651

Failed 'jenkins-worker : install tap2junit'

test-digitalocean-freebsd11-x64-1 ansible: fixes for freebsd #2652

An exception occurred during task execution. To see the full traceback, use -vvv. The error was: ModuleNotFoundError: No module named 'pkg_resources'
{"changed": false, "msg": "Failed to import the required Python library (setuptools) on test-digitalocean-freebsd11-x64-1's Python /usr/local/bin/python. Please read module documentation and install in the appropriate location. If the required library is installed, but Ansible is using the wrong Python interpreter, please consult the documentation on ansible_python_interpreter"}

Failed uninstall `node`

test-digitalocean-ubuntu1604-x86-1 Errors running Ansible across current cluster #2531 (comment)

FAILED! => {"changed": false, "cmd": "node -v", "delta": "0:00:01.462308", "end": "2021-01-27 20:56:35.373427", "failed_when_result": true, "rc": 0, "start": "2021-01-27 20:56:33.911119", "stderr": "", "stderr_lines": [], "stdout": "v9.11.2", "stdout_lines": ["v9.11.2"]}

No "containers" config

(these probably need to be removed, they were part of my next-gen containerisation experiment iirc)

test-digitalocean-ubuntu2004_docker-x64-1
test-digitalocean-ubuntu2004_docker-x64-2

The text was updated successfully, but these errors were encountered:

AshCripps · 2021-01-29T11:41:57Z

Running into some ansible issues myself currently with this error

* Failed to parse /Users/ash/github/build/ansible/plugins/inventory/nodejs_yaml.py with ini plugin: /Users/ash/github/build/ansible/plugins/inventory/nodejs_yaml.py:25: Expected key=value host variable assignment, got: __future__

But once i get it working ill get on to ansibling the macs.

richardlau · 2021-01-29T11:53:31Z

FTR IBM i and z/OS do not have sudo available/installed.

richardlau · 2021-01-29T17:04:20Z

Ran the ansible scripts against our LinuxONE (rhel-s390x) hosts and they run into an error in the baselayout : run ccache installer task. I've opened #2533 to cover separately. The scripts did get past the package update task so sudo has been updated on all of the LinuxONE hosts (to sudo-1.8.23-10.el7_9.1.s390x)

richardlau · 2021-02-02T14:11:11Z

I've updated the AIX hosts. All IBM platforms that have sudo have been updated to patch the advisory.

AshCripps · 2021-02-16T15:45:56Z

Ive run against all the macs only error I faced is this on release-nearform-macos10.15-x64-1 which im currently investigating:

TASK [jenkins-worker : download slave.jar] ********************************************************************************************************************************************************************************************
task path: /Users/ash/github/build/ansible/roles/jenkins-worker/tasks/main.yml:165
fatal: [release-nearform-macos10.15-x64-1]: FAILED! => {"changed": false, "dest": "/Users/iojs/slave.jar", "elapsed": 0, "gid": 0, "group": "wheel", "mode": "0644", "msg": "Request failed: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1056)>", "owner": "root", "size": 877037, "state": "file", "uid": 0, "url": "https://ci.nodejs.org/jnlpJars/slave.jar"}

richardlau · 2021-02-19T11:32:22Z

I'm looking at test-nearform_intel-ubuntu1604-x64-1.

Manually running `apt get update` fails with an expired gpg key:

# apt-get update
Hit:1 http://ppa.launchpad.net/ubuntu-toolchain-r/test/ubuntu xenial InRelease
Get:2 http://security.ubuntu.com/ubuntu xenial-security InRelease [109 kB]
Hit:3 http://ie.archive.ubuntu.com/ubuntu xenial InRelease
Hit:4 http://ie.archive.ubuntu.com/ubuntu xenial-updates InRelease
Hit:5 http://ie.archive.ubuntu.com/ubuntu xenial-backports InRelease
Get:6 https://ftp.heanet.ie/mirrors/cran.r-project.org/bin/linux/ubuntu xenial/ InRelease [3,607 B]
Err:6 https://ftp.heanet.ie/mirrors/cran.r-project.org/bin/linux/ubuntu xenial/ InRelease
  The following signatures were invalid: KEYEXPIRED 1602869253  KEYEXPIRED 1602869253  KEYEXPIRED 1602869253
Fetched 113 kB in 0s (124 kB/s)
Reading package lists... Done
W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: https://ftp.heanet.ie/mirrors/cran.r-project.org/bin/linux/ubuntu xenial/ InRelease: The following signatures were invalid: KEYEXPIRED 1602869253  KEYEXPIRED 1602869253  KEYEXPIRED 1602869253
W: Failed to fetch https://ftp.heanet.ie/mirrors/cran.r-project.org/bin/linux/ubuntu/xenial/InRelease  The following signatures were invalid: KEYEXPIRED 1602869253  KEYEXPIRED 1602869253  KEYEXPIRED 1602869253
W: Some index files failed to download. They have been ignored, or old ones used instead.
#

richardlau · 2021-02-19T11:56:11Z

I'm looking at test-nearform_intel-ubuntu1604-x64-1.

Manually running apt get update fails with an expired gpg key:

root@test-nearform--intel-ubuntu1604-x64-1:~# apt-key list | grep "expired: "
pub   2048R/E084DAB9 2010-10-19 [expired: 2020-10-16]
root@test-nearform--intel-ubuntu1604-x64-1:~# apt-key adv --keyserver keyserver.ubuntu.com --refresh-keys
Executing: /tmp/tmp.mckX5xKLZr/gpg.1.sh --keyserver
keyserver.ubuntu.com
--refresh-keys
gpg: refreshing 7 keys from hkp://keyserver.ubuntu.com
gpg: requesting key 437D05B5 from hkp server keyserver.ubuntu.com
gpg: requesting key C0B21F32 from hkp server keyserver.ubuntu.com
gpg: requesting key EFE21092 from hkp server keyserver.ubuntu.com
gpg: requesting key FBB75451 from hkp server keyserver.ubuntu.com
gpg: requesting key E084DAB9 from hkp server keyserver.ubuntu.com
gpg: requesting key AF4F7421 from hkp server keyserver.ubuntu.com
gpg: requesting key BA9EF27F from hkp server keyserver.ubuntu.com
gpg: key 437D05B5: "Ubuntu Archive Automatic Signing Key <ftpmaster@ubuntu.com>" 43 new signatures
gpg: key C0B21F32: "Ubuntu Archive Automatic Signing Key (2012) <ftpmaster@ubuntu.com>" 17 new signatures
gpg: key EFE21092: "Ubuntu CD Image Automatic Signing Key (2012) <cdimage@ubuntu.com>" 65 new signatures
gpg: key FBB75451: "Ubuntu CD Image Automatic Signing Key <cdimage@ubuntu.com>" 93 new signatures
gpg: key E084DAB9: "Michael Rutter <marutter@gmail.com>" 2 new signatures
gpg: key AF4F7421: "Sylvestre Ledru - Debian LLVM packages <sylvestre@debian.org>" 1 new signature
gpg: key BA9EF27F: "Launchpad Toolchain builds" not changed
gpg: Total number processed: 7
gpg:              unchanged: 1
gpg:         new signatures: 221
root@test-nearform--intel-ubuntu1604-x64-1:~# apt-key list | grep "expired: "
root@test-nearform--intel-ubuntu1604-x64-1:~#

Refreshing keys has allowed the create worker ansible playbook to successfully run on test-nearform_intel-ubuntu1604-x64-1.

richardlau · 2021-03-05T12:59:14Z

Removed the smartos 15/16 hosts from the list in the description as they've been removed: #2552

richardlau · 2021-03-17T19:42:50Z

Failed update

test-digitalocean-freebsd11-x64-2 ("pkg: repository meta /var/db/pkg/FreeBSD.meta has wrong version 2")

Fixed by running

$ pkg --version
1.10.5
$ sudo pkg bootstrap -f
The package management tool is not yet installed on your system.
Do you want to fetch and install it now? [y/N]: y
Bootstrapping pkg from pkg+http://pkg.FreeBSD.org/FreeBSD:11:amd64/latest, please wait...
Verifying signature with trusted certificate pkg.freebsd.org.2013102301... done
Installing pkg-1.16.3...
Newer FreeBSD version for package pkg:
To ignore this error set IGNORE_OSVERSION=yes
- package: 1104001
- running kernel: 1102000
Ignore the mismatch and continue? [y/N]:
package pkg is already installed, forced install
Extracting pkg-1.16.3: 100%
$ pkg --version
1.16.3
$

1.16.3 matches the version of pkg on test-digitalocean-freebsd11-x64-1.

Running the playbook on test-digitalocean-freebsd11-x64-2 now errors at the same point as test-digitalocean-freebsd11-x64-1

An exception occurred during task execution. To see the full traceback, use -vvv. The error was: ModuleNotFoundError: No module named 'pkg_resources'
fatal: [test-digitalocean-freebsd11-x64-2]: FAILED! => {"changed": false, "msg": "Failed to import the required Python library (setuptools) on test-digitalocean-freebsd11-x64-2's Python /usr/local/bin/python. Please read the module documentation and install it in the appropriate location. If the required library is installed, but Ansible is using the wrong Python interpreter, please consult the documentation on ansible_python_interpreter"}

richardlau · 2021-03-22T15:04:19Z

Failed 'baselayout : centos7 | install ius'

"Name or service not known", is IUS still a thing?

test-rackspace-centos7-x64-1

release-digitalocean-centos7-x64-1

build/ansible/roles/baselayout/tasks/partials/repo/centos7.yml

Lines 8 to 12 in 722fdc1

    
           - name: centos7 | install ius 
        
             when: "arch != 'arm64' and arch != 'ppc64'" 
        
             yum: 
        
               name: "https://centos{{ ansible_distribution_major_version }}.iuscommunity.org/ius-release.rpm" 
        
               state: present

I'm not sure what we're installing from there, but the "centos*.iuscommunity.org" URLs were retired in 2020: iusrepo/announce#18

richardlau · 2021-04-13T16:34:56Z

Re. "Failed 'not secret' (Jenkins secret not in secrets/inventory.yml)" -- Having double checked we no longer have them in neither public nor release Jenkins, I've removed all the centos5 entries from secrets/inventory.yml in addition to the ones marked "should remove", and added the missing secrets for the others in the list.

Edit: Centos 5 was removed from Jenkins in #1984

Removed: release-requireio-osx1010-x64-1 release-scaleway-ubuntu1604-armv7l-1 release-scaleway-ubuntu1604-armv7l-2 test-requireio_rvagg-ubuntu1404-arm64_odroidxu-1 test-requireio_rvagg-ubuntu1404-arm64_odroidxu-2 test-requireio_rvagg-ubuntu1404-arm64_odroidxu3-1 test-requireio-osx1010-x64-1 Refs: #2531

Remove: test-mininodes-ubuntu1604-arm64_odroid_c2-1 test-mininodes-ubuntu1604-arm64_odroid_c2-2 test-mininodes-ubuntu1604-arm64_odroid_c2-3 test-scaleway-ubuntu1804-armv7l-1 test-scaleway-ubuntu1804-armv7l-2 test-scaleway-ubuntu1804-armv7l-3 Refs: #2531

richardlau · 2021-05-13T14:10:38Z

Failed uninstall `node`

test-digitalocean-ubuntu1604-x86-1

FAILED! => {"changed": false, "cmd": "node -v", "delta": "0:00:01.462308", "end": "2021-01-27 20:56:35.373427", "failed_when_result": true, "rc": 0, "start": "2021-01-27 20:56:33.911119", "stderr": "", "stderr_lines": [], "stdout": "v9.11.2", "stdout_lines": ["v9.11.2"]}

This was a snap install of Node.js.

root@test-digitalocean-ubuntu1604-x86-1:~# which node
/snap/bin/node
root@test-digitalocean-ubuntu1604-x86-1:~#

Removed via snap remove node.

- Update `python` to point to `python2` (for consistency with other platforms in our CI; use `python3` for Python 3). - Use `py38-pip` to match the version of Python 3 installed. - Create a `pip3` symlink as Ansible's `pip` task fails without it. - `-slaveLog` Jenkins agent parameter has been obsoleted. Use the `-o` parameter to `daemon` to redirect stdout/stderr to the log file. Refs: #2531

Tasks registering variables in Ansible will always set the variable regardless of any `when` clause(s). This means if more than one task registers the same variable, the last one "wins". Fix the bootstrap role on Fedora by correctly registering `has_libselinux` in both `fedora30` and non-`fedora30` cases. Refs: #2531

github-actions · 2022-03-11T00:15:25Z

This issue is stale because it has been open many days with no activity. It will be closed soon unless the stale label is removed or a comment is made.

This was referenced Feb 3, 2021

AIX libuv build fails with "Member libiconv.so.2 is not found in archive" #2534

Closed

AIX builds fail with "curl: not found" #2536

Closed

richardlau mentioned this issue Feb 19, 2021

coverage stats have plummeted nodejs/node#37368

Closed

richardlau mentioned this issue Mar 12, 2021

"baselayout : run ccache installer" fails running ansible against LinuxONE/smartos #2533

Closed

richardlau mentioned this issue Mar 23, 2021

Ansible errors on centos7-x64 #2587

Closed

richardlau mentioned this issue Apr 9, 2021

ansible: add libcap-dev package to Linux distros #2607

Closed

richardlau mentioned this issue Apr 13, 2021

ansible: remove obsolete entries #2622

Merged

This was referenced Apr 21, 2021

ansible: add extra M1 macs #2628

Merged

ansible: remove obsolete hosts #2632

Merged

This was referenced May 13, 2021

ansible: fixes for freebsd #2652

Merged

ansible: fix multiple register vars on Fedora #2654

Merged

github-actions bot added the stale label Mar 11, 2022

github-actions bot closed this as completed Apr 11, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Errors running Ansible across current cluster #2531

Errors running Ansible across current cluster #2531

rvagg commented Jan 28, 2021 •

edited by richardlau

Loading

AshCripps commented Jan 29, 2021

richardlau commented Jan 29, 2021

richardlau commented Jan 29, 2021

richardlau commented Feb 2, 2021

AshCripps commented Feb 16, 2021

richardlau commented Feb 19, 2021

richardlau commented Feb 19, 2021 •

edited

Loading

richardlau commented Mar 5, 2021

richardlau commented Mar 17, 2021 •

edited

Loading

Failed update

richardlau commented Mar 22, 2021

Failed 'baselayout : centos7 | install ius'

richardlau commented Apr 13, 2021 •

edited

Loading

richardlau commented May 13, 2021

Failed uninstall `node`

github-actions bot commented Mar 11, 2022

Errors running Ansible across current cluster #2531

Errors running Ansible across current cluster #2531

Comments

rvagg commented Jan 28, 2021 • edited by richardlau Loading

Failed 'not secret' (Jenkins secret not in secrets/inventory.yml)

Unreachable

Failed on 'baselayout : run ccache installer'

Failed update

Failed 'bootstrap : install libselinux-python bindings'

Failed 'set hostname'

Failed 'baselayout : centos7 | install ius'

Failed 'jenkins-worker : install tap2junit'

Failed uninstall node

No "containers" config

AshCripps commented Jan 29, 2021

richardlau commented Jan 29, 2021

richardlau commented Jan 29, 2021

richardlau commented Feb 2, 2021

AshCripps commented Feb 16, 2021

richardlau commented Feb 19, 2021

richardlau commented Feb 19, 2021 • edited Loading

richardlau commented Mar 5, 2021

richardlau commented Mar 17, 2021 • edited Loading

Failed update

richardlau commented Mar 22, 2021

Failed 'baselayout : centos7 | install ius'

richardlau commented Apr 13, 2021 • edited Loading

richardlau commented May 13, 2021

Failed uninstall node

github-actions bot commented Mar 11, 2022

rvagg commented Jan 28, 2021 •

edited by richardlau

Loading

Failed uninstall `node`

richardlau commented Feb 19, 2021 •

edited

Loading

richardlau commented Mar 17, 2021 •

edited

Loading

richardlau commented Apr 13, 2021 •

edited

Loading

Failed uninstall `node`