CentOS 7 is end of life as of June 30, 2024, need to upgrade OS on freezingsaddles.org

[obscurerichard]

CentOS 7 had its end of life on June 30, 2024. We need to upgrade to a newer, supported operating system, such as Rocky Linux 8 or 9, or maybe Amazon Linux 2023.

You usually need to install a fresh system and reinstall all the packages but it is really tempting in this case to try an upgrade path. Normally that's not supported but there is a path to do it using the AlmaLinux ELevate project and the Leapp utility. You have to do an upgrade between 7 and 8, and then to 9, with a supported RHEL-compatible distribution.

[obscurerichard]

To prepare a dry run for this, I wanted to try an OS upgrade procedure using did the following:

• Created a new KMS customer managed key unmanaged-freezing-backup-key that is multi-region enabled and replicated to us-west-2, ap-northeast-1, eu-central-1, and us-west-2, managed by FreezingSaddlesAdmin and gave it the right permissions for the AWS Backup service in the same region to use it.
• Deleted the Default backup vault using the AWS CLI since it is not configured to use our KMS key
• Configured a new AWS Backup vault to use the unmanaged-freezing-backup-key
• Granted AWSBackupDefaultServiceRole the right to pass role to the IAM role that the instance runs with, see here for the details
• Made an on-demand backup of the EC2 instance
• Restored the EC2 instance to a slightly larger instance type to facilitate easy upgrade

[obscurerichard]

I restored the backup to unmanaged-fs-ec2-restore-tmp, which worked smoothly.

Upgrading to Rocky Linux 8 with a dry run using leapp worked pretty well, only minor changes were needed to the docker-compose files. Monitoring the upgrade process with the EC2 serial console support worked really well.

The basic services came over OK. I added some host name variants to test with, such as tmp-www.freezingsaddles.org and tweaked the environment file so those hosts would resolve.

Next up: upgrade to Rocky Linux 9, after making another backup of the temporary instance with AWS Backup.

[obscurerichard]

To get from CentOS 8 to 9

• To get the system up to date tried to issue sudo dnf update -y --allowerasing as there was one recalcitrant package:

Error:
 Problem: cannot install both nagios-plugins-2.4.9-1.el8.x86_64 from epel and nagios-plugins-2.4.9-1.el7.x86_64 from @System
  - package nagios-plugins-game-2.4.9-1.el7.x86_64 from @System requires nagios-plugins = 2.4.9-1.el7, but none of the providers can be installed
  - cannot install the best update candidate for package nagios-plugins-2.4.9-1.el7.x86_64
  - problem with installed package nagios-plugins-game-2.4.9-1.el7.x86_64
(try to add '--allowerasing' to command line to replace conflicting packages or '--skip-broken' to skip uninstallable packages or '--nobest' to use not only best candidate packages)

But then the system upgrade bombed with errors about not finding some python2 packages in the preupgrade area...

That was a dead end so I discovered I had to edit /etc/dnf/dnf.conf and remove the excludes

exclude=
#exclude=python2-leapp,snactor,leapp-upgrade-el7toel8,leapp

Then I was able to do sudo dnf remove -y python2-leapp python3-leapp and proceed to doing sudo yum install -y leapp-upgrade leapp-data-rocky

It looks like a bunch of things are still installed with el7 packages, including Docker, Nagios, and more. I'm not so sure this is a good idea, and I can't seem to get a clean upgrade to Rocky Linux 9 with this path, so I think it might be time to abandon it for now...

[obscurerichard]

Instead of doing an upgrade, it's probably time to rebuild the system on a fresh Rocky Linux 9 system.

I subscribed to the Rocky Linux 9 AWS Marketplace offering in the Freezing Saddles AWS account to make this easier. I'll put the scripts in freezingsaddles/freezing-compose#36 for this.

I'm going to adapt the provisioning scripts from obscure-scripts and Docker-for-Developers to get the Rocky Linux 9 provisioned a bit more cleanly.

[obscurerichard]

It's done! The new server is provisioned

The old server running CentOs 7 had over 1000 days of uptime, it had not been rebooted since January of 2021:

reboot   system boot  3.10.0-1160.53.1 Fri Jan 28 02:03 - 03:48 (1032+01:45)

Most of the provisioning was done in a scripted way, but there are some fixups I made by hand, this bash history gives the gist of how it went down. I had made backups by hand on the old server and selectively restored them.

freezingsaddles-bash-history.txt

To finish this off I should probably fix up the Icinga monitoring - that's severed into freezingsaddles/freezing-compose#37