Kingfisher Process Docker app deploy

[jpmckinney]

closes #200, addresses #402, closes #427

Main:

• Remove configuration for Kingfisher Process v1
• @yolile will delete some of the DB users
  • If any others are removed from Pillar data after June 9, remember to run states to delete the SQL users
• Copy things over for RBC Qlik BI: the ocdskingfishercollect DB (to be renamed kingfisher_collect) and /home/collect/data directory (to be moved to /home/incremental/data). We would need to comment-out the cron jobs until those two have been moved.
• Once the kingfisher_collect database and /home/incremental/data directory are populated, change cron.absent to cron.present in incremental.sls
• @RobHooper to add network configuration, review bullets below, etc.
• Decide on set -euo pipefail for firewall_reset.sh Kingfisher Process Docker app deploy #418 (comment)

Notes:

• I haven't checked pgbackrest at all, e.g. postgres.backup in Pillar, salt/postgres/files/pgbackrest, etc. Do we need to change the pgbackrest configuration (kingfishser_common.sls) to avoid conflict with the old DB backups?
• I think we'll want to remove ocp04 from Salt, but not sure what we'll do about ocp05 (current replica), since we want to keep it up for up to a year at most, in case we want to access that database.
• I haven't checked replication at all.

---

• Add these steps to create_server.rst under "5. Migrate from the old server" once clarified whether needed

Once main server is online:

• Check uid of docker user patches uid in pillar/kingfisher_process.sls
• Update tinyproxy IPs in pillar/tinyproxy.sls
• Re-enable backups (restore configuration commented out in 9fca6f3 and 1332186 and in pillar/private/kingfisher_common.sls)
• Update firewall IPs in pillar/kingfisher_replica.sls

Once replica server is online:

• Update replica_ipv4 (and replica_ipv6) in pillar/kingfisher_main.sls

One either server is online:

• Run https://github.com/jfcoz/postgresqltuner to see if any changes necessary
• Double-check vm.nr_hugepages https://www.postgresql.org/docs/current/kernel-resources.html#LINUX-HUGE-PAGES
  • This is based on peak memory usage (VmPeak from /proc/123/status). That said, the new server has 64GB not 128GB RAM, and postgresqltuner recommends halving this number, so I did that.

[jpmckinney]

@RobHooper I've made the changes I wanted to make. There is a corresponding PR for private Pillar.

[jpmckinney]

If you upgrade to Salt 3006 for this work, please check whether {% include 'postgres/files/conf/shared.include' %} still works (I opened #432 for a possibly related issue).

[ghost]

what [to] do about ocp05 (current replica), since we want to keep it up for up to a year at most, in case we want to access that database.

To help reduce the risks with Ubuntu end of life we can install Ubuntu Pro on ocp05.
This would provide further security updates for most packages, in particularly core system software.
The only software this would not cover is Postgres which depends on their custom apt repo support.

https://ubuntu.com/pro
https://wiki.postgresql.org/wiki/Apt

This would be best for if the plan is for ocp05 to be an online "archived" database.

Alternatively, we can turn off the server but keep it available, then we are one server boot away from accessing the data.
This way we won't need a solution like Ubuntu Pro.

pgbackrest
We are best creating a new stanza in pgbackrest to avoid clashing with the old backups.
Especially since we are upgrading Postgres versions.

I can create the new stanza and update everywhere. kingfisher-2023 or kingfisherv15 (version number matching Postgres) perhaps, let me know if you have a preference.

[jpmckinney]

For Ubuntu Pro, the only server option seems to be "Physical servers with unlimited VMs" (we have no VMs).

Would we need to go with that one, or is "Desktop" the option to take?

I'm happy with either kingfisher-2023 or kingfisherv15. We can go with kingfisher-2023 :)

[ghost]

The "Ubuntu Pro Server with unlimited VMs" option is indeed the best option for us because it is a physical server.
We need Ubuntu Universe repository coverage included because we are using a number of packages installed from there.

It should cost $500 for the year.

[jpmckinney]

Okay, I'll get that approved, but let's go ahead with that as the plan.

[ghost]

Happy being led by you if we bring the replica online now or later.

We will loose the failover / disaster recovery that the replica server currently provides.
We have database backups but it is a slower process to recover with them.
The replica server is also significantly quicker to failover to in the case of hardware failures (i.e. the worst-case scenario).

I am also conscious that the new server is smaller so this is another factor to consider around load.

If we are not starting the replica now I will configure pgbackrest before this is merged.

[jpmckinney]

Let's start without the replica.

[jpmckinney]

I added some comments to the file to help check whether this grep still matches text.

I think the 1 needs to be changed to \d+ (in which case grep needs a -E option)

I couldn't find what outputs We encountered an internal error\. Please try again\.

Is the \| meant to match a literal pipe character? (I'm not sure how pgbackrest concatenates messages).

[ghost]

\| is an OR operator for grep, we are using this to ignore multiple lines we have seen are false positives.

Checking our records, "expire command encountered" has only ever reported "1" error.
I would like to know when multiple files fail so \d+ would be too greedy from this point of view.

[jpmckinney]

Aha, sounds good!

[jpmckinney]

@RobHooper This PR is good to go. I added one comment, but it can be looked into separately.