Provisioning & Deployment Plan

[mekarpeles]

This is a planning epic for our Provisioning and Deployment project and doesn't require direct attention.
-- To be closed when all underlying issues are resolved.

Top-level Goals

1. Migrate ol-home to ol-home0 #4060 Deprecate & decommission ol-home in favor of ol-home0
   1. Migrate legacy ol-home services (cron jobs, solr-updater, import-bot) to run in Docker on ol-home0
   2. Improve docker-based environment/deployment #4382, Deployments should be triggered from ol-home0 #4318, Create cron jobs docker-compose service #4383, Scripts for production deployments #4395, Reboot Docker services on specified servers #4457 Automate Deployment (on ol-home0)
   3. Migrate ol-home to ol-home0 #4060 Final decomission steps for ol-home
2. Recomission dev.openlibrary.org like staging.openlibrary.org #4229 Reprovision dev.openlibrary.org like staging.openlibrary.org (py3)
3. Upgrade All Trusty Nodes to Ubuntu Latest #2036 Re-provision all necessary hosts with focal (ol-www1, ol-db[1,2], ol-mem[3-5], ol-backup0)
4. Replace ol-www1 with docker-based ol-www0 #4252 Move all necessary hosts to docker (ol-www1, maybe ol-mem[3-5])
5. remove jenkins from ojf #4363 Add jenkins to help automate & schedule deploys, branch staging, and solr rebuilding
6. Automate Deploy Notes #4724 Make our deploys create a detailed GitHub release using https://github.com/github-tools/github-release-notes

Steps

Many of these still require issues & breakdown

• Create importbot docker service #4324 Move ImportBot to ol-home0 --> make-a-service - Port ID?, Nagios
• Deploy new dockerized solr-updater (to prod)
• Scripts for production deployments #4395 Scripts for production deployments -- Automated (4h -> 15min)
• Reboot Docker services on specified servers #4457 Reboot Docker services on specified servers
• internetarchive/olsystem#110 Replace references of ol-home with ol-home0
• Create cron jobs docker-compose service #4383 Move cron jobs to ol-home0 (Make one service for all cron jobs -- Caution on the double run)
• Remove OL and Infogami volume mounts from production #4719 rm volume mounts (for deployments) -- needs issue
• Replace ol-www1 with docker-based ol-www0 #4252 ol-www1 epic (edge cases: logs, ssl-termination, sitemaps) @mekarpeles
• Replace ol-www1 with docker-based ol-www0 #4252 need to spin up a ol-www0 @mekarpeles
• deploy strategy needs to address ol-www0, solr updater, import bot, cron service + volume_mounts
• Migrate ol-home to ol-home0 #4060 eventual ol-home deprecation
  • remove ol-home proxypass to ol-home0 for infobase + update config
  • Investigate if we can remove stats_solr #4426 can we dump stats-solr and move from ol-home to ol-home0
  • can we reclaim 4gb of ram from ol-home (until we can decommission)
• ol-db[1,2] and ol-backup?

Stakeholders

@bfalling, @cdrini, @cclauss

[cclauss]

* dumps are generated at /1/var/tmp/dumps
/olsystem/bin/uploaditem.py:222: DeprecationWarning: This method will be removed in future versions.  Use 'parser.read_file()' instead.
  p.readfp(f)
uploading with parameters: {'item_identifier': 'ol_dump_2021-02-26', 'from_url': 'rsync://ol-home0.us.archive.org/var_1/tmp/dumps/ol_dump_2021-02-26', 'submitter': 'openlibrary@archive.org'}
  Submitted.
/olsystem/bin/uploaditem.py:222: DeprecationWarning: This method will be removed in future versions.  Use 'parser.read_file()' instead.
  p.readfp(f)
uploading with parameters: {'item_identifier': 'ol_cdump_2021-02-26', 'from_url': 'rsync://ol-home0.us.archive.org/var_1/tmp/dumps/ol_cdump_2021-02-26', 'submitter': 'openlibrary@archive.org'}
  Submitted.