Ubuntu Bionic rollout to solr host(s)

[adborden]

User Story

As an operator, I need new Ubuntu Bionic hosts provisioned in all of our environments so that I can continue receiving security updates automatically from Ubuntu.

Part of #571

Ubuntu 14.04 support ends April 2019. We should move to Ubuntu 16.04 or 18.04. The latest 18.04 is preferred, but we should consider what kind of support exists for it.

Details

Hosts are provisioned in these environments and play books are successfully applied.

1. bionic sandbox
2. staging
3. production

To rebuild the solr index:

1. configure a single harvest instance to use solr, disable any cron jobs
2. rebuild the index ckan seach-index rebuild_fast
3. update catalog inventory and redeploy so all catalog instances are using the new solr
4. do the same for inventory

Acceptance Criteria

• Bionic hosts are provisioned and operating correctly
• Bionic hosts reporting to our monitoring systems
• Solr index is rebuilt
• Old Trusty hosts are decommissioned and removed from Ansible inventory
• New Relic solr alarm is updated for the new process name

Definition of Done

The above acceptance criteria must be met.

[adborden]

BSP ticket opened, waiting for them to provision hosts.

[adborden]

Hosts were provisioned, but with incorrect EBS volumes configured. Waiting for them to be re-provisioned.

[adborden]

EBS volumes have been attached to the instance and I initialized them. I've added steps for rebuilding the solr index and I'm currently testing them on staging.

[adborden]

Waiting on SecOps to provide a trendmicro package for Bionic.

[adborden]

Trendmicro issue is resolved

I noticed that the solr replicas aren't sized correctly. They are t2.medium but should be m4.xlarge.

[adborden]

Inventory index is rebuilt in staging and production. Inventory hosts are now running against these new solr instances 🎉

[adborden]

For catalog, currently rebuilding the index in staging. For production, harvester and web hosts cannot connect over port 8983, looks like the original firewall change request didn't go through. I opened a ticket with SecOps.

[adborden]

Updated the NewRelic alert to look for java processes.