Renew OpenSearch reserved instance leases

[hannes-ucsc]

Apparently it is not possible to purchase a lease and pick the starting date so we need to make the purchase on the day of expiration of the existing lease.

dev December 20, 2024, 03:30 pm
anvildev December 20, 2024, 03:26 pm
prod January 01, 2025, 09:59 am
anvilprod December 20, 2024, 03:06 pm

The dev lease should be reduced to four nodes (from six). All leases should be changed to the r7gd.xlarge.search instance type (from r6gd.xlarge.search).

• Prepare PR for the instance type and node count changes for all lower deployments
• Prepare hotfix PR for the instance type change for prod, anvilprod and hammerbox

The PR checklist should be amended to include the steps outlined below.

On December 20

• Push PR branch to GitLab dev but cancel deploy job of the sandbox pipeline
• sandbox pipeline is stopped at the deploy job
• Modify instance type for azul-index-sandbox manually (it should trigger a blue/green deployment)
• Blue/green deployment of azul-index-sandbox is complete and domain status is green
• Resume the sandbox pipeline
• sandbox pipeline is complete
• Merge the PR and push merge commit to GitLab anvildev but cancel deploy job of the anvilbox pipeline
• anvilbox pipeline is stopped at the deploy job
• Modify instance type for azul-index-anvilbox manually (it should trigger a blue/green deployment)
• Blue/green deployment of azul-index-anvilbox is complete and domain status is green
• Resume the anvilbox pipeline
• anvilbox pipeline is complete

December 21

• File backport PR for hotfix PR

TBC

[hannes-ucsc]

Spike to check if instance type and count change requires resource replacement in TF, using terraform plan or equivalent.