... More Solr improvements

[nickumia-reisys]

These changes cover our most recent, most successful test case for SolrCloud which implements the following changes,

• Zookeeper timeout options
  • Theory: this is the timeout between Solr and Zookeeper, if a Solr node times out with ZK, it gets out of sync and ZK can't communicate with it too well after that.
  • Action: Increase from 30,000 to 600,000. (Borrowed from the other team, no real derivation).
  • Allow more tolerance for ZK latency GSA-TTS/datagov-brokerpak-solr#35
  • Update solrcloud brokerpak v1.0.7 to v1.0.8 datagov-ssb#138
• Solr Java Memory/Garbage Collection tuning
  • Theory: If there is a very expensive operation happening in Solr that causes it's memory to spike/crash alongside with garbage collection cleanup, Solr goes down and if CKAN keeps trying to do that with all of the nodes and then there's no leader because of this, it causes the collection to crash
  • Action: Increase Memory from 14GB to 32GB (Note: I didn't do anything with changing the type of garbage collection because it seems like the most optimal GC is implemented already)
  • Implemented in this PR
• Latency within AWS Availability Zones that causes sync issues between Solr/Zookeeper
  • Theory: Solr and ZK require a very very low latency connection to stay in sync and for ZK to do proper node management. The guidance is to not run ZK across multiple AZs.
  • Action: Deploy managed node group to just one availability zone
  • Single AZ Support GSA-TTS/datagov-brokerpak-eks#93
• Solr Performance Specs for our Larger Document Sizes
  • Theory: Based on this guidance, there's different values for autoCommit and autoSoftCommit based on Larger numbers of small documents vs. Very Large Documents.
  • Action: I decreased this to help keep Java Memory utilization low for our larger documents.
  • Implemented in this PR
• Disable Solr Restarts
  • Theory: Performing a Solr restart while index writes are occurring causes problems when the nodes start coming back up again. Read queries are fine to be occurring during restarts. It's just if index writes aren't coordinated with ZK properly during leader transitions or transaction logging, it causes the collection to become unresponsive.
  • Action: "Disable" it by setting the next restart to be a very long time in the future (Jan 1st, 2027).
  • Implemented in this PR
  • Despite the following documentation, Solr does not handle restart recoveries for our data well,
    • https://solr.apache.org/guide/8_11/solrcloud-recoveries-and-write-tolerance.html#write-side-fault-tolerance
    • https://apache.github.io/solr-operator/docs/solr-cloud/solr-cloud-crd.html#update-strategy
    • https://apache.github.io/solr-operator/docs/solr-cloud/managed-updates.html

Things that we should do (but are not covered in this PR):

• Solr Performance with NewRelic
  • We've talked about this before, but if we really want to inspect Solr, we'll probably have to do the work to get this set up.

[nickumia-reisys]

Also note that we need an EC2 with proper CPU/RAM to support this. The current EC2 type is c5.9xlarge