Solr: load balancing, fault tolerance, and high availability

[pdurbin]

As of Dataverse 4.0 it is assumed that Solr is only running on a single host. In #1083 we added a configuration option so that Solr doesn't have to run on localhost but there are several problems with our current approach:

• The server running Solr is a single point of failure: When solr down, we should say Dataverse is down #1042
• Only having one server for indexing might be a performance bottleneck: Solr: Index all performance is too slow with full production data. #50

Ideally, this ticket would encompass the following deliverables:

• Fault tolerance: no more single point of failure, greater availability
• Load balancing: distribution of indexing and search load across multiple servers

From a quick look it seems like SolrCloud is what we want: https://cwiki.apache.org/confluence/display/solr/SolrCloud . It makes use of ZooKeeper, which we previously wrote briefly about at #1083 (comment) .

[pdurbin]

I just realized that http://lucene.apache.org/solr/quickstart.html is totally new for Solr 5 vs. Solr 4.

To launch Solr, run: bin/solr start -e cloud -noprompt

So maybe it will be easier to us SolrCloud as part of an upgrade from Solr 4.6.0 (#456). It seem like they don't guide you to use java -jar start.jar anymore.

[pdurbin]

@bionary just wrote to the mailing list with a nice writeup having do with SolrCloud, and I just copied and pasted his list of issues to think about into a Google Doc called Integrating Dataverse 4.x with SolrCloud.

[pdurbin]

As I wrote at #2985 (comment) I still feel bad for encouraging @bionary to create pull request #2985 and then being unable to follow up on it and shepard it through completion.

No one else seems to have expressed much interest in load balancing, fault tolerance, and high availability for Solr as it's used in Dataverse, so I'm closing this issue. Please get in touch if you're interested after all.