Running solr_wrapper a 2nd time with "persist" option set does not find the existing core

[afred]

Versions used:

• Using solr_wrapper 0.12.0
• Using Solr 6.4.1 (the version selected by solr_wrapper)

Expected Result:

I should be able to:

1. run solr_wrapper for a 1st time, with a configuration that has the persist flag set to true.
2. stop solr_wrapper
3. run solr_wrapper a 2nd time, with the same config, and not get an error.

Actual Results:

Running solr_wrapper a 2nd time, with the same config, gives an error (see below).

To Replicate:

1. create a config file named .solr_wrapper.yml containing the following:

   collection:
     dir: solr/config/
     name: my-core
     persist: true

2. Run solr_wrapper for the first time (without the core having been created yet) with the specified config, like this...

   solr_wrapper --config .solr_wrapper.yml
   

3. At this point, Solr starts.

4. Stop Solr using Ctrl+C

5. Run solr_wrapper again, with the same command and config file...

   solr_wrapper --config .solr_wrapper.yml
   

6. At this point, I get the following error...

   ERROR: 
   Core 'my-core' already exists!
   Checked core existence using Core API command:
   http://localhost:8983/solr/admin/cores?action=STATUS&core=my-core
   

Background:

• I've tracked this down to the fact that SolrWrapper::Client#core? is returning false when checking for the core named my-core.
• In my case, the core was created (during the first start of solr_wrapper) in /tmp/solr-6.4.1/server/solr/my-core, and after getting the error, I can still see the directory corresponding to my-core on the filesystem.
• The Solr api call being made to check for the core inside of SolrWrapper::Client#core? looks like this: admin/cores?action=STATUS&wt=json&core=my-core'
• The response returned by Solr is:

   {"responseHeader":{"status":0,"QTime":1},"initFailures":{},"status":{"my-core":{}}}

• This would indicate that SolrWrapper::Client#core? is actually behaving correctly and returning false, given the response from Solr. However, Solr itself is not finding the core that it created previously.
• This causes SolrWrapper::Client#exists? to return false, which in turn causes this conditional to also return false, which in turn runs the line exec('create', create_options).
• And it is the call to exec('create', create_options) where Solr is somehow able to find the core it couldn't find before, and throws the error for trying to create a core that already exists.

[jcoyne]

This is the same as #69

[afred]

But #69 doesn't refer to the persist option. If the persist option is set to false, and you try to create a core that already exists, then it's my understanding that an error should be raised. IOW, you need to be explicit when you want to re-use an existing core.

[jcoyne]

@afred I think that #69 was just less well described. I think the same fix will wipe out both issues.

[mark-dce]

It appears to me that this is a timing related issue.

I can in no way reliably trigger or eliminate the error using the gem as released. On a low-resource VM, the issue seems to be more prevalent, i.e. I get the "Core ... already exists" error multiple successive times before the solr starts as expected. See example runs where I did nothing but try launching the wrapper again in this gist https://gist.github.com/mark-dce/0a6bb38adde0d5b7645ed58778158575

If I add a status check at the beginning of the create method something like

def create(options = {})
      sleep 5 unless started?
      options[:name] ||= SecureRandom.hex
# etc.

the solr service appears to start completely reliably - see this gist of behavior after the change https://gist.github.com/mark-dce/7aaede62802adea79ab826a48af23ad8

[conorom]

As I buried at the end of a massive comment on the very-related #69, if this gem does undergo future development it might be best to find a better core existence check. This page has a couple of newer answers that indicate the accepted solution (as used in this gem) is not reliable and/or has been superseded in newer versions of Solr.

But perhaps it will just wane as more and more people develop in containers instead. I still use and appreciate it though! :)