Validating databases (splunkd validatedb) failed with code '-1' on 9.1.2

[yaroslav-nakonechnikov]

Hello,

we see, that on 9.1.2 only on indexer nodes there is an error in splunkd_stderr.log:
Validating databases (splunkd validatedb) failed with code '-1'

and it fails to start containers:

TASK [splunk_indexer : Convert Extrenal Cluster Master Name into Internal URI] ***
ok: [localhost]
Wednesday 13 December 2023  09:12:47 +0000 (0:00:00.091)       0:00:52.105 ****

TASK [splunk_indexer : Setup Peers with Associated Site] ***********************
changed: [localhost]
Wednesday 13 December 2023  09:12:48 +0000 (0:00:01.296)       0:00:53.402 ****

TASK [splunk_indexer : include_tasks] ******************************************
included: /opt/ansible/roles/splunk_indexer/tasks/../../../roles/splunk_common/tasks/check_for_required_restarts.yml for localhost
Wednesday 13 December 2023  09:12:48 +0000 (0:00:00.034)       0:00:53.437 ****

TASK [splunk_indexer : Check for required restarts] ****************************
changed: [localhost]
Wednesday 13 December 2023  09:12:49 +0000 (0:00:00.652)       0:00:54.090 ****
Wednesday 13 December 2023  09:12:49 +0000 (0:00:00.022)       0:00:54.112 ****

TASK [Check all instances for required restarts] *******************************
included: /opt/ansible/roles/splunk_common/tasks/check_for_required_restarts.yml for localhost
Wednesday 13 December 2023  09:12:49 +0000 (0:00:00.038)       0:00:54.151 ****

TASK [Check for required restarts] *********************************************
changed: [localhost]
Wednesday 13 December 2023  09:12:50 +0000 (0:00:00.609)       0:00:54.761 ****
Wednesday 13 December 2023  09:12:50 +0000 (0:00:00.000)       0:00:54.761 ****

RUNNING HANDLER [splunk_common : Restart the splunkd service] ******************
included: /opt/ansible/roles/splunk_common/tasks/../handlers/restart_splunk.yml for localhost
Wednesday 13 December 2023  09:12:50 +0000 (0:00:00.029)       0:00:54.791 ****
FAILED - RETRYING: Restart the splunkd service - Via CLI (60 retries left).
FAILED - RETRYING: Restart the splunkd service - Via CLI (59 retries left).
FAILED - RETRYING: Restart the splunkd service - Via CLI (58 retries left).

it was reported to splunk in case 3366553

[yaroslav-nakonechnikov]

still exist in 9.1.3

[yaroslav-nakonechnikov]

looks like solved in 9.2