Internal index unrecoverable when upgrade from 5.6.x fails to reindex #39339
Description
When following the steps mentioned in the upgrade guide,
if we disable the cluster shard allocation but fail to enable it after
upgrading the nodes and plugins, the next step of upgrading internal
indices fails. As we did not check the bulk response for reindexing,
we delete the old index assuming it has been reindexed successfully.
This is fatal as we cannot recover from this state since index to be upgraded
has been deleted.
Steps to follow:-
Following the upgrade guide:
https://www.elastic.co/guide/en/elastic-stack/6.6/upgrading-elastic-stack.html
- Before the upgrade, we disable cluster shard allocation (Step 1)
- And after the upgrade of the node(s) to 6.6, plugins upgrade we need to enable the cluster shard allocation (Step 8 )
- And then run the upgrade internal indices step if you are upgrading from 5.6 (https://www.elastic.co/guide/en/elastic-stack/6.6/upgrading-elastic-stack.html#upgrade-internal-indices)
If we miss the step of enabling cluster shard allocation, during reindex operation UnavailableShardsException is thrown.
But as we do not check the BulkByScrollResponse if there were any failures and proceed as a successful operation thereby deleting the internal index.
Solution:-
There should be a pre-upgrade check before proceeding with the upgrade. Also, in the case where reindex fails for some reason, it should not delete the original index.