[RW Separation] Updates to restore from snapshot with added search replicas

[mch2]

With #15368 we add the ability to create and update an index to have search replicas. We need to also support this configuration on restore. With this issue we should accomplish this while honoring the same validations for search replicas:

1. Search replicas allowed with Segrep only
2. Auto expand replica count not allowed
3. Feature flag enforced

[vinaykpud]

Hi @mch2
I would like to pickup this issue

[vinaykpud]

Reproduced this issue and found that:

1. Search replicas not getting restored from the snapshot
2. We need to add validations while restoring the snapshot with replication type SEGMENT, so that search replicas do not get created when replication type is DOCUMENT.

[vinaykpud]

Extending above mentioned validation adding a detailed validations bellow and marking which needs to be allowed and and which needs to be blocked.

Scenario 1:

Snapshot created with index.replication.type: DOCUMENT with no search replica (note: Search replicas are allowed only in SEGMENT replication type), can be restored with bellow settings:
Case 1. No explicit index.replication.type and index.number_of_search_only_replicas in the settings: Allow
Case 2. No explicit index.replication.type and index.number_of_search_only_replicas = 0: Allow
Case 3. index.replication.type: DOCUMENT and no explicit index.number_of_search_only_replicas : Allow
Case 4. index.replication.type: DOCUMENT and index.number_of_search_only_replicas = 0 : Allow
Case 5. index.replication.type: DOCUMENT and index.number_of_search_only_replicas > 0 : Block
Case 6. index.replication.type: SEGMENT and no explicit index.number_of_search_only_replicas : Allow
Case 7. index.replication.type: SEGMENT and index.number_of_search_only_replicas = 0 : Allow
Case 8. index.replication.type: SEGMENT and index.number_of_search_only_replicas > 0 : Allow

Scenario 2:

Snapshot created with index.replication.type: SEGMENT with no search replica.
Case 1. No explicit index.replication.type and index.number_of_search_only_replicas in the settings: Allow
Case 2. No explicit index.replication.type and index.number_of_search_only_replicas = 0: Allow
Case 3. index.replication.type: DOCUMENT and no explicit index.number_of_search_only_replicas : Allow
Case 4. index.replication.type: DOCUMENT and index.number_of_search_only_replicas = 0 : Allow
Case 5. index.replication.type: DOCUMENT and index.number_of_search_only_replicas > 0 : Block
Case 6. index.replication.type: SEGMENT and no explicit index.number_of_search_only_replicas : Allow
Case 7. index.replication.type: SEGMENT and index.number_of_search_only_replicas = 0 : Allow
Case 8. index.replication.type: SEGMENT and index.number_of_search_only_replicas > 0 : Allow

Scenario 3:

Snapshot created with index.replication.type: SEGMENT with index.number_of_search_only_replicas > 0
Case 1. No explicit index.replication.type and index.number_of_search_only_replicas in the settings: Allow
Case 2. No explicit index.replication.type and index.number_of_search_only_replicas = 0: Allow
Case 3. index.replication.type: DOCUMENT and no explicit index.number_of_search_only_replicas : Block
Case 4. index.replication.type: DOCUMENT and index.number_of_search_only_replicas = 0 : Allow
Case 5. index.replication.type: DOCUMENT and index.number_of_search_only_replicas > 0 : Block
Case 6. index.replication.type: SEGMENT and no explicit index.number_of_search_only_replicas : Allow
Case 7. index.replication.type: SEGMENT and index.number_of_search_only_replicas = 0 : Allow
Case 8. index.replication.type: SEGMENT and index.number_of_search_only_replicas > 0 : Allow