Deleting an index during concurrent taking of more than one snapshot causes future restore to fail

[aYukiSekiguchi]

Describe the bug
Elasticsearch 7.10.2 which OpenSearch forked has a known issue about snapshot and restore:
https://www.elastic.co/guide/en/elasticsearch/reference/7.10/release-notes-7.10.2.html#known-issues-7.10.2

If an index is deleted while the cluster is concurrently taking more than one snapshot then there is a risk that one of the snapshots may never complete and also that some shard data may be lost from the repository, causing future restore operations to fail.

To Reproduce
Steps to reproduce the behavior:
I haven't reproduced the behavior, but I guess...

Delete index while the cluster is concurrently taking more than one snapshot.
There is a risk that one of the snapshots may never complete and also that some shard data may be lost from the repository, causing future restore operations to fail.

Expected behavior
Concurrent snapshot and restore work.

Plugins
I haven't reproduced the behavior, but I guess no plugin is needed.

Screenshots
None

Host/Environment (please complete the following information):
I haven't reproduced the behavior, but I guess all environment are affected.

Additional context
I checked the patch in Elasticsearch and the same file in OpenSearch main branch. It looks like OpenSearch has the same issue.

[Bukhtawar]

This is specific to Amazon OpenSearch Service which has been well documented here https://docs.aws.amazon.com/opensearch-service/latest/developerguide/supported-operations.html#version_7_10

[dblock]

Is there something we should/can do in OpenSearch itself? If not close this?

[reta]

@dblock the bare bone OpenSearch works just fine:

{
    "acknowledged": true,
    "persistent": {
        "snapshot": {
            "max_concurrent_operations": "1"
        }
    },
    "transient": {}
}

As per @Bukhtawar , it is specific to AWS OpenSearch offering: only limited settings are supported.

[aYukiSekiguchi]

I understand it is the problem of Amazon OpenSearch Service that the mitigation setting doesn't work.

However, my question is "Does OpenSearch have the know issue which Elasticsearch 7.10.2 has?"

To clarify, I quote the known issue from Elasticsearch 7.10.2 Release Note:

Snapshot and restore: If an index is deleted while the cluster is concurrently taking more than one snapshot then there is a risk that one of the snapshots may never complete and also that some shard data may be lost from the repository, causing future restore operations to fail.

...

This issue is fixed in Elasticsearch versions 7.13.1 and later. It is not possible to repair a repository once it is affected by this issue, so you must restore the repository from a backup, or clear the repository by executing DELETE _snapshot//*, or move to a fresh repository. For more details, see #73456.

[reta]

@aYukiSekiguchi OpenSearch is a fork of Elasticsearch as of 7.10.2, so it is very likely that the issue is still present

[dblock]

@aYukiSekiguchi @Bukhtawar lets reopen an re-describe the original problem? It sounds like the root issue is that “If an index is deleted while the cluster is concurrently taking more than one snapshot then there is a risk that one of the snapshots may never complete and also that some shard data may be lost from the repository, causing future restore operations to fail.” We should fix this. Please note that we cannot take non-AL2 code from ES.

[aYukiSekiguchi]

I updated the description and removed about the mitigation setting because it confused some people.

Note for future readers:
The snapshot.max_concurrent_operations mitigation in Elasticsearch 7.10.2 Release Note should work on the bare bone OpenSearch. However, it looks like Amazon OpenSearch Service doesn't support the setting.

[xuezhou25]

Updating my thoughts on the root cause:
Snapshot creation and deletion can not be proceed at the same time.
After a snapshot deletion, actions will be triggered on in-progress snapshots, see:

OpenSearch/server/src/main/java/org/opensearch/snapshots/SnapshotsService.java

Lines 2897 to 2905 in 658f7a6

	* The removal of a delete from the cluster state can trigger two possible actions on in-progress snapshots:
	* <ul>
	* <li>Snapshots that had unfinished shard snapshots in state {@link ShardSnapshotStatus#UNASSIGNED_QUEUED} that
	* could not be started because the delete was running can have those started.</li>
	* <li>Snapshots that had all their shards reach a completed state while a delete was running (e.g. as a result of
	* nodes dropping out of the cluster or another incoming delete aborting them) need not be updated in the cluster
	* state but need to have their finalization triggered now that it's possible with the removal of the delete
	* from the state.</li>
	* </ul>

If there is a snapshot creation in the queue during the snapshot deletion, which means there are unfinished shard snapshots in UNASSIGNED_QUEUED state, there will be a step to check the current shard status in updatedSnapshotsInProgress from this line:

OpenSearch/server/src/main/java/org/opensearch/snapshots/SnapshotsService.java

Line 2957 in 6f6e84e

// We don't have a new assignment for this shard because its index was concurrently deleted

When an index is deleted before the check, ShardSnapshotStatus will be put into MISSING state, which is taken as shard snapshot "completed". Then there is an incorrect use in updating snapshotEntries, in this line:

OpenSearch/server/src/main/java/org/opensearch/snapshots/SnapshotsService.java

Line 2968 in 6f6e84e

snapshotEntries.add(entry.withStartedShards(updatedAssignmentsBuilder.build()));

The method withStartedShards assumes that the ShardSnapshotStatus is completed, see:

OpenSearch/server/src/main/java/org/opensearch/cluster/SnapshotsInProgress.java

Lines 552 to 556 in 6f6e84e

	/**
	* Same as {@link #withShardStates} but does not check if the snapshot completed and thus is only to be used when starting new
	* shard snapshots on data nodes for a running snapshot.
	*/
	public Entry withStartedShards(ImmutableOpenMap<ShardId, ShardSnapshotStatus> shards) {

It lends to the completed snapshots not being finalized, and this is the code for finalizing the completed snapshots:

OpenSearch/server/src/main/java/org/opensearch/snapshots/SnapshotsService.java

Lines 2972 to 2974 in 6f6e84e

	// Entry is already completed so we will finalize it now that the delete doesn't block us after
	// this CS update finishes
	newFinalizations.add(entry);

The code defect may cause state consistency issue during restoring the snapshot: the snapshot status is uncompleted, while the ShardSnapshotStatus is completed.

AVERTISSEMENT: Uncaught exception in thread: Thread[opensearch[node_t0][clusterManagerService#updateTask][T#1],5,TGRP-DedicatedClusterSnapshotRestoreIT]
java.lang.AssertionError: Completed state must imply all shards completed but saw state [STARTED] and shards [[test-index][0]=>ShardSnapshotStatus[state=MISSING, nodeId=null, reason=missing index, generation=null]]