Assertion failure in replica promotion for a closed remote translog enabled index while writing Integ tests

[ashking94]

Is your feature request related to a problem? Please describe.
While writing tests for recovery flow as part of #8476, discovered that NoOpEngine is created when the index is closed. In this test, I am creating a remote translog enabled with 0 replica, index some docs, then closing the index. Post that, updating index settings to increase replica count to 1. Then stopping the node which has the

OpenSearch/server/src/main/java/org/opensearch/indices/IndicesService.java

Lines 838 to 843 in c1c23b4

	private EngineFactory getEngineFactory(final IndexSettings idxSettings) {
	final IndexMetadata indexMetadata = idxSettings.getIndexMetadata();
	if (indexMetadata != null && indexMetadata.getState() == IndexMetadata.State.CLOSE) {
	// NoOpEngine takes precedence as long as the index is closed
	return NoOpEngine::new;
	}

During replica to primary promotion, below method is hit on the replica shard -

OpenSearch/server/src/main/java/org/opensearch/index/shard/IndexShard.java

Lines 550 to 556 in c1c23b4

	public void updateShardState(
	final ShardRouting newRouting,
	final long newPrimaryTerm,
	final BiConsumer<IndexShard, ActionListener<ResyncTask>> primaryReplicaSyncer,
	final long applyingClusterStateVersion,
	final Set<String> inSyncAllocationIds,
	final IndexShardRoutingTable routingTable

At the end of this method, the control reaches toturnOffTranslogRetention(); -
https://github.com/opensearch-project/OpenSearch/blob/c1c23b42e335bb337c668c9cf9dccd8b71dfdbab/server/src/main/java/org/opensearch/index/shard/IndexShard.java#L748C4-L759

This ultimately hits the below assertion which fails -

OpenSearch/server/src/main/java/org/opensearch/index/engine/NoOpEngine.java

Lines 209 to 216 in c1c23b4

	translog.trimUnreferencedReaders();
	// refresh the translog stats
	translogStats = translog.stats();
	assert translog.currentFileGeneration() == translog.getMinFileGeneration() : "translog was not trimmed "
	+ " current gen "
	+ translog.currentFileGeneration()
	+ " != min gen "
	+ translog.getMinFileGeneration();

Describe the solution you'd like
Need to think over on this.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.