[Remote Store] Handle translog upload during primary to primary relocation for remote-backed indexes

[ashking94]

Is your feature request related to a problem? Please describe.
Currently, when peer recovery happens for primary-primary relocation, both the older & newer primary bootstraps engine and translog such that the upload happens from both the shards.

Following things need to be ensured -

• When index.translog.durability is set as async, the translog sync happens every index.translog.sync_interval periodically. With this change, we decide to upload translog to remote store basis the primary mode. Since translog sync is async, in current status quo state, the upload is not determinstic to happen before primaryMode is marked as false. We need to ensure that the upload happens (successfully or it fails) deterministically before the primaryMode turns false on the old primary and before the new primary resets the engine (resetToWriteableEngine).
• Reset the new primary engine after all translogs have been uploaded from the older primary to the remote store. This will ensure that the new primary gets all the translogs from remote store and then only it starts accepting any new write requests. This is already handled with ForceSyncTransportRequestHandler.

There is possible issue with data loss in relocation flow for seg rep based indexes. Issue - #5848

Describe the solution you'd like
Since the complete handoff process ensures that the primary mode switch happens such that the any inflight request in older primary is first drained off and then the newer primary gets promoted with primary mode set to true as part of handoff.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

[sachinpkale]

Fixed in #5804