[Feature] Support for weighted zonal search request routing policy

[Bukhtawar]

Is your feature request related to a problem? Please describe.
The search requests at the coordinator performs a round robin to route requests to shard copies, with adaptive replica selection it might choose to route request to copies that rank lower in preference based on certain parameters. However there seems to be some use cases where weighted routing policy adds value

1. Heterogeneous(zone wise) instance types -- Certain instance capacities are sometimes available only in certain zones and not others, which means customers can choose to run 4xl instances in 1 zone and 2xl instances in others. Weighted routing, 2:1 might help heterogeneous deplyoments.
2. Zonal deployment model --Software Deployments can go slow and might choose to perform 1 zone deployment at a time, which means cutting off traffic to one zone under deployment might be needed. Setting the policy to 1:0 should effectively cut-off all search shard requests to go to copies in the AZ under deployment
3. Zonal failures -- Zonal failures are common and there is no mechanism to weigh away shard request traffic off unhealthy zone, even though HTTP traffic is weighted away

Describe the solution you'd like
Support for a weighted routing policy that can help to incrementally weigh away traffic or route traffic based on the routing policy. To start off small we can have a manual mechanism to configure policies and provide smarter defaults and guard rails to prevent acting up on bad configurations.

[dblock]

If I read this correctly, you're trying to solve for availability scenarios with AZ failures by using weighted routing. If that's the case, you might want to clarify above (maybe explain where you come from a little better?). Then, does it ever make sense to have heterogeneous instances within the same zone? in which case weighted routing may also be a good idea for better throughput and not just availability?

[Bukhtawar]

If I read this correctly, you're trying to solve for availability scenarios with AZ failures by using weighted routing. If that's the case, you might want to clarify above (maybe explain where you come from a little better?). Then, does it ever make sense to have heterogeneous instances within the same zone? in which case weighted routing may also be a good idea for better throughput and not just availability?

Modified the issue to reflect zonal routing policy.

[dblock]

So is the answer to does it ever make sense to have heterogeneous instances within the same AZ a no?

[Bukhtawar]

It does make sense, but do you think we can build that capability incrementally. While building the zonal policy, we should see how it could be extended for these uses cases in future as well

[dblock]

Take a look at #2877 (comment), can we solve both this and that problem the same way or in the same path?

[andrross]

I agree with @dblock that there may be a common mechanism here that would solve many different use cases. Just for my own clarification though I have a couple questions :)

re: zonal failures - Are you referring to partial failures here where hosts in the failed zone are still responsive but have degraded performance? If the failed zone is fully partitioned away from the rest of the cluster and all network connections are broken then would weighting away be necessary?

re: zonal deployment model - Can this be solved by graceful shutdowns during deployments so that new traffic is not accepted and existing requests are allowed to complete? It seems like it would be preferable to solve this in a way that doesn't require the operator to orchestrate weighting policies during deployments, if that's possible.

[Bukhtawar]

zonal failures - Are you referring to partial failures here where hosts in the failed zone are still responsive but have degraded performance? If the failed zone is fully partitioned away from the rest of the cluster and all network connections are broken then would weighting away be necessary

Yes weigh away would guarantee that transient network faults don't cause a flip flop till the zonal failure complete heals. For predictability it might be desired we stop routing any traffic to the impacted zone/rack.

zonal deployment model - Can this be solved by graceful shutdowns during deployments so that new traffic is not accepted and existing requests are allowed to complete? It seems like it would be preferable to solve this in a way that doesn't require the operator to orchestrate weighting policies during deployments, if that's possible.

Yes the intent is to introduce graceful shutdowns. While I am not sure what you meant by "orchestrating weighting policies during deployments", the idea would be to allow controls to incrementally(fe 5% -> 20% -> 50% -> 100%) weigh away traffic which might require operator/automated orchestration

[elfisher]

Can we add labels for "roadmap" and the version of OpenSearch this is targeting? I can add it to the overall project roadmap in the right column once that is done.

[saratvemulapalli]

@Bukhtawar are we good to go for 2.5 ?
Code freeze is tomorrow.

[kotwanikunal]

@Bukhtawar are we good to go for 2.5 ? Code freeze is tomorrow.

@Bukhtawar Any updates?