[BUG] Monitor search query not compatible with older OpenSearch versions (remote queries)

[zolboo1024]

What is the bug?
Not sure if this is considered a bug but the search query made by the monitor under alerting tab returns an error when querying from a remote index where the OS version is 2.3.
The error returned:
Error: Failed fetching inputs: RemoteTransportException[[error while communicating with remote cluster [rcluster2]]]; nested: RemoteTransportException[[opensearch-rnode2][10.89.0.36:9300][indices:data/read/search]]; nested: IllegalArgumentException[no Preference for [_primary_first]];; RemoteTransportException[[opensearch-rnode2][10.89.0.36:9300][indices:data/read/search]]; nested: IllegalArgumentException[no Preference for [_primary_first]];; java.lang.IllegalArgumentException: no Preference for [_primary_first]

How can one reproduce the bug?
Steps to reproduce the behavior:

1. Have a cluster running OS version 2.3.
2. Have a cluster running the latest OS version, 2.11.
3. From the 2.11 cluster, make a query from the monitor to the 2.3 cluster.

What is the expected behavior?
Our team was using 2.8 on the central local cluster last week and everything was working fine. We updated to 2.11 this week and it works fine with remote clusters where the version if 2.8 or higher. However, on 2.3 remote clusters, the error above occurs.
After some digging, we found 2.3 does not support the search preference _primary_first (according to the documentation). We cannot downgrade since our new data is indexed using the newer version of Lucene.
After further digging, it relates to this change:
#1205
It was merged for 2.11 and changes the preference on all Search Requests inside monitor to have the PRIMARY_FIRST preference.

I fixed this problem temporarily by modifying the Alerting source code and undoing the changes in 1205. However, for backwards compatibility, it would be useful to check the cluster version before adding headers into the search request.

What is your host/environment?

• OS: Linux
• Version: 2.11 and 2.3
• Plugins: Alerting