add eligible node role settings

[ylwu-amzn]

Description

Per PM's suggestion, remote model should be able to run on data or ML node by default. But we should also keep local model run on ML node by default. So we still keep plugins.ml_commons.only_run_on_ml_node as true, but only control local model. Add more setting to provide granular level control on remote and local model.

1. Add plugins.ml_commons.task_dispatcher.eligible_node_role.remote_model which controls only remote model can run on which node. Default value is ["data", "ml"] which means the remote model can run on data node and ML node by default.
2. Add plugins.ml_commons.task_dispatcher.eligible_node_role.local_model which controls only local model can run on which node. But we have plugins.ml_commons.only_run_on_ml_node which controls the model can only run on ML node or not. To provide BWC, For local model, 1/ if plugins.ml_commons.only_run_on_ml_node is true, we will always run it on ML node. 2/ if plugins.ml_commons.only_run_on_ml_node is false, will run model on nodes defined in plugins.ml_commons.task_dispatcher.eligible_node_role.local_model.

Issues Resolved

[List any issues this PR will resolve]

Check List

• New functionality includes testing.
  • All tests pass
• New functionality has been documented.
  • New functionality has javadoc added
• Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[codecov]

Codecov Report

Merging #1197 (677d47c) into 2.9 (4ac47f6) will increase coverage by 0.06%.
Report is 1 commits behind head on 2.9.
The diff coverage is 82.05%.

@@             Coverage Diff              @@
##                2.9    #1197      +/-   ##
============================================
+ Coverage     78.75%   78.81%   +0.06%     
- Complexity     2122     2132      +10     
============================================
  Files           167      167              
  Lines          8678     8714      +36     
  Branches        871      874       +3     
============================================
+ Hits           6834     6868      +34     
- Misses         1448     1449       +1     
- Partials        396      397       +1     

Flag	Coverage Δ
ml-commons	78.81% <82.05%> (+0.06%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files Changed	Coverage Δ
...rg/opensearch/ml/plugin/MachineLearningPlugin.java	99.04% <ø> (ø)
...n/java/org/opensearch/ml/cluster/MLSyncUpCron.java	78.94% <50.00%> (+0.10%)	⬆️
.../java/org/opensearch/ml/task/MLTaskDispatcher.java	32.94% <50.00%> (ø)
...org/opensearch/ml/cluster/DiscoveryNodeHelper.java	87.61% <68.57%> (-7.62%)	⬇️
...h/ml/action/deploy/TransportDeployModelAction.java	81.02% <100.00%> (+0.13%)	⬆️
.../ml/action/execute/TransportExecuteTaskAction.java	100.00% <100.00%> (ø)
...arch/ml/action/forward/TransportForwardAction.java	95.65% <100.00%> (+0.04%)	⬆️
...tion/prediction/TransportPredictionTaskAction.java	77.77% <100.00%> (+1.03%)	⬆️
.../action/register/TransportRegisterModelAction.java	96.22% <100.00%> (ø)
...l/action/training/TransportTrainingTaskAction.java	100.00% <100.00%> (ø)
... and 6 more

... and 2 files with indirect coverage changes

[dhrubo-os]

Just have a minor comment. Overall looks good to me.