Migrate Start Detector job to SDK using SDKRestClient

[dbwiddis]

Is your feature request related to a problem?

Part of opensearch-project/OpenSearch#5224

What solution would you like?

Migrate the Start Detector job Rest API to Extensions SDK using SDKRestClient.

Use the implementation of #353 as a reference. Add new issues for any steps you skip.

Implement the following API endpoints:

• POST _plugins/_anomaly_detection/detectors/<detectorId>/_start

[joshpalis]

Completed Tasks for Real time analysis:

• Implement create components for Anomaly Detection Extension
• Register AD Rest Handlers for start detector : RestAnomalyDetectorJobAction
• Register AD Action Handlers for start detector : AnomalyDetectorJobAction
• Register extension Job Details with Job Scheduler when start detector API is invoked
• Integrate AcquireLock API with AnomalyDetectorJobRunner
• Integrate ReleaseLock API with AnomalyDetectorJobRunner

Replace the following instances in which client.execute() is used to trigger actions registered within OpenSearch with corresponding methods made available by the Java Client (OpenSearchAsyncClient).

• AnomalyDetectionIndicies.updateJobIndexSettingIfNecessary()
  • Note : Once this client.execute is replaced, uncomment anomalyDetectionIndices.update() in the following two places
    • IndexAnomalyDetectorTransportAction
    • ADTaskManager
• ADTaskManager.resetEntityTasksAsStopped()
• ADTaskManager.updateLatestFlagOfOldTasksAndCreateNewTask()
• ADTaskManager.resetLatestFlagAsFalse()
• ADTaskManager.deleteTaskDocs()
• ADTaskManager.deleteADTasks()
• ADTaskManager.deleteADResultOfDetector()
• ADTaskManager.cleanChildTasksAndADResultsOfDeletedTask()
  • First Action
  • Second Action

Remaining Tasks :

• Modify Job Scheduler Proxy ScheduledJobParser / ScheduledJobRunner to extend from ActionRequest. This is in response to a modification to the ExtensionProxyAction workflow from [FEATURE] [DISCUSS] Implement Proxy Actions #525. [Extensions] Makes JobRunner/Parameter/Request/Response classes extend from ActionRequest/Response job-scheduler#352
• Migrate AnomalyResultAction/TransportAction to SDK using the SDKClient #626

Blockers :

#624

[joshpalis]

Tasks To Enable Historical Analysis

• Integrate ForwardADTaskAction with the SDK and register within getActions
• Modify ADTaskManager.forwardRequestToLeadNode to remove hashring and redirect the ForwardADTaskRequest to the extension node
• Remove hashring from ADTaskManager.checkTaskSlots and execute ADStatsNodesAction for extension node only
• Remove hashring from ADTaskManager.startHistoricalAnalysis and forwardDetectRequestToCoordinationNode for only the extension node. ADVersion should just be defaulted to current
• Remove hashring from ADTaskManager.scaleTaskLaneOnCoordinatingNode
• Remove hashRing from ADTaskManager.detectorTaskSlotScaleDelta so that the extension node is the only eligible data node
• Register ad-batch-task-threadpool as an executor builder. Already added in by [Extensions] Migrate ADResultBulkAction/TransportAction and Multi-Entity real time analysis anomaly-detection#876
• Integrate ADBatchAnomalyResultAction with the SDK and register within getActions. Called by ADTaskManager.runNextEntityForHCADHistorical (This just calls ADBatchTaskRunner.run for each request).
• Modify ADBatchTaskRunner.getTopEntitiesListener and remove hashring to direct all high cardinality tasks back to the extension node
• Modify ADBatchTaskRunner.forwardOrExecuteEntityTask and ensure that ADBatch Task is executed locally on the extension node. (This negates the need to register ADBatchTaskRemoteExecutionAction as all tasks will be executed on the extension node)
• Comment out ADBatchTaskRunner.dispatchTask() and return cluster.localNode() directly as querying for node stats is not necessary for extensions.
• Modify AbstractRetriever and replace all instances of Internal(Aggregation/Bucket/Etc) with corresponding Parsed(Aggregation/Bucket/Etc)
• Potential Issue : double check SearchFeatureDao.parseBucketAggregationResponse as the aggregation class InternalComposite may not be supported by HLRC (May need to replace with ParsedComposite)
• Potential issue : double check AbstractRetirever.parseAggregation as the aggregation class InternalSingleBucketAggregation may not be supported by HLRC
• Replace StringTerms with ParsedStringTerms in ADBatchTaskRunner.searchTopEntitiesForSingleCategoryHC as the SDKRestClient uses the HLRC internally
• Remove transportService from ADTaskManager.forwardADTaskToCoordinatingNode()and ADTaskManager.forwardStaleRunningEntitiesToCoordinatingNode() and replace with client execute call for ForwardADTaskAction
• Remove hashring from ADTaskManager.stopHistoricalAnalysis to enable stoping historical analysis tasks
• Integrate ADCancelTaskAction with the SDK and register within getActions
• Fix affected test classes

[minalsha]

@joshpalis PR:opensearch-project/anomaly-detection#837 is merged for start detector. Can we close this issue?