Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Feature/extensions] Migrates Start Detector #837

Merged
merged 54 commits into from
Apr 3, 2023
Merged

[Feature/extensions] Migrates Start Detector #837

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
54 commits
Select commit Hold shift + click to select a range
8d0cee5
Initial start detector commit, sans test classes
joshpalis Mar 2, 2023
9d40a94
Adding XContentParser import to IndexAnomalyDetectorJobActionHandler
joshpalis Mar 2, 2023
a663f12
spotlessApply
joshpalis Mar 2, 2023
ae5cd1f
Changing inject dependency to guice for AnomalyDetectorJobTransportAc…
joshpalis Mar 3, 2023
d5840be
Pulling settings from extensionRunner in AnomalyDetectorJobTransportA…
joshpalis Mar 3, 2023
b6b139d
Making AnomalyDetectorJobTransportAction doExecute protected
joshpalis Mar 3, 2023
59cff9c
Adding RestAnomalyDetectorJobAction to getExtensionRestHandlers
joshpalis Mar 3, 2023
c8e6b86
Updating test constructors for ADJobRunner/JobParameter
joshpalis Mar 7, 2023
d5fa24e
merge feature/extensions
joshpalis Mar 7, 2023
8e24c6f
fixing merge conflict, spotlessApply
joshpalis Mar 7, 2023
1ba5852
adding restClient()
joshpalis Mar 7, 2023
333153c
Merge feature/extensions into startdetector
joshpalis Mar 7, 2023
076d14c
uncommenting AnomalyDetectorJobTransportAction test classes and addin…
joshpalis Mar 7, 2023
a4a7e9f
Added job details registration to RestAnomalyDetectorJobAction, added…
joshpalis Mar 8, 2023
509e3c4
Triggering job details registration with Job Scheduler within RestAn…
joshpalis Mar 8, 2023
33610e8
Inital Job Scheduler communication mechanism integration commit
joshpalis Mar 9, 2023
ca3914c
Fixing SDKRestClient null action map, commenting out test classes, cr…
joshpalis Mar 9, 2023
2df0224
Adding class names of commented out test classes to jacocoExclusions
joshpalis Mar 9, 2023
089d334
jacocoExclusion
joshpalis Mar 9, 2023
1e097c9
Integrates AcquireLockRequest/ReleaseLockRequest in AnomalyDetectorJo…
joshpalis Mar 10, 2023
384d4f8
spotlessApply
joshpalis Mar 10, 2023
2832875
Adds OpenSearchAsyncClient to AnomalyDetectorExtension
joshpalis Mar 20, 2023
1ed4a3f
Adds OpenSearchAsyncClient to AnomalyDetectorExtension / rest actions…
joshpalis Mar 21, 2023
668d7e1
Removing logs from RestAnomalyDetectorJobAction
joshpalis Mar 21, 2023
7a62ef1
Merge branch 'feature/extensions' into startdetector
joshpalis Mar 21, 2023
e0e1bb0
Renamed opensearchAsyncClient to sdkJavaAsyncClient. Returning sdkJav…
joshpalis Mar 21, 2023
7867c97
Sending fully qualified class names of ADJobRunner/Parser to Job Sche…
joshpalis Mar 24, 2023
c96ed36
Fixes serde logic for the ADJobParameter/Runner TransportActions
joshpalis Mar 28, 2023
07e8695
Initial StopDetector commit, prepares required created components for…
joshpalis Mar 22, 2023
a5074e0
Fixing related tests
joshpalis Mar 23, 2023
9ed9a20
Commented out DeleteTests.java until we have support for SDKRestClien…
joshpalis Mar 23, 2023
4df32c8
Adding required created components for AnomalyResultTransportAction
joshpalis Mar 28, 2023
4a3bb84
Fixing affected test classes
joshpalis Mar 29, 2023
9a6136f
Refactoring ADJobRunnerTransportAction to use the document ID from th…
joshpalis Mar 29, 2023
966e565
fixing ADJobRunnerTransportActionTests
joshpalis Mar 29, 2023
6f312af
Implements getExecutorBuilders and comments out the AnomalyResultActi…
joshpalis Mar 30, 2023
ce52378
Adding issues to commented test classes
joshpalis Mar 30, 2023
5bb188a
Merge branch 'feature/extensions' into startdetector
joshpalis Mar 30, 2023
64d0d9f
Replacing ExtensionRestRequest with RestRequest, removed SDKNamedXCon…
joshpalis Mar 30, 2023
e5c5640
Stubbing clusterSettings.addSettingsUpdateConsumer to fix failing tests
joshpalis Mar 30, 2023
68f1b28
Adding classes to the jacoco explusions config
joshpalis Mar 30, 2023
89c4db2
Adding more classes to jacoco test coverage
joshpalis Mar 30, 2023
2a84fd1
Addressing PR comments, still need to work on replacing match with ma…
joshpalis Mar 31, 2023
cc37a69
Removing stray semicolon
joshpalis Mar 31, 2023
1a593f4
Replacing string with ADJobIndex constant for initAnomalyDetectorJobI…
joshpalis Mar 31, 2023
c40b7b8
Fixing updateByQueryRequests
joshpalis Mar 31, 2023
0ac4e4d
Fixing DeleteByQueryRequests in ADTaskManager
joshpalis Mar 31, 2023
9a26880
Using adminClient to call .indices() in AnomalyDetectionIndices
joshpalis Apr 1, 2023
a45d303
Fixing comments
joshpalis Apr 1, 2023
5002340
Fixing failing tests that havent failed before, weird
joshpalis Apr 1, 2023
0380d9a
spotless
joshpalis Apr 1, 2023
06a1fab
Addressing PR comments
joshpalis Apr 3, 2023
60a2348
Fixing OpenSearchRejectedExecutionException import
joshpalis Apr 3, 2023
bb740db
Addressing PR comments, changing ES to OS
joshpalis Apr 3, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
28 changes: 27 additions & 1 deletion build.gradle
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -695,6 +695,7 @@ List<String> jacocoExclusions = [
'org.opensearch.ad.transport.AnomalyDetectorJobAction',
'org.opensearch.ad.transport.AnomalyResultTransportAction',
'org.opensearch.ad.transport.GetAnomalyDetectorAction',
'org.opensearch.ad.transport.AnomalyDetectorJobTransportAction',
'org.opensearch.ad.transport.AnomalyDetectorJobResponse',
'org.opensearch.ad.transport.GetAnomalyDetectorRequest',
'org.opensearch.ad.transport.AnomalyResultTransportAction',
Expand Down Expand Up @@ -743,7 +744,32 @@ List<String> jacocoExclusions = [
'org.opensearch.ad.task.ADBatchTaskCache',
'org.opensearch.ad.caching.DoorKeeper',
'org.opensearch.ad.caching.PriorityCache.1',
'org.opensearch.ad.caching.CacheProvider'
'org.opensearch.ad.caching.CacheProvider',
'org.opensearch.ad.util.IndexUtils',
'org.opensearch.ad.cluster.ClusterManagerEventListener',
'org.opensearch.ad.cluster.ClusterManagerEventListener.1',
'org.opensearch.ad.cluster.ClusterManagerEventListener.2',
'org.opensearch.ad.transport.ADStatsNodesAction',
'org.opensearch.ad.transport.BackPressureRouting',
'org.opensearch.ad.transport.ADStatsNodesTransportAction',
'org.opensearch.ad.stats.InternalStatNames',
'org.opensearch.ad.NodeState',
'org.opensearch.ad.transport.DeleteModelRequest',
'org.opensearch.ad.transport.DeleteModelNodeResponse',
'org.opensearch.ad.transport.DeleteModelResponse',
'org.opensearch.ad.transport.StopDetectorTransportAction',
'org.opensearch.ad.transport.StopDetectorRequest',
'org.opensearch.ad.ratelimit.RateLimitedRequestWorker',
'org.opensearch.ad.ratelimit.QueuedRequest',
'org.opensearch.ad.ratelimit.CheckpointReadWorker',
'org.opensearch.ad.ratelimit.ConcurrentWorker',
'org.opensearch.ad.ratelimit.EntityFeatureRequest',
'org.opensearch.ad.ratelimit.RateLimitedRequestWorker.RequestQueue',
'org.opensearch.ad.stats.StatNames',
'org.opensearch.ad.stats.ADStats',
'org.opensearch.ad.MaintenanceState',
'org.opensearch.ad.AnomalyDetectorExtension.1',
'org.opensearch.ad.AnomalyDetectorExtension.1.1'
owaiskazi19 marked this conversation as resolved.
Show resolved Hide resolved
]


Expand Down
399 changes: 382 additions & 17 deletions src/main/java/org/opensearch/ad/AnomalyDetectorExtension.java
View file
Open in desktop

Large diffs are not rendered by default.

192 changes: 87 additions & 105 deletions src/main/java/org/opensearch/ad/AnomalyDetectorJobRunner.java
View file
Open in desktop

Large diffs are not rendered by default.

10 changes: 5 additions & 5 deletions src/main/java/org/opensearch/ad/NodeStateManager.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,14 +37,14 @@
import org.opensearch.ad.transport.BackPressureRouting;
import org.opensearch.ad.util.ClientUtil;
import org.opensearch.ad.util.ExceptionUtil;
import org.opensearch.client.Client;
import org.opensearch.cluster.service.ClusterService;
import org.opensearch.common.lease.Releasable;
import org.opensearch.common.settings.Settings;
import org.opensearch.common.unit.TimeValue;
import org.opensearch.common.xcontent.LoggingDeprecationHandler;
import org.opensearch.common.xcontent.XContentType;
import org.opensearch.core.xcontent.XContentParser;
import org.opensearch.sdk.SDKClient.SDKRestClient;
import org.opensearch.sdk.SDKClusterService;
import org.opensearch.sdk.SDKNamedXContentRegistry;

/**
Expand All @@ -56,7 +56,7 @@ public class NodeStateManager implements MaintenanceState, CleanState {
private static final Logger LOG = LogManager.getLogger(NodeStateManager.class);
public static final String NO_ERROR = "no_error";
private ConcurrentHashMap<String, NodeState> states;
private Client client;
private SDKRestClient client;
private SDKNamedXContentRegistry xContentRegistry;
private ClientUtil clientUtil;
// map from detector id to the map of ES node id to the node's backpressureMuter
Expand All @@ -78,13 +78,13 @@ public class NodeStateManager implements MaintenanceState, CleanState {
* @param clusterService Cluster service accessor
*/
public NodeStateManager(
Client client,
SDKRestClient client,
SDKNamedXContentRegistry xContentRegistry,
Settings settings,
ClientUtil clientUtil,
Clock clock,
Duration stateTtl,
ClusterService clusterService
SDKClusterService clusterService
) {
this.states = new ConcurrentHashMap<>();
this.client = client;
Expand Down
4 changes: 2 additions & 2 deletions src/main/java/org/opensearch/ad/caching/PriorityCache.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -53,9 +53,9 @@
import org.opensearch.ad.model.ModelProfile;
import org.opensearch.ad.ratelimit.CheckpointWriteWorker;
import org.opensearch.ad.settings.AnomalyDetectorSettings;
import org.opensearch.cluster.service.ClusterService;
import org.opensearch.common.Strings;
import org.opensearch.common.unit.TimeValue;
import org.opensearch.sdk.SDKClusterService;
import org.opensearch.threadpool.ThreadPool;

import com.google.common.cache.Cache;
Expand Down Expand Up @@ -95,7 +95,7 @@ public PriorityCache(
MemoryTracker memoryTracker,
int numberOfTrees,
Clock clock,
ClusterService clusterService,
SDKClusterService clusterService,
Duration modelTtl,
ThreadPool threadPool,
CheckpointWriteWorker checkpointWriteQueue,
Expand Down
12 changes: 6 additions & 6 deletions src/main/java/org/opensearch/ad/indices/ADIndex.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@

import org.opensearch.ad.constant.CommonName;
import org.opensearch.ad.model.AnomalyDetector;
import org.opensearch.ad.model.AnomalyDetectorJob;
import org.opensearch.ad.util.ThrowingSupplierWrapper;

/**
Expand All @@ -34,12 +35,11 @@ public enum ADIndex {
false,
ThrowingSupplierWrapper.throwingSupplierWrapper(AnomalyDetectionIndices::getAnomalyDetectorMappings)
),
// @anomaly-detection.create-detector Commented this code until we have support of Job Scheduler for extensibility
// JOB(
// AnomalyDetectorJob.ANOMALY_DETECTOR_JOB_INDEX,
// false,
// ThrowingSupplierWrapper.throwingSupplierWrapper(AnomalyDetectionIndices::getAnomalyDetectorJobMappings)
// ),
JOB(
AnomalyDetectorJob.ANOMALY_DETECTOR_JOB_INDEX,
false,
ThrowingSupplierWrapper.throwingSupplierWrapper(AnomalyDetectionIndices::getAnomalyDetectorJobMappings)
),
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No complaint here, just pointing to the diff above as a reason we shouldn't be using // comments on blocks of code :)

CHECKPOINT(
CommonName.CHECKPOINT_INDEX_NAME,
false,
Expand Down
Loading