Migrate Search Detector result to SDK using SDKRestClient #377

dbwiddis · 2023-01-29T02:36:18Z

Is your feature request related to a problem?

Part of opensearch-project/OpenSearch#5224

What solution would you like?

Migrate the Search Detector result Rest API to Extensions SDK using SDKRestClient.

Use the implementation of #353 as a reference. Add new issues for any steps you skip.

Implement the following API endpoints:

POST _plugins/_anomaly_detection/detectors/results/_search/
POST _plugins/_anomaly_detection/detectors/results/_search/<custom_result_index>
POST _plugins/_anomaly_detection/detectors/results/_search/<custom_result_index>?only_query_custom_result_index=false
POST _plugins/_anomaly_detection/detectors/results/_search/<custom_result_index>?only_query_custom_result_index=true

The text was updated successfully, but these errors were encountered:

owaiskazi19 · 2023-03-31T09:32:12Z

This is blocked as currently the result index is not created while creating the detector.

 /_extensions/_ad-extension/detectors/results/_search/{resultIndex}

Will be covered in #626

joshpalis · 2023-03-31T17:07:33Z

@owaiskazi19 The anomaly result index is created as part of the AnomalyDetectorJobRunner.runADJob workflow. This is instantiated as part of this PR. Currently, the default anomaly result index is created and only anomaly result exceptions are indexed to this index. Issue #626 will handle indexing the actual anomaly result.

owaiskazi19 · 2023-04-07T00:32:43Z

This is currently failing with the below error:

17:31:19.108 [httpclient-dispatch-1] ERROR org.opensearch.ad.transport.SearchAnomalyResultTransportAction - Failed to search result indices for all detectors
java.lang.ClassCastException: class org.opensearch.search.aggregations.bucket.terms.ParsedStringTerms cannot be cast to class org.opensearch.search.aggregations.bucket.terms.StringTerms (org.opensearch.search.aggregations.bucket.terms.ParsedStringTerms and org.opensearch.search.aggregations.bucket.terms.StringTerms are in unnamed module of loader 'app')
        at org.opensearch.ad.transport.SearchAnomalyResultTransportAction.processSingleSearchResponse(SearchAnomalyResultTransportAction.java:136) ~[main/:?]
        at org.opensearch.ad.transport.SearchAnomalyResultTransportAction.lambda$searchADResultIndex$3(SearchAnomalyResultTransportAction.java:240) ~[main/:?]
        at org.opensearch.action.ActionListener$1.onResponse(ActionListener.java:80) [opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.client.RestHighLevelClient$1.onSuccess(RestHighLevelClient.java:2093) [opensearch-rest-high-level-client-3.0.0-SNAPSHOT.jar:?]
        at org.opensearch.client.RestClient$FailureTrackingResponseListener.onSuccess(RestClient.java:721) [opensearch-rest-client-3.0.0-SNAPSHOT.jar:?]
        at org.opensearch.client.RestClient$1.completed(RestClient.java:440) [opensearch-rest-client-3.0.0-SNAPSHOT.jar:?]
        at org.opensearch.client.RestClient$1.completed(RestClient.java:434) [opensearch-rest-client-3.0.0-SNAPSHOT.jar:?]
        at org.apache.hc.core5.concurrent.BasicFuture.completed(BasicFuture.java:123) [httpcore5-5.1.5.jar:5.1.5]
        at org.apache.hc.core5.concurrent.ComplexFuture.completed(ComplexFuture.java:79) [httpcore5-5.1.5.jar:5.1.5]
        at org.apache.hc.client5.http.impl.async.InternalAbstractHttpAsyncClient$1$3$1.completed(InternalAbstractHttpAsyncClient.java:299) [httpclient5-5.1.4.jar:5.1.4]
        at org.apache.hc.core5.http.nio.support.AbstractAsyncResponseConsumer$2.completed(AbstractAsyncResponseConsumer.java:108) [httpcore5-5.1.5.jar:5.1.5]
        at org.apache.hc.core5.http.nio.entity.AbstractBinAsyncEntityConsumer.completed(AbstractBinAsyncEntityConsumer.java:84) [httpcore5-5.1.5.jar:5.1.5]
        at org.apache.hc.core5.http.nio.entity.AbstractBinDataConsumer.streamEnd(AbstractBinDataConsumer.java:81) [httpcore5-5.1.5.jar:5.1.5]
        at org.apache.hc.core5.http.nio.support.AbstractAsyncResponseConsumer.streamEnd(AbstractAsyncResponseConsumer.java:149) [httpcore5-5.1.5.jar:5.1.5]
        at org.apache.hc.client5.http.impl.async.HttpAsyncMainClientExec$1.streamEnd(HttpAsyncMainClientExec.java:235) [httpclient5-5.1.4.jar:5.1.4]
        at org.apache.hc.core5.http.impl.nio.ClientHttp1StreamHandler.dataEnd(ClientHttp1StreamHandler.java:280) [httpcore5-5.1.5.jar:5.1.5]
        at org.apache.hc.core5.http.impl.nio.ClientHttp1StreamDuplexer.dataEnd(ClientHttp1StreamDuplexer.java:366) [httpcore5-5.1.5.jar:5.1.5]
        at org.apache.hc.core5.http.impl.nio.AbstractHttp1StreamDuplexer.onInput(AbstractHttp1StreamDuplexer.java:335) [httpcore5-5.1.5.jar:5.1.5]
        at org.apache.hc.core5.http.impl.nio.AbstractHttp1IOEventHandler.inputReady(AbstractHttp1IOEventHandler.java:64) [httpcore5-5.1.5.jar:5.1.5]
        at org.apache.hc.core5.http.impl.nio.ClientHttp1IOEventHandler.inputReady(ClientHttp1IOEventHandler.java:41) [httpcore5-5.1.5.jar:5.1.5]
        at org.apache.hc.core5.reactor.InternalDataChannel.onIOEvent(InternalDataChannel.java:133) [httpcore5-5.1.5.jar:5.1.5]
        at org.apache.hc.core5.reactor.InternalChannel.handleIOEvent(InternalChannel.java:51) [httpcore5-5.1.5.jar:5.1.5]
        at org.apache.hc.core5.reactor.SingleCoreIOReactor.processEvents(SingleCoreIOReactor.java:178) [httpcore5-5.1.5.jar:5.1.5]
        at org.apache.hc.core5.reactor.SingleCoreIOReactor.doExecute(SingleCoreIOReactor.java:127) [httpcore5-5.1.5.jar:5.1.5]
        at org.apache.hc.core5.reactor.AbstractSingleCoreIOReactor.execute(AbstractSingleCoreIOReactor.java:85) [httpcore5-5.1.5.jar:5.1.5]
        at org.apache.hc.core5.reactor.IOReactorWorker.run(IOReactorWorker.java:44) [httpcore5-5.1.5.jar:5.1.5]
        at java.lang.Thread.run(Thread.java:832) [?:?]

owaiskazi19 · 2023-04-12T21:58:40Z

Solved the above error by changing the return type to ParsedStringTerms from StringTerms and updated the migration guide as well: #673

owaiskazi19 · 2023-04-12T22:02:45Z

API: 127.0.0.1:9200/_extensions/_ad-extension/_opendistro/_anomaly_detection/detectors/results/_search

Query:

{
  "query": {
    "bool": {
      "filter": [
        {
          "term": {
            "detector_id": "MyARbocB98_wy8zPgAMY"
          }
        },
        {
          "range": {
            "anomaly_grade": {
              "gt": 0
            }
          }
        }
      ],
      "must_not": [
        {
          "exists": {
            "field": "task_id"
          }
        }
      ]
    }
  }
}

Response:

{
    "took": 0,
    "timed_out": false,
    "_shards": {
        "total": 0,
        "successful": 0,
        "skipped": 0,
        "failed": 0
    },
    "hits": {
        "total": {
            "value": 0,
            "relation": "eq"
        },
        "max_score": 0.0,
        "hits": []
    }
}

owaiskazi19 · 2023-04-12T22:03:50Z

API: /_anomaly_detection/detectors/results/_search

Query:

{
  "query": {
    "bool": {
      "filter": [
        {
          "term": {
            "detector_id": "MyARbocB98_wy8zPgAMY"
          }
        },
        {
          "range": {
            "anomaly_grade": {
              "gt": 0
            }
          }
        }
      ],
      "must_not": [
        {
          "exists": {
            "field": "task_id"
          }
        }
      ]
    }
  }
}

Response:

{
    "took": 0,
    "timed_out": false,
    "_shards": {
        "total": 0,
        "successful": 0,
        "skipped": 0,
        "failed": 0
    },
    "hits": {
        "total": {
            "value": 0,
            "relation": "eq"
        },
        "max_score": 0.0,
        "hits": []
    }
}

owaiskazi19 · 2023-04-12T22:04:30Z

API: 127.0.0.1:9200/_extensions/_ad-extension/detectors/results/_search/opensearch-ad-plugin-result-test1?only_query_custom_result_index=true

Query:

{
  "query": {
    "bool": {
      "filter": [
        {
          "term": {
            "detector_id": "MyARbocB98_wy8zPgAMY"
          }
        },
        {
          "range": {
            "anomaly_grade": {
              "gt": 0
            }
          }
        }
      ],
      "must_not": [
        {
          "exists": {
            "field": "task_id"
          }
        }
      ]
    }
  }
}

Response:

{
    "took": 1,
    "timed_out": false,
    "_shards": {
        "total": 1,
        "successful": 1,
        "skipped": 0,
        "failed": 0
    },
    "hits": {
        "total": {
            "value": 0,
            "relation": "eq"
        },
        "max_score": null,
        "hits": []
    }
}

owaiskazi19 · 2023-04-12T22:05:15Z

API: 127.0.0.1:9200/_extensions/_ad-extension/detectors/results/_search/opensearch-ad-plugin-result-test1?only_query_custom_result_index=false

Query:

{
  "query": {
    "bool": {
      "filter": [
        {
          "term": {
            "detector_id": "MyARbocB98_wy8zPgAMY"
          }
        },
        {
          "range": {
            "anomaly_grade": {
              "gt": 0
            }
          }
        }
      ],
      "must_not": [
        {
          "exists": {
            "field": "task_id"
          }
        }
      ]
    }
  }
}

Response:

{
    "took": 1,
    "timed_out": false,
    "_shards": {
        "total": 1,
        "successful": 1,
        "skipped": 0,
        "failed": 0
    },
    "hits": {
        "total": {
            "value": 0,
            "relation": "eq"
        },
        "max_score": null,
        "hits": []
    }
}

dbwiddis added the enhancement New feature or request label Jan 29, 2023

dbwiddis mentioned this issue Jan 29, 2023

[META] Migrate AD backend plugin to AD Extension opensearch-project/OpenSearch#5224

Open

21 tasks

minalsha assigned owaiskazi19 Mar 10, 2023

owaiskazi19 mentioned this issue Mar 29, 2023

[Extensions] Migrates Search, SearchResults and SearchADTasks action to extensions opensearch-project/anomaly-detection#849

Merged

3 tasks

owaiskazi19 mentioned this issue Apr 5, 2023

[FEATURE] Create custom result index with Create Detector #651

Closed

owaiskazi19 closed this as completed May 1, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Migrate Search Detector result to SDK using SDKRestClient #377

Migrate Search Detector result to SDK using SDKRestClient #377

dbwiddis commented Jan 29, 2023 •

edited by owaiskazi19

Loading

owaiskazi19 commented Mar 31, 2023

joshpalis commented Mar 31, 2023 •

edited

Loading

owaiskazi19 commented Apr 7, 2023

owaiskazi19 commented Apr 12, 2023

owaiskazi19 commented Apr 12, 2023

owaiskazi19 commented Apr 12, 2023

owaiskazi19 commented Apr 12, 2023

owaiskazi19 commented Apr 12, 2023

Migrate Search Detector result to SDK using SDKRestClient #377

Migrate Search Detector result to SDK using SDKRestClient #377

Comments

dbwiddis commented Jan 29, 2023 • edited by owaiskazi19 Loading

Is your feature request related to a problem?

What solution would you like?

owaiskazi19 commented Mar 31, 2023

joshpalis commented Mar 31, 2023 • edited Loading

owaiskazi19 commented Apr 7, 2023

owaiskazi19 commented Apr 12, 2023

owaiskazi19 commented Apr 12, 2023

owaiskazi19 commented Apr 12, 2023

owaiskazi19 commented Apr 12, 2023

owaiskazi19 commented Apr 12, 2023

dbwiddis commented Jan 29, 2023 •

edited by owaiskazi19

Loading

joshpalis commented Mar 31, 2023 •

edited

Loading