[FEATURE] AD 2.15 & 2.17 Improvements

[jackiehanyang]

Memory Circuit Breaker Threshold Adjustment (targeting 2.15 release)

Problem Statement:
Currently, the memory circuit breaker threshold is hardcoded to 85%. There is a need for a dynamic setting that allows adjustment of this threshold. The system should dynamically read the open memory circuit breaker threshold on each node, particularly since the customer occasionally experiences memory usage spikes up to 88%.

Action Items:

• Update default threshold to 95%.
• Update upper threshold limit to 98%.
• Make JVM_HEAP_USAGE_THRESHOLD a dynamic setting, where customers can adjust it dynamically.
• e2e testing

---

Result Index Lifecycle Management (targeting 2.15 release)

Problem Statement:
Currently, customers must use ISM to manage their custom result indices. We aim to implement lifecycle management for result indices within AD for customers, which includes creating daily result indices and setting a retention period for the data.

Action Items:

• On AD-Dashboard detector creation page, add three result index lifecycle management conditions: max_index_age_condition, max_index_size_condition, and index_cleanup_intervals
• On AD-Dashboard detector details page, detector configurations tab, display these three conditions after Custom result index.
• Update AD detector config to read these three conditions at backend
• Update the current index management logic in AD to include managing the lifecycle of custom result indices based on three newly added conditions.
• Update all related UTs and ITs

---

Flattening result index Nested/array Values for Dashboard Viewing (targeting 2.17 release)

Problem Statement:
Many values are not flattened, making it difficult to view them on the dashboard. For instance, entity values are nested objects, and features are arrays. The requirement is to reference a feature by name and apply conditions like f1 > 3. Additionally, there is a need to perform terms aggregation on categorical fields. This will require adjustments to the mapping and the addition of new fields in the result index.

Solution:
We flatten all four nested fields, and update the result index mapping to add those flattened fields. For example, taking the sample result index provided in the documentation here - https://opensearch.org/docs/latest/observing-your-data/ad/result-mapping/. After flattening, we update the result index with the following flattened fields:

{
    ......SAME ORIGINAL CONTENT AS ABOVE......
 
    // flattened feature_data fields
    "feature_data_processing_bytes_max": 2322,
    "feature_data_processing_bytes_avg": 1718.6666666666667,
    "feature_data_processing_bytes_min": 1375,
    "feature_data_processing_bytes_sum": 5156,
    "feature_data_processing_time_max": 31198，
    
    // flattened entity fields
    "entity_process_name_value": "process_3",
    
    // flattened relevant_attribution fields
    "relevant_attribution_processing_bytes_max": 0.03628638020431366,
    "relevant_attribution_processing_bytes_avg": 0.03384479053991436,
    "relevant_attribution_processing_bytes_min": 0.058812549572819096,
    "relevant_attribution_processing_bytes_sum": 0.10154576265526988,
    "relevant_attribution_processing_time_max": 0.7695105170276828,
    
    // flattened expected_values fields
    "expected_values_processing_bytes_max": 2291,
    "expected_values_processing_bytes_avg": 1677.3333333333333,
    "expected_values_processing_bytes_min": 1054,
    "expected_values_processing_bytes_sum": 6062,
    "expected_values_processing_time_max": 23379
    
    // flattened past_values fields
    "past_values_processing_bytes_max": 905,
    "past_values_processing_bytes_avg": 479,
    "past_values_processing_bytes_min": 128,
    "past_values_processing_bytes_sum": 1437,
    "past_values_processing_time_max": 8440
}

Action Items:

• add a checkbox on frontend to allow customer to turn on/off this feature
• update AD Config file to include this new field
• create a method that updates result index by generating flattened fields and appending it in the original result index if there're nested fields exist
• update places where we need to lookup result index mapping
  • create result index -https://github.com/opensearch-project/anomaly-detection/blob/main/src/main/java/org/opensearch/timeseries/indices/IndexManagement.java#L1334
  • rollover result index - https://github.com/opensearch-project/anomaly-detection/blob/main/src/main/java/org/opensearch/timeseries/indices/IndexManagement.java#L1234
  • when node starts, it updates result index mapping based on version - https://github.com/opensearch-project/anomaly-detection/blob/main/src/main/java/org/opensearch/timeseries/indices/IndexManagement.java#L1075
  • in validate api where it verifies result index mapping
  • in job interval where it validates result index mapping - ADJobProcessor.validateResultIndexAndRunJob
• add/update UTs and ITs

[dblock]

Catch All Triage - 1 2 3 4 5 6