[Alerting]: harden APIs of built-in alert index-threshold #60702

pmuellr · 2020-03-19T21:49:14Z

resolves #59889

The index threshold APIs - used by both the index threshold UI and the
alert executor - were returning errors (500's from http endpoints) when
getting errors from ES.

These have been changed so that the error is logged as a warning, and the
relevant API returns an "empty" result.

Another 500 response was found while experimenting with this. Apparently
the date_range agg requires a date format to be passed in if the date format
in ES is not the format sent in the date range values - in this case an ISO date.
The repro on this was to select the .security alias (or it's index) within the
index threshold alert UI, and then select one of it's date fields, which are
formatted as epoch_millis, not ISO.

Checklist

Delete any items that are not applicable to this PR.

Unit or functional tests were updated or added to match the most common scenarios

elasticmachine · 2020-03-19T21:49:17Z

Pinging @elastic/kibana-alerting-services (Team:Alerting Services)

resolves elastic#59889 The index threshold APIs - used by both the index threshold UI and the alert executor - were returning errors (500's from http endpoints) when getting errors from ES. These have been changed so that the error is logged as a warning, and the relevant API returns an "empty" result. Another 500 response was found while experimenting with this. Apparently the date_range agg requires a date format to be passed in if the date format in ES is not an ISO date. The repro on this was to select the `.security` alias (or it's index) within the index threshold alert UI, and then select one of it's date fields.

pmuellr · 2020-03-20T12:46:47Z

x-pack/plugins/alerting_builtins/server/alert_types/index_threshold/lib/time_series_query.ts

@@ -94,6 +94,7 @@ export async function timeSeriesQuery(
    dateAgg: {
      date_range: {
        field: timeField,
+        format: 'strict_date_time',


This was added to indicate the format of the dates that we pass in as the date range values. Without it, an error will occur if the dates in the index are not in this format, eg epoch_millis.

pmuellr · 2020-03-20T12:47:50Z

x-pack/plugins/alerting_builtins/server/alert_types/index_threshold/lib/time_series_query.ts

@@ -134,8 +135,8 @@ export async function timeSeriesQuery(
    esResult = await callCluster('search', esQuery);
  } catch (err) {
    // console.log('time_series_query.ts error\n', JSON.stringify(err, null, 4));
-    logger.warn(`${logPrefix} error: ${JSON.stringify(err.message)}`);
-    throw new Error('error running search');


Instead of throwing an error, just return an empty result. Also improve the log message (doesn't need to be stringified).

pmuellr · 2020-03-20T12:49:17Z

x-pack/plugins/alerting_builtins/server/alert_types/index_threshold/routes/time_series_query.ts


-    let result: TimeSeriesResult;
-    try {


try/catch was removed, as we now catch the only directly throwable API (above), so anything else would be a JS runtime error.

pmuellr · 2020-03-20T12:50:22Z

...tegration/spaces_only/tests/alerting/builtin_alert_types/index_threshold/create_test_data.ts

@@ -53,6 +53,7 @@ async function createEsDocument(es: any, epochMillis: number, testedValue: numbe
    source: DOCUMENT_SOURCE,
    reference: DOCUMENT_REFERENCE,
    date: new Date(epochMillis).toISOString(),
+    date_epoch_millis: epochMillis,


added a new date field, but in epoch millis format, to test using it as a time field

pmuellr · 2020-03-20T12:53:20Z

...ntegration/spaces_only/tests/alerting/builtin_alert_types/index_threshold/fields_endpoint.ts

@@ -139,6 +139,12 @@ export default function fieldsEndpointTests({ getService }: FtrProviderContext)
      expect(field.name).to.eql('updated_at');
      expect(field.type).to.eql('date');
    });
+
+    // TODO: the pattern '*a:b,c:d*' throws an exception in dev, but not ci!


This is a bit weird. Seems like FT is running ES in some mode where bad cluster names won't result in exceptions, but running yarn es will. If you enter the pattern above when running in dev (yarn es, yarn start), you'll see the exception logged in the console, but you don't see it in in FT.

mikecote

LGTM!

YulNaumenko

LGTM

pmuellr · 2020-03-20T16:12:23Z

@elasticmachine merge upstream

pmuellr · 2020-03-20T18:01:40Z

@elasticmachine merge upstream

kibanamachine · 2020-03-20T19:48:55Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request
Commit: 3f8d421

History

💔 Build #34966 failed c36a95c
💚 Build #34788 succeeded 8e78f89
💔 Build #34768 failed 51303276868544062e7f3faa36143fd2e0ff0653
💚 Build #34748 succeeded eb1540a85ec2123d146113749cd4b133ddedc0ed
💔 Build #34725 failed 21b90562baab53877868a77f0dd05db4e6110f88

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

) resolves elastic#59889 The index threshold APIs - used by both the index threshold UI and the alert executor - were returning errors (500's from http endpoints) when getting errors from ES. These have been changed so that the error is logged as a warning, and the relevant API returns an "empty" result. Another 500 response was found while experimenting with this. Apparently the date_range agg requires a date format to be passed in if the date format in ES is not an ISO date. The repro on this was to select the `.security` alias (or it's index) within the index threshold alert UI, and then select one of it's date fields.

…60813) resolves #59889 The index threshold APIs - used by both the index threshold UI and the alert executor - were returning errors (500's from http endpoints) when getting errors from ES. These have been changed so that the error is logged as a warning, and the relevant API returns an "empty" result. Another 500 response was found while experimenting with this. Apparently the date_range agg requires a date format to be passed in if the date format in ES is not an ISO date. The repro on this was to select the `.security` alias (or it's index) within the index threshold alert UI, and then select one of it's date fields.

pmuellr added Feature:Alerting v8.0.0 release_note:skip Skip the PR/issue when compiling release notes v7.7.0 Team:ResponseOps Label for the ResponseOps team (formerly the Cases and Alerting teams) labels Mar 19, 2020

pmuellr force-pushed the alerting/it-index-pattern-500 branch from 389055e to 5130327 Compare March 20, 2020 00:49

pmuellr force-pushed the alerting/it-index-pattern-500 branch from 5130327 to 8e78f89 Compare March 20, 2020 05:29

pmuellr commented Mar 20, 2020

View reviewed changes

pmuellr marked this pull request as ready for review March 20, 2020 12:53

pmuellr requested a review from a team as a code owner March 20, 2020 12:53

mikecote self-requested a review March 20, 2020 13:33

mikecote approved these changes Mar 20, 2020

View reviewed changes

YulNaumenko approved these changes Mar 20, 2020

View reviewed changes

Merge branch 'master' into alerting/it-index-pattern-500

c36a95c

Merge branch 'master' into alerting/it-index-pattern-500

3f8d421

pmuellr merged commit 5efd59b into elastic:master Mar 20, 2020

pmuellr mentioned this pull request Mar 20, 2020

[7.x] [Alerting]: harden APIs of built-in alert index-threshold (#60702) #60813

Merged

pmuellr added the backported label Mar 20, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Alerting]: harden APIs of built-in alert index-threshold #60702

[Alerting]: harden APIs of built-in alert index-threshold #60702

pmuellr commented Mar 19, 2020 •

edited

Loading

elasticmachine commented Mar 19, 2020

pmuellr Mar 20, 2020

pmuellr Mar 20, 2020

pmuellr Mar 20, 2020

pmuellr Mar 20, 2020

pmuellr Mar 20, 2020

mikecote left a comment

YulNaumenko left a comment

pmuellr commented Mar 20, 2020

pmuellr commented Mar 20, 2020

kibanamachine commented Mar 20, 2020

[Alerting]: harden APIs of built-in alert index-threshold #60702

[Alerting]: harden APIs of built-in alert index-threshold #60702

Conversation

pmuellr commented Mar 19, 2020 • edited Loading

Checklist

elasticmachine commented Mar 19, 2020

pmuellr Mar 20, 2020

Choose a reason for hiding this comment

pmuellr Mar 20, 2020

Choose a reason for hiding this comment

pmuellr Mar 20, 2020

Choose a reason for hiding this comment

pmuellr Mar 20, 2020

Choose a reason for hiding this comment

pmuellr Mar 20, 2020

Choose a reason for hiding this comment

mikecote left a comment

Choose a reason for hiding this comment

YulNaumenko left a comment

Choose a reason for hiding this comment

pmuellr commented Mar 20, 2020

pmuellr commented Mar 20, 2020

kibanamachine commented Mar 20, 2020

💚 Build Succeeded

History

pmuellr commented Mar 19, 2020 •

edited

Loading