[Maps] Fix threshold alert issue resolving nested fields #83577

kindsun · 2020-11-17T19:20:54Z

Geo threshold alerts incorrectly handle nested date and geo fields. Since ES queries use dot notation for single keys, the following lodash usage works incorrectly for something like geo.coordinates:

❌ entityHits.hits.hits[0].fields.${geoField}[0] -> entityHits.hits.hits[0].fields.geo.coordinates[0]

Instead, to treat the entire dot-separated field as one field, as is the case in the response, the following should be used:

✔️ entityHits.hits.hits[0].fields["${geoField}"][0] -> entityHits.hits.hits[0].fields["geo.coordinates"][0]

…rt-issue-resolving-nested-fields

elasticmachine · 2020-11-17T19:20:56Z

Pinging @elastic/kibana-gis (Team:Geo)

thomasneirynck

thx for quick fix.

question wrt. test, maybe I'm missing something there though :)

thomasneirynck · 2020-11-17T20:12:01Z

x-pack/plugins/stack_alerts/server/alert_types/geo_threshold/geo_threshold.ts

@@ -38,7 +38,7 @@ export function transformResults(
        return _.map(subBuckets, (subBucket) => {
          const locationFieldResult = _.get(
            subBucket,
-            `entityHits.hits.hits[0].fields.${geoField}[0]`,
+            `entityHits.hits.hits[0].fields["${geoField}"][0]`,


clever hack. I had no idea _.get could work with this syntax.

...ins/stack_alerts/server/alert_types/geo_threshold/tests/es_sample_response_with_nesting.json

nreese · 2020-11-17T20:26:50Z

Does your fix work if the field is an object and if the field has a dot in the name? We had to handle similar problems when pulling fields out of hits for documents source. Below are the test cases we ensured worked.

https://github.com/elastic/kibana/blob/master/x-pack/plugins/maps/common/elasticsearch_util/elasticsearch_geo_utils.test.js#L167

kindsun · 2020-11-17T21:46:30Z

Does your fix work if the field is an object and if the field has a dot in the name? We had to handle similar problems when pulling fields out of hits for documents source. Below are the test cases we ensured worked.

https://github.com/elastic/kibana/blob/master/x-pack/plugins/maps/common/elasticsearch_util/elasticsearch_geo_utils.test.js#L167

Thanks for the heads up! Did some investigation, it works with both fields using a dot, like geo.coordinates or nested like { geo: { coordinates: ...}}. In this case we're grabbing data from the returned values assigned to docvalue_fields. These are specified upfront in the initial query and match both the dot and nested forms on the ES side. By the time we see it for this function, it's always just what we specified: geo.coordinates.

kindsun · 2020-11-17T22:43:21Z

@elasticmachine merge upstream

…-fields

kibanamachine · 2020-11-18T00:29:10Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request
Commit: 96afd8f

Metrics [docs]

Distributable file count

id	before	after	diff
`default`	42848	42849	+1

History

💔 Build #88444 failed ba31a5c
💚 Build #88408 succeeded 2b7e8fb

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>

… (#83605) Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com> Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>

…) (#83606) Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>

…o-node-details * 'master' of github.com:elastic/kibana: (65 commits) update chromedriver dependency to 87 (elastic#83624) [TSVB] use new Search API for rollup search (elastic#83275) [TSVB] Y-axis has number formatting not considering all series formatters in the group (elastic#83438) [Logs UI] Update <LogStream /> internal state when its props change (elastic#83302) Add tag bulk action context menu (elastic#82816) [code coverage] adding plugin to flush coverage data (elastic#83447) [UsageCollection] Expose `KibanaRequest` to explicitly opted-in collectors (elastic#83413) Added eventBus to trigger and listen plotHandler event (elastic#83435) [Runtime fields] Editor phase 1 (elastic#81472) [Maps] Fix threshold alert issue resolving nested fields (elastic#83577) chore(NA): remove usage of unverified es snapshots (elastic#83589) [DOCS] Adds Elastic Contributor Program link (elastic#83561) Upgrade EUI to v30.2.0 (elastic#82730) Don't show loading screen during auto-reload (elastic#83376) Functional tests - fix esArchive mappings with runtime fields (elastic#83530) [deb/rpm] Create keystore after installation (elastic#76465) [rpm] Create default environment file at "/etc/sysconfig/kibana" (elastic#82144) [docker] removes workaround for missing crypto-policies-scripts subpackage (elastic#83455) [ML] Persisted URL state for the Data frame analytics jobs and models pages (elastic#83439) adds xpack.security.authc.selector.enabled setting (elastic#83551) ...

bhavyarm · 2020-12-03T13:56:18Z

@aaronjcaldwell how do I test this please? Thanks!

kindsun · 2020-12-03T14:13:36Z

Hello @bhavyarm! Spinning up a Geo Threshold Alert environment (required to test this fix) is currently a very involved task requiring several steps outlined here. Since this functionality is still experimental, I'm not sure if it's critical for you to test or not. Happy to walk you through the steps offline if needed. We'd likely need about an hour.

bhavyarm · 2020-12-03T15:45:28Z

@aaronjcaldwell oh wow ok. It looks super interesting. will talk to Lee and buzz you. Thanks!

Aaron Caldwell added 3 commits November 17, 2020 12:08

Handle elasticsearch resp dot notation correctly in lodash

c196174

Add test for nested field response data handling

680a5e9

Merge remote-tracking branch 'upstream/master' into fix-threshold-ale…

2b7e8fb

…rt-issue-resolving-nested-fields

kindsun added [Deprecated-Use Team:Presentation]Team:Geo Former Team Label for Geo Team. Now use Team:Presentation Feature:Alerting labels Nov 17, 2020

kindsun requested a review from thomasneirynck November 17, 2020 19:20

kindsun added v7.10.1 v7.11.0 v8.0.0 release_note:fix labels Nov 17, 2020

thomasneirynck approved these changes Nov 17, 2020

View reviewed changes

Review feedback. Remove unneeded hits content in test files

ba31a5c

Merge branch 'master' into fix-threshold-alert-issue-resolving-nested…

96afd8f

…-fields

kindsun merged commit 982639f into elastic:master Nov 18, 2020

kindsun deleted the fix-threshold-alert-issue-resolving-nested-fields branch November 18, 2020 00:48

This was referenced Nov 18, 2020

[7.x] [Maps] Fix threshold alert issue resolving nested fields (#83577) #83605

Merged

[7.10] [Maps] Fix threshold alert issue resolving nested fields (#83577) #83606

Merged

kindsun pushed a commit to kindsun/kibana that referenced this pull request Nov 18, 2020

[Maps] Fix threshold alert issue resolving nested fields (elastic#83577)

910155a

Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>

kindsun pushed a commit that referenced this pull request Nov 18, 2020

[7.10] [Maps] Fix threshold alert issue resolving nested fields (#83577…

7a0330c

…) (#83606) Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>

kindsun mentioned this pull request Nov 18, 2020

[Maps] Geo containment alert V1 completion tasks #80749

Closed

20 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Maps] Fix threshold alert issue resolving nested fields #83577

[Maps] Fix threshold alert issue resolving nested fields #83577

kindsun commented Nov 17, 2020

elasticmachine commented Nov 17, 2020

thomasneirynck left a comment

thomasneirynck Nov 17, 2020

nreese commented Nov 17, 2020

kindsun commented Nov 17, 2020

kindsun commented Nov 17, 2020

kibanamachine commented Nov 18, 2020

bhavyarm commented Dec 3, 2020

kindsun commented Dec 3, 2020

bhavyarm commented Dec 3, 2020

[Maps] Fix threshold alert issue resolving nested fields #83577

[Maps] Fix threshold alert issue resolving nested fields #83577

Conversation

kindsun commented Nov 17, 2020

elasticmachine commented Nov 17, 2020

thomasneirynck left a comment

Choose a reason for hiding this comment

thomasneirynck Nov 17, 2020

Choose a reason for hiding this comment

nreese commented Nov 17, 2020

kindsun commented Nov 17, 2020

kindsun commented Nov 17, 2020

kibanamachine commented Nov 18, 2020

💚 Build Succeeded

Metrics [docs]

Distributable file count

History

bhavyarm commented Dec 3, 2020

kindsun commented Dec 3, 2020

bhavyarm commented Dec 3, 2020