[DOCS] Document range enrich policy #79607

mjmbischoff · 2021-10-21T08:35:54Z

Adding the docs similar to the match policy for the new range policy

Fixed wording

martijnvg · 2021-10-25T11:46:06Z

The yaml test snippets fail with an error and assertion error:

java.lang.AssertionError: java.lang.AssertionError: callback must handle its own exceptions
»  	at org.elasticsearch.client.node.NodeClient.lambda$executeLocally$0(NodeClient.java:105) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»  	at org.elasticsearch.tasks.TaskManager$1.onResponse(TaskManager.java:169) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»  	at org.elasticsearch.tasks.TaskManager$1.onResponse(TaskManager.java:163) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»  	at org.elasticsearch.action.ActionListener$DelegatingActionListener.onResponse(ActionListener.java:184) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»  	at org.elasticsearch.action.admin.indices.get.TransportGetIndexAction.doMasterOperation(TransportGetIndexAction.java:113) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»  	at org.elasticsearch.action.admin.indices.get.TransportGetIndexAction.doMasterOperation(TransportGetIndexAction.java:39) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»  	at org.elasticsearch.action.support.master.info.TransportClusterInfoAction.masterOperation(TransportClusterInfoAction.java:45) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»  	at org.elasticsearch.action.support.master.info.TransportClusterInfoAction.masterOperation(TransportClusterInfoAction.java:24) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»  	at org.elasticsearch.action.support.master.TransportMasterNodeAction.executeMasterOperation(TransportMasterNodeAction.java:97) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»  	at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction.lambda$doStart$3(TransportMasterNodeAction.java:185) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»  	at org.elasticsearch.action.ActionRunnable$2.doRun(ActionRunnable.java:62) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»  	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»  	at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:178) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»  	at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction.doStart(TransportMasterNodeAction.java:185) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»  	at org.elasticsearch.action.support.master.TransportMasterNodeAction.doExecute(TransportMasterNodeAction.java:128) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»  	at org.elasticsearch.action.support.master.TransportMasterNodeAction.doExecute(TransportMasterNodeAction.java:52) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»  	at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:77) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»  	at org.elasticsearch.action.support.ActionFilter$Simple.apply(ActionFilter.java:42) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»  	at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:75) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»  	at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:53) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»  	at org.elasticsearch.tasks.TaskManager.registerAndExecute(TaskManager.java:163) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»  	at org.elasticsearch.client.node.NodeClient.executeLocally(NodeClient.java:100) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»  	at org.elasticsearch.client.node.NodeClient.doExecute(NodeClient.java:80) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»  	at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:375) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»  	at org.elasticsearch.client.FilterClient.doExecute(FilterClient.java:54) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»  	at org.elasticsearch.xpack.enrich.EnrichPolicyRunner$3.doExecute(EnrichPolicyRunner.java:613) ~[?:?]
»  	at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:375) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»  	at org.elasticsearch.client.support.AbstractClient$IndicesAdmin.execute(AbstractClient.java:1254) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»  	at org.elasticsearch.client.support.AbstractClient$IndicesAdmin.getIndex(AbstractClient.java:1304) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»  	at org.elasticsearch.xpack.enrich.EnrichPolicyRunner.run(EnrichPolicyRunner.java:127) ~[?:?]
»  	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:678) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»  	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
»  	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
»  	at java.lang.Thread.run(Thread.java:834) [?:?]
»  Caused by: java.lang.AssertionError: callback must handle its own exceptions
»  	... 34 more
»  Caused by: org.elasticsearch.ElasticsearchException: Field 'range' has type [text] which doesn't appear to be a range type
»  	at org.elasticsearch.xpack.enrich.EnrichPolicyRunner.createRangeEnrichMappingBuilder(EnrichPolicyRunner.java:304) ~[?:?]
»  	at org.elasticsearch.xpack.enrich.EnrichPolicyRunner.resolveEnrichMapping(EnrichPolicyRunner.java:251) ~[?:?]
»  	at org.elasticsearch.xpack.enrich.EnrichPolicyRunner.prepareAndCreateEnrichIndex(EnrichPolicyRunner.java:375) ~[?:?]
»  	at org.elasticsearch.xpack.enrich.EnrichPolicyRunner.lambda$run$0(EnrichPolicyRunner.java:134) ~[?:?]
»  	at org.elasticsearch.action.ActionListener$DelegatingFailureActionListener.onResponse(ActionListener.java:217) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»  	at org.elasticsearch.client.node.NodeClient.lambda$executeLocally$0(NodeClient.java:103) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»  	... 33 more

The assertion error can be fixed by moving prepareAndCreateEnrichIndex(...) method call inside the try block in the run method of EnrichPolicyRunner (to line 130). Fixing this should leave us just with the actual error (Field 'range' has type [text] which doesn't appear to be a range type) and not with many failing docs tests (node gets killed because of the assertion error).

I'm not sure why the actual error occurs. I will need to take a better look.

elasticmachine · 2021-10-25T11:46:30Z

Pinging @elastic/es-data-management (Team:Data Management)

elasticmachine · 2021-10-25T11:46:50Z

Pinging @elastic/es-docs (Team:Docs)

mjmbischoff · 2021-10-25T14:01:55Z

I left the call in the spot where it originally was, interesting as it does look like te other paths could raise exceptions as well. Moving it makes sense, feels a little bit weird to do it as part of this PR?

The remaining error is likely due to // TEST[continued] is missing on the second call, breaking the chain making it 3 separate tests. Since it relies on the mapping to indicate it was an ip range (10.100.0.0/16 gets conservatively mapped as a string by default) Because the chain is broken the document gets indexed with a text field, leading to the error.

… to try catch. Fixes "java.lang.AssertionError: java.lang.AssertionError: callback must handle its own exceptions"

…, which breaks the test.

…y moving to try catch. Fixes "java.lang.AssertionError: java.lang.AssertionError: callback must handle its own exceptions"" -> moving to separate PR. This reverts commit 4d78c8e.

martijnvg · 2021-10-25T14:13:45Z

Moving it makes sense, feels a little bit weird to do it as part of this PR?

Yes, let's do this in a separate PR.

martijnvg · 2021-10-25T14:14:43Z

The remaining error is likely due to // TEST[continued] is missing on the second call, breaking the chain making it 3 separate tests. Since it relies on the mapping to indicate it was an ip range (10.100.0.0/16 gets conservatively mapped as a string by default) Because the chain is broken the document gets indexed with a text field, leading to the error.

Yes, it could be that actual error is caused by another test snippet.

jrodewig

Thanks @mjmbischoff.

I pushed some commits to update the API docs, remove some unneeded snippet comments, publish your example, and clean up some of my own mistakes.

I left some suggestions and comments. My only major outstanding question is what happens if the processor's field contains an array? The docs aren't clear right now.

docs/reference/ingest/range-enrich-policy-type-ex.asciidoc

docs/reference/ingest/apis/enrich/put-enrich-policy.asciidoc

jrodewig

Thanks for clarifying @mjmbischoff. I pushed some additional commits to clean up some of my additions based on your comment and to move the include statements to the right place.

If those changes look okay to you, this LGTM. Since I contributed, I'd like to get final approval from @martijnvg before merging.

docs/reference/ingest/range-enrich-policy-type-ex.asciidoc

martijnvg

Thanks @mjmbischoff and @jrodewig!
This LGTM

* upstream/master: (209 commits) Enforce license expiration (elastic#79671) TSDB: Automatically add timestamp mapper (elastic#79136) [DOCS] `_id` is required for bulk API's `update` action (elastic#79774) EQL: Add optional fields and limit joining keys on non-null values only (elastic#79677) [DOCS] Document range enrich policy (elastic#79607) [DOCS] Fix typos in 8.0 security migration (elastic#79802) Allow listing older repositories (elastic#78244) [ML] track inference model feature usage per node (elastic#79752) Remove IncrementalClusterStateWriter & related code (elastic#79738) Reuse previous indices lookup when possible (elastic#79004) Reduce merging in PersistedClusterStateService (elastic#79793) SQL: Adjust JDBC docs to use milliseconds for timeouts (elastic#79628) Remove endpoint for freezing indices (elastic#78918) [ML] add timeout parameter for DELETE trained_models API (elastic#79739) [ML] wait for .ml-state-write alias to be readable (elastic#79731) [Docs] Update edgengram-tokenizer.asciidoc (elastic#79577) [Test][Transform] fix UpdateTransformActionRequestTests failure (elastic#79787) Limit CS Update Task Description Size (elastic#79443) Apply the reader wrapper on can_match source (elastic#78988) [DOCS] Adds new transform limitation item and a note to the tutorial (elastic#79479) ... # Conflicts: # server/src/main/java/org/elasticsearch/index/IndexMode.java # server/src/test/java/org/elasticsearch/index/TimeSeriesModeTests.java

Adding docs for the range enrich policy Co-authored-by: James Rodewig <40268737+jrodewig@users.noreply.github.com> Co-authored-by: Michael Bischoff <michael.bischoff@elastic.co>

Adding docs for the range enrich policy Co-authored-by: James Rodewig <40268737+jrodewig@users.noreply.github.com>

Adding docs for the range enrich policy

92c5b4c

elasticsearchmachine added v8.0.0 external-contributor Pull request authored by a developer outside the Elasticsearch team labels Oct 21, 2021

mjmbischoff and others added 2 commits October 21, 2021 10:49

Fixing reference to match instead of range

f30a014

Fixed wording

Merge branch 'elastic:master' into range_enrich_docs

f2b6628

mjmbischoff assigned martijnvg Oct 25, 2021

martijnvg added the :Data Management/Ingest Node Execution or management of Ingest Pipelines including GeoIP label Oct 25, 2021

elasticmachine added the Team:Data Management Meta label for data/management team label Oct 25, 2021

martijnvg self-requested a review October 25, 2021 11:46

martijnvg removed their assignment Oct 25, 2021

martijnvg added the >docs General docs changes label Oct 25, 2021

elasticmachine added the Team:Docs Meta label for docs team label Oct 25, 2021

martijnvg added the v7.16.1 label Oct 25, 2021

mjmbischoff added 3 commits October 25, 2021 16:05

Addressing assertion failure, 'downgrading' to normal error by moving…

4d78c8e

… to try catch. Fixes "java.lang.AssertionError: java.lang.AssertionError: callback must handle its own exceptions"

Missing // TEST[continued] breaks chain leading to 3 distinct tests…

6db6165

…, which breaks the test.

Revert "Addressing assertion failure, 'downgrading' to normal error b…

af56557

…y moving to try catch. Fixes "java.lang.AssertionError: java.lang.AssertionError: callback must handle its own exceptions"" -> moving to separate PR. This reverts commit 4d78c8e.

mjmbischoff mentioned this pull request Oct 25, 2021

Addressing assertion failure, 'downgrading' to exception - enrich #79717

Merged

martijnvg requested a review from jrodewig October 25, 2021 15:26

mjmbischoff and others added 6 commits October 25, 2021 18:15

Fixing TESTRESPONSE

471b245

Improving cleanup of other enrich examples

3765972

Remove unneeded snippet comments

58c3773

Update create enrich policy API docs

2aba898

Fix typo

30dda53

Fix xref

81d7117

include docs

48b19cd

jrodewig changed the title ~~Adding docs for the range enrich policy~~ [DOCS] Document range enrich policy Oct 25, 2021

clarify API docs

42df797

jrodewig reviewed Oct 25, 2021

View reviewed changes

jrodewig reviewed Oct 26, 2021

View reviewed changes

docs/reference/ingest/apis/enrich/put-enrich-policy.asciidoc Outdated Show resolved Hide resolved

mjmbischoff and others added 6 commits October 26, 2021 08:01

Processing review comments

348c1a3

Fix API docs desc

935f796

Move includes to right place

450c97d

Clean up API desc + example intro

a43a9c0

fix typos

84e997b

cleanup

335716f

jrodewig approved these changes Oct 26, 2021

View reviewed changes

docs/reference/ingest/range-enrich-policy-type-ex.asciidoc Show resolved Hide resolved

martijnvg approved these changes Oct 26, 2021

View reviewed changes

mjmbischoff merged commit c30ab86 into elastic:master Oct 26, 2021

mjmbischoff deleted the range_enrich_docs branch October 26, 2021 13:15

jrodewig mentioned this pull request Oct 26, 2021

[7.16] [DOCS] Document range enrich policy (#79607) #79829

Merged

jakelandis added v8.0.0-beta1 and removed v8.0.0 labels Oct 27, 2021

danhermann added v7.16.0 and removed v7.16.1 labels Oct 27, 2021

lockewritesdocs pushed a commit to lockewritesdocs/elasticsearch that referenced this pull request Oct 28, 2021

[DOCS] Document range enrich policy (elastic#79607)

7ba79ee

Adding docs for the range enrich policy Co-authored-by: James Rodewig <40268737+jrodewig@users.noreply.github.com>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[DOCS] Document range enrich policy #79607

[DOCS] Document range enrich policy #79607

mjmbischoff commented Oct 21, 2021

martijnvg commented Oct 25, 2021

elasticmachine commented Oct 25, 2021

elasticmachine commented Oct 25, 2021

mjmbischoff commented Oct 25, 2021

martijnvg commented Oct 25, 2021

martijnvg commented Oct 25, 2021 •

edited

Loading

jrodewig left a comment

jrodewig left a comment

martijnvg left a comment

[DOCS] Document range enrich policy #79607

[DOCS] Document range enrich policy #79607

Conversation

mjmbischoff commented Oct 21, 2021

martijnvg commented Oct 25, 2021

elasticmachine commented Oct 25, 2021

elasticmachine commented Oct 25, 2021

mjmbischoff commented Oct 25, 2021

martijnvg commented Oct 25, 2021

martijnvg commented Oct 25, 2021 • edited Loading

jrodewig left a comment

Choose a reason for hiding this comment

jrodewig left a comment

Choose a reason for hiding this comment

martijnvg left a comment

Choose a reason for hiding this comment

martijnvg commented Oct 25, 2021 •

edited

Loading