Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

INGEST: Document Processor Conditional #33388

Merged
merged 6 commits into from
Oct 23, 2018
Merged

INGEST: Document Processor Conditional #33388

Changes from 5 commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
04d1013
INGEST: Document Processor Conditional
original-brownbear Sep 4, 2018
b0d3eb8
Merge remote-tracking branch 'elastic/master' into ingest-conditional…
original-brownbear Sep 5, 2018
1472452
CR: Fix whitespace around comma
original-brownbear Sep 5, 2018
a45e0f6
Move to set proc
original-brownbear Sep 5, 2018
8add065
CR: wording 'map' -> 'object'
original-brownbear Sep 5, 2018
f33f36f
Merge remote-tracking branch 'elastic/master' into ingest-conditional…
original-brownbear Oct 23, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 19 additions & 1 deletion docs/reference/ingest/ingest-node.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -721,12 +721,30 @@ All processors are defined in the following way within a pipeline definition:
// NOTCONSOLE

Each processor defines its own configuration parameters, but all processors have
the ability to declare `tag` and `on_failure` fields. These fields are optional.
the ability to declare `tag`, `on_failure` and `if` fields. These fields are optional.

A `tag` is simply a string identifier of the specific instantiation of a certain
processor in a pipeline. The `tag` field does not affect the processor's behavior,
but is very useful for bookkeeping and tracing errors to specific processors.

The `if` field must contain a script that returns a boolean value. If the script evaluates to `true`
then the processor will be executed for the given document otherwise it will be skipped.
The `if` field takes an object with the script fields defined in <<script-processor, script-options>>
and accesses a read only version of the document via the same `ctx` variable used by scripts in the
<<script-processor>>.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we link to the painless doc ?

Do you think it would useful to also have additional examples (not necessarily the full the processor) of how to implement logstash's ~= ? It seemed to be the most common operator to use for this type of check.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm yea it could make sense to add more examples (I agree, it's way too hard to figure out how to do that in Painless from what we currently have in the docs).

WDYT @rjernst ?


[source,js]
--------------------------------------------------
{
"set": {
"if": "ctx.bar == 'expectedValue'",
"field": "foo",
"value": "bar"
}
}
--------------------------------------------------
// NOTCONSOLE
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is this omitted from tests? Can we make an example that will work?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I figured just with the example config we don't have enough to go by for running anything, we'd need to actually index a document against a concrete pipeline here to make a test out of it right?
(that may be a little confusing given we only want an example of the if field here? Idk, you decide :))

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IIRC we can have extra setup that is hidden from the generated documentation. I think having the examples always "work" is key to keeping the documentation up to date as apis change.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok :) Will look into that tomorrow morning :)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can't add a test for a snipped like this it seems. I get this error:

Execution failed for task ':docs:buildRestTests'.
> reference/ingest/ingest-node.asciidoc[737:745](js)// CONSOLE: Didn't match (?:(?<comment>#.+)|(?:(?:(?<method>GET|PUT|POST|HEAD|OPTIONS|DELETE)\s+(?<pathAndQuery>[^\n]+)(?<body>(?:\n(?!GET|PUT|POST|HEAD|OPTIONS|DELETE|startyaml|#)[^\n]+)+)?)|(?:startyaml(?s)(?<yaml>.+?)(?-s)endyaml)))\n+: {
    "set": {
      "if": "ctx.bar == 'expectedValue'",
      "field": "foo",
      "value": "bar"
    }
  }

if I try anything here. => Seems I at least need a snippet that includes an actual request. That would break with the style of the following docs that also just show a quick outline of the configuration (without tests) for each processor?


See <<handling-failure-in-pipelines>> to learn more about the `on_failure` field and error handling in pipelines.

The <<ingest-info,node info API>> can be used to figure out what processors are available in a cluster.
Expand Down