(new troubleshoot) Elasticsearch Ingest Pipelines #1727
+137
−0
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
🔍 Preview links for changed docs: 🔔 The preview site may take up to 3 minutes to finish building. These links will become live once it completes. |
theletterf
requested changes
Jun 24, 2025
| - id: elasticsearch | ||
| --- | ||
|
|
||
| # Troubleshoot Ingest Pipelines [troubleshooting-pipelines] |
There was a problem hiding this comment.
Suggested change
| # Troubleshoot Ingest Pipelines [troubleshooting-pipelines] | |
| # Troubleshoot Ingest pipelines [troubleshooting-pipelines] |
We use sentence casing for headings.
There was a problem hiding this comment.
Unless this is the proper noun for a feature. Is it?
|
|
||
| {{es}} [Ingest Pipelines](https://www.elastic.co/docs/manage-data/ingest/transform-enrich/ingest-pipelines) allow you to transform data during ingest. Per [write model](https://www.elastic.co/docs/deploy-manage/distributed-architecture/reading-and-writing-documents#basic-write-model), they run from `ingest` [node roles](https://www.elastic.co/docs/deploy-manage/distributed-architecture/clusters-nodes-shards/node-roles) under the `write` [thread pool](https://www.elastic.co/docs/reference/elasticsearch/configuration-reference/thread-pool-settings). | ||
|
|
||
| They can be edited under {{kib}}'s **Stack Management > Ingest Pipelines** and/or {{es}}'s [Modify Pipeline API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-ingest-put-pipeline). They store under {{es}}'s [cluster state](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-cluster-state) as accessed from [List Pipelines](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-ingest-get-pipeline). |
There was a problem hiding this comment.
Suggested change
| They can be edited under {{kib}}'s **Stack Management > Ingest Pipelines** and/or {{es}}'s [Modify Pipeline API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-ingest-put-pipeline). They store under {{es}}'s [cluster state](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-cluster-state) as accessed from [List Pipelines](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-ingest-get-pipeline). | |
| You can edit ingest pipelines under {{kib}}'s **Stack Management > Ingest Pipelines** or from {{es}}'s [Modify Pipeline API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-ingest-put-pipeline). They store under {{es}}'s [cluster state](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-cluster-state) as accessed from [List Pipelines](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-ingest-get-pipeline). |
|
|
||
| They can be edited under {{kib}}'s **Stack Management > Ingest Pipelines** and/or {{es}}'s [Modify Pipeline API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-ingest-put-pipeline). They store under {{es}}'s [cluster state](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-cluster-state) as accessed from [List Pipelines](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-ingest-get-pipeline). | ||
|
|
||
| Ingest Piplines can be [Simulated](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-ingest-simulate) during testing, but after go-live are triggered during event ingest from |
There was a problem hiding this comment.
Suggested change
| Ingest Piplines can be [Simulated](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-ingest-simulate) during testing, but after go-live are triggered during event ingest from | |
| Ingest Pipelines can be [Simulated](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-ingest-simulate) during testing, but after go-live are triggered during event ingest from: |
Comment on lines
+22
to
+24
| * the query parameter `pipeline` flag the [create doc](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-create) or [update doc](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-update_) or [bulk modify docs](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-bulk) API request | ||
| * the ingest target's backing [index setting](https://www.elastic.co/docs/reference/elasticsearch/index-settings/index-modules#dynamic-index-settings) for `index.default_pipeline` and/or `index.final_pipeline` | ||
| * an Ingest Pipeline may sub-call another as a [pipelinen processor](https://www.elastic.co/docs/reference/enrich-processor/pipeline-processor) |
There was a problem hiding this comment.
Suggested change
| * the query parameter `pipeline` flag the [create doc](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-create) or [update doc](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-update_) or [bulk modify docs](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-bulk) API request | |
| * the ingest target's backing [index setting](https://www.elastic.co/docs/reference/elasticsearch/index-settings/index-modules#dynamic-index-settings) for `index.default_pipeline` and/or `index.final_pipeline` | |
| * an Ingest Pipeline may sub-call another as a [pipelinen processor](https://www.elastic.co/docs/reference/enrich-processor/pipeline-processor) | |
| * The query parameter `pipeline` flag the [create doc](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-create) or [update doc](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-update_) or [bulk modify docs](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-bulk) API request. | |
| * Rhe ingest target's backing [index setting](https://www.elastic.co/docs/reference/elasticsearch/index-settings/index-modules#dynamic-index-settings) for `index.default_pipeline` and/or `index.final_pipeline`. | |
| * An Ingest Pipeline may sub-call another as a [pipelinen processor](https://www.elastic.co/docs/reference/enrich-processor/pipeline-processor). |
| You might notice an Ingest Pipeline is not running as performant as possible under load testing from one of the following symptoms. | ||
|
|
||
|
|
||
| ### High CPU [troubleshooting-pipelines-symptoms-cpu] |
There was a problem hiding this comment.
Suggested change
| ### High CPU [troubleshooting-pipelines-symptoms-cpu] | |
| ### High CPU usage [troubleshooting-pipelines-symptoms-cpu] |
|
|
||
| ### Delayed timestamps [troubleshooting-pipelines-symptoms-delayed] | ||
|
|
||
| There can be multiple timestamps associated with a single data event. By default in Elastic-built [Integrations](https://www.elastic.co/docs/reference/integrations), every event [records](https://www.elastic.co/docs/reference/ecs/ecs-principles-implementation#_timestamps) |
There was a problem hiding this comment.
Suggested change
| There can be multiple timestamps associated with a single data event. By default in Elastic-built [Integrations](https://www.elastic.co/docs/reference/integrations), every event [records](https://www.elastic.co/docs/reference/ecs/ecs-principles-implementation#_timestamps) | |
| There can be multiple timestamps associated with a single data event. By default in Elastic-built [Integrations](https://www.elastic.co/docs/reference/integrations), every event [records](https://www.elastic.co/docs/reference/ecs/ecs-principles-implementation#_timestamps) the following: |
Comment on lines
+58
to
+60
| * `@timestamp` for when an event originated | ||
| * `event.created` for when an event first reached an Elastic product | ||
| * `event.ingested` for when an event finished processing through an {{es}} Ingest Pipeline |
There was a problem hiding this comment.
Suggested change
| * `@timestamp` for when an event originated | |
| * `event.created` for when an event first reached an Elastic product | |
| * `event.ingested` for when an event finished processing through an {{es}} Ingest Pipeline | |
| * `@timestamp` for when an event originated. | |
| * `event.created` for when an event first reached an Elastic product. | |
| * `event.ingested` for when an event finished processing through an {{es}} Ingest Pipeline. |
|
|
||
| This reports the overall Ingest Pipelines's statistics as well as statistics for each of its processors. If a pipeline calls a sub-pipeline the parent's statistics will record the total time and not subtract time spent waiting on the other. | ||
|
|
||
| Storing this output into `nodes_stats.json` and then using [thid-party tool JQ](https://jqlang.github.io/jq/) to parse through this JSON, some common views reported during troubleshooting include |
There was a problem hiding this comment.
Suggested change
| Storing this output into `nodes_stats.json` and then using [thid-party tool JQ](https://jqlang.github.io/jq/) to parse through this JSON, some common views reported during troubleshooting include | |
| Storing this output into `nodes_stats.json` and then using [third-party tool JQ](https://jqlang.github.io/jq/) to parse through this JSON, some common views reported during troubleshooting include: |
|
|
||
| Storing this output into `nodes_stats.json` and then using [thid-party tool JQ](https://jqlang.github.io/jq/) to parse through this JSON, some common views reported during troubleshooting include | ||
|
|
||
| * Ingest Pipeline's processor has more than 10% failed events |
There was a problem hiding this comment.
Consider turning these into subheadings of level 4.
| $ cat nodes_stats.json | jq -c '.nodes[]|.name as $node| .ingest.pipelines|to_entries[]| .key as $pipeline| .value.processors[] | to_entries[]|.key as $process| { pipeline:$pipeline, process:$process, node:$node, total:.value.stats.count, failed:.value.stats.failed, failed_percent:(try (100*.value.stats.failed/.value.stats.count|round) catch 0)}| select(.total>0 and .failed_percent>10)' | ||
| ``` | ||
|
|
||
| Especially where `ignore_failure` is enabled, this might reflect an incomplete setup and/or wasted processing time. |
There was a problem hiding this comment.
Suggested change
| Especially where `ignore_failure` is enabled, this might reflect an incomplete setup and/or wasted processing time. | |
| Especially where `ignore_failure` is enabled, this might reflect an incomplete setup or wasted processing time. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
👋 howdy, team!
This adds a new page for troubleshooting Elasticsearch's Ingest Pipelines. I tried to avoid repeating existing docs (only a tad in the first section) & then dive into which symptoms you'd notice & then how you'd find your problem child via Node Stats.
This is my second PR in new repo, so I imagine it's going to have pipeline issues, sorry. Kindly confirm link references / such. This guide should also be Dev approved before merging if you'll kindly tag the appropriate team. 🙏
TIA! Stef 🦕