From 46b1a418a12550ff3a0c2a405927301ad6893251 Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Fri, 22 Dec 2023 14:38:03 -0700 Subject: [PATCH 1/9] Add urldecode processor documentation Signed-off-by: Melissa Vagi --- _ingest-pipelines/processors/urldecode.md | 86 +++++++++++++++++++++++ 1 file changed, 86 insertions(+) create mode 100644 _ingest-pipelines/processors/urldecode.md diff --git a/_ingest-pipelines/processors/urldecode.md b/_ingest-pipelines/processors/urldecode.md new file mode 100644 index 0000000000..41f1d167c7 --- /dev/null +++ b/_ingest-pipelines/processors/urldecode.md @@ -0,0 +1,86 @@ +--- +layout: default +title: URL decode +parent: Ingest processors +nav_order: 320 +--- + +# URL decode processor + +The `url_decode` processor is used to . + +The following is the syntax for the `url_decode` processor: + +```json + +``` +{% include copy-curl.html %} + +## Configuration parameters + +The following table lists the required and optional parameters for the `url_decode` processor. + +Parameter | Required/Optional | Description | +|-----------|-----------|-----------| + + +## Using the processor + +Follow these steps to use the processor in a pipeline. + +### Step 1: Create a pipeline + +The following query creates a pipeline, named , that uses the `url_decode` processor to : + +```json + +``` +{% include copy-curl.html %} + +### Step 2 (Optional): Test the pipeline + +It is recommended that you test your pipeline before you ingest documents. +{: .tip} + +To test the pipeline, run the following query: + +```json + +``` +{% include copy-curl.html %} + +#### Response + +The following example response confirms that the pipeline is working as expected: + +```json + +``` + +### Step 3: Ingest a document + +The following query ingests a document into an index named `testindex1`: + +```json + +``` +{% include copy-curl.html %} + +#### Response + +The request indexes the document into the index and will index all documents with . + +```json + +``` + +### Step 4 (Optional): Retrieve the document + +To retrieve the document, run the following query: + +```json + +``` +{% include copy-curl.html %} + + \ No newline at end of file From 8162c3669fe5b3e1288c745a6ece58953b145f8d Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Thu, 25 Apr 2024 17:26:35 -0600 Subject: [PATCH 2/9] Add text and examples Signed-off-by: Melissa Vagi --- _ingest-pipelines/processors/urldecode.md | 110 +++++++++++++++++++--- 1 file changed, 96 insertions(+), 14 deletions(-) diff --git a/_ingest-pipelines/processors/urldecode.md b/_ingest-pipelines/processors/urldecode.md index 41f1d167c7..1d57a49a1f 100644 --- a/_ingest-pipelines/processors/urldecode.md +++ b/_ingest-pipelines/processors/urldecode.md @@ -7,22 +7,35 @@ nav_order: 320 # URL decode processor -The `url_decode` processor is used to . +The `urldecode` processor is useful for decoding URL-encoded strings that may be present in log data or other text fields. This can make the data more readable and easier to analyze, especially when dealing with URLs or query parameters that contain special characters or spaces. -The following is the syntax for the `url_decode` processor: +The following is the syntax for the `urldecode` processor: ```json - +{ + "urldecode": { + "field": "field_to_decode", + "target_field": "decoded_field" + } +} ``` {% include copy-curl.html %} ## Configuration parameters -The following table lists the required and optional parameters for the `url_decode` processor. +The following table lists the required and optional parameters for the `urldecode` processor. Parameter | Required/Optional | Description | |-----------|-----------|-----------| - +`field` | Required | The field containing the URL-encoded string to be decoded. Supports [template snippets]({{site.url}}{{site.baseurl}}/ingest-pipelines/create-ingest/#template-snippets). | +`target_field` | Required | The field where the decoded string is stored. If not specified, the decoded string is stored in the same field as the original encoded string. Supports [template snippets]({{site.url}}{{site.baseurl}}/ingest-pipelines/create-ingest/#template-snippets). | +`ignore_missing` | Optional | Specifies whether the processor should ignore documents that do not contain the specified `field`. If set to `true`, the processor ignores missing values in the `field` and leaves the `target_field` unchanged. Default is `false`. | +`override_target` | Optional | Determines what happens when `target_field` exists in the document. If set to `true`, the processor overwrites the existing `target_field` value with the new value. If set to `false`, the existing value remains and the processor does not overwrite it. Default is `false`. | +`description` | Optional | A brief description of the processor. | +`if` | Optional | A condition for running the processor. | +`ignore_failure` | Optional | Specifies whether the processor continues execution even if it encounters an error. If set to `true`, failures are ignored. Default is `false`. | +`on_failure` | Optional | A list of processors to run if the processor fails. | +`tag` | Optional | An identifier tag for the processor. Useful for debugging in order to distinguish between processors of the same type. | ## Using the processor @@ -30,10 +43,21 @@ Follow these steps to use the processor in a pipeline. ### Step 1: Create a pipeline -The following query creates a pipeline, named , that uses the `url_decode` processor to : +The following query creates a pipeline named `urldecode_pipeline` that uses the `urldecode` processor to decode the URL-encoded string in the `encoded_url` field and store the decoded string in the `decoded_url` field: ```json - +PUT _ingest/pipeline/urldecode_pipeline +{ + "description": "Decode URL-encoded strings", + "processors": [ + { + "urldecode": { + "field": "encoded_url", + "target_field": "decoded_url" + } + } + ] +} ``` {% include copy-curl.html %} @@ -45,7 +69,16 @@ It is recommended that you test your pipeline before you ingest documents. To test the pipeline, run the following query: ```json - +POST _ingest/pipeline/urldecode_pipeline/_simulate +{ + "docs": [ + { + "_source": { + "encoded_url": "https://example.com/search?q=hello%20world" + } + } + ] +} ``` {% include copy-curl.html %} @@ -54,7 +87,23 @@ To test the pipeline, run the following query: The following example response confirms that the pipeline is working as expected: ```json - +{ + "docs": [ + { + "doc": { + "_index": "_index", + "_id": "_id", + "_source": { + "decoded_url": "https://example.com/search?q=hello world", + "encoded_url": "https://example.com/search?q=hello%20world" + }, + "_ingest": { + "timestamp": "2024-04-25T23:16:44.886165001Z" + } + } + } + ] +} ``` ### Step 3: Ingest a document @@ -62,16 +111,32 @@ The following example response confirms that the pipeline is working as expected The following query ingests a document into an index named `testindex1`: ```json - +PUT testindex1/_doc/1?pipeline=url_decode_pipeline +{ + "encoded_url": "https://example.com/search?q=url%20decode%20test" +} ``` {% include copy-curl.html %} #### Response -The request indexes the document into the index and will index all documents with . +The request indexes the document into the index `testindex1` and will index all documents with the +`encoded_url` field, which will be processed by the `urldecode_pipeline` to populate the `decoded_url` field. ```json - +{ + "_index": "testindex1", + "_id": "1", + "_version": 67, + "result": "updated", + "_shards": { + "total": 2, + "successful": 1, + "failed": 0 + }, + "_seq_no": 68, + "_primary_term": 47 +} ``` ### Step 4 (Optional): Retrieve the document @@ -79,8 +144,25 @@ The request indexes the document into the index and will index all To retrieve the document, run the following query: ```json - +GET testindex1/_doc/1 ``` {% include copy-curl.html %} - \ No newline at end of file +#### Response + +The response includes the original `encoded_url` field and the `decoded_url` field: + +```json +{ + "_index": "testindex1", + "_id": "1", + "_version": 67, + "_seq_no": 68, + "_primary_term": 47, + "found": true, + "_source": { + "decoded_url": "https://example.com/search?q=url decode test", + "encoded_url": "https://example.com/search?q=url%20decode%20test" + } +} +``` From f716a811c8445fea12013adafbe737b20b27acbd Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Fri, 26 Apr 2024 14:24:16 -0600 Subject: [PATCH 3/9] Update urldecode.md Signed-off-by: Melissa Vagi Signed-off-by: Melissa Vagi --- _ingest-pipelines/processors/urldecode.md | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/_ingest-pipelines/processors/urldecode.md b/_ingest-pipelines/processors/urldecode.md index 1d57a49a1f..6f8bfcdfcc 100644 --- a/_ingest-pipelines/processors/urldecode.md +++ b/_ingest-pipelines/processors/urldecode.md @@ -27,10 +27,9 @@ The following table lists the required and optional parameters for the `urldecod Parameter | Required/Optional | Description | |-----------|-----------|-----------| -`field` | Required | The field containing the URL-encoded string to be decoded. Supports [template snippets]({{site.url}}{{site.baseurl}}/ingest-pipelines/create-ingest/#template-snippets). | -`target_field` | Required | The field where the decoded string is stored. If not specified, the decoded string is stored in the same field as the original encoded string. Supports [template snippets]({{site.url}}{{site.baseurl}}/ingest-pipelines/create-ingest/#template-snippets). | +`field` | Required | The field containing the URL-encoded string to be decoded. | +`target_field` | Optional | The field where the decoded string is stored. If not specified, the decoded string is stored in the same field as the original encoded string. | `ignore_missing` | Optional | Specifies whether the processor should ignore documents that do not contain the specified `field`. If set to `true`, the processor ignores missing values in the `field` and leaves the `target_field` unchanged. Default is `false`. | -`override_target` | Optional | Determines what happens when `target_field` exists in the document. If set to `true`, the processor overwrites the existing `target_field` value with the new value. If set to `false`, the existing value remains and the processor does not overwrite it. Default is `false`. | `description` | Optional | A brief description of the processor. | `if` | Optional | A condition for running the processor. | `ignore_failure` | Optional | Specifies whether the processor continues execution even if it encounters an error. If set to `true`, failures are ignored. Default is `false`. | From 59b160ff60151a2834b63d86bfdec674550c9d6d Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Tue, 30 Apr 2024 10:26:36 -0600 Subject: [PATCH 4/9] Update _ingest-pipelines/processors/urldecode.md Co-authored-by: Nathan Bower Signed-off-by: Melissa Vagi --- _ingest-pipelines/processors/urldecode.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_ingest-pipelines/processors/urldecode.md b/_ingest-pipelines/processors/urldecode.md index 6f8bfcdfcc..6a1deeb6ca 100644 --- a/_ingest-pipelines/processors/urldecode.md +++ b/_ingest-pipelines/processors/urldecode.md @@ -7,7 +7,7 @@ nav_order: 320 # URL decode processor -The `urldecode` processor is useful for decoding URL-encoded strings that may be present in log data or other text fields. This can make the data more readable and easier to analyze, especially when dealing with URLs or query parameters that contain special characters or spaces. +The `urldecode` processor is useful for decoding URL-encoded strings in log data or other text fields. This can make the data more readable and easier to analyze, especially when working with URLs or query parameters that contain special characters or spaces. The following is the syntax for the `urldecode` processor: From 85d757d422f6b8390690f65f7f392d3ee2c83ec0 Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Tue, 30 Apr 2024 10:26:47 -0600 Subject: [PATCH 5/9] Update _ingest-pipelines/processors/urldecode.md Co-authored-by: Nathan Bower Signed-off-by: Melissa Vagi --- _ingest-pipelines/processors/urldecode.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_ingest-pipelines/processors/urldecode.md b/_ingest-pipelines/processors/urldecode.md index 6a1deeb6ca..eca747cfde 100644 --- a/_ingest-pipelines/processors/urldecode.md +++ b/_ingest-pipelines/processors/urldecode.md @@ -28,7 +28,7 @@ The following table lists the required and optional parameters for the `urldecod Parameter | Required/Optional | Description | |-----------|-----------|-----------| `field` | Required | The field containing the URL-encoded string to be decoded. | -`target_field` | Optional | The field where the decoded string is stored. If not specified, the decoded string is stored in the same field as the original encoded string. | +`target_field` | Optional | The field in which the decoded string is stored. If not specified, then the decoded string is stored in the same field as the original encoded string. | `ignore_missing` | Optional | Specifies whether the processor should ignore documents that do not contain the specified `field`. If set to `true`, the processor ignores missing values in the `field` and leaves the `target_field` unchanged. Default is `false`. | `description` | Optional | A brief description of the processor. | `if` | Optional | A condition for running the processor. | From 6e0d4b975b2a86a265ce09bc406fb45b1aa48f29 Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Tue, 30 Apr 2024 10:26:56 -0600 Subject: [PATCH 6/9] Update _ingest-pipelines/processors/urldecode.md Co-authored-by: Nathan Bower Signed-off-by: Melissa Vagi --- _ingest-pipelines/processors/urldecode.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_ingest-pipelines/processors/urldecode.md b/_ingest-pipelines/processors/urldecode.md index eca747cfde..91181b3a9f 100644 --- a/_ingest-pipelines/processors/urldecode.md +++ b/_ingest-pipelines/processors/urldecode.md @@ -29,7 +29,7 @@ Parameter | Required/Optional | Description | |-----------|-----------|-----------| `field` | Required | The field containing the URL-encoded string to be decoded. | `target_field` | Optional | The field in which the decoded string is stored. If not specified, then the decoded string is stored in the same field as the original encoded string. | -`ignore_missing` | Optional | Specifies whether the processor should ignore documents that do not contain the specified `field`. If set to `true`, the processor ignores missing values in the `field` and leaves the `target_field` unchanged. Default is `false`. | +`ignore_missing` | Optional | Specifies whether the processor should ignore documents that do not contain the specified `field`. If set to `true`, then the processor ignores missing values in the `field` and leaves the `target_field` unchanged. Default is `false`. | `description` | Optional | A brief description of the processor. | `if` | Optional | A condition for running the processor. | `ignore_failure` | Optional | Specifies whether the processor continues execution even if it encounters an error. If set to `true`, failures are ignored. Default is `false`. | From cde05f3ea6ae072be11cbd91f36e8d545c365e6c Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Tue, 30 Apr 2024 10:27:07 -0600 Subject: [PATCH 7/9] Update _ingest-pipelines/processors/urldecode.md Co-authored-by: Nathan Bower Signed-off-by: Melissa Vagi --- _ingest-pipelines/processors/urldecode.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_ingest-pipelines/processors/urldecode.md b/_ingest-pipelines/processors/urldecode.md index 91181b3a9f..e22131298f 100644 --- a/_ingest-pipelines/processors/urldecode.md +++ b/_ingest-pipelines/processors/urldecode.md @@ -119,7 +119,7 @@ PUT testindex1/_doc/1?pipeline=url_decode_pipeline #### Response -The request indexes the document into the index `testindex1` and will index all documents with the +The request indexes the document into the index `testindex1` and will index all documents containing the `encoded_url` field, which will be processed by the `urldecode_pipeline` to populate the `decoded_url` field. ```json From 4d84af81e04b9960bed67a8bb230b52ac4ee5b7c Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Tue, 30 Apr 2024 10:27:19 -0600 Subject: [PATCH 8/9] Update _ingest-pipelines/processors/urldecode.md Co-authored-by: Nathan Bower Signed-off-by: Melissa Vagi --- _ingest-pipelines/processors/urldecode.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_ingest-pipelines/processors/urldecode.md b/_ingest-pipelines/processors/urldecode.md index e22131298f..5593306105 100644 --- a/_ingest-pipelines/processors/urldecode.md +++ b/_ingest-pipelines/processors/urldecode.md @@ -32,7 +32,7 @@ Parameter | Required/Optional | Description | `ignore_missing` | Optional | Specifies whether the processor should ignore documents that do not contain the specified `field`. If set to `true`, then the processor ignores missing values in the `field` and leaves the `target_field` unchanged. Default is `false`. | `description` | Optional | A brief description of the processor. | `if` | Optional | A condition for running the processor. | -`ignore_failure` | Optional | Specifies whether the processor continues execution even if it encounters an error. If set to `true`, failures are ignored. Default is `false`. | +`ignore_failure` | Optional | Specifies whether the processor continues to run even if it encounters an error. If set to `true`, then failures are ignored. Default is `false`. | `on_failure` | Optional | A list of processors to run if the processor fails. | `tag` | Optional | An identifier tag for the processor. Useful for debugging in order to distinguish between processors of the same type. | From 832e0cf6833f852a95c4d4e183937570a646c8c4 Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Tue, 30 Apr 2024 10:52:32 -0600 Subject: [PATCH 9/9] Update urldecode.md Signed-off-by: Melissa Vagi Signed-off-by: Melissa Vagi --- _ingest-pipelines/processors/urldecode.md | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/_ingest-pipelines/processors/urldecode.md b/_ingest-pipelines/processors/urldecode.md index 5593306105..1736a8469b 100644 --- a/_ingest-pipelines/processors/urldecode.md +++ b/_ingest-pipelines/processors/urldecode.md @@ -104,6 +104,7 @@ The following example response confirms that the pipeline is working as expected ] } ``` +{% include copy-curl.html %} ### Step 3: Ingest a document @@ -119,8 +120,7 @@ PUT testindex1/_doc/1?pipeline=url_decode_pipeline #### Response -The request indexes the document into the index `testindex1` and will index all documents containing the -`encoded_url` field, which will be processed by the `urldecode_pipeline` to populate the `decoded_url` field. +The preceding request indexes the document into the index `testindex1` and indexes all documents containing the `encoded_url` field, which is processed by the `urldecode_pipeline` to populate the `decoded_url` field, as shown in the following response: ```json { @@ -137,6 +137,7 @@ The request indexes the document into the index `testindex1` and will index all "_primary_term": 47 } ``` +{% include copy-curl.html %} ### Step 4 (Optional): Retrieve the document @@ -165,3 +166,4 @@ The response includes the original `encoded_url` field and the `decoded_url` fie } } ``` +{% include copy-curl.html %}