From 5fcd9f865ff15e4c08af804e01520ec393fb5bf1 Mon Sep 17 00:00:00 2001 From: David Kilfoyle Date: Thu, 7 Dec 2023 12:47:31 -0500 Subject: [PATCH 1/3] Document presets for performance tuning to ES output --- ...eet-settings-output-elasticsearch.asciidoc | 112 ++++++++++++++++++ 1 file changed, 112 insertions(+) diff --git a/docs/en/ingest-management/fleet/fleet-settings-output-elasticsearch.asciidoc b/docs/en/ingest-management/fleet/fleet-settings-output-elasticsearch.asciidoc index 6a5b0bfdf..046718822 100644 --- a/docs/en/ingest-management/fleet/fleet-settings-output-elasticsearch.asciidoc +++ b/docs/en/ingest-management/fleet/fleet-settings-output-elasticsearch.asciidoc @@ -81,6 +81,16 @@ output is set in the <>. | When this setting is on, {agent}s use this output to send <> if no other output is set in the <>. Sending monitoring data to a remote {es} cluster is currently not supported. + +// ============================================================================= + +| +[id="es-agent-performance-tuning"] +**Performance tuning** + +| Choose one of the menu options to tune your {agent} performace when sending data to an {es} output. You can optimize for throughput, scale, latency, or select choose a balanced set of performance specifications. Refer to [Performance tuning settings] for details about the setting values and the potential impact on performance. + +You can also use the <> field to set custom values. Note that if you adjust performance settings through the advanced YAML configuration, the **Performance tuning** option automatically changes to `Custom`. |=== [[es-output-settings-yaml-config]] @@ -118,4 +128,106 @@ include::../elastic-agent/configuration/outputs/output-shared-settings.asciidoc[ |=== + +[[es-output-settings-performance-tuning-settings]] +== Performance tuning settings + +.Performance tuning preset values +[cols="1,1,1,1,1"] +|=== +|Configuration |Balanced|Optimized for Throughput |Optimized for Scale |Optimized for Latency + +|`bulk_max_size` +|1600 +|1600 +|1600 +|50 + +|`worker` +|1 +|4 +|1 +|1 + +|`queue.mem.events` +|3200 +|12800 +|3200 +|4100 + +|`flush.min_events` +|1600 +|1600 +|1600 +|2050 + +|`flush.timeout` +|10 +|5 +|20 +|1 + +|`compression` +|1 +|1 +|1 +|1 + +|`idle_timeout` +|3 +|15 +|1 +|60 +|=== + +For descriptions of each setting, refer to <>. For the `queue.mem.events`, `flush.min_events` and `flush.timeout` settings, refer to the {filebeat-ref}/configuring-internal-queue.html[internal queue configuration settings] in the {filebeat} documentation. + +.Potential performance effect +[cols="1,1,1,1,1"] +|=== +|Configuration |Balanced|Optimized for Throughput |Optimized for Scale |Optimized for Latency + +|Stateful throughput +|3x +|5x +|3x +|1x + +|Serverless throughput +|5-10x +|10-20x +|5-10x +|1x + +|Serverless throughput (relative to stateful) +|0.2-0.3x +|0.3-0.5x +|0.2-0.3x +|0.1x + +|Connections +|0.3x +|4x +|0.04x +|1x + +|Network traffic +|0.1x +|0.1x +|0.05x +|0.1x + +|High-throughput queue latency +|1x +|1x +|1x +|1x + +|Low-throughput queue latency +|10x +|5x +|20x +|1x +|=== + :type!: \ No newline at end of file From 94233c55269f69189f4de39454fad01a9b0d7e37 Mon Sep 17 00:00:00 2001 From: David Kilfoyle Date: Thu, 7 Dec 2023 15:53:29 -0500 Subject: [PATCH 2/3] Typo and link fix --- .../fleet/fleet-settings-output-elasticsearch.asciidoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/en/ingest-management/fleet/fleet-settings-output-elasticsearch.asciidoc b/docs/en/ingest-management/fleet/fleet-settings-output-elasticsearch.asciidoc index 046718822..f7f0ab3d4 100644 --- a/docs/en/ingest-management/fleet/fleet-settings-output-elasticsearch.asciidoc +++ b/docs/en/ingest-management/fleet/fleet-settings-output-elasticsearch.asciidoc @@ -88,7 +88,7 @@ Sending monitoring data to a remote {es} cluster is currently not supported. [id="es-agent-performance-tuning"] **Performance tuning** -| Choose one of the menu options to tune your {agent} performace when sending data to an {es} output. You can optimize for throughput, scale, latency, or select choose a balanced set of performance specifications. Refer to [Performance tuning settings] for details about the setting values and the potential impact on performance. +| Choose one of the menu options to tune your {agent} performace when sending data to an {es} output. You can optimize for throughput, scale, latency, or you can choose a balanced (the default) set of performance specifications. Refer to <> for details about the setting values and their potential impact on performance. You can also use the <> field to set custom values. Note that if you adjust performance settings through the advanced YAML configuration, the **Performance tuning** option automatically changes to `Custom`. |=== From 1df7941f6e9ead235001dd4eb85d90e33389a2fe Mon Sep 17 00:00:00 2001 From: David Kilfoyle Date: Mon, 11 Dec 2023 09:51:49 -0500 Subject: [PATCH 3/3] Address review comments --- .../fleet/fleet-settings-output-elasticsearch.asciidoc | 10 ++-------- 1 file changed, 2 insertions(+), 8 deletions(-) diff --git a/docs/en/ingest-management/fleet/fleet-settings-output-elasticsearch.asciidoc b/docs/en/ingest-management/fleet/fleet-settings-output-elasticsearch.asciidoc index f7f0ab3d4..dc952257e 100644 --- a/docs/en/ingest-management/fleet/fleet-settings-output-elasticsearch.asciidoc +++ b/docs/en/ingest-management/fleet/fleet-settings-output-elasticsearch.asciidoc @@ -88,9 +88,9 @@ Sending monitoring data to a remote {es} cluster is currently not supported. [id="es-agent-performance-tuning"] **Performance tuning** -| Choose one of the menu options to tune your {agent} performace when sending data to an {es} output. You can optimize for throughput, scale, latency, or you can choose a balanced (the default) set of performance specifications. Refer to <> for details about the setting values and their potential impact on performance. +| Choose one of the menu options to tune your {agent} performance when sending data to an {es} output. You can optimize for throughput, scale, latency, or you can choose a balanced (the default) set of performance specifications. Refer to <> for details about the setting values and their potential impact on performance. -You can also use the <> field to set custom values. Note that if you adjust performance settings through the advanced YAML configuration, the **Performance tuning** option automatically changes to `Custom`. +You can also use the <> field to set custom values. Note that if you adjust any of the performance settings described in the following **Advanced YAML configuration** section, the **Performance tuning** option automatically changes to `Custom` and cannot be changed. |=== [[es-output-settings-yaml-config]] @@ -199,12 +199,6 @@ For descriptions of each setting, refer to <