Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove Monitoring ingest pipelines #77459

Merged
merged 9 commits into from
Sep 28, 2021
2 changes: 2 additions & 0 deletions docs/reference/migration/migrate_8_0.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,7 @@ coming[8.0.0]
* <<breaking_80_indices_changes>>
* <<breaking_80_java_changes>>
* <<breaking_80_mappings_changes>>
* <<breaking_80_monitoring_changes>>
* <<breaking_80_network_changes>>
* <<breaking_80_node_changes>>
* <<breaking_80_packaging_changes>>
Expand Down Expand Up @@ -100,6 +101,7 @@ include::migrate_8_0/ilm.asciidoc[]
include::migrate_8_0/indices.asciidoc[]
include::migrate_8_0/java.asciidoc[]
include::migrate_8_0/mappings.asciidoc[]
include::migrate_8_0/monitoring.asciidoc[]
include::migrate_8_0/network.asciidoc[]
include::migrate_8_0/node.asciidoc[]
include::migrate_8_0/packaging.asciidoc[]
Expand Down
33 changes: 33 additions & 0 deletions docs/reference/migration/migrate_8_0/monitoring.asciidoc
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
[discreet]
[[breaking_80_monitoring_changes]]
=== Monitoring changes

//NOTE: The notable-breaking-changes tagged regions are re-used in the
//Installation and Upgrade Guide

//tag::notable-breaking-changes[]
.The `use_ingest` setting on Monitoring exporter configurations has been removed.
[%collapsible]
====
*Details* +
The `xpack.monitoring.exporters.*.use_ingest` property was deprecated in 7.16.0 and
has been removed. This parameter controlled the creation of pipelines for monitoring
indices that previously had no function.

*Impact* +
Discontinue the use of the `xpack.monitoring.exporters.*.use_ingest` setting.
====

.The `index.pipeline.master_timeout` setting on Monitoring HTTP exporter configurations has been removed.
[%collapsible]
====
*Details* +
The `xpack.monitoring.exporters.*.index.pipeline.master_timeout` property was
deprecated in 7.16.0. This parameter set the timeout when waiting for the remote
Monitoring cluster to create pipelines. Those pipelines for monitoring indices previously
had no function and are now removed in 8.0.0.

*Impact* +
Discontinue the use of the `xpack.monitoring.exporters.*.index.pipeline.master_timeout` setting.
====
//end::notable-breaking-changes[]
Original file line number Diff line number Diff line change
Expand Up @@ -7,15 +7,10 @@
package org.elasticsearch.xpack.core.monitoring.exporter;

import org.elasticsearch.Version;
import org.elasticsearch.common.Strings;
import org.elasticsearch.common.bytes.BytesReference;
import org.elasticsearch.common.time.DateFormatter;
import org.elasticsearch.common.xcontent.XContentBuilder;
import org.elasticsearch.common.xcontent.XContentType;
import org.elasticsearch.xpack.core.monitoring.MonitoredSystem;
import org.elasticsearch.xpack.core.template.TemplateUtils;

import java.io.IOException;
import java.time.Instant;
import java.util.Locale;

Expand Down Expand Up @@ -60,11 +55,6 @@ public final class MonitoringTemplateUtils {
*/
public static final String[] OLD_TEMPLATE_IDS = { "data", "es", "kibana", "logstash" }; //excluding alerts since 6.x watches use it

/**
* IDs of pipelines that can be used with
*/
public static final String[] PIPELINE_IDS = { TEMPLATE_VERSION, OLD_TEMPLATE_VERSION };

private MonitoringTemplateUtils() { }

/**
Expand Down Expand Up @@ -107,104 +97,6 @@ public static String createEmptyTemplate(final String id) {
return "{\"index_patterns\":[\".monitoring-" + id + "-" + OLD_TEMPLATE_VERSION + "*\"],\"version\":" + LAST_UPDATED_VERSION + "}";
}

/**
* Get a pipeline name for any template ID.
*
* @param id The template identifier.
* @return Never {@code null} {@link String} prefixed by "xpack_monitoring_" and the {@code id}.
* @see #TEMPLATE_IDS
*/
public static String pipelineName(String id) {
return "xpack_monitoring_" + id;
}

/**
* Create a pipeline that allows documents for different template versions to be upgraded.
* <p>
* The expectation is that you will call either {@link Strings#toString(XContentBuilder)} or
* {@link BytesReference#bytes(XContentBuilder)}}.
*
* @param id The API version (e.g., "6") to use
* @param type The type of data you want to format for the request
* @return Never {@code null}. Always an ended-object.
* @throws IllegalArgumentException if {@code apiVersion} is unrecognized
* @see #PIPELINE_IDS
*/
public static XContentBuilder loadPipeline(final String id, final XContentType type) {
switch (id) {
case TEMPLATE_VERSION:
return emptyPipeline(type);
case OLD_TEMPLATE_VERSION:
return pipelineForApiVersion6(type);
}

throw new IllegalArgumentException("unrecognized pipeline API version [" + id + "]");
}

/**
* Create a pipeline to upgrade documents from {@link MonitoringTemplateUtils#OLD_TEMPLATE_VERSION}
* The expectation is that you will call either {@link Strings#toString(XContentBuilder)} or
* {@link BytesReference#bytes(XContentBuilder)}}.
*
* @param type The type of data you want to format for the request
* @return Never {@code null}. Always an ended-object.
* @see #LAST_UPDATED_VERSION
*/
static XContentBuilder pipelineForApiVersion6(final XContentType type) {
try {
return XContentBuilder.builder(type.xContent()).startObject()
.field("description", "This pipeline upgrades documents from the older version of the Monitoring API to " +
"the newer version (" + TEMPLATE_VERSION + ") by fixing breaking " +
"changes in those older documents before they are indexed from the older version (" +
OLD_TEMPLATE_VERSION + ").")
.field("version", LAST_UPDATED_VERSION)
.startArray("processors")
.startObject()
// remove the type
.startObject("script")
.field("source","ctx._type = null" )
.endObject()
.endObject()
.startObject()
// ensure the data lands in the correct index
.startObject("gsub")
.field("field", "_index")
.field("pattern", "(.monitoring-\\w+-)6(-.+)")
.field("replacement", "$1" + TEMPLATE_VERSION + "$2")
.endObject()
.endObject()
.endArray()
.endObject();
} catch (final IOException e) {
throw new RuntimeException("Failed to create pipeline to upgrade from older version [" + OLD_TEMPLATE_VERSION +
"] to the newer version [" + TEMPLATE_VERSION + "].", e);
}
}

/**
* Create an empty pipeline.
* The expectation is that you will call either {@link Strings#toString(XContentBuilder)} or
* {@link BytesReference#bytes(XContentBuilder)}}.
*
* @param type The type of data you want to format for the request
* @return Never {@code null}. Always an ended-object.
* @see #LAST_UPDATED_VERSION
*/
public static XContentBuilder emptyPipeline(final XContentType type) {
try {
// For now: We prepend the API version to the string so that it's easy to parse in the future; if we ever add metadata
// to pipelines, then it would better serve this use case
return XContentBuilder.builder(type.xContent()).startObject()
.field("description", "This is a placeholder pipeline for Monitoring API version " + TEMPLATE_VERSION +
" so that future versions may fix breaking changes.")
.field("version", LAST_UPDATED_VERSION)
.startArray("processors").endArray()
.endObject();
} catch (final IOException e) {
throw new RuntimeException("Failed to create empty pipeline", e);
}
}

/**
* Get the index name given a specific date format, a monitored system and a timestamp.
*
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -210,8 +210,10 @@ public void testDeleteExistingPipeline() throws Exception {
);
assertTrue(
exc.getMessage()
.contains("Could not delete policy [my_policy] because" + " a pipeline is referencing it [my_pipeline, another_pipeline]")
.contains("Could not delete policy [my_policy] because a pipeline is referencing it [")
);
assertTrue(exc.getMessage().contains("another_pipeline"));
assertTrue(exc.getMessage().contains("my_pipeline"));

// delete the pipelines so the policies can be deleted
client().performRequest(new Request("DELETE", "/_ingest/pipeline/my_pipeline"));
Expand Down
Loading