Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update autoconfigure to append signal path to otlp http endpoint if n… #3666

Merged
merged 6 commits into from
Oct 5, 2021
Merged
Show file tree
Hide file tree
Changes from 4 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions sdk-extensions/autoconfigure/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -50,9 +50,9 @@ The [OpenTelemetry Protocol (OTLP)](https://github.com/open-telemetry/openteleme
|------------------------------|-----------------------------|---------------------------------------------------------------------------|
| otel.traces.exporter=otlp (default) | OTEL_TRACES_EXPORTER=otlp | Select the OpenTelemetry exporter for tracing (default) |
| otel.metrics.exporter=otlp | OTEL_METRICS_EXPORTER=otlp | Select the OpenTelemetry exporter for metrics |
| otel.exporter.otlp.endpoint | OTEL_EXPORTER_OTLP_ENDPOINT | The OTLP traces and metrics endpoint to connect to. Must be a URL with a scheme of either `http` or `https` based on the use of TLS. Default is `http://localhost:4317`. |
| otel.exporter.otlp.traces.endpoint | OTEL_EXPORTER_OTLP_TRACES_ENDPOINT | The OTLP traces endpoint to connect to. Must be a URL with a scheme of either `http` or `https` based on the use of TLS. Default is `http://localhost:4317`. |
| otel.exporter.otlp.metrics.endpoint | OTEL_EXPORTER_OTLP_METRICS_ENDPOINT | The OTLP metrics endpoint to connect to. Must be a URL with a scheme of either `http` or `https` based on the use of TLS. Default is `http://localhost:4317`. |
| otel.exporter.otlp.endpoint | OTEL_EXPORTER_OTLP_ENDPOINT | The OTLP traces and metrics endpoint to connect to. Must be a URL with a scheme of either `http` or `https` based on the use of TLS. If protocol is `http/protobuf` the version and signal will be appended to the path (e.g. `v1/traces` or `v1/metrics`), if not present already. Default is `http://localhost:4317` when protocol is `grpc`, and `http://localhost:4317/v1/{signal}` when protocol is `http/protobuf`. |
| otel.exporter.otlp.traces.endpoint | OTEL_EXPORTER_OTLP_TRACES_ENDPOINT | The OTLP traces endpoint to connect to. Must be a URL with a scheme of either `http` or `https` based on the use of TLS. Default is `http://localhost:4317` when protocol is `grpc`, and `http://localhost:4317/v1/traces` when protocol is `http/protobuf`. |
| otel.exporter.otlp.metrics.endpoint | OTEL_EXPORTER_OTLP_METRICS_ENDPOINT | The OTLP metrics endpoint to connect to. Must be a URL with a scheme of either `http` or `https` based on the use of TLS. Default is `http://localhost:4317` when protocol is `grpc`, and `http://localhost:4317/v1/metrics` when protocol is `http/protobuf`. |
| otel.exporter.otlp.certificate | OTEL_EXPORTER_OTLP_CERTIFICATE | The path to the file containing trusted certificates to use when verifying an OTLP trace or metric server's TLS credentials. The file should contain one or more X.509 certificates in PEM format. By default the host platform's trusted root certificates are used. |
| otel.exporter.otlp.traces.certificate | OTEL_EXPORTER_OTLP_TRACES_CERTIFICATE | The path to the file containing trusted certificates to use when verifying an OTLP trace server's TLS credentials. The file should contain one or more X.509 certificates in PEM format. By default the host platform's trusted root certificates are used. |
| otel.exporter.otlp.metrics.certificate | OTEL_EXPORTER_OTLP_METRICS_CERTIFICATE | The path to the file containing trusted certificates to use when verifying an OTLP metric server's TLS credentials. The file should contain one or more X.509 certificates in PEM format. By default the host platform's trusted root certificates are used. |
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,8 @@
package io.opentelemetry.sdk.autoconfigure;

import static io.opentelemetry.sdk.autoconfigure.OtlpConfigUtil.DATA_TYPE_METRICS;
import static io.opentelemetry.sdk.autoconfigure.OtlpConfigUtil.PROTOCOL_GRPC;
import static io.opentelemetry.sdk.autoconfigure.OtlpConfigUtil.PROTOCOL_HTTP_PROTOBUF;

import io.opentelemetry.exporter.logging.LoggingMetricExporter;
import io.opentelemetry.exporter.otlp.http.metrics.OtlpHttpMetricExporter;
Expand Down Expand Up @@ -82,7 +84,7 @@ static MetricExporter configureOtlpMetrics(
String protocol = OtlpConfigUtil.getOtlpProtocol(DATA_TYPE_METRICS, config);

MetricExporter exporter;
if (protocol.equals("http/protobuf")) {
if (protocol.equals(PROTOCOL_HTTP_PROTOBUF)) {
try {
ClasspathUtil.checkClassExists(
"io.opentelemetry.exporter.otlp.http.metrics.OtlpHttpMetricExporter",
Expand All @@ -104,7 +106,7 @@ static MetricExporter configureOtlpMetrics(
builder::setTrustedCertificates);

exporter = builder.build();
} else if (protocol.equals("grpc")) {
} else if (protocol.equals(PROTOCOL_GRPC)) {
try {
ClasspathUtil.checkClassExists(
"io.opentelemetry.exporter.otlp.metrics.OtlpGrpcMetricExporter",
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -8,18 +8,23 @@
import io.opentelemetry.sdk.autoconfigure.spi.ConfigProperties;
import io.opentelemetry.sdk.autoconfigure.spi.ConfigurationException;
import java.io.IOException;
import java.net.MalformedURLException;
import java.net.URL;
import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.Paths;
import java.time.Duration;
import java.util.Map;
import java.util.function.BiConsumer;
import java.util.function.Consumer;
import javax.annotation.Nullable;

final class OtlpConfigUtil {

static final String DATA_TYPE_TRACES = "traces";
static final String DATA_TYPE_METRICS = "metrics";
static final String PROTOCOL_GRPC = "grpc";
static final String PROTOCOL_HTTP_PROTOBUF = "http/protobuf";

static String getOtlpProtocol(String dataType, ConfigProperties config) {
String protocol = config.getString("otel.exporter.otlp." + dataType + ".protocol");
Expand All @@ -32,7 +37,7 @@ static String getOtlpProtocol(String dataType, ConfigProperties config) {
if (protocol == null) {
protocol = config.getString("otel.experimental.exporter.otlp.protocol");
}
return (protocol == null) ? "grpc" : protocol;
return (protocol == null) ? PROTOCOL_GRPC : protocol;
}

static void configureOtlpExporterBuilder(
Expand All @@ -43,12 +48,32 @@ static void configureOtlpExporterBuilder(
Consumer<String> setCompression,
Consumer<Duration> setTimeout,
Consumer<byte[]> setTrustedCertificates) {
String endpoint = config.getString("otel.exporter.otlp." + dataType + ".endpoint");
String protocol = getOtlpProtocol(dataType, config);
boolean isHttpProtobuf = protocol.equals(PROTOCOL_HTTP_PROTOBUF);
URL endpoint =
validateEndpoint(
config.getString("otel.exporter.otlp." + dataType + ".endpoint"), isHttpProtobuf);
if (endpoint == null) {
endpoint = config.getString("otel.exporter.otlp.endpoint");
endpoint = validateEndpoint(config.getString("otel.exporter.otlp.endpoint"), isHttpProtobuf);
if (endpoint != null && isHttpProtobuf) {
String path = endpoint.getPath();
String signalPath = signalPath(dataType);
if (!path.endsWith(signalPath)) {
if (!path.endsWith("/")) {
path += "/";
}
path += signalPath;
}
try {
endpoint = new URL(endpoint, path);
} catch (MalformedURLException e) {
throw new ConfigurationException(
"Unexpected exception appending signal path to OTLP endpoint", e);
}
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I took a little liberty from the spec here by appending a / before the signal path if it doesn't exist.

A strict interpretation would cause http://localhost:4317/foo to be transformed to http://localhost:4317/foov1/traces

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think that's OK. If someone needs a path without slash (or something entirely different), they can configure it with the per-signal configuration.
Still, it should probably also be fixed in the spec.

}
}
if (endpoint != null) {
setEndpoint.accept(endpoint);
setEndpoint.accept(endpoint.toString());
}

Map<String, String> headers = config.getMap("otel.exporter.otlp." + dataType + ".headers");
Expand Down Expand Up @@ -92,5 +117,47 @@ static void configureOtlpExporterBuilder(
}
}

@Nullable
private static URL validateEndpoint(@Nullable String endpoint, boolean allowPath) {
if (endpoint == null) {
return null;
}
URL endpointUrl;
try {
endpointUrl = new URL(endpoint);
} catch (MalformedURLException e) {
throw new ConfigurationException("OTLP endpoint must be a valid URL: " + endpoint, e);
}
if (!endpointUrl.getProtocol().equals("http") && !endpointUrl.getProtocol().equals("https")) {
throw new ConfigurationException(
"OTLP endpoint scheme must be http or https: " + endpointUrl.getProtocol());
}
if (endpointUrl.getQuery() != null) {
throw new ConfigurationException(
"OTLP endpoint must not have a query string: " + endpointUrl.getQuery());
}
if (endpointUrl.getRef() != null) {
throw new ConfigurationException(
"OTLP endpoint must not have a fragment: " + endpointUrl.getRef());
}
if (!allowPath && (!endpointUrl.getPath().isEmpty() && !endpointUrl.getPath().equals("/"))) {
throw new ConfigurationException(
"OTLP endpoint must not have a path: " + endpointUrl.getPath());
}
return endpointUrl;
}

private static String signalPath(String dataType) {
switch (dataType) {
case DATA_TYPE_METRICS:
return "v1/metrics";
case DATA_TYPE_TRACES:
return "v1/traces";
default:
throw new IllegalArgumentException(
"Cannot determine signal path for unrecognized data type: " + dataType);
}
}

private OtlpConfigUtil() {}
}
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,8 @@
package io.opentelemetry.sdk.autoconfigure;

import static io.opentelemetry.sdk.autoconfigure.OtlpConfigUtil.DATA_TYPE_TRACES;
import static io.opentelemetry.sdk.autoconfigure.OtlpConfigUtil.PROTOCOL_GRPC;
import static io.opentelemetry.sdk.autoconfigure.OtlpConfigUtil.PROTOCOL_HTTP_PROTOBUF;
import static java.util.stream.Collectors.counting;
import static java.util.stream.Collectors.groupingBy;
import static java.util.stream.Collectors.joining;
Expand Down Expand Up @@ -107,7 +109,7 @@ static SpanExporter configureExporter(
static SpanExporter configureOtlp(ConfigProperties config) {
String protocol = OtlpConfigUtil.getOtlpProtocol(DATA_TYPE_TRACES, config);

if (protocol.equals("http/protobuf")) {
if (protocol.equals(PROTOCOL_HTTP_PROTOBUF)) {
ClasspathUtil.checkClassExists(
"io.opentelemetry.exporter.otlp.http.trace.OtlpHttpSpanExporter",
"OTLP HTTP Trace Exporter",
Expand All @@ -124,7 +126,7 @@ static SpanExporter configureOtlp(ConfigProperties config) {
builder::setTrustedCertificates);

return builder.build();
} else if (protocol.equals("grpc")) {
} else if (protocol.equals(PROTOCOL_GRPC)) {
ClasspathUtil.checkClassExists(
"io.opentelemetry.exporter.otlp.trace.OtlpGrpcSpanExporter",
"OTLP gRPC Trace Exporter",
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -5,11 +5,20 @@

package io.opentelemetry.sdk.autoconfigure;

import static io.opentelemetry.sdk.autoconfigure.OtlpConfigUtil.DATA_TYPE_METRICS;
import static io.opentelemetry.sdk.autoconfigure.OtlpConfigUtil.DATA_TYPE_TRACES;
import static io.opentelemetry.sdk.autoconfigure.OtlpConfigUtil.PROTOCOL_GRPC;
import static io.opentelemetry.sdk.autoconfigure.OtlpConfigUtil.PROTOCOL_HTTP_PROTOBUF;
import static org.assertj.core.api.Assertions.assertThat;
import static org.assertj.core.api.Assertions.assertThatCode;
import static org.assertj.core.api.Assertions.assertThatThrownBy;

import com.google.common.collect.ImmutableMap;
import io.opentelemetry.sdk.autoconfigure.spi.ConfigurationException;
import java.util.Collections;
import java.util.Map;
import java.util.concurrent.atomic.AtomicReference;
import org.assertj.core.api.ThrowableAssert.ThrowingCallable;
import org.junit.jupiter.api.Test;

class OtlpConfigUtilTest {
Expand All @@ -19,7 +28,7 @@ void getOtlpProtocolDefault() {
assertThat(
OtlpConfigUtil.getOtlpProtocol(
DATA_TYPE_TRACES, DefaultConfigProperties.createForTest(Collections.emptyMap())))
.isEqualTo("grpc");
.isEqualTo(PROTOCOL_GRPC);

assertThat(
OtlpConfigUtil.getOtlpProtocol(
Expand Down Expand Up @@ -58,4 +67,121 @@ void getOtlpProtocolDefault() {
"otel.exporter.otlp.traces.protocol", "qux"))))
.isEqualTo("qux");
}

@Test
void configureOtlpExporterBuilder_ValidEndpoints() {
assertThatCode(
configureEndpoint(
ImmutableMap.of("otel.exporter.otlp.endpoint", "http://localhost:4317")))
.doesNotThrowAnyException();
assertThatCode(
configureEndpoint(
ImmutableMap.of("otel.exporter.otlp.endpoint", "http://localhost:4317/")))
.doesNotThrowAnyException();
assertThatCode(
configureEndpoint(ImmutableMap.of("otel.exporter.otlp.endpoint", "http://localhost")))
.doesNotThrowAnyException();
assertThatCode(
configureEndpoint(ImmutableMap.of("otel.exporter.otlp.endpoint", "https://localhost")))
.doesNotThrowAnyException();
assertThatCode(
configureEndpoint(
ImmutableMap.of("otel.exporter.otlp.endpoint", "http://foo:bar@localhost")))
.doesNotThrowAnyException();
assertThatCode(
configureEndpoint(
ImmutableMap.of(
"otel.exporter.otlp.endpoint", "http://localhost:4317/path",
"otel.exporter.otlp.protocol", "http/protobuf")))
.doesNotThrowAnyException();
}

@Test
void configureOtlpExporterBuilder_InvalidEndpoints() {
assertThatThrownBy(
configureEndpoint(ImmutableMap.of("otel.exporter.otlp.endpoint", "/foo/bar")))
.isInstanceOf(ConfigurationException.class)
.hasMessageContaining("OTLP endpoint must be a valid URL:");
assertThatThrownBy(
configureEndpoint(
ImmutableMap.of("otel.exporter.otlp.endpoint", "file://localhost:4317")))
.isInstanceOf(ConfigurationException.class)
.hasMessageContaining("OTLP endpoint scheme must be http or https:");
assertThatThrownBy(
configureEndpoint(
ImmutableMap.of("otel.exporter.otlp.endpoint", "http://localhost:4317?foo=bar")))
.isInstanceOf(ConfigurationException.class)
.hasMessageContaining("OTLP endpoint must not have a query string:");
assertThatThrownBy(
configureEndpoint(
ImmutableMap.of("otel.exporter.otlp.endpoint", "http://localhost:4317#fragment")))
.isInstanceOf(ConfigurationException.class)
.hasMessageContaining("OTLP endpoint must not have a fragment:");
assertThatThrownBy(
configureEndpoint(
ImmutableMap.of(
"otel.exporter.otlp.endpoint", "http://localhost:4317/path",
"otel.exporter.otlp.protocol", "grpc")))
.isInstanceOf(ConfigurationException.class)
.hasMessageContaining("OTLP endpoint must not have a path:");
}

private static ThrowingCallable configureEndpoint(Map<String, String> properties) {
return () ->
OtlpConfigUtil.configureOtlpExporterBuilder(
DATA_TYPE_TRACES,
DefaultConfigProperties.createForTest(properties),
value -> {},
(value1, value2) -> {},
value -> {},
value -> {},
value -> {});
}

@Test
void configureOtlpExporterBuilder_HttpProtobufEndpoint() {
assertThat(configureHttpProtobufEndpoint(DATA_TYPE_TRACES, "http://localhost:4317"))
.isEqualTo("http://localhost:4317/v1/traces");
assertThat(configureHttpProtobufEndpoint(DATA_TYPE_TRACES, "http://localhost:4317/"))
.isEqualTo("http://localhost:4317/v1/traces");
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you add a test for http://localhost:4317/collector/v1/traces -> http://localhost:4317/collector/v1/traces and http://localhost:4317/collector -> http://localhost:4317/collector/v1/traces ?

Copy link
Member

@Oberon00 Oberon00 Sep 28, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think that would violate the spec. Configuring a per-signal endpoint should not append anything, so http://localhost:4317/collector should stay as just that. As worded in the spec, the "/v1/traces" suffix is merely convention for the collector. If I have a backend that does not follow this convention, I should be able to override the per-signal endpoint with the per-signal configuration, without having anything changed in the URL.

assertThat(configureHttpProtobufEndpoint(DATA_TYPE_TRACES, "http://localhost:4317/foo"))
.isEqualTo("http://localhost:4317/foo/v1/traces");
assertThat(configureHttpProtobufEndpoint(DATA_TYPE_TRACES, "http://localhost:4317/foo/"))
.isEqualTo("http://localhost:4317/foo/v1/traces");
assertThat(configureHttpProtobufEndpoint(DATA_TYPE_TRACES, "http://localhost:4317/v1/traces"))
.isEqualTo("http://localhost:4317/v1/traces");
assertThat(configureHttpProtobufEndpoint(DATA_TYPE_TRACES, "http://localhost:4317/v1/metrics"))
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks - this is a descriptive and good case to have here

.isEqualTo("http://localhost:4317/v1/metrics/v1/traces");
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this one makes sense to me (I mean...it's almost certainly wrong, but it's what the user requested). However, the one where we add /v1/traces when it's already on the end seems surprising to me. Yes, it's precisely what the user is requesting, but I suspect this will happen and users will complain. Are we were we don't want to check to make sure that the generic endpoint hasn't already had the /v1/<signal> tacked onto it before duplicating it?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is exactly what the spec is forbidding. Better the users notice the error earlier than much later when they have deployed that config everywhere and want to start using metrics and it ends up at /v1/traces/v1/metrics.
If you suggest that the configuration should be "smart" and detect any signal path (e.g., replacing /v1/metrics with /v1/traces for traces, leaving it in-place for metrics) I suggest a spec PR. I wouldn't be against it, if we can manage to log some warning at least...

Copy link
Member

@Oberon00 Oberon00 Oct 5, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe I misunderstood and you meant the signal-specific thing? That seems like a bug indeed. I'll comment above. #3666 (comment)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, I was suggesting that if the user provides a trace endpoint as the "generic" endpoint, that we don't mess with it for traces, since it's likely(?) that they are only using tracing, and don't care about metrics and what endpoint might have been populated for it.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's currently against the spec and I think delaying the error until the point in time where the user uses any other signal is not good. It would be better to ensure that the (404?) error that occurs with the doubled signal path is easy to find (e.g. by logging).
I could be convinced of the "fully smart" method, stripping off any known trailing signal path from the generic URL, but making only the configured path work seems to be a bad balance.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok. I understand that side of the argument. This is potentially a breaking change for some users, so we should be very sure to call it out very clearly in our release notes/CHANGELOG. @jack-berg Can you craft a CHANGELOG entry for this PR? Thanks!


assertThat(configureHttpProtobufEndpoint(DATA_TYPE_METRICS, "http://localhost:4317"))
.isEqualTo("http://localhost:4317/v1/metrics");
assertThat(configureHttpProtobufEndpoint(DATA_TYPE_METRICS, "http://localhost:4317/"))
.isEqualTo("http://localhost:4317/v1/metrics");
assertThat(configureHttpProtobufEndpoint(DATA_TYPE_METRICS, "http://localhost:4317/foo"))
.isEqualTo("http://localhost:4317/foo/v1/metrics");
assertThat(configureHttpProtobufEndpoint(DATA_TYPE_METRICS, "http://localhost:4317/foo/"))
.isEqualTo("http://localhost:4317/foo/v1/metrics");
assertThat(configureHttpProtobufEndpoint(DATA_TYPE_METRICS, "http://localhost:4317/v1/metrics"))
.isEqualTo("http://localhost:4317/v1/metrics");
assertThat(configureHttpProtobufEndpoint(DATA_TYPE_METRICS, "http://localhost:4317/v1/traces"))
.isEqualTo("http://localhost:4317/v1/traces/v1/metrics");
}

private static String configureHttpProtobufEndpoint(String dataType, String configuredEndpoint) {
AtomicReference<String> endpoint = new AtomicReference<>("");

OtlpConfigUtil.configureOtlpExporterBuilder(
dataType,
DefaultConfigProperties.createForTest(
ImmutableMap.of(
"otel.exporter.otlp.protocol", PROTOCOL_HTTP_PROTOBUF,
"otel.exporter.otlp.endpoint", configuredEndpoint)),
endpoint::set,
(value1, value2) -> {},
value -> {},
value -> {},
value -> {});

return endpoint.get();
}
}
Loading