Ratpack/Netty connect errors not being emitted correctly

[necrolyte2]

Describe the bug
Netty http connections that have connect errors are being reported as a generic connect span tagged with an error vs being reported as a normal http span with details about the http connection and tagged with an error(screenshots attached in code linked below)

Steps to reproduce
https://github.com/necrolyte2/ratpack-kotlin-otel/releases/tag/6.0.0

What did you expect to see?
Netty http spans that have an error related to the connection should emit a regular http span with an error and maybe a message about why it failed.

What did you see instead?
Only a connect span with an error attached

What version are you using?
1.5.0

Environment
Compiler: AdoptOpenJDK 11.0.6

Additional context
Add any other context about the problem here.

[necrolyte2]

I did finally find that you can add OTEL_INSTRUMENTATION_NETTY_ALWAYS_CREATE_CONNECT_SPAN=true to get connect errors with better information, but it still seems that http spans should be emitted with the error vs creating a completely different span name.

[iNikem]

I did finally find that you can add OTEL_INSTRUMENTATION_NETTY_ALWAYS_CREATE_CONNECT_SPAN=true to get connect errors with better information, but it still seems that http spans should be emitted with the error vs creating a completely different span name.

If I remember correctly, we create a separate generic span because in case of connection errors we don't have enough information to create a proper http span with all required semantic attributes.

[trask]

hi @necrolyte2, I think this may be resolved by #4787 which introduced real ratpack http client instrumentation instead of relying on the underlying netty instrumentation which has some limitations like this, can you give this a try with a recent javaagent version and let us know how it goes?

[necrolyte2]

I'm going to be on vacation for few weeks, but will see if somebody on my team can do it otherwise when I get back I'll try to replicate/validate

[necrolyte2]

It seems there is plenty of information available on the error now so 👍

One interesting thing though is that if I use the zipkin exporter you do not get the same information as the Jaeger or OTLP exporters. Not sure if that is a limitation of zipkin or if we need a bug reported for that.

Zipkin

Jaeger

[mateuszrzeszutek]

That is a limitation of Zipkin. Zipkin spans cannot contain event attributes (exceptions are span events in opentelemetry, with e.g. stacktrace attached as an attribute), links or resource information.

[necrolyte2]

Is the expectation that separate CONNECT and RESOLVE spans are emitted vs adding that information into the actual HTTP span?

Probably fine either way at this point, but wondering what other's thoughts are

[mateuszrzeszutek]

We thought that separate spans make slightly more sense in this case because the connection, DNS resolution, SSL handshake are not events, but processes -- they take time, are not instantaneous; and they themselves might contain event information (exceptions).
(There are also netty-specific problems, where a CONNECT span appears before the HTTP span and thus you can't add any info to the HTTP span, but you won't experience it unless you're using raw netty)