Duplicate HTTP dependency with JAX-RS Client

[umutesen]

Expected behavior

There should not be any duplicate spans exported for a single http request with JAX-RS Client with Jersey.

Actual behavior

Duplicate spans exported for a sing HTTP request. As far as I can see, the only notable difference is InstrumentationLibraryInfo{name=io.opentelemetry.http-url-connection vs InstrumentationLibraryInfo{name=io.opentelemetry.jaxrs-client-2.0

To Reproduce

Steps to reproduce the behavior:

Configure custom http client with javax.ws.rs.client.ClientBuilder.newClient and invoke a GET request.

final ClientConfig clientConfig = new ClientConfig();
    clientConfig.register(MultiPartFeature.class);
    clientConfig.register(json);
    clientConfig.register(JacksonFeature.class);
    clientConfig.property(HttpUrlConnectorProvider.SET_METHOD_WORKAROUND, true);
    if (debugging) {
      clientConfig.register(new LoggingFeature(java.util.logging.Logger.getLogger(LoggingFeature.DEFAULT_LOGGER_NAME), java.util.logging.Level.INFO, LoggingFeature.Verbosity.PAYLOAD_ANY, 1024*50 /* Log payloads up to 50K */));
      clientConfig.property(LoggingFeature.LOGGING_FEATURE_VERBOSITY, LoggingFeature.Verbosity.PAYLOAD_ANY);
      // Set logger to ALL
      java.util.logging.Logger.getLogger(LoggingFeature.DEFAULT_LOGGER_NAME).setLevel(java.util.logging.Level.ALL);
    }
    performAdditionalClientConfiguration(clientConfig);
    return ClientBuilder.newClient(clientConfig);

System information

Please provide the following information:

• SDK Version: Java Agent 3.2.6
• OS type and version: Windows 10
• Application Server type and version (if applicable): JBoss EAP 6.1
• Using spring-boot? Yes
• Additional relevant libraries (with version, if applicable): n/a

Logs

2022-02-18 22:56:31.593Z DEBUG c.m.a.a.internal.exporter.Exporter - exporting span: SpanData{spanContext=ImmutableSpanContext{traceId=e0ea3cdd8bd4e9c27a6daa22b11a9829, spanId=cbb3bc24ed4d8198, traceFlags=01, traceState=ArrayBasedTraceState{entries=[ai-internal-sp, 100]}, remote=false, valid=true}, parentSpanContext=ImmutableSpanContext{traceId=e0ea3cdd8bd4e9c27a6daa22b11a9829, spanId=9f1b0a5d95a74f48, traceFlags=01, traceState=ArrayBasedTraceState{entries=[ai-internal-sp, 100]}, remote=false, valid=true}, resource=Resource{schemaUrl=https://opentelemetry.io/schemas/1.8.0, attributes={host.arch="amd64", host.name="XXXX", os.description="Windows 10 10.0", os.type="windows", process.command_line="C:\Program Files\Zulu\zulu-8\jre;bin;java.exe -Dprogram.name=JBossTools: JBoss Enterprise Application Platform 6.1+ -Xms1024m -Xmx1024m -XX:MaxPermSize=256m -Dorg.jboss.resolver.warning=true -Djava.net.preferIPv4Stack=true -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInterval=3600000 -Djava.awt.headless=true -Dorg.jboss.boot.log.file=C:\Development\svn\JBoss\jboss-eap-6.4\standalone\log\boot.log -Dlogging.configuration=file:C:\Development\svn\JBoss\jboss-eap-6.4\standalone\configuration\logging.properties -Djboss.home.dir=C:\Development\svn\JBoss\jboss-eap-6.4 -Dorg.jboss.logmanager.nocolor=true -Djboss.bind.address.management=localhost -Djava.util.logging.manager=org.jboss.logmanager.LogManager -Xbootclasspath/p:$JBOSS_HOME/modules/system/layers/base/org/jboss/logmanager/main/jboss-logmanager-1.5.4.Final-redhat-1.jar -Djboss.modules.system.pkgs=org.jboss.byteman,com.sun.crypto.provider,org.jboss.logmanager -javaagent:C:\applicationinsights-agent-3.2.6.jar -Dfile.encoding=Cp1252 -Xbootclasspath:C:\Program Files\Zulu\zulu-8\jre\lib\resources.jar;C:\Program Files\Zulu\zulu-8\jre\lib\rt.jar;C:\Program Files\Zulu\zulu-8\jre\lib\jsse.jar;C:\Program Files\Zulu\zulu-8\jre\lib\jce.jar;C:\Program Files\Zulu\zulu-8\jre\lib\charsets.jar;C:\Program Files\Zulu\zulu-8\jre\lib\jfr.jar", process.executable.path="C:\Program Files\Zulu\zulu-8\jre;bin;java.exe", process.pid=21008, process.runtime.description="Azul Systems, Inc. OpenJDK 64-Bit Server VM 25.311-b02", process.runtime.name="OpenJDK Runtime Environment", process.runtime.version="1.8.0_311-b02", service.name="AdviserDashboard.API", telemetry.sdk.language="java", telemetry.sdk.name="opentelemetry", telemetry.sdk.version="1.10.0"}}, instrumentationLibraryInfo=InstrumentationLibraryInfo{name=io.opentelemetry.http-url-connection, version=null, schemaUrl=null}, name=HTTP GET, kind=CLIENT, startEpochNanos=1645224991573912700, endEpochNanos=1645224991581594500, attributes=AttributesMap{data={net.transport=ip_tcp, net.peer.name=xxxxx.test, http.status_code=200, http.user_agent=OpenAPI-Generator/1.0/java, http.flavor=1.1, http.url=http://xxxxxx/, applicationinsights.internal.operation_name=GET /XXXXX/*, [thread.id](http://thread.id/)=97, thread.name=http-localhost/127.0.0.1:8080-1, http.method=GET}, capacity=128, totalAddedValues=10}, totalAttributeCount=10, events=[], totalRecordedEvents=0, links=[], totalRecordedLinks=0, status=ImmutableStatusData{statusCode=UNSET, description=}, hasEnded=true}

2022-02-18 22:56:31.594Z DEBUG c.m.a.a.internal.exporter.Exporter - exporting span: SpanData{spanContext=ImmutableSpanContext{traceId=e0ea3cdd8bd4e9c27a6daa22b11a9829, spanId=6e3d1c041421135f, traceFlags=01, traceState=ArrayBasedTraceState{entries=[ai-internal-sp, 100]}, remote=false, valid=true}, parentSpanContext=ImmutableSpanContext{traceId=e0ea3cdd8bd4e9c27a6daa22b11a9829, spanId=9f1b0a5d95a74f48, traceFlags=01, traceState=ArrayBasedTraceState{entries=[ai-internal-sp, 100]}, remote=false, valid=true}, resource=Resource{schemaUrl=https://opentelemetry.io/schemas/1.8.0, attributes={host.arch="amd64", host.name="XXXX", os.description="Windows 10 10.0", os.type="windows", process.command_line="C:\Program Files\Zulu\zulu-8\jre;bin;java.exe -Dprogram.name=JBossTools: JBoss Enterprise Application Platform 6.1+ -Xms1024m -Xmx1024m -XX:MaxPermSize=256m -Dorg.jboss.resolver.warning=true -Djava.net.preferIPv4Stack=true -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInterval=3600000 -Djava.awt.headless=true -Dorg.jboss.boot.log.file=C:\Development\svn\JBoss\jboss-eap-6.4\standalone\log\boot.log -Dlogging.configuration=file:C:\Development\svn\JBoss\jboss-eap-6.4\standalone\configuration\logging.properties -Djboss.home.dir=C:\Development\svn\JBoss\jboss-eap-6.4 -Dorg.jboss.logmanager.nocolor=true -Djboss.bind.address.management=localhost -Djava.util.logging.manager=org.jboss.logmanager.LogManager -Xbootclasspath/p:$JBOSS_HOME/modules/system/layers/base/org/jboss/logmanager/main/jboss-logmanager-1.5.4.Final-redhat-1.jar -Djboss.modules.system.pkgs=org.jboss.byteman,com.sun.crypto.provider,org.jboss.logmanager -javaagent:C:\applicationinsights-agent-3.2.6.jar -Dfile.encoding=Cp1252 -Xbootclasspath:C:\Program Files\Zulu\zulu-8\jre\lib\resources.jar;C:\Program Files\Zulu\zulu-8\jre\lib\rt.jar;C:\Program Files\Zulu\zulu-8\jre\lib\jsse.jar;C:\Program Files\Zulu\zulu-8\jre\lib\jce.jar;C:\Program Files\Zulu\zulu-8\jre\lib\charsets.jar;C:\Program Files\Zulu\zulu-8\jre\lib\jfr.jar", process.executable.path="C:\Program Files\Zulu\zulu-8\jre;bin;java.exe", process.pid=21008, process.runtime.description="Azul Systems, Inc. OpenJDK 64-Bit Server VM 25.311-b02", process.runtime.name="OpenJDK Runtime Environment", process.runtime.version="1.8.0_311-b02", service.name="AdviserDashboard.API", telemetry.sdk.language="java", telemetry.sdk.name="opentelemetry", telemetry.sdk.version="1.10.0"}}, instrumentationLibraryInfo=InstrumentationLibraryInfo{name=io.opentelemetry.jaxrs-client-2.0, version=null, schemaUrl=null}, name=HTTP GET, kind=CLIENT, startEpochNanos=1645224991573647600, endEpochNanos=1645224991583595900, attributes=AttributesMap{data={http.response_content_length=105, net.transport=ip_tcp, net.peer.name=xxxxx.test, http.status_code=200, http.flavor=1.1, http.url=http://xxxxxx/, applicationinsights.internal.operation_name=GET /XXXXX/*, [thread.id](http://thread.id/)=97, thread.name=http-localhost/127.0.0.1:8080-1, http.method=GET}, capacity=128, totalAddedValues=10}, totalAttributeCount=10, events=[], totalRecordedEvents=0, links=[], totalRecordedLinks=0, status=ImmutableStatusData{statusCode=UNSET, description=}, hasEnded=true}

[trask]

@umutesen thanks for reporting this! I've sent a proposal for addressing this upstream open-telemetry/opentelemetry-java-instrumentation#5430

[trask]

this has been merged into the vnext branch, and will be included in 3.3.0-BETA, let me know if you'd like a snapshot build and I can post one here

[umutesen]

Yes, please! Thank you

[trask]

here you go: https://github.com/microsoft/ApplicationInsights-Java/suites/5656154172/artifacts/185207848 see below

[trask]

sorry! wrong download, here you go: https://github.com/microsoft/ApplicationInsights-Java/suites/5660094239/artifacts/185433031

[umutesen]

Thanks! I can confirm this issue is fixed in 3.3.0-BETA. When are we likely to get the stable version?

[trask]

great to hear! just a guess right now, probably end of April or into May, there are lots of changes in 3.3.0, so needs some time to bake