[🐛 Bug]: Issues when using selenium-remote-driver and opentelemetry sdk

[valodzka]

What happened?

Selenium java remote web driver currently initialize OptenTelemetrySdk instance by itself and that makes impossible to use pre configured OpenTelemetry instances (exception thrown).

Also it's impossible to use opentelementry-agent with selenium.

How can we reproduce the issue?

// configure opentelementry instance and set it as global
OpenTelemetrySdk.builder()
     .setTracerProvider(sdkTracerProvider)                 
     .setPropagators(ContextPropagators.create(W3CTraceContextPropagator.getInstance()))
     .buildAndRegisterGlobal();

// then try use selenium
new RemoteWebDriver(remoteAddress, capabilities)

To reproduce issue with agent add "-javaagent:path/to/opentelemetry-javaagent.jar" to project using selenium remote and JPMS.

Relevant log output

Wrapped by: java.lang.IllegalStateException: GlobalOpenTelemetry.set has already been called. GlobalOpenTelemetry.set must be called only once before any calls to GlobalOpenTelemetry.get. If you are using the OpenTelemetrySdk, use OpenTel
emetrySdkBuilder.buildAndRegisterGlobal instead. Previous invocation set to cause of this exception.
        at io.opentelemetry.api@1.9.0/io.opentelemetry.api.GlobalOpenTelemetry.set(GlobalOpenTelemetry.java:84)
        at io.opentelemetry.sdk.autoconfigure@1.9.0-alpha/io.opentelemetry.sdk.autoconfigure.OpenTelemetrySdkAutoConfiguration.newOpenTelemetrySdk(OpenTelemetrySdkAutoConfiguration.java:113)
        at io.opentelemetry.sdk.autoconfigure@1.9.0-alpha/io.opentelemetry.sdk.autoconfigure.AutoConfiguredOpenTelemetrySdkBuilder.build(AutoConfiguredOpenTelemetrySdkBuilder.java:177)
        at io.opentelemetry.sdk.autoconfigure@1.9.0-alpha/io.opentelemetry.sdk.autoconfigure.AutoConfiguredOpenTelemetrySdk.initialize(AutoConfiguredOpenTelemetrySdk.java:29)
        at io.opentelemetry.sdk.autoconfigure@1.9.0-alpha/io.opentelemetry.sdk.autoconfigure.OpenTelemetrySdkAutoConfiguration.initialize(OpenTelemetrySdkAutoConfiguration.java:47)
        at org.seleniumhq.selenium.remote_driver/org.openqa.selenium.remote.tracing.opentelemetry.OpenTelemetryTracer.createTracer(OpenTelemetryTracer.java:77)
        at org.seleniumhq.selenium.remote_driver/org.openqa.selenium.remote.tracing.opentelemetry.OpenTelemetryTracer.getInstance(OpenTelemetryTracer.java:59)
        at org.seleniumhq.selenium.remote_driver/org.openqa.selenium.remote.RemoteWebDriver.createTracedExecutorWithTracedHttpClient(RemoteWebDriver.java:151)
        at org.seleniumhq.selenium.remote_driver/org.openqa.selenium.remote.RemoteWebDriver.<init>(RemoteWebDriver.java:146)

Trace when using with java agent:

java.lang.reflect.InvocationTargetException
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:568)
	at io.opentelemetry.javaagent.bootstrap.AgentInitializer.initialize(AgentInitializer.java:40)
	at io.opentelemetry.javaagent.OpenTelemetryAgent.agentmain(OpenTelemetryAgent.java:51)
	at io.opentelemetry.javaagent.OpenTelemetryAgent.premain(OpenTelemetryAgent.java:44)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:568)
	at java.instrument/sun.instrument.InstrumentationImpl.loadClassAndStartAgent(InstrumentationImpl.java:491)
	at java.instrument/sun.instrument.InstrumentationImpl.loadClassAndCallPremain(InstrumentationImpl.java:503)
Caused by: java.lang.NoSuchMethodError: 'io.opentelemetry.sdk.resources.Resource io.opentelemetry.sdk.resources.Resource.create(io.opentelemetry.javaagent.shaded.io.opentelemetry.api.common.Attributes)'
	at io.opentelemetry.javaagent.tooling.AutoVersionResourceProvider.createResource(AutoVersionResourceProvider.java:26)
	at io.opentelemetry.sdk.autoconfigure@1.10.0-alpha/io.opentelemetry.sdk.autoconfigure.ResourceConfiguration.configureResource(ResourceConfiguration.java:42)
	at io.opentelemetry.sdk.autoconfigure@1.10.0-alpha/io.opentelemetry.sdk.autoconfigure.AutoConfiguredOpenTelemetrySdkBuilder.build(AutoConfiguredOpenTelemetrySdkBuilder.java:186)
	at io.opentelemetry.javaagent.tooling.OpenTelemetryInstaller.installAgentTracer(OpenTelemetryInstaller.java:63)
	at io.opentelemetry.javaagent.tooling.OpenTelemetryInstaller.beforeAgent(OpenTelemetryInstaller.java:35)
	at io.opentelemetry.javaagent.tooling.AgentInstaller.runBeforeAgentListeners(AgentInstaller.java:192)
	at io.opentelemetry.javaagent.tooling.AgentInstaller.installBytebuddyAgent(AgentInstaller.java:122)
	at io.opentelemetry.javaagent.tooling.AgentInstaller.installBytebuddyAgent(AgentInstaller.java:100)
	... 13 more

Selenium version

java 4.1.0

[github-actions]

@valodzka, thank you for creating this issue. We will troubleshoot it as soon as we can.

---

Info for maintainers

Triage this issue by using labels.

If information is missing, add a helpful comment and then I-issue-template label.

If the issue is a question, add the I-question label.

If the issue is valid but there is no time to troubleshoot it, consider adding the help wanted label.

If the issue requires changes or fixes from an external project (e.g., ChromeDriver, GeckoDriver, W3C), add the applicable G-* label, and it will provide the correct link and auto-close the issue.

After troubleshooting the issue, please add the R-awaiting answer label.

Thank you!

[valodzka]

Issues with agent was resolved in open-telemetry/opentelemetry-java-instrumentation#5169

Issue with preconfigured still remains. This is two line full reproducer that throws error:

  public static void main(String[] args) {
       GlobalOpenTelemetry.set(OpenTelemetry.noop());
       new RemoteWebDriver(new DesiredCapabilities());
   }

[pujagani]

Thank you for providing the details to help reproduce the problem. I understand the requirement to use OpenTelemetry in your own testing setup.
Selenium uses the AutoConfiguredOpenTelemetrySdk since Selenium 4.0.0-Beta-2. This will allow the user to pass any OpenTelemetry configuration options and they will be picked up the OpenTelemetrySdk instance.
Sharing an example below:

public static void main(String[] args) throws MalformedURLException, InterruptedException {
  //  beta 2 onwards
   System.setProperty("otel.traces.exporter", "jaeger");
   System.setProperty("otel.exporter.jaeger.endpoint","http://localhost:14250");
   System.setProperty("JAEGER_SERVICE_NAME", "client");
   System.setProperty("otel.resource.attributes","service.name=selenium-java-try");

   ImmutableCapabilities capabilities = new ImmutableCapabilities("browserName", "chrome");

   WebDriver driver = new RemoteWebDriver(new URL("http://localhost:4444"), capabilities);
   driver.get("http://www.google.com");
   driver.quit();
 }

}

If you want to instrument Spans and link them to the Selenium spans then the Tracer for the same can be obtained and used as follows:
OpenTelemetryTracer.getInstance().setOpenTelemetryContext(Context.current().with(span));
Please ensure context is set to see properly linked traces.

I hope this helps. Apologies if I have missed a use case you are trying to accomplish, in that case, please provide details about it and I can have look again to help figure it out.

[valodzka]

I understand the requirement to use OpenTelemetry in your own testing setup.

Requirement is to be able configure OpenTelemetry in some advanced way. AutoConfiguredOpenTelemetrySdk is nice but very limited. For example I'm using two traces exporters, but it isn't possible to configure it via auto configuration.

Based on this https://opentelemetry.io/docs/instrumentation/java/manual_instrumentation/

Libraries that want to export telemetry data using OpenTelemetry MUST only depend on the opentelemetry-api package and should never configure or depend on the OpenTelemetry SDK.

I suggest remote driver (which is a library) shouldn't configure opentelemetry at all (or at least provide option to disable this behaviour).

[pujagani]

Thank you for providing the feedback. I now understand the situation better. I was working to ensure we can disable tracing when using the RemoteWebDriver as needed. The changes are present in the pull request shared above.

[ham1]

Thanks for the rapid fix.
I've just discovered this is the cause of our manual + auto-instrumented tests failing when using a Grid in CI but working locally.
Are "snapshots" of trunk published so I can test it out? Or is there a guide on how we might use trunk?
Edit: Snapshots aren't published but 4.1.2 should be out soon, (thanks for the answer on IRC)

[ham1]

I've just tried v4.1.2, however get the following issue when trying to auto-instrument tests using a RemoteWebDriver.
I've tried opentelemetry-java-instrumentation versions 1.10.1, 1.10.0, 1.9.0 and 1.6.0 but all give the same issue.
Is there some config I'm missing?

Jan 31 11:26:28 [ERROR] OpenJDK 64-Bit Server VM warning: Sharing is only supported for boot loader classes because bootstrap classpath has been appended
Jan 31 11:26:29 [ERROR] [otel.javaagent 2022-01-31 11:26:29:055 +0000] [main] INFO io.opentelemetry.javaagent.tooling.VersionLogger - opentelemetry-javaagent - version: 1.10.0
Jan 31 11:26:30 [INFO] Running TestSuite
Jan 31 11:26:35 11:26:31.587 [main] INFO  - Initialized test config envs: java.util.stream.ReferencePipeline$3@1131aead 
log4j:WARN No appenders could be found for logger (org.apache.http.impl.conn.BasicClientConnectionManager).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
Jan 31, 2022 11:26:34 AM org.openqa.selenium.remote.tracing.opentelemetry.OpenTelemetryTracer createTracer
INFO: Using OpenTelemetry for tracing
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
...
Jan 31 11:26:35 [ERROR] Tests run: 9, Failures: 3, Errors: 0, Skipped: 6, Time elapsed: 4.794 s <<< FAILURE! - in TestSuite
Jan 31 11:26:35 [ERROR] z.y.x.Test  Time elapsed: 4.679 s  <<< FAILURE!
Jan 31 11:26:35 org.openqa.selenium.SessionNotCreatedException: 
Jan 31 11:26:35 Could not start a new session. Possible causes are invalid address of the remote server or browser start-up failure.
Jan 31 11:26:35 Build info: version: '4.1.2', revision: '9a5a329c5a'
Jan 31 11:26:35 System info: host: '53fd5145132f', ip: '172.18.0.9', os.name: 'Linux', os.arch: 'amd64', os.version: '5.4.58-32.125.amzn2.x86_64', java.version: '17.0.2'
Jan 31 11:26:35 Driver info: org.openqa.selenium.remote.RemoteWebDriver
Jan 31 11:26:35 Command: [null, newSession {capabilities=[Capabilities {acceptInsecureCerts: true, browserName: chrome, goog:chromeOptions: {args: [--window-size=1200,900, --headless], extensions: []}}], desiredCapabilities=Capabilities {acceptInsecureCerts: true, browserName: chrome, goog:chromeOptions: {args: [--window-size=1200,900, --headless], extensions: []}}}]
Jan 31 11:26:35 Capabilities {}
Jan 31 11:26:35 Caused by: java.lang.IllegalStateException: Context has not been changed

Should I raise a new bug or should this be reopened?

[diemol]

that seems to be a different issue, a new bug report would be nice

[ham1]

Ok, thanks, I'll try to get a smaller repro this afternoon and raise tomorrow.
I'll also try to set enableTracing=false when creating the RemoteWebDriver