Error: Port already in use. Version 1.43.0 or higher.

[ajax-boiko-y]

Description

After updating the agent version, the application occasionally fails to start due to a port conflict error. This issue appears to be a race condition where the application’s configured JMX remote port (8856) is temporarily used by jps during startup. This problem was not observed before the agent update.

Observed Behavior

• The application logs indicate a Port already in use error when attempting to bind to the JMX remote port.
• The issue seems transient—sometimes the port becomes available, and the application starts successfully.

Error Details

Error: JMX connector server communication error: service:jmx:rmi://stage-sp-tcp-ip-10-193-8-163:8856 jdk.internal.agent.AgentConfigurationError: java.rmi.server.ExportException: Port already in use: 8856; nested exception is: java.net.BindException: Address already in use at jdk.management.agent/sun.management.jmxremote.ConnectorBootstrap.exportMBeanServer(ConnectorBootstrap.java:841) at jdk.management.agent/sun.management.jmxremote.ConnectorBootstrap.startRemoteConnectorServer(ConnectorBootstrap.java:479) at jdk.management.agent/jdk.internal.agent.Agent.startAgent(Agent.java:447) at jdk.management.agent/jdk.internal.agent.Agent.startAgent(Agent.java:603) Caused by: java.rmi.server.ExportException: Port already in use: 8856; nested exception is: java.net.BindException: Address already in use at java.rmi/sun.rmi.transport.tcp.TCPTransport.listen(TCPTransport.java:356) at java.rmi/sun.rmi.transport.tcp.TCPTransport.exportObject(TCPTransport.java:251) at java.rmi/sun.rmi.transport.tcp.TCPEndpoint.exportObject(TCPEndpoint.java:415) at java.rmi/sun.rmi.transport.LiveRef.exportObject(LiveRef.java:147) at java.rmi/sun.rmi.server.UnicastServerRef.exportObject(UnicastServerRef.java:235) at jdk.management.agent/sun.management.jmxremote.ConnectorBootstrap$PermanentExporter.exportObject(ConnectorBootstrap.java:204) at java.management.rmi/javax.management.remote.rmi.RMIJRMPServerImpl.export(RMIJRMPServerImpl.java:153) at java.management.rmi/javax.management.remote.rmi.RMIJRMPServerImpl.export(RMIJRMPServerImpl.java:138) at java.management.rmi/javax.management.remote.rmi.RMIConnectorServer.start(RMIConnectorServer.java:450) at jdk.management.agent/sun.management.jmxremote.ConnectorBootstrap.exportMBeanServer(ConnectorBootstrap.java:837) ... 3 more

Configuration Details

JAVA_TOOL_OPTIONS includes the following relevant options:
-Dcom.sun.management.jmxremote=true
-Dcom.sun.management.jmxremote.ssl=false
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.port=8855
-Dcom.sun.management.jmxremote.rmi.port=8856
-Dcom.sun.management.jmxremote.local.only=false
-Djava.rmi.server.hostname=localhost
-Djava.net.preferIPv4Stack=true

-javaagent:/usr/local/dd-java-agent/dd-java-agent.jar
-Ddd.env=stage-sp
-Ddd.service=tcp
-Ddd.version=1.1.0.12.RELEASE
-Ddd.trace.enabled=false
-Ddd.jmxfetch.enabled=false
-Ddd.profiling.enabled=true
-Ddd.profiling.ddprof.cpu.enabled=true
-Ddd.profiling.ddprof.wall.enabled=true
-Ddd.profiling.ddprof.alloc.enabled=true
-Ddd.profiling.ddprof.liveheap.enabled=true

Investigation Notes

• During startup, jps temporarily uses ports 8855 and 8856:

COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
jps 31369 root 9u IPv4 102917 0t0 TCP *:8856 (LISTEN)
jps 31369 root 11u IPv4 102918 0t0 TCP *:8855 (LISTEN)

• Once jps releases the ports, the application starts normally.

Possible Cause

The problem might have been introduced by recent changes in the agent or its dependencies. A potential related change: dd-trace-java PR #7971.

Expected Behavior

The application should start consistently without encountering port conflicts.

Additional Information

• Does not occur with the previous agent version.
• The port conflict issue is intermittent and resolves if jps releases the ports in time.

[sebastianfeduniak]

I can confirm. The same issue. The latest working version is 1.42.2

[sebastianfeduniak]

Hello, in the 1.46.0 version the same bug exists.

[mcculls]

Hi @sebastianfeduniak can you confirm that the port conflict goes away when profiling is disabled?

[sebastianfeduniak]

@mcculls Hard to say now TBH. In the previous version, there was a flood of exceptions while starting containers. In the latest version, it's 1 or 2 if any for the same size of fleet. I will observe it more and will get back to you.

[sebastianfeduniak]

Hi @mcculls ! I did more testing today and I see the same issue with profiling disabled.
To be more precise, I set

            {
              "name": "DD_APM_ENABLED",
              "value": "false",
            },

and still observe the same error with port conflict.

[mcculls]

Hi @sebastianfeduniak, where did you set that? That would turn off APM in the main agent, but it wouldn't affect dd-java-agent

Profiling is off by default in dd-java-agent so you likely have either DD_PROFILING_ENABLED=true in your environment or -Ddd.profiling.enabled=true added as a JVM option somewhere: https://docs.datadoghq.com/profiler/enabling/java/?tab=datadogprofiler - changing that setting to false would turn off the profiler.

(The only place in the codebase that might launch jps is in profiling's TempLocationManager)

[sebastianfeduniak]

Hi @mcculls I followed your instructions and can confirm that the port conflict goes away when profiling is disabled.

[mcculls]

Thanks for confirming @sebastianfeduniak - we are making some changes to help mitigate this in the next release.