[jmxfetch] add list_jvms command

[yannmh]

New list_jvms command.

List available Java virtual machines on the system using the Attach
API.

See DataDog/jmxfetch#112

[olivielpeau]

👍

[remh]

Shouldn't the jmxfetch version be updated ?

[olivielpeau]

Won't merge this for 5.10.0 unless there are other JMXFetch-related changes we want to ship with 5.10.0

[iancward]

Any chance of getting this released soon?
I applied the patch to an agent running 5.13.2 and it seems to work; however, /etc/init.d/datadog-agent jmx list_jvms only shows java processes running as the same user (e.g. dd-agent). Does this mean that the agent can only use the JMX Attach connection method for processes that run as the same user as the collector or can the agent monitor processes started by other users and this is simply a limitation of the jvms_list command?

Thanks.

[olivielpeau]

Thanks for your feedback @iancward! I'll work on merging and testing this for the next minor release of the agent (5.17.0).

About the JMXAttach connection method: list_jvms does list all the JVMs it can attach to. JMXFetch can only attach to the jvms run by the same user as the JMXFetch process (dd-agent by default).

Although we don't recommend it, if you need JMXFetch to be able to attach to other processes you can try to run JMXFetch as that other user by:

• changing the agent's supervisor configuration file (/etc/dd-agent/supervisor.conf) and setting that user as the user under [program:jmxfetch]
• setting the correct write permissions on the agent's run/ and log/ directory for that user
• running supervisord as root in /etc/dd-agent/supervisor.conf: user=root under [supervisord] (this has security implications, which is why we don't recommend this)

[iancward]

Thanks for the info @olivielpeau. I did find https://gist.github.com/olivielpeau/80b8c8f9d13d4644faee328ed178fb36 and had tried to use that, but ran into some issues. I added the user for the monitored process into the datadog group (and the datadog user to the monitored process user), and updated the permissions on the /tmp/hsperfdata directories and list_jvms would list the process IDs, but couldn't see the full path, which I need for the search to work.

I ended up publishing JMX via RMI and configuring Datadog to monitor the JVMs that way. It was more configuration to manage, but definitely a lot less screwy business with files, permissions, and users.

[olivielpeau]

Oh, thanks for linking to that gist @iancward, I completely forgot I had written this...

Using RMI probably makes more sense in the long run indeed. Unfortunately the Attach API has a few bugs on Oracle's JVM even when set up correctly, I think the bug you found was one of them...