Missing metrics when integrating jmx_prometheus_javaagent:1.0.1 with WildFly

[alexciouca]

Hello,

After upgrading the jmx_prometheus_javaagent from v0.20.0 to v1.0.1, I noticed that most of the metrics provided by jboss are missing. For example, all the metrics from jboss.as domain are missing, plus many others.

Would you have any idea what extra configuration do I need to set with the upgrade to v1.0.1 ?

The used configuration YAML is:

rules:
  - pattern: ".*"

[dhoard]

@alexciouca the rules you are using should capture everything.

Are you seeing other metrics?
Is the exporter scrape error metric > 0?
Can you either define an environment variable or Java system property to capture developer debug?

environment variable

JMX_PROMETHEUS_EXPORTER_DEVELOPER_DEBUG=true

Java system property

jmx.prometheus.exporter.developer.debug=true

The output will go to system out.

[alexciouca]

@dhoard the rules should capture everything, but still I am missing quite a few of the metrics after the upgrade.

I am attaching the logs and the metrics after enabling debug, for both the v0.20.0 and the v1.0.1 versions. The difference can be seen, and for example, the metrics for jboss.as:subsystem=datasources ... are missing

logs-jmx1.0.1.txt
metrics-jmx1.0.1.txt
logs-jmx0.20.0.txt
metrics-jmx0.20.0.txt

[alexciouca]

@dhoard In the meantime we have come across #128, which is quite and old ticket, but it is very similar to what we are experiencing here and the workaround presented there seems to work. I confirm that after I added -Djavax.management.builder.initial=io.prometheus.jmx.ContainerNotReadyYet parameter, the metrics are back.

Since this is mainly a workaround, would there be anything that can be done on the jmx exporter side?

[dhoard]

@alexciouca I would try using startDelaySeconds.

This will delay initial metrics collection/prevent calls to ManagementFactory.getPlatformMBeanServer() to allow Wildfly to initialize their custom MBeanServer

startDelaySeconds: 60
rules:
  - pattern: ".*"

The code that gets the MBeanServer hasn't changed between 0.20.0 and 1.0.1.