java.rmi.UnmarshalException while scraping metrics

[faenschi]

hello all

we upgraded the jmx-exporter and collector in our application to a newer version:

• collector from 0.10 to 0.16.1
• jmx_prometheus_javaagent from 0.11.0 to 0.16.1

the goal of our application is to scrape metric of different servers (tomcat, weblogic). it's a standalone app and we do not use any tomcat or weblogic dependencies .

now we are facing some issues while scraping bean metrics. if we use the old dependency, we are able to get the thread metrics from our weblogic server like:

[2022-03-21 15:28:25] FINE    scrape: 'com.bea:ServerRuntime=node1,Name=ThreadPoolRuntime,Type=ThreadPoolRuntime'_'StuckThreadCount': process
[2022-03-21 15:28:25] FINE    scrape: 'com.bea{ServerRuntime=node1, Name=ThreadPoolRuntime, Type=ThreadPoolRuntime}StuckThreadCount': 0
[2022-03-21 15:28:25] FINE    add metric sample: weblogic_threadpool_stuck_thread_count [] [] 0.0

With the new dependencies we get an error:

"java.rmi.UnmarshalException: error unmarshalling return; nested exception is: \n\tjava.lang.ClassNotFoundException: weblogic.management.runtime.ExecuteThread (no security manager: RMI class loader disabled)"
 detail = {ClassNotFoundException@43203} "java.lang.ClassNotFoundException: weblogic.management.runtime.ExecuteThread (no security manager: RMI class loader disabled)"

btw: if i use jconsole i sucessfully retrieve the metrics

Any help is very appreciated!
Regards Angela

[fstab]

Please let me know how exactly you start your application and how you use the javaagent and the collector dependency.

[faenschi]

I defined the dependencies in pom.xml:

<prometheus.jmx.version>0.16.1</prometheus.jmx.version>
<dependency>
	<groupId>io.prometheus.jmx</groupId>
	<artifactId>collector</artifactId>
	<version>${prometheus.jmx.version}</version>
	<exclusions>
		<exclusion>
			<artifactId>simpleclient</artifactId>
			<groupId>io.prometheus</groupId>
		</exclusion>
	</exclusions>
</dependency>
    <dependency>
        <groupId>io.prometheus.jmx</groupId>
        <artifactId>jmx_prometheus_javaagent</artifactId>
<version>${prometheus.jmx.version}</version>
    </dependency>

then the application is started like that:
packagename.LauncherKt -Dvertx-config-path=conf/config.json -Djava.util.logging.config.file=config/logging-linux.properties run packagename.MainVerticle

we also tried using the javaagent directly in our weblogic process like it is described here: https://github.com/prometheus/jmx_exporter
this worked fine. so it could be a classloading problem. I also debugged the JmxScraper class, the difference i saw between the versions is that the attributes are loaded differently. In the old version the method
private void scrapeBean(MBeanServerConnection beanConn, ObjectName mbeanName) loads attribute by attribut and the new one collects them and loads them at once.

With the new version the method protected static Object unmarshalValue(Class<?> type, ObjectInput in)in class UnicastRef
has a javax.management.AttributeList as a type/class argument and then it fails to load the class. the old version uses a primitive type because not a list of attributes is loaded, but the value itself.

[fstab]

jmx_prometheus_javaagent implements a Java agent that can be attached to your application. It is not intended to be used as a library within your application.

You write

we also tried using the javaagent directly in our weblogic process ... this worked fine

So I suggest you use it like this, instead of trying to use it as a dependencies in your application.

[antobaldu]

Hello, sorry but there was a misunderstanding. In out project we don't use javaagent, Angela added in the pox.xml just for test. The pom.xml is:

<prometheus.jmx.version>0.16.1</prometheus.jmx.version>
<dependency>
	<groupId>io.prometheus.jmx</groupId>
	<artifactId>collector</artifactId>
	<version>${prometheus.jmx.version}</version>
</dependency>

And we want to use the jmx exporter like a independent HTTP server , not a java agent.

So with the new version, like Angela wrote, we have this UnmarshalException :

In the new version the attributes are loaded differently. In the old version the method
private void scrapeBean(MBeanServerConnection beanConn, ObjectName mbeanName) loads attribute by attribut and the new one collects them and loads them at once.
With the new version the method protected static Object unmarshalValue(Class<?> type, ObjectInput in)in class UnicastRef
has a javax.management.AttributeList as a type/class argument and then it fails to load the class. the old version uses a primitive type because not a list of attributes is loaded, but the value itself. "

Is it possible to fix this problem?

Regards
Antonella

[faenschi]

@fstab yes antonella is right. this was a wrong config i sent you. we do not use this dependency in our pom.xml . is this an issue you could fix?

[fstab]

Hi, there is no need to implement your own standalone HTTP server. You can download jmx_prometheus_httpserver-0.16.1-jar-with-dependencies.jar from https://repo.maven.apache.org/maven2/io/prometheus/jmx/jmx_prometheus_httpserver/0.16.1/ and run it with

java -jar jmx_prometheus_httpserver-0.16.1-jar-with-dependencies.jar 9000 config.yaml

An example config looks like this (replace hostPort with the host and port to connect to via remote JMX):

hostPort: localhost:7072
rules:
- pattern: ".*"

[faenschi]

Hello @fstab

So i tried to override the JMXScraper class from the io.prometheus.jmx.collector library. I used the old mechanism to load the attributes of a Bean step by step instead of loading a whole list.
And it works like that!

So the problem is, that with the new version of the library 0.16.1 a whole list of attributes of a Mbean is loaded. And if one of these attributes is of a type which is unknown or not a primitive type (in my case a
weblogic class), then loading of all of these values in the list fails. With the old code, just the one attribute fails.

Here you see all attributes and two of them are Weblogic-specific classes, for example weblogic.management.runtime.ExecuteThreads:

So with the new mechanism of loading i will not get any values of these attributes at all.
Hope this helps a bit?

Angela

[faenschi]

@fstab I think this pull request caused the issue above: #238