Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

OpenTelemetry integration #981

Closed
wants to merge 78 commits into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
78 commits
Select commit Hold shift + click to select a range
a59fd52
Initial work to integrate OpenTelemtry
dhoard Jun 19, 2024
1ee23d1
Initial OpenTelemetry integration tests
dhoard Jun 19, 2024
3f87236
Added Prometheus container
dhoard Jun 19, 2024
7755e5a
Removed unused code
dhoard Jun 19, 2024
1b56648
Added code to start required containers and test that Prometheus is r…
dhoard Jun 20, 2024
bee7bef
Renamed methods
dhoard Jun 20, 2024
acf0009
Refactored OpenTelemetryTest to allow multi-threaded execution
dhoard Jun 20, 2024
d2a1cb8
Adding more test functionality
dhoard Jun 20, 2024
ea94e19
Updated AntuBLUE test engine
dhoard Jun 20, 2024
c9548ea
Updated AntuBLUE test engine
dhoard Jun 21, 2024
c16c05e
Formatting
dhoard Jun 21, 2024
55e76c5
Removed groupId since it's inherited
dhoard Jun 21, 2024
1655b3a
Added shading configuration. Formatting
dhoard Jun 21, 2024
58bfc2b
Added AntuBLUE Test Engine Extras
dhoard Jun 21, 2024
b78468d
Fixed tests
dhoard Jun 22, 2024
e4a660a
Formatting
dhoard Jun 22, 2024
887db75
Removed unused member variable
dhoard Jun 22, 2024
11779b8
Renamed method. Add code to close response
dhoard Jun 25, 2024
0421bd2
Updated AntuBLUE Test Engine
dhoard Jun 27, 2024
3b02389
Refactored OpenTelemetry tests
dhoard Jun 27, 2024
d8dce81
Code cleanup
dhoard Jun 27, 2024
789649b
Code cleanup
dhoard Jun 27, 2024
038ad85
Fixed spelling
dhoard Jun 27, 2024
2702d30
Renamed test methods
dhoard Jun 27, 2024
96396a1
Refactored common code to abstract base class. Renamed DockerImageNam…
dhoard Jun 27, 2024
0748089
Refactored common code to abstract base class. Renamed DockerImageNam…
dhoard Jun 27, 2024
9fe6822
Updated Javadoc
dhoard Jun 27, 2024
5f80863
Refactored Docker Java image handling
dhoard Jun 27, 2024
6e2002d
Updated README.md
dhoard Jun 27, 2024
a846e5c
Changed script name
dhoard Jun 28, 2024
356cd79
Removed old script
dhoard Jun 28, 2024
b4349f7
Updated README.md
dhoard Jun 28, 2024
66a8eea
Updated AntuBLUE Test Engine
dhoard Jun 28, 2024
09561e0
Code cleanup
dhoard Jun 28, 2024
82bcedf
Updated Javadoc
dhoard Jun 28, 2024
ac9a46a
Code cleanup
dhoard Jun 28, 2024
6710f18
Changed default interval to 60 seconds
dhoard Jun 28, 2024
ec0ecb9
Formatting
dhoard Jun 28, 2024
874c3ea
Changed default protocol to grpc
dhoard Jun 28, 2024
d97a074
Changed code to use default OpenTelemetry default endpoint
dhoard Jun 28, 2024
6db01da
Fixed test engine version
dhoard Jun 28, 2024
5330c27
Updated AntuBLUE Test Engine
dhoard Jul 3, 2024
3717030
Code cleanup
dhoard Jul 3, 2024
39a7ec5
Added OpenTelemtry test for Prometheus 2.51.2
dhoard Jul 3, 2024
50bb660
Changed JavaDockerImages.name() to return a List
dhoard Jul 4, 2024
9800f50
Updated AntuBLUE test engine. Refactored OpenTelemetry tests to use n…
dhoard Jul 9, 2024
dedcacd
Changed poms to apply Spotless pluging during compilation
dhoard Jul 9, 2024
f9b5759
Fixed grammar
dhoard Jul 9, 2024
2852d59
Updated AntuBLUE test engine
dhoard Jul 11, 2024
a96b9c7
Refactored/simplified integration tests
dhoard Jul 12, 2024
e2d46e1
Added support for OpenTelemetry headers
dhoard Jul 12, 2024
4475bb7
Fixed duplicate plugin definition
dhoard Jul 12, 2024
4cac019
Refactored test classes to test OpenTelemetryExporter authentication
dhoard Jul 12, 2024
616ad22
Converted integration tests to use Verifyica
dhoard Jul 17, 2024
fa64310
Added shell scripts
dhoard Jul 20, 2024
e24afac
Changed to use Verifyica test engine
dhoard Jul 20, 2024
4409507
Added Verifyica configuration properties
dhoard Jul 20, 2024
59e18e6
Merge branch 'refactor-integration-tests' into otel-integration
dhoard Jul 20, 2024
d64323a
Updated Verifyica test engine version
dhoard Jul 22, 2024
734d51a
Changed code to use OpenTelemetry default is configuration is not spe…
dhoard Jul 24, 2024
f86cf36
Removed unused static references
dhoard Jul 24, 2024
4c2e541
Changed code so that HTTP / OpenTelemetry modes are exclusive
dhoard Jul 24, 2024
6ef71f6
Added missing class
dhoard Jul 24, 2024
4888cf4
Added new Java test containers
dhoard Jul 24, 2024
ab05361
Added missing resource
dhoard Jul 24, 2024
7ea84cc
Changed jmx_prometheus_httpserver module to jmx_prometheus_standalone
dhoard Jul 25, 2024
566478b
Changed methods for consistency
dhoard Jul 25, 2024
9c61465
Code cleanup
dhoard Jul 26, 2024
50dce7d
Increase max throttle time
dhoard Jul 26, 2024
add9256
Updated Verifyica test engine version
dhoard Jul 26, 2024
6f8a334
Cleaned up argument parsing. Updated tests
dhoard Jul 27, 2024
746ba2e
Updated Verifyica test engine version
dhoard Aug 2, 2024
01bfc92
Changed/fix shell script
dhoard Aug 4, 2024
4cc1369
Fixed usage information
dhoard Aug 4, 2024
a08970b
Changed per code review
dhoard Aug 5, 2024
6c84cbb
Rolled Verifyica version
dhoard Aug 6, 2024
f577cf0
Updated Verifyica test engine version
dhoard Aug 7, 2024
7b98afe
Fixed container startup check
dhoard Aug 8, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .circleci/config.yml
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ jobs:
- checkout
- restore_cache:
key: maven-dependencies-{{ checksum "pom.xml" }}
- run: ./integration_test_suite/docker-pull-images.smoke-test.sh
- run: ./integration_test_suite/pull-smoke-test-docker-images.sh
- run: ./mvnw clean install
- run: ./mvnw javadoc:jar
- save_cache:
Expand Down
1 change: 0 additions & 1 deletion .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -17,4 +17,3 @@ output.log
stress-test.log
test.sh
test.log
.antublue-test-engine.properties
6 changes: 3 additions & 3 deletions MAINTAINER_NOTES.md
Original file line number Diff line number Diff line change
Expand Up @@ -57,19 +57,19 @@ Example

```shell
/home/dhoard/Downloads/jmx_prometheus_javaagent-0.20.0.jar
/home/dhoard/Downloads/jmx_prometheus_httpserver-0.20.0.jar
/home/dhoard/Downloads/jmx_prometheus_standalone-0.20.0.jar
```

Command

```shell
./tools/patch-and-run-integration-test-suite.sh <javaagent.jar> <httpserver.jar>
./tools/patch-and-run-integration-test-suite.sh <javaagent.jar> <standalone.jar>
```

Example

```shell
./tools/patch-and-run-integration-test-suite.sh /home/dhoard/Downloads/jmx_prometheus_javaagent-0.20.0.jar /home/dhoard/Downloads/jmx_prometheus_httpserver-0.20.0.jar
./tools/patch-and-run-integration-test-suite.sh /home/dhoard/Downloads/jmx_prometheus_javaagent-0.20.0.jar /home/dhoard/Downloads/jmx_prometheus_standalone-0.20.0.jar
```

### Step 3
Expand Down
18 changes: 12 additions & 6 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,16 +1,22 @@
[![Build Status](https://circleci.com/gh/prometheus/jmx_exporter.svg?style=svg)](https://circleci.com/gh/prometheus/jmx_exporter)

# JMX Exporter

[![Build Status](https://circleci.com/gh/prometheus/jmx_exporter.svg?style=svg)](https://circleci.com/gh/prometheus/jmx_exporter)
JMX to Prometheus exporter:

JMX to Prometheus exporter: a collector that can configurable scrape and
expose MBeans of a JMX target.
A collector that can configurable scrape and expose MBeans of a JMX target.

This exporter is intended to be run as a Java Agent, exposing a HTTP server
and serving metrics of the local JVM. It can be also run as a standalone
HTTP server and scrape remote JMX targets, but this has various
This exporter is intended to be run as a Java Agent, exposing either
an HTTP endpoint or pushing Open Telemetry metrics of the local JVM.

It can be also run as a standalone server and scrape remote JMX targets, but this has various
disadvantages, such as being harder to configure and being unable to expose
process metrics (e.g., memory and CPU usage).

In particular all the `jvm_*` metrics like `jvm_classes_loaded_total`, `jvm_threads_current`,
`jvm_threads_daemon` and `jvm_memory_bytes_used` won't be available when
using the standalone server.

**Running the exporter as a Java agent is strongly encouraged.**

# Documentation
Expand Down
23 changes: 22 additions & 1 deletion collector/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -48,6 +48,28 @@
</systemPropertyVariables>
</configuration>
</plugin>
<plugin>
<groupId>com.diffplug.spotless</groupId>
<artifactId>spotless-maven-plugin</artifactId>
<version>2.43.0</version>
<configuration>
<java>
<googleJavaFormat>
<version>1.22.0</version>
<style>AOSP</style>
<reflowLongStrings>true</reflowLongStrings>
</googleJavaFormat>
</java>
</configuration>
<executions>
<execution>
<goals>
<goal>apply</goal>
</goals>
<phase>compile</phase>
</execution>
</executions>
</plugin>
</plugins>
</build>

Expand Down Expand Up @@ -89,7 +111,6 @@
<artifactId>prometheus-metrics-instrumentation-jvm</artifactId>
<scope>test</scope>
</dependency>

</dependencies>

</project>
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@
* Metrics being exported:
*
* <pre>
* jmx_exporter_build_info{version="3.2.0",name="jmx_prometheus_httpserver",} 1.0
* jmx_exporter_build_info{version="3.2.0",name="jmx_prometheus_standalone",} 1.0
* </pre>
*/
public class BuildInfoMetrics {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -181,7 +181,7 @@ private void exitOnConfigError() {
LOGGER.log(
SEVERE,
"Configuration error: When running jmx_exporter in standalone mode (using"
+ " jmx_prometheus_httpserver-*.jar) you must configure 'jmxUrl' or"
+ " jmx_prometheus_standalone-*.jar) you must configure 'jmxUrl' or"
+ " 'hostPort'.");
System.exit(-1);
}
Expand Down
39 changes: 22 additions & 17 deletions docs/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,17 +5,22 @@
JMX Exporter
=====

JMX to Prometheus exporter: a collector that can configurable scrape and
expose mBeans of a JMX target.
JMX to Prometheus exporter:

This exporter is intended to be run as a Java Agent, exposing a HTTP server
and serving metrics of the local JVM. It can be also run as a standalone
HTTP server and scrape remote JMX targets, but this has various
A collector that can configurable scrape and expose MBeans of a JMX target.

This exporter is intended to be run as a Java Agent, exposing either
an HTTP endpoint or pushing Open Telemetry metrics of the local JVM.

It can be also run as a standalone server and scrape remote JMX targets, but this has various
disadvantages, such as being harder to configure and being unable to expose
process metrics (e.g., memory and CPU usage). In particular all the
`jvm_*` metrics like `jvm_classes_loaded_total`, `jvm_threads_current`,
`jvm_threads_daemon` and `jvm_memory_bytes_used` won't be availabe if
using the standalone http server.
process metrics (e.g., memory and CPU usage).

In particular all the `jvm_*` metrics like `jvm_classes_loaded_total`, `jvm_threads_current`,
`jvm_threads_daemon` and `jvm_memory_bytes_used` won't be available when
using the standalone server.

**Running the exporter as a Java agent is strongly encouraged.**

### **NOTES**

Expand Down Expand Up @@ -52,21 +57,21 @@ rules:

Example configurations can be found in the `example_configs/` directory.

## Running the Standalone HTTP Server
## Running the Standalone Server

- [jmx_prometheus_httpserver-1.0.1.jar](https://repo1.maven.org/maven2/io/prometheus/jmx/jmx_prometheus_httpserver/1.0.1/jmx_prometheus_httpserver-1.0.1.jar)
- [jmx_prometheus_standalone-1.0.1.jar](https://repo1.maven.org/maven2/io/prometheus/jmx/jmx_prometheus_standalone/1.0.1/jmx_prometheus_standalone-1.0.1.jar)

To run the standalone HTTP server, download one of the JARs and run:
To run the standalone server, download one of the JARs and run:

```
java -jar jmx_prometheus_httpserver-1.0.1.jar 12345 config.yaml
java -jar jmx_prometheus_standalone-1.0.1.jar 12345 config.yaml
```

Metrics will now be accessible at [http://localhost:12345/metrics](http://localhost:12345/metrics).
To bind the java agent to a specific IP change the port number to `host:port`.

The standalone HTTP server will read JMX remotely over the network. Therefore, you need to specify
either `hostPort` or `jmxUrl` in `config.yaml` to tell the HTTP server where the JMX beans can be accessed.
The standalone server will read JMX remotely over the network. Therefore, you need to specify
either `hostPort` or `jmxUrl` in `config.yaml` to tell the server where the JMX beans can be accessed.

A minimal `config.yaml` looks like this:

Expand All @@ -76,7 +81,7 @@ rules:
- pattern: ".*"
```

As stated above, it is recommended to run JMX exporter as a Java agent and not as a standalone HTTP server.
As stated above, it is recommended to run JMX exporter as a Java agent and not as a standalone server.

**NOTES**

Expand Down Expand Up @@ -332,7 +337,7 @@ httpServer:

## Integration Test Suite

The JMX exporter uses the [AntuBLUE Test Engine](https://github.com/antublue/test-engine) and [Testcontainers](https://www.testcontainers.org/) to run integration tests with different Java versions.
The JMX exporter project uses [Verifyica](https://github.com/antublue/verifyica) and [Testcontainers](https://www.testcontainers.org/) to run integration tests with different Java versions.

You need to have Docker installed to run the integration test suite.

Expand Down
53 changes: 41 additions & 12 deletions integration_test_suite/README.md
Original file line number Diff line number Diff line change
@@ -1,14 +1,12 @@
Integration Test Suite
---
---
# Integration Test Suite

### Smoke test Docker images tested
### Smoke test Java Docker images tested

[Smoke test Docker images](https://github.com/prometheus/jmx_exporter/blob/main/integration_test_suite/integration_tests/src/test/resources/docker-image-names.smoke-test.txt)
[Smoke test Docker images](https://github.com/prometheus/jmx_exporter/blob/main/integration_test_suite/integration_tests/src/test/resources/smoke-test-java-docker-images.txt)

### Docker images tested (all)
### All Java Docker images tested (all)

[All Docker images](https://github.com/prometheus/jmx_exporter/blob/main/integration_test_suite/integration_tests/src/test/resources/docker-image-names.all.txt)
[All Docker images](https://github.com/prometheus/jmx_exporter/blob/main/integration_test_suite/integration_tests/src/test/resources/java-docker-images.txt)

### Running the integration test suite (smoke test Docker images)

Expand All @@ -19,21 +17,21 @@ Integration Test Suite
### Run the integration test suite (all Docker images)

```shell
export DOCKER_IMAGE_NAMES=ALL
export JAVA_DOCKER_IMAGES=ALL
./mvnw clean verify
```

### Run the integration test suite on a specific Docker image

```shell
export DOCKER_IMAGE_NAMES="<your custom Docker image>"
export JAVA_DOCKER_IMAGES="<your custom Docker image>"
./mvnw clean verify
```

Example:

```shell
export DOCKER_IMAGE_NAMES="azul/zulu-openjdk:17"
export JAVA_DOCKER_IMAGES="azul/zulu-openjdk:17"
./mvnw clean verify
```

Expand All @@ -44,15 +42,46 @@ Pulling Docker images (not required, but you may see request timeouts/pull failu
Smoke test Docker images

```shell
./integration_test_suite/docker-pull-images.smoke-test.sh
./integration_test_suite/pull-smoke-test-java-docker-images.sh
```

All Docker images

```shell
./integration_test_suite/docker-pull-images.all.sh
./integration_test_suite/pull-java-docker-images.sh
```

## Notes

- You may need to set up Docker hub login to pull images

# Docker networks

By default, Docker is not configured to run a large number images simultaneously.

You can increase Docker networks by creating a Docker network configuration file...

```
/etc/docker/daemon.json
```

... with the content...

```yaml
{
"default-address-pools": [
{
"base": "172.16.0.0/16",
"size": 24
},
{
"base": "192.168.0.0/16",
"size": 24
}
]
}
```

## Notes

- Docker will need to be restarted
Loading