Using opentelemetry java agent for exporting prometheus for core java based application in the production environment

[AbhinandRaj08]

Hello everyone,
I am trying to set up opentelemetry for my core java based application. I am using the opentelemetry java agent for instrumentation. I am using docker to launch containers for Grafana, Tempo, Loki and Prometheus services. I want to use Prometheus as the metrics exporter Prometheus exporter and want to see the metrics in Prometheus datasource configured in Grafana. I have created all the config files and docker compose file and tested it out with the following configurations.

In order to configure that in the development environment, I have added a scrape job in the Prometheus config like below:

- job_name: 'my-service' static_configs: - targets: [ 'host.docker.internal:9464']

Also providing you the VM options in the run configuration of my main class,

-javaagent:C:\Users\A***.***\Downloads\opentelemetry-javaagent.jar -Dotel.exporter.otlp.endpoint=http://localhost:4317 -Dotel.service.name=my-service -Dotel.exporter.otlp.insecure=true -Dotel.traces.exporter=otlp -Dotel.metrics.exporter=prometheus -Dotel.instrumentation.runtime-metrics.enabled=true -Dotel.exporter.prometheus.port=9464

Now it is working fine. I want to test this in the production environment. I have read the docker desktop documentation and got to know that host.docker.internal cannot be used in the production environment.https://docs.docker.com/desktop/networking/#i-want-to-connect-from-a-container-to-a-service-on-the-host

1. What should I specify instead of ‘host.docker.internal’ as the hostname in the production environment?

2. And I am not specifying the host in my VM options currently. So can I go with the same approach in production environment as well? i.e., can I give <my-host-ip>:9464 in the targets ?

3. Or else should I specify the IP address of my machine as the host in VM options?

4. Also is there any better way to collect and export the metrics from my core java application?

Please feel free to give your suggestions and views on this. Thank you in advance.