Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

How to suppress export logs in open-telemetry agent? #5347

Closed
VelDeveloper opened this issue Apr 4, 2023 · 9 comments
Closed

How to suppress export logs in open-telemetry agent? #5347

VelDeveloper opened this issue Apr 4, 2023 · 9 comments
Labels
Bug Something isn't working

Comments

@VelDeveloper
Copy link

VelDeveloper commented Apr 4, 2023

I am using "aws-opentelemetry-agent" for my spring-boot application and we are exporting logs & traces via gRPC.
whenever it exports it generates a log line like below. These logs line blow up the logging system since we export them more often.
Can I suppress this log message? I want error logs whenever the export fails.

[2023-04-04T10:16:20.401Z] "POST /opentelemetry.proto.collector.trace.v1.TraceService/Export HTTP/2" 200 - via_upstream - "-" 2452 7 2 1 "-" "OTel-OTLP-Exporter-Java/1.23.1" "d3f20df4-13b0-956f-9202-72cd00fc3566" "xray-collector.opentelemetry-operator-system:4317" "10.10.200.164:4317" outbound|4317||xray-collector.sample.local - default

[2023-04-04T10:16:24.897Z] "POST /opentelemetry.proto.collector.metrics.v1.MetricsService/Export HTTP/2" 200 - via_upstream - "-" 23319 7 60 17 "-" "OTel-OTLP-Exporter-Java/1.23.1" "b8ed6dc9-c3ac-9f90-abef-e161e8282b2f" "prometheus-amp-collector.opentelemetry-operator-system:4317" "10.10.235.74:4317" outbound|4317||prometheus-amp-sample.svc.cluster.local  - default

Steps to reproduce
FROM maven:3.8.3-openjdk-17-slim

RUN mkdir -p /agent
RUN apt-get update && apt-get install -y wget
RUN wget -O /agent/aws-opentelemetry-agent.jar https://github.com/aws-observability/aws-otel-java-instrumentation/releases/latest/download/aws-opentelemetry-agent.jar

ENV JAVA_TOOL_OPTIONS="-javaagent:/agent/aws-opentelemetry-agent.jar
-Dotel.resource.attributes=service.name=sample
-Dotel.propagators=tracecontext
-Dotel.traces.exporter=otlp
-Dotel.exporter.metrics.endpoint=http://sample:4317
-Dotel.exporter.traces.endpoint=http://sample:4317
-Dotel.metrics.exporter=otlp"

VOLUME /srv/logs

ARG JAR_FILE
COPY ${JAR_FILE} sample.jar
ENTRYPOINT ["java","-jar","/sample.jar"]

What version and what artifacts are you using?
we are using the latest aws-opentelemetry-sdk

Environment
Environment: AWS EKS

@VelDeveloper VelDeveloper added the Bug Something isn't working label Apr 4, 2023
@jkwatson
Copy link
Contributor

jkwatson commented Apr 4, 2023

This looks like an issue for AWS, not for the general opentelemetry java libraries. Or possibly for the opentelemetry-java-instrumentation project where the vanilla agent code lives.

@mateuszrzeszutek
Copy link
Member

The logs samples you posted don't look like they are produced by the upstream agent. Consider filing an issue in the https://github.com/aws-observability/aws-otel-java-instrumentation project first, maybe they'll recognize what emits these logs.

@VelDeveloper
Copy link
Author

VelDeveloper commented Apr 4, 2023

@mateuszrzeszutek @jkwatson Just I created a ticket in aws repository. But a generic question, setting "OTEL_LOG_LEVEL=ERROR" doesn't fix the above issue? This ticket discussed the log_level. I want to see the log line only if there is any error in exporting otherwise I don't need log_level I believe we can achieve this using "OTEL_LOG_LEVEL=ERROR"

@jkwatson
Copy link
Contributor

jkwatson commented Apr 4, 2023

@mateuszrzeszutek @jkwatson Just I created a ticket in aws repository. But a generic question, setting "OTEL_LOG_LEVEL=ERROR" doesn't fix the above issue? This ticket discussed the log_level. I want to see the log line only if there is any error in exporting otherwise I don't need log_level I believe we can achieve this using "OTEL_LOG_LEVEL=ERROR"

The Java SDK doesn't support that property, nor will it probably going forward, since the SDK uses JUL and the log level can be configured for that directly.

@rapphil
Copy link

rapphil commented Apr 6, 2023

[2023-04-04T10:16:20.401Z] "POST /opentelemetry.proto.collector.trace.v1.TraceService/Export HTTP/2" 200 - via_upstream - "-" 2452 7 2 1 "-" "OTel-OTLP-Exporter-Java/1.23.1" "d3f20df4-13b0-956f-9202-72cd00fc3566" "xray-collector.opentelemetry-operator-system:4317" "10.10.200.164:4317" outbound|4317||xray-collector.sample.local - default

[2023-04-04T10:16:24.897Z] "POST /opentelemetry.proto.collector.metrics.v1.MetricsService/Export HTTP/2" 200 - via_upstream - "-" 23319 7 60 17 "-" "OTel-OTLP-Exporter-Java/1.23.1" "b8ed6dc9-c3ac-9f90-abef-e161e8282b2f" "prometheus-amp-collector.opentelemetry-operator-system:4317" "10.10.235.74:4317" outbound|4317||prometheus-amp-sample.svc.cluster.local  - default

These log lines look like access logs generated in server side. I'm suspecting that the logs are being generated from a different workload that is sharing the same pod as the java application.

I'm posting here just to request help in one matter: does java http client libraries like okhttp would be able to produce access logs like these?

@rapphil
Copy link

rapphil commented Apr 6, 2023

I just confirmed with @VelDeveloper and the log entries are coming from istio.

Ref: https://istio.io/latest/docs/tasks/observability/logs/access-log/#test-the-access-log

@jkwatson
Copy link
Contributor

jkwatson commented Apr 6, 2023

I just confirmed with @VelDeveloper and the log entries are coming from istio.

Ref: https://istio.io/latest/docs/tasks/observability/logs/access-log/#test-the-access-log

can we close this issue, then?

@rapphil
Copy link

rapphil commented Apr 6, 2023

The issue aws-observability/aws-otel-java-instrumentation#369 (comment) was closed, so I assume yes.

@jkwatson
Copy link
Contributor

jkwatson commented Apr 6, 2023

closing as not-relevant for this repository.

@jkwatson jkwatson closed this as completed Apr 6, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Bug Something isn't working
Projects
None yet
Development

No branches or pull requests

4 participants