Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Helm chart default telemetry config causes router to crashloop #6104

Open
vatsalpatel opened this issue Oct 2, 2024 · 0 comments · May be fixed by #6105
Open

Helm chart default telemetry config causes router to crashloop #6104

vatsalpatel opened this issue Oct 2, 2024 · 0 comments · May be fixed by #6105

Comments

@vatsalpatel
Copy link

vatsalpatel commented Oct 2, 2024

Describe the bug

The expcted configuration is telemetry.exporters.metrics.common.resource.service.name, which is correct if exporters are provided.
However, when exporters are not provided, the mistake in helm chart causes the resource part of the config to become resources
Above only causes a configuration warning, but the server doesn't start with the following error
could not create the HTTP server: Permission denied (os error 13)

Here is the part that needs to be fixed:
In the file https://github.com/apollographql/router/blob/dev/helm/chart/router/templates/configmap.yaml, line 8 needs to be
{{- $configuration := dict "telemetry" (dict "metrics" (dict "common" (dict "resource" (dict "service.name" $routerFullName)))) -}}

To Reproduce

Steps to reproduce the behavior:
Install router through helm without specifying any telemetry configurations

Expected behavior

The server starts normally

Output

{"timestamp":"2024-10-01T14:06:24.342393458Z","level":"INFO","message":"Apollo Router v1.55.0 // (c) Apollo Graph, Inc. // Licensed as ELv2 (https://go.apollo.dev/elv2)","target":"apollo_router::executable","resource":{}}
{"timestamp":"2024-10-01T14:06:24.342495242Z","level":"INFO","message":"Anonymous usage data is gathered to inform Apollo product development.  See https://go.apollo.dev/o/privacy for details.","target":"apollo_router::executable","resource":{}}
{"timestamp":"2024-10-01T14:06:24.642461025Z","level":"WARN","message":"router configuration contains deprecated options: \n\n  1. telemetry.metrics.common.resources has been renamed to telemetry.metrics.common.resource\n\n  2. telemetry.(tracing|metrics|logging) has been moved to to telemetry.exporters.(tracing|metrics|logging)\n\nThese will become errors in the future. Run `router config upgrade <path_to_router.yaml>` to see a suggested upgraded configuration.","target":"apollo_router::configuration::upgrade","resource":{}}
{"timestamp":"2024-10-01T14:06:24.645538926Z","level":"WARN","message":"router configuration contains deprecated options: \n\n  1. telemetry.metrics.common.resources has been renamed to telemetry.metrics.common.resource\n\n  2. telemetry.(tracing|metrics|logging) has been moved to to telemetry.exporters.(tracing|metrics|logging)\n\nThese will become errors in the future. Run `router config upgrade <path_to_router.yaml>` to see a suggested upgraded configuration.","target":"apollo_router::configuration::upgrade","resource":{}}
{"timestamp":"2024-10-01T14:06:25.246461693Z","level":"INFO","message":"Apollo Studio usage reporting is enabled. See https://go.apollo.dev/o/data for details","target":"apollo_router::plugins::telemetry::metrics::apollo","resource":{}}
{"timestamp":"2024-10-01T14:06:25.254918327Z","level":"WARN","message":"telemetry.instrumentation.spans.mode is currently set to 'deprecated', either explicitly or via defaulting. Set telemetry.instrumentation.spans.mode explicitly in your router.yaml to 'spec_compliant' for log and span attributes that follow OpenTelemetry semantic conventions. This option will be defaulted to 'spec_compliant' in a future release and eventually removed altogether","target":"apollo_router::plugins::telemetry","resource":{}}
{"timestamp":"2024-10-01T14:06:47.844982263Z","level":"INFO","message":"Health check exposed at http://0.0.0.0:8088/health","target":"apollo_router::axum_factory::axum_http_server_factory","resource":{"service.version":"1.55.0","service.name":"unknown_service:router","process.executable.name":"router"}}
{"timestamp":"2024-10-01T14:06:47.933381642Z","level":"INFO","message":"stopped","target":"apollo_router::state_machine","resource":{"service.version":"1.55.0","service.name":"unknown_service:router","process.executable.name":"router"}}
{"timestamp":"2024-10-01T14:06:47.934142484Z","level":"ERROR","message":"could not create the HTTP server: Permission denied (os error 13)","target":"apollo_router::executable","resource":{"service.version":"1.55.0","service.name":"unknown_service:router","process.executable.name":"router"}}
could not create the HTTP server: Permission denied (os error 13)
Stream closed EOF for <namespace>/<pod-name> (router)

Desktop (please complete the following information):

  • OS: As specified in the Dockerfile
  • Version As specified in the Dockerfile

Additional context

We are upgrading the router from v1.20.0. WIthout changing our config, it works until v1.33.0, from v1.34.0 it starts breaking.
v1.34.0 happens to be the version where the config change was made. It doesn't work on latest version either due to the mistake in helm chart

@vatsalpatel vatsalpatel linked a pull request Oct 2, 2024 that will close this issue
6 tasks
@vatsalpatel vatsalpatel changed the title Mistake in helm chart causes router to crashloop Helm chart default telemetry config causes router to crashloop Nov 26, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant