Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Alertmanager config doesn't apply #279

Closed
natalytvinova opened this issue Aug 7, 2024 · 1 comment · Fixed by #284
Closed

Alertmanager config doesn't apply #279

natalytvinova opened this issue Aug 7, 2024 · 1 comment · Fixed by #284
Labels
Status: Triage Type: Bug

Comments

@natalytvinova
Copy link

natalytvinova commented Aug 7, 2024 •
edited
Loading

Bug Description

Hi team,
I'm trying to pass this config for email configuration and getting this error:

route:
  receiver: 'Mail Alert'
  # group_by: [ alertname ]
  repeat_interval: 30s
  group_wait: 15s
  group_interval: 15s


receivers:
  - name: 'Mail Alert'
    email_configs:
      - smarthost: '<smtp>'
        auth_username: '<mail>'
        auth_password: "<password>"
        from: 'mail'
        to: '<email>'
        headers:
          subject: 'Prometheus Mail Alerts'

Error:

root@alertmanager-0:/var/lib/juju/agents/unit-alertmanager-0/charm# ./dispatch
2024-08-07 08:47:33,598 DEBUG    load_verify_locations cafile='/var/run/secrets/kubernetes.io/serviceaccount/ca.crt'
2024-08-07 08:47:33,644 DEBUG    no relation on 'tracing': tracing not ready
2024-08-07 08:47:33,650 DEBUG    <class '__main__.AlertmanagerCharm'>.<property object at 0x7fe9c790d770> returned None; quietly disabling charm_tracing for the run.
2024-08-07 08:47:33,657 DEBUG    Emitting Juju event config_changed.
2024-08-07 08:47:33,722 DEBUG    start_tls.complete return_value=<httpcore._backends.sync.SyncStream object at 0x7fe9c78e1d90>
2024-08-07 08:47:33,730 DEBUG    send_request_headers.started request=<Request [b'GET']>
2024-08-07 08:47:33,738 DEBUG    send_request_headers.complete
2024-08-07 08:47:33,745 DEBUG    send_request_body.started request=<Request [b'GET']>
2024-08-07 08:47:33,755 DEBUG    send_request_body.complete
2024-08-07 08:47:33,768 DEBUG    receive_response_headers.started request=<Request [b'GET']>
2024-08-07 08:47:33,779 DEBUG    receive_response_headers.complete return_value=(b'HTTP/1.1', 200, b'OK', [(b'Audit-Id', b'e9b011f8-5e73-4312-bf7a-2b6f5607f63c'), (b'Cache-Control', b'no-cache, private'), (b'Content-Type', b'application/json'), (b'X-Kubernetes-Pf-Flowschema-Uid', b'42abedad-eb08-4d66-a830-1cce313dfe10'), (b'X-Kubernetes-Pf-Prioritylevel-Uid', b'20325b98-c934-4299-9e36-39
2024-08-07 08:47:33,811 DEBUG    response_closed.complete
2024-08-07 08:47:33,824 INFO     Kubernetes resources for app 'alertmanager', container 'alertmanager' patched successfully: ResourceRequirements(claims=None, limits={}, requests={'cpu': '0.25', 'memory': '200Mi'})
2024-08-07 08:47:33,832 DEBUG    send_request_headers.started request=<Request [b'GET']>
2024-08-07 08:47:33,839 DEBUG    send_request_headers.complete
2024-08-07 08:47:33,845 DEBUG    send_request_body.started request=<Request [b'GET']>
-Pf-Flowschema-Uid', b'42abedad-eb08-4d66-a830-1cce313dfe10'), (b'X-Kubernetes-Pf-Prioritylevel-Uid', b'20325b98-c934-4299-9e36-39
P/1.1 200 OK"
2024-08-07 08:47:34,087 DEBUG    receive_response_headers.started request=<Request [b'GET']>
2024-08-07 08:47:34,095 DEBUG    receive_response_headers.complete return_value=(b'HTTP/1.1', 200, b'OK', [(b'Audit-Id', b'8ea8457b-c7ee-42ef-9124-c796a9b1e9cf'), (b'Cache-Control', b'no-cache, private'), (b'Content-Type', b'application/json'), (b'X-Kubernetes-Pf-Flowschema-Uid', b'42abedad-eb08-4d66-a830-1cce313dfe10'), (b'X-Kubernetes-Pf-Prioritylevel-Uid', b'20325b98-c934-4299-9e36-39b0dca9fdf9'), (b'Date', b'Wed, 07 Aug 2024 08:47:34 GMT'), (b'Transfer-Encoding', b'chunked')])
2024-08-07 08:47:34,102 INFO     HTTP Request: GET https://10.152.183.1/api/v1/namespaces/cos/pods/alertmanager-0 "HTTP/1.1 200 OK"
2024-08-07 08:47:34,108 DEBUG    receive_response_body.started request=<Request [b'GET']>
2024-08-07 08:47:34,115 DEBUG    receive_response_body.complete
2024-08-07 08:47:34,121 DEBUG    response_closed.started
2024-08-07 08:47:34,127 DEBUG    response_closed.complete
2024-08-07 08:47:34,393 DEBUG    Could not locate cos-tool at: "cos-tool-amd64"
2024-08-07 08:47:34,399 DEBUG    Skipping injection of juju topology as label matchers
2024-08-07 08:47:34,406 DEBUG    `cos-tool` unavailable. Leaving expression unchanged: rate(alertmanager_notifications_failed_total{integration=~".*"}[5m]) > 0
2024-08-07 08:47:34,412 DEBUG    Reading alert rule from /var/lib/juju/agents/unit-alertmanager-0/charm/src/prometheus_alert_rules/alertmanager_notifications_failed.rule
2024-08-07 08:47:34,420 DEBUG    `cos-tool` unavailable. Leaving expression unchanged: vector(1)
2024-08-07 08:47:34,426 DEBUG    Reading alert rule from /var/lib/juju/agents/unit-alertmanager-0/charm/src/prometheus_alert_rules/heartbeat.rule
2024-08-07 08:47:34,435 DEBUG    `cos-tool` unavailable. Leaving expression unchanged: absent(up{})
2024-08-07 08:47:34,441 DEBUG    Reading alert rule from /var/lib/juju/agents/unit-alertmanager-0/charm/src/prometheus_alert_rules/alertmanager_missing.rule
2024-08-07 08:47:34,449 DEBUG    `cos-tool` unavailable. Leaving expression unchanged: alertmanager_config_last_reload_successful{} != 1
2024-08-07 08:47:34,456 DEBUG    Reading alert rule from /var/lib/juju/agents/unit-alertmanager-0/charm/src/prometheus_alert_rules/alertmanager_configuration_reload_failure.rule
2024-08-07 08:47:34,538 ERROR    Uncaught exception while in charm code:
Traceback (most recent call last):
  File "./src/charm.py", line 590, in <module>
    main(AlertmanagerCharm)
  File "/var/lib/juju/agents/unit-alertmanager-0/charm/venv/ops/main.py", line 548, in main
    manager.run()
  File "/var/lib/juju/agents/unit-alertmanager-0/charm/venv/ops/main.py", line 527, in run
    self._emit()
  File "/var/lib/juju/agents/unit-alertmanager-0/charm/venv/ops/main.py", line 516, in _emit
    _emit_charm_event(self.charm, self.dispatcher.event_name)
  File "/var/lib/juju/agents/unit-alertmanager-0/charm/venv/ops/main.py", line 147, in _emit_charm_event
    event_to_emit.emit(*args, **kwargs)
  File "/var/lib/juju/agents/unit-alertmanager-0/charm/venv/ops/framework.py", line 348, in emit
    framework._emit(event)
  File "/var/lib/juju/agents/unit-alertmanager-0/charm/venv/ops/framework.py", line 860, in _emit
    self._reemit(event_path)
  File "/var/lib/juju/agents/unit-alertmanager-0/charm/venv/ops/framework.py", line 950, in _reemit
    custom_handler(event)
  File "/var/lib/juju/agents/unit-alertmanager-0/charm/lib/charms/tempo_k8s/v1/charm_tracing.py", line 548, in wrapped_function
    return callable(*args, **kwargs)  # type: ignore
  File "./src/charm.py", line 468, in _on_config_changed
    self._common_exit_hook(update_ca_certs=True)
  File "/var/lib/juju/agents/unit-alertmanager-0/charm/lib/charms/tempo_k8s/v1/charm_tracing.py", line 548, in wrapped_function
    return callable(*args, **kwargs)  # type: ignore
  File "./src/charm.py", line 437, in _common_exit_hook
    self.alertmanager_workload.update_config(self._render_manifest())
  File "/var/lib/juju/agents/unit-alertmanager-0/charm/lib/charms/tempo_k8s/v1/charm_tracing.py", line 548, in wrapped_function
    return callable(*args, **kwargs)  # type: ignore
  File "./src/charm.py", line 370, in _render_manifest
    ConfigBuilder(api_port=self.api_port)
  File "/var/lib/juju/agents/unit-alertmanager-0/charm/src/config_builder.py", line 136, in build
    alertmanager=self._alertmanager_config,
  File "/var/lib/juju/agents/unit-alertmanager-0/charm/src/config_builder.py", line 85, in _alertmanager_config
    config = self._config.copy()
AttributeError: 'str' object has no attribute 'copy'

To Reproduce

  1. Juju deploy cos-lite with alertmanager latest/stable rev 119
  2. juju config config_file=alertmanager.yml

Environment

$ juju status -m microk8s-controller:cos 
Model  Controller           Cloud/Region              Version  SLA          Timestamp
cos    microk8s-controller  microk8s_cloud/localhost  3.4.4    unsupported  13:08:12Z

App                              Version  Status   Scale  Charm                         Channel        Rev  Address         Exposed  Message
alertmanager                     0.27.0   waiting      1  alertmanager-k8s              latest/stable  119  10.152.183.218  no       installing agent
catalogue                                 active       1  catalogue-k8s                 latest/stable   43  10.152.183.33   no       
grafana                          9.5.3    active       1  grafana-k8s                   latest/stable  114  10.152.183.86   no       
loki                             2.9.6    active       1  loki-k8s                      latest/stable  151  10.152.183.87   no       
prometheus                       2.50.1   active       1  prometheus-k8s                latest/stable  189  10.152.183.121  no       
scrape-interval-config-metrics   n/a      active       1  prometheus-scrape-config-k8s  latest/stable   51  10.152.183.212  no       
scrape-interval-config-monitors  n/a      waiting      1  prometheus-scrape-config-k8s  latest/stable   51  10.152.183.144  no       installing agent
traefik                          v2.11.0  active       1  traefik-k8s                   latest/stable  194  10.152.183.36   no       Serving at 172.27.80.50

Unit                                Workload  Agent  Address       Ports  Message
alertmanager/0*                     error     idle   10.1.138.224         hook failed: "config-changed"
catalogue/0*                        active    idle   10.1.103.166         
grafana/0*                          active    idle   10.1.138.223         
loki/0*                             active    idle   10.1.103.171         
prometheus/0*                       active    idle   10.1.113.150         
scrape-interval-config-metrics/0*   active    idle   10.1.103.161         
scrape-interval-config-monitors/0*  blocked   idle   10.1.103.160         missing metrics provider (relate to upstream charm?)
traefik/0*                          active    idle   10.1.103.169         Serving at 172.27.80.50

Offer                            Application                      Charm                         Rev  Connected  Endpoint                  Interface                Role
alertmanager                     alertmanager                     alertmanager-k8s              119  0/0        karma-dashboard           karma_dashboard          provider
grafana                          grafana                          grafana-k8s                   114  13/13      grafana-dashboard         grafana_dashboard        requirer
loki                             loki                             loki-k8s                      151  12/12      logging                   loki_push_api            provider
prometheus                       prometheus                       prometheus-k8s                189  12/12      receive-remote-write      prometheus_remote_write  provider
scrape-interval-config-metrics   scrape-interval-config-metrics   prometheus-scrape-config-k8s  51   1/1        configurable-scrape-jobs  prometheus_scrape        requirer
scrape-interval-config-monitors  scrape-interval-config-monitors  prometheus-scrape-config-k8s  51   0/0        configurable-scrape-jobs  prometheus_scrape        requirer

Relevant log output

$ juju debug-log -m microk8s-controller:cos
unit-alertmanager-0: 13:06:51 ERROR unit.alertmanager/0.juju-log Uncaught exception while in charm code:
Traceback (most recent call last):
  File "./src/charm.py", line 590, in <module>
    main(AlertmanagerCharm)
  File "/var/lib/juju/agents/unit-alertmanager-0/charm/venv/ops/main.py", line 548, in main
    manager.run()
  File "/var/lib/juju/agents/unit-alertmanager-0/charm/venv/ops/main.py", line 527, in run
    self._emit()
  File "/var/lib/juju/agents/unit-alertmanager-0/charm/venv/ops/main.py", line 516, in _emit
    _emit_charm_event(self.charm, self.dispatcher.event_name)
  File "/var/lib/juju/agents/unit-alertmanager-0/charm/venv/ops/main.py", line 147, in _emit_charm_event
    event_to_emit.emit(*args, **kwargs)
  File "/var/lib/juju/agents/unit-alertmanager-0/charm/venv/ops/framework.py", line 348, in emit
    framework._emit(event)
  File "/var/lib/juju/agents/unit-alertmanager-0/charm/venv/ops/framework.py", line 860, in _emit
    self._reemit(event_path)
  File "/var/lib/juju/agents/unit-alertmanager-0/charm/venv/ops/framework.py", line 950, in _reemit
    custom_handler(event)
  File "/var/lib/juju/agents/unit-alertmanager-0/charm/lib/charms/tempo_k8s/v1/charm_tracing.py", line 548, in wrapped_function
    return callable(*args, **kwargs)  # type: ignore
  File "./src/charm.py", line 468, in _on_config_changed
    self._common_exit_hook(update_ca_certs=True)
  File "/var/lib/juju/agents/unit-alertmanager-0/charm/lib/charms/tempo_k8s/v1/charm_tracing.py", line 548, in wrapped_function
    return callable(*args, **kwargs)  # type: ignore
  File "./src/charm.py", line 437, in _common_exit_hook
    self.alertmanager_workload.update_config(self._render_manifest())
  File "/var/lib/juju/agents/unit-alertmanager-0/charm/lib/charms/tempo_k8s/v1/charm_tracing.py", line 548, in wrapped_function
    return callable(*args, **kwargs)  # type: ignore
  File "./src/charm.py", line 370, in _render_manifest
    ConfigBuilder(api_port=self.api_port)
  File "/var/lib/juju/agents/unit-alertmanager-0/charm/src/config_builder.py", line 136, in build
    alertmanager=self._alertmanager_config,
  File "/var/lib/juju/agents/unit-alertmanager-0/charm/src/config_builder.py", line 85, in _alertmanager_config
    config = self._config.copy()
AttributeError: 'str' object has no attribute 'copy'
unit-alertmanager-0: 13:06:51 ERROR juju.worker.uniter.operation hook "config-changed" (via hook dispatching script: dispatch) failed: exit status 1
unit-alertmanager-0: 13:06:51 INFO juju.worker.uniter awaiting error resolution for "config-changed" hook
unit-grafana-0: 13:07:10 INFO juju.worker.uniter.operation ran "update-status" hook (via hook dispatching script: dispatch)
unit-prometheus-0: 13:07:36 WARNING unit.prometheus/0.update-status rehash: warning: skipping ca-certificates.crt,it does not contain exactly one certificate or CRL
unit-prometheus-0: 13:07:37 INFO juju.worker.uniter.operation ran "update-status" hook (via hook dispatching script: dispatch)
unit-loki-0: 13:08:34 WARNING unit.loki/0.update-status rehash: warning: skipping ca-certificates.crt,it does not contain exactly one certificate or CRL
unit-loki-0: 13:08:35 INFO juju.worker.uniter.operation ran "update-status" hook (via hook dispatching script: dispatch)
unit-scrape-interval-config-metrics-0: 13:08:46 INFO juju.worker.uniter.operation ran "update-status" hook (via hook dispatching script: dispatch)
unit-catalogue-0: 13:08:50 INFO juju.worker.uniter.operation ran "update-status" hook (via hook dispatching script: dispatch)

Additional context

No response
@Abuelodelanada
Copy link
Contributor

Hello @natalytvinova !

The problem is that if we want to send a file content to a config we have to add the @, for example:

juju config alertmanager config_file=@alertmanager.yml

Anyway I'm working to better handle this exception.

@Abuelodelanada Abuelodelanada mentioned this issue Aug 19, 2024
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Status: Triage Type: Bug
Projects
None yet
Milestone
No milestone
2 participants
@Abuelodelanada @natalytvinova