Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

prom2teams support for alertmanager 0.21.0 #189

Closed
1 task
marcinbojko opened this issue Jun 24, 2020 · 12 comments
Closed
1 task

prom2teams support for alertmanager 0.21.0 #189

marcinbojko opened this issue Jun 24, 2020 · 12 comments
Labels

Comments

@marcinbojko
Copy link

Prerequisites

Description

[Description of the issue]
prom2teams in version 2.5.5 doesn't support alertmanager in version 0.21.

Steps to Reproduce

  1. [First Step]
    run prom2teams in v 2.5.5 and alertmanager in version 0.21.x
  2. [Second Step]
    send notification
  3. prom2teams crashes as several fields are 'unknown'

Expected behavior: [What you expect to happen]
ignore unknown fields, parse and send notification
Actual behavior: [What actually happens]
crash
Reproduces how often: [What percentage of the time does it reproduce?]
always

Versions

The version/s you notice the behavior.
2.5.4/2.5.5

Additional Information

Any additional information, configuration or data that might be necessary to reproduce the issue.

@aretokas
Copy link

aretokas commented Jul 1, 2020

As an example, it fails to process truncatedAlerts - It's entirely probable this will be an issue again in the future. The service should ensure that the bare minimum is there for the alert, process that, and ignore any extra fields unless configured to do something with them.

It should not crash just because the JSON changes (unless of course it's missing a critical field).

{
  "receiver": "teams",
  "status": "firing",
  "alerts": [
    {
      "status": "firing",
      "labels": {
        "alertname": "monitor_service_down",
        "instance": "nodeexporter:9100",
        "job": "nodeexporter",
        "severity": "critical"
      },
      "annotations": {
        "description": "Service nodeexporter:9100 is down.",
        "summary": "Monitor service non-operational"
      },
      "startsAt": "2020-06-29T08:38:40.873441835Z",
      "endsAt": "0001-01-01T00:00:00Z",
      "generatorURL": "http://dca7eb0ee6bd:9090/graph?g0.expr=up+%3D%3D+0&g0.tab=1",
      "fingerprint": "63d160a31e26eeb5"
    }
  ],
  "groupLabels": {},
  "commonLabels": {
    "alertname": "monitor_service_down",
    "instance": "nodeexporter:9100",
    "job": "nodeexporter",
    "severity": "critical"
  },
  "commonAnnotations": {
    "description": "Service nodeexporter:9100 is down.",
    "summary": "Monitor service non-operational"
  },
  "externalURL": "http://ca5a0c57a917:9093",
  "version": "4",
  "groupKey": "{}:{}",
  "truncatedAlerts": 0
}

@lazyBisa
Copy link
Contributor

lazyBisa commented Jul 9, 2020

@frantsao, @dortegau, could you review and merge this asap? It is currently breaking our whole Alerting setup.

Thank You.

@lazyBisa
Copy link
Contributor

@frantsao @dortegau sorry for nagging, but this is really an issue for our system right now.

Unfortunately, the Docker build seems to not produce a working image so I (as a python noob) am also not able to build my own image to push to our cluster.

dortegau added a commit that referenced this issue Jul 13, 2020
#189 Fixed handling of additional json properties of alertmanager 0.21.0.
@dortegau
Copy link
Member

Hi all,

I´ve published a 2.5.6 version including your changes.

Could you check if everything is OK?

@marcinbojko
Copy link
Author

Unfortunately no:

                                                                                                                                                                                                                                                     │
│   2020-07-13 15:04:56,374 - prom2teams_app - ERROR - An unhandled exception occurred. name 'EXCLUDE' is not defined                                                                                                                                    │
│   Traceback (most recent call last):                                                                                                                                                                                                                   │
│     File "/usr/local/lib/python3.5/site-packages/flask/app.py", line 2292, in wsgi_app                                                                                                                                                                 │
│       response = self.full_dispatch_request()                                                                                                                                                                                                          │
│     File "/usr/local/lib/python3.5/site-packages/flask/app.py", line 1815, in full_dispatch_request                                                                                                                                                    │
│       rv = self.handle_user_exception(e)                                                                                                                                                                                                               │
│     File "/usr/local/lib/python3.5/site-packages/flask_restplus/api.py", line 583, in error_router                                                                                                                                                     │
│       return original_handler(e)                                                                                                                                                                                                                       │
│     File "/usr/local/lib/python3.5/site-packages/flask_restplus/api.py", line 583, in error_router                                                                                                                                                     │
│       return original_handler(e)                                                                                                                                                                                                                       │
│     File "/usr/local/lib/python3.5/site-packages/flask/app.py", line 1718, in handle_user_exception                                                                                                                                                    │
│       reraise(exc_type, exc_value, tb)                                                                                                                                                                                                                 │
│     File "/usr/local/lib/python3.5/site-packages/flask/_compat.py", line 35, in reraise                                                                                                                                                                │
│       raise value                                                                                                                                                                                                                                      │
│     File "/usr/local/lib/python3.5/site-packages/flask/app.py", line 1813, in full_dispatch_request                                                                                                                                                    │
│       rv = self.dispatch_request()                                                                                                                                                                                                                     │
│     File "/usr/local/lib/python3.5/site-packages/flask/app.py", line 1799, in dispatch_request                                                                                                                                                         │
│       return self.view_functions[rule.endpoint](**req.view_args)                                                                                                                                                                                       │
│     File "/usr/local/lib/python3.5/site-packages/flask_restplus/api.py", line 325, in wrapper                                                                                                                                                          │
│       resp = resource(*args, **kwargs)                                                                                                                                                                                                                 │
│     File "/usr/local/lib/python3.5/site-packages/flask/views.py", line 87, in view                                                                                                                                                                     │
│       self = view.view_class(*class_args, **class_kwargs)                                                                                                                                                                                              │
│     File "/usr/local/lib/python3.5/site-packages/prom2teams/app/versions/v2/namespace.py", line 18, in __init__                                                                                                                                        │
│       self.schema = MessageSchema(exclude_fields=app.config['LABELS_EXCLUDED'], exclude_annotations=app.config['ANNOTATIONS_EXCLUDED'], unknown=EXCLUDE)                                                                                               │
│   NameError: name 'EXCLUDE' is not defined                                                                                                                                                                                                             │
│   2020-07-13 15:04:56,376 - werkzeug - INFO - 172.16.253.24 - - [13/Jul/2020 15:04:56] "POST /v2/Connector HTTP/1.1" 500 -                                                                                                                             │
│   2020-07-13 15:04:56,383 - werkzeug - ERROR - Error on request:                                                                                                                                                                                       │
│   Traceback (most recent call last):                                                                                                                                                                                                                   │
│     File "/usr/local/lib/python3.5/site-packages/flask/app.py", line 2292, in wsgi_app                                                                                                                                                                 │
│       response = self.full_dispatch_request()                                                                                                                                                                                                          │
│     File "/usr/local/lib/python3.5/site-packages/flask/app.py", line 1815, in full_dispatch_request                                                                                                                                                    │
│       rv = self.handle_user_exception(e)                                                                                                                                                                                                               │
│     File "/usr/local/lib/python3.5/site-packages/flask_restplus/api.py", line 583, in error_router                                                                                                                                                     │
│       return original_handler(e)                                                                                                                                                                                                                       │
│     File "/usr/local/lib/python3.5/site-packages/flask_restplus/api.py", line 583, in error_router                                                                                                                                                     │
│       return original_handler(e)                                                                                                                                                                                                                       │
│     File "/usr/local/lib/python3.5/site-packages/flask/app.py", line 1718, in handle_user_exception                                                                                                                                                    │
│       reraise(exc_type, exc_value, tb)                                                                                                                                                                                                                 │
│     File "/usr/local/lib/python3.5/site-packages/flask/_compat.py", line 35, in reraise                                                                                                                                                                │
│       raise value                                                                                                                                                                                                                                      │
│     File "/usr/local/lib/python3.5/site-packages/flask/app.py", line 1813, in full_dispatch_request                                                                                                                                                    │
│       rv = self.dispatch_request()                                                                                                                                                                                                                     │
│     File "/usr/local/lib/python3.5/site-packages/flask/app.py", line 1799, in dispatch_request                                                                                                                                                         │
│       return self.view_functions[rule.endpoint](**req.view_args)                                                                                                                                                                                       │
│     File "/usr/local/lib/python3.5/site-packages/flask_restplus/api.py", line 325, in wrapper                                                                                                                                                          │
│       resp = resource(*args, **kwargs)                                                                                                                                                                                                                 │
│     File "/usr/local/lib/python3.5/site-packages/flask/views.py", line 87, in view                                                                                                                                                                     │
│       self = view.view_class(*class_args, **class_kwargs)                                                                                                                                                                                              │
│     File "/usr/local/lib/python3.5/site-packages/prom2teams/app/versions/v2/namespace.py", line 18, in __init__                                                                                                                                        │
│       self.schema = MessageSchema(exclude_fields=app.config['LABELS_EXCLUDED'], exclude_annotations=app.config['ANNOTATIONS_EXCLUDED'], unknown=EXCLUDE)                                                                                               │
│   NameError: name 'EXCLUDE' is not defined                                                                                                                                                                                                             │
│   2020-07-13T15:04:56.384441218Z                                                                                                                                                                                                                       │
│   During handling of the above exception, another exception occurred:                                                                                                                                                                                  │
│   2020-07-13T15:04:56.384449818Z                                                                                                                                                                                                                       │
│   Traceback (most recent call last):                                                                                                                                                                                                                   │
│     File "/usr/local/lib/python3.5/site-packages/werkzeug/serving.py", line 306, in run_wsgi                                                                                                                                                           │
│       execute(self.server.app)                                                                                                                                                                                                                         │
│     File "/usr/local/lib/python3.5/site-packages/werkzeug/serving.py", line 294, in execute                                                                                                                                                            │
│       application_iter = app(environ, start_response)                                                                                                                                                                                                  │
│     File "/usr/local/lib/python3.5/site-packages/flask/app.py", line 2309, in __call__                                                                                                                                                                 │
│       return self.wsgi_app(environ, start_response)                                                                                                                                                                                                    │
│     File "/usr/local/lib/python3.5/site-packages/flask/app.py", line 2295, in wsgi_app                                                                                                                                                                 │
│       response = self.handle_exception(e)                                                                                                                                                                                                              │
│     File "/usr/local/lib/python3.5/site-packages/flask_restplus/api.py", line 583, in error_router                                                                                                                                                     │
│       return original_handler(e)                                                                                                                                                                                                                       │
│     File "/usr/local/lib/python3.5/site-packages/flask_restplus/api.py", line 583, in error_router                                                                                                                                                     │
│       return original_handler(e)                                                                                                                                                                                                                       │
│     File "/usr/local/lib/python3.5/site-packages/flask/app.py", line 1748, in handle_exception                                                                                                                                                         │
│       return self.finalize_request(handler(e), from_error_handler=True)                                                                                                                                                                                │
│     File "/usr/local/lib/python3.5/site-packages/prom2teams/app/api.py", line 26, in error_handler                                                                                                                                                     │
│       return str(e), e.code                                                                                

@lazyBisa
Copy link
Contributor

@dortegau @marcinbojko sorry for the issues, seems I forgot an Import.
Unfortunately, I am a complete Python noob and was not able to start the project on my machine. docker build . -t prom2teams && docker run --rm prom2teams unfortunately gives me the following log output and then terminates:

': [Errno 2] No such file or directoryms/replace_config.py
': No such file or directory

also, from what I can understand there is no test starting the whole app? That might be usefull too. Or some hints in the README how to build and run locally. python setup.py bdist_wheel (taken from dockerfile) unfortunately didn't do the job.

@lazyBisa
Copy link
Contributor

Can anyone merge this or help me build a runnable image? Please?

@lazyBisa
Copy link
Contributor

lazyBisa commented Jul 22, 2020

solved it like this:

fix.patch

--- prometheus/message_schema.py
+++ prometheus/message_schema.py
@@ -5,6 +5,8 @@ log = logging.getLogger('prom2teams')


 class MessageSchema(Schema):
+    class Meta:
+        unknown = EXCLUDE

     def __init__(self, exclude_fields=tuple(), exclude_annotations=tuple()):
         super().__init__()
@@ -68,7 +70,8 @@ class AlertSchema(Schema):
     endsAt = fields.DateTime()
     generatorURL = fields.Str()
     fingerprint = fields.Str()
-
+    class Meta:
+        unknown = EXCLUDE

 class LabelSchema(Schema):
     alertname = fields.Str(default='unknown', missing='unknown')

Dockerfile

FROM idealista/prom2teams:2.5.5

RUN pip install pypatch
COPY fix.patch /fix.patch

RUN pypatch apply /fix.patch prom2teams

and then
docker build . -t <your-image-name>

@wrossmann
Copy link

Honestly, until this gets ironed out I just bumped my Alertmanager version back to 0.20.0 and it's working just fine.

Unless you have a specific dependency on something in 0.21.0 it's a painless change.

@marcinbojko
Copy link
Author

tested with image 2.5.7 - still same error

@dortegau
Copy link
Member

Fixed in #210

@stale
Copy link

stale bot commented Oct 18, 2020

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging a pull request may close this issue.

5 participants