[AIRFLOW-4069] Add Opsgenie Alert Hook and Operator

[nritholtz]

Make sure you have checked all steps below.

Jira

• My PR addresses the following Airflow Jira issues and references them in the PR title. For example, "[AIRFLOW-XXX] My Airflow PR"
  • In case you are fixing a typo in the documentation you can prepend your commit with [AIRFLOW-XXX], code changes always need a Jira issue.
  • In case you are proposing a fundamental code change, you need to create an Airflow Improvement Proposal(AIP).

Description

• Here are some details about my PR, including screenshots of any UI changes:
  Add a hook and operator for the Opsgenie Alert API to create alerts.

Tests

• My PR adds the following unit tests OR does not need testing for this extremely good reason:
  tests/contrib/hooks/test_opsgenie_alert_hook.py and tests/contrib/operators/test_opsgenie_alert_operator.py

Commits

• My commits all reference Jira issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "How to write a good git commit message":
  1. Subject is separated from body by a blank line
  2. Subject is limited to 50 characters (not including Jira issue reference)
  3. Subject does not end with a period
  4. Subject uses the imperative mood ("add", not "adding")
  5. Body wraps at 72 characters
  6. Body explains "what" and "why", not "how"

Documentation

• In case of new functionality, my PR adds documentation that describes how to use it.
  • When adding new operators/hooks/sensors, the autoclass documentation generation needs to be added.
  • All the public functions and the classes in the PR contain docstrings that explain what it does
  • If you implement backwards incompatible changes, please leave a note in the Updating.md so we can assign it to a appropriate release

Code Quality

• Passes flake8

[codecov-io]

Codecov Report

Merging #4903 into master will increase coverage by 0.05%.
The diff coverage is 96.49%.

@@            Coverage Diff             @@
##           master    #4903      +/-   ##
==========================================
+ Coverage   76.23%   76.29%   +0.05%     
==========================================
  Files         466      468       +2     
  Lines       30101    30188      +87     
==========================================
+ Hits        22949    23031      +82     
- Misses       7152     7157       +5

Impacted Files	Coverage Δ
airflow/contrib/hooks/opsgenie_alert_hook.py	100% <100%> (ø)
airflow/utils/db.py	90.38% <100%> (+0.09%)	⬆️
...rflow/contrib/operators/opsgenie_alert_operator.py	93.93% <93.93%> (ø)
airflow/utils/log/file_processor_handler.py	87.25% <0%> (+1.14%)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update dbed51e...778c919. Read the comment docs.

[nritholtz]

@zhongjiajie Fair point, I removed the ability to explicitly pass the api_key to the hook.

[zhongjiajie]

@nritholtz Well I notice that you add new hook without operator. should we add operator to this new hook?

[nritholtz]

@zhongjiajie On second thought, Ill add an operator as well since I'm already in this code.

[nritholtz]

Added operator for the hook, and also converted the api_key field to a connection password. This way, the BaseHook debug_info call will obfuscate the key in the logs.

[zhongjiajie]

My 2 cents.

[zhongjiajie]

cc @mik-laj we still have discuss in #4903 (comment) and #4903 (comment).
after we finish we could ask committer to review this PR.

[nritholtz]

I've updated the PR as per the suggestions and also added additional test coverage for the hook.

[nritholtz]

Made suggested changes.

Just an FYI, while reviewing the Dingding code as per your suggestion, I found an issue with the doc generation on https://github.com/apache/airflow/blob/master/airflow/contrib/hooks/dingding_hook.py#L104 where there needs to be a newline for the param/type to be parsed correctly.

[ashb]

Yes it does, good catch.

[nritholtz]

@ashb Only one job in the build failed due to unrelated tests:

======================================================================
1) ERROR: test_integration_run_dag (tests.contrib.minikube.test_kubernetes_executor.KubernetesExecutorTest)
----------------------------------------------------------------------
   Traceback (most recent call last):
    tests/contrib/minikube/test_kubernetes_executor.py line 173 in test_integration_run_dag
      result_json = self.start_dag(dag_id=dag_id, host=host)
    tests/contrib/minikube/test_kubernetes_executor.py line 134 in start_dag
      'dags/{dag_id}/paused/false'.format(host=host, dag_id=dag_id)
    .tox/py35-backend_postgres-env_kubernetes/lib/python3.5/site-packages/requests/api.py line 75 in get
      return request('get', url, params=params, **kwargs)
    .tox/py35-backend_postgres-env_kubernetes/lib/python3.5/site-packages/requests/api.py line 60 in request
      return session.request(method=method, url=url, **kwargs)
    .tox/py35-backend_postgres-env_kubernetes/lib/python3.5/site-packages/requests/sessions.py line 533 in request
      resp = self.send(prep, **send_kwargs)
    .tox/py35-backend_postgres-env_kubernetes/lib/python3.5/site-packages/requests/sessions.py line 646 in send
      r = adapter.send(request, **kwargs)
    .tox/py35-backend_postgres-env_kubernetes/lib/python3.5/site-packages/requests/adapters.py line 516 in send
      raise ConnectionError(e, request=request)
   ConnectionError: HTTPConnectionPool(host='10.20.1.120', port=30809): Max retries exceeded with url: /api/experimental/dags/example_kubernetes_executor_config/paused/false (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f3181a65cc0>: Failed to establish a new connection: [Errno 111] Connection refused',))
   -------------------- >> begin captured stdout << ---------------------
   [2019-04-05 17:34:09,772] {connectionpool.py:205} DEBUG - Starting new HTTP connection (1): 10.20.1.120:30809
   
   --------------------- >> end captured stdout << ----------------------
   -------------------- >> begin captured logging << --------------------
   airflow.utils.log.logging_mixin.LoggingMixin: INFO: Creating new Airflow config file in: /home/airflow/airflow.cfg
   airflow.utils.log.logging_mixin.LoggingMixin: INFO: Reading the config from /home/airflow/airflow.cfg
   airflow.utils.log.logging_mixin.LoggingMixin: INFO: Creating new FAB webserver config file in: /home/airflow/webserver_config.py
   airflow.settings: INFO: Configured default timezone <Timezone [UTC]>
   root: INFO: Generating grammar tables from /usr/lib/python3.5/lib2to3/Grammar.txt
   root: INFO: Generating grammar tables from /usr/lib/python3.5/lib2to3/PatternGrammar.txt
   airflow.utils.log.logging_mixin.LoggingMixin: DEBUG: Cannot import  due to  doesn't look like a module path
   airflow.logging_config: DEBUG: Unable to load custom logging, using default config instead
   urllib3.connectionpool: DEBUG: Starting new HTTP connection (1): 10.20.1.120:30809
   --------------------- >> end captured logging << ---------------------
======================================================================
2) ERROR: test_integration_run_dag_with_scheduler_failure (tests.contrib.minikube.test_kubernetes_executor.KubernetesExecutorTest)
----------------------------------------------------------------------
   Traceback (most recent call last):
    tests/contrib/minikube/test_kubernetes_executor.py line 196 in test_integration_run_dag_with_scheduler_failure
      result_json = self.start_dag(dag_id=dag_id, host=host)
    tests/contrib/minikube/test_kubernetes_executor.py line 134 in start_dag
      'dags/{dag_id}/paused/false'.format(host=host, dag_id=dag_id)
    .tox/py35-backend_postgres-env_kubernetes/lib/python3.5/site-packages/requests/api.py line 75 in get
      return request('get', url, params=params, **kwargs)
    .tox/py35-backend_postgres-env_kubernetes/lib/python3.5/site-packages/requests/api.py line 60 in request
      return session.request(method=method, url=url, **kwargs)
    .tox/py35-backend_postgres-env_kubernetes/lib/python3.5/site-packages/requests/sessions.py line 533 in request
      resp = self.send(prep, **send_kwargs)
    .tox/py35-backend_postgres-env_kubernetes/lib/python3.5/site-packages/requests/sessions.py line 646 in send
      r = adapter.send(request, **kwargs)
    .tox/py35-backend_postgres-env_kubernetes/lib/python3.5/site-packages/requests/adapters.py line 516 in send
      raise ConnectionError(e, request=request)
   ConnectionError: HTTPConnectionPool(host='10.20.1.120', port=30809): Max retries exceeded with url: /api/experimental/dags/example_kubernetes_executor_config/paused/false (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f3181a65208>: Failed to establish a new connection: [Errno 111] Connection refused',))

All suggested changes have been made. PTAL

[ashb]

Taking a look. Yeah, the kube tests timeout sometimes :(

[ashb]

@nritholtz Two small changes - if you can make them soon I can get this in to 1.10.3

[nritholtz]

@ashb looks like unrelated test failures again on 2 of the jobs. Otherwise made all suggested changes.

Thanks.

[nritholtz]

@ashb Thanks for the rebuild, looks all green now. 🎉

[zhongjiajie]

Made suggested changes.

Just an FYI, while reviewing the Dingding code as per your suggestion, I found an issue with the doc generation on https://github.com/apache/airflow/blob/master/airflow/contrib/hooks/dingding_hook.py#L104 where there needs to be a newline for the param/type to be parsed correctly.

Good catch, I will add new line tomorrow, Thanks. @nritholtz