Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[AIRFLOW-3556] Add cross join set downstream function #4356

Merged
merged 1 commit into from
Dec 26, 2018
Merged

[AIRFLOW-3556] Add cross join set downstream function #4356

merged 1 commit into from
Dec 26, 2018

Conversation

BasPH
Copy link
Contributor

@BasPH BasPH commented Dec 21, 2018

Make sure you have checked all steps below.

Jira

  • My PR addresses the following Airflow Jira issues and references them in the PR title. For example, "[AIRFLOW-XXX] My Airflow PR"

Description

  • Here are some details about my PR, including screenshots of any UI changes:

Add function to set "cross join style" downstream dependencies between two list of tasks. For example:

cross_downstream(from_tasks=[t1, t2, t3], to_tasks=[t4, t5, t6])

Sets dependencies:
    t1 --> t4
       \ /
    t2 -X> t5
       / \
    t3 --> t6

Equivalent to:
t1.set_downstream(t4)
t1.set_downstream(t5)
t1.set_downstream(t6)
t2.set_downstream(t4)
t2.set_downstream(t5)
t2.set_downstream(t6)
t3.set_downstream(t4)
t3.set_downstream(t5)
t3.set_downstream(t6)

Tests

  • My PR adds the following unit tests OR does not need testing for this extremely good reason:

HelpersTest.test_cross_downstream()

Commits

  • My commits all reference Jira issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "How to write a good git commit message":
    1. Subject is separated from body by a blank line
    2. Subject is limited to 50 characters (not including Jira issue reference)
    3. Subject does not end with a period
    4. Subject uses the imperative mood ("add", not "adding")
    5. Body wraps at 72 characters
    6. Body explains "what" and "why", not "how"

Documentation

  • In case of new functionality, my PR adds documentation that describes how to use it.
    • When adding new operators/hooks/sensors, the autoclass documentation generation needs to be added.
    • All the public functions and the classes in the PR contain docstrings that explain what it does

Code Quality

  • Passes flake8

@BasPH BasPH changed the title [AIRFLOW-3556] Add cross join set dependency function [AIRFLOW-3556] Add cross join set downstream function Dec 21, 2018
@codecov-io
Copy link

codecov-io commented Dec 21, 2018

Codecov Report

Merging #4356 into master will increase coverage by 62%.
The diff coverage is 100%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master    #4356      +/-   ##
==========================================
+ Coverage   16.13%   78.14%     +62%     
==========================================
  Files         202      202              
  Lines       16483    16486       +3     
==========================================
+ Hits         2660    12883   +10223     
+ Misses      13823     3603   -10220
Impacted Files Coverage Δ
airflow/utils/helpers.py 82.6% <100%> (+55.94%) ⬆️
airflow/plugins_manager.py 93.1% <0%> (+1.14%) ⬆️
airflow/executors/dask_executor.py 2% <0%> (+2%) ⬆️
airflow/exceptions.py 100% <0%> (+2.85%) ⬆️
airflow/utils/operator_resources.py 86.95% <0%> (+4.34%) ⬆️
airflow/executors/__init__.py 63.46% <0%> (+5.76%) ⬆️
airflow/utils/module_loading.py 100% <0%> (+10.52%) ⬆️
airflow/utils/decorators.py 91.66% <0%> (+14.58%) ⬆️
airflow/settings.py 80.41% <0%> (+14.68%) ⬆️
airflow/hooks/oracle_hook.py 15.47% <0%> (+15.47%) ⬆️
... and 162 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update f211390...0bbf833. Read the comment docs.

@BasPH BasPH closed this Dec 22, 2018
@BasPH BasPH deleted the cross-downstream branch December 22, 2018 09:25
@BasPH BasPH restored the cross-downstream branch December 22, 2018 09:28
@BasPH BasPH reopened this Dec 22, 2018
Copy link
Contributor

@Fokko Fokko left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @BasPH

@Fokko Fokko merged commit c014324 into apache:master Dec 26, 2018
aliceabe pushed a commit to aliceabe/incubator-airflow that referenced this pull request Jan 3, 2019
ashb pushed a commit to ashb/airflow that referenced this pull request Feb 2, 2019
wmorris75 pushed a commit to modmed/incubator-airflow that referenced this pull request Jul 29, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants