Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add two operators in AWS Providers: RedshiftResumeClusterOperator and RedshiftPauseClusterOperator #19665

Merged
merged 29 commits into from
Dec 13, 2021
Merged

Add two operators in AWS Providers: RedshiftResumeClusterOperator and RedshiftPauseClusterOperator #19665

merged 29 commits into from
Dec 13, 2021

Commits on Nov 17, 2021

  1. This commit adds new features to the airflow aws redshift module. Sim… 

    …ilar to the aws ec2 hook, this commit implements a wait_for_state function in the redshift hook module. Additionally we are implementing two new Operators: RedshiftResumeClusterOperator and RedshiftPauseClusterOperator. These operators will let developers leverage Airflow to resume and pause Redshift clusters before/after loading data to optimize for costs.
    dbarrundia3 committed Nov 17, 2021
    Configuration menu
    Copy the full SHA
    580dd93 View commit details
    Browse the repository at this point in the history

Commits on Nov 18, 2021

  1. Add missing requirements for Static checks. Add license to all python… 

    … files, run isort, and add new operators to the amazon provider.yaml
    dbarrundia3 committed Nov 18, 2021
    Configuration menu
    Copy the full SHA
    7a921c5 View commit details
    Browse the repository at this point in the history

  2. Adding changes for pre-commit success

    dbarrundia3 committed Nov 18, 2021
    Configuration menu
    Copy the full SHA
    0edb98d View commit details
    Browse the repository at this point in the history

Commits on Nov 20, 2021

  1. Adding ClusterStates Enum to redshift hook to avoid having magic st… 

    …rings and remove `wait_for_state` method in favor of the sensor at `airflow/providers/amazon/aws/sensors/redshift.py`
    dbarrundia3 committed Nov 20, 2021
    Configuration menu
    Copy the full SHA
    9911c72 View commit details
    Browse the repository at this point in the history

Commits on Nov 22, 2021

  1. Adding documentation to existing AWS operator docs

    dbarrundia3 committed Nov 22, 2021
    Configuration menu
    Copy the full SHA
    a68cbbf View commit details
    Browse the repository at this point in the history

  2. Moving the cast to ClusterStates Enum to inside the `cluster_status… 

    …` method instead of doing the cast in all Operators that use the function.
    dbarrundia3 committed Nov 22, 2021
    Configuration menu
    Copy the full SHA
    b20a38e View commit details
    Browse the repository at this point in the history

  3. Fixed logging to redshift sensor to show the actual value of the ENUM… 

    … and change type of `target_status`
    dbarrundia3 committed Nov 22, 2021
    Configuration menu
    Copy the full SHA
    81f2f75 View commit details
    Browse the repository at this point in the history

Commits on Nov 29, 2021

  1. Fixed logging to redshift sensor to show the actual value of the ENUM… 

    … and change type of `target_status`
    dbarrundia3 committed Nov 29, 2021
    Configuration menu
    Copy the full SHA
    61b2935 View commit details
    Browse the repository at this point in the history

Commits on Nov 30, 2021

  1. Adding seealso to Operator documentation

    dbarrundia3 committed Nov 30, 2021
    Configuration menu
    Copy the full SHA
    3b3f1e0 View commit details
    Browse the repository at this point in the history

  2. Update airflow/providers/amazon/aws/operators/redshift_resume_cluster.py 

    Co-authored-by: Daniel Standish <15932138+dstandish@users.noreply.github.com>
    @dbarrundiag @dstandish
    dbarrundiag and dstandish authored Nov 30, 2021
    Configuration menu
    Copy the full SHA
    b151337 View commit details
    Browse the repository at this point in the history

  3. Update airflow/providers/amazon/aws/operators/redshift_pause_cluster.py 

    Co-authored-by: Daniel Standish <15932138+dstandish@users.noreply.github.com>
    @dbarrundiag @dstandish
    dbarrundiag and dstandish authored Nov 30, 2021
    Configuration menu
    Copy the full SHA
    ec92273 View commit details
    Browse the repository at this point in the history

  4. Removing unused check_interval from operator

    dbarrundia3 committed Nov 30, 2021
    Configuration menu
    Copy the full SHA
    3ce29da View commit details
    Browse the repository at this point in the history

  5. Merge remote-tracking branch 'origin/redshift-resume-pause-cluster-op… 

    …erators' into redshift-resume-pause-cluster-operators
    dbarrundia3 committed Nov 30, 2021
    Configuration menu
    Copy the full SHA
    456cb73 View commit details
    Browse the repository at this point in the history

  6. Update airflow/providers/amazon/aws/sensors/redshift.py 

    Co-authored-by: Daniel Standish <15932138+dstandish@users.noreply.github.com>
    @dbarrundiag @dstandish
    dbarrundiag and dstandish authored Nov 30, 2021
    Configuration menu
    Copy the full SHA
    4d6e402 View commit details
    Browse the repository at this point in the history

  7. Update docs/apache-airflow-providers-amazon/operators/redshift.rst 

    Co-authored-by: Daniel Standish <15932138+dstandish@users.noreply.github.com>
    @dbarrundiag @dstandish
    dbarrundiag and dstandish authored Nov 30, 2021
    Configuration menu
    Copy the full SHA
    57c4c35 View commit details
    Browse the repository at this point in the history

  8. Update docs/apache-airflow-providers-amazon/operators/redshift.rst 

    Co-authored-by: Daniel Standish <15932138+dstandish@users.noreply.github.com>
    @dbarrundiag @dstandish
    dbarrundiag and dstandish authored Nov 30, 2021
    Configuration menu
    Copy the full SHA
    e03031f View commit details
    Browse the repository at this point in the history

  9. Update tests/providers/amazon/aws/sensors/test_redshift.py 

    Co-authored-by: Daniel Standish <15932138+dstandish@users.noreply.github.com>
    @dbarrundiag @dstandish
    dbarrundiag and dstandish authored Nov 30, 2021
    Configuration menu
    Copy the full SHA
    92cfcf2 View commit details
    Browse the repository at this point in the history

Commits on Dec 7, 2021

  1. Rolling back changes to not use enum and move all Operators into one … 

    …unified file.
    dbarrundia3 committed Dec 7, 2021
    Configuration menu
    Copy the full SHA
    06a7e3d View commit details
    Browse the repository at this point in the history

  2. Merge remote-tracking branch 'origin/redshift-resume-pause-cluster-op… 

    …erators' into redshift-resume-pause-cluster-operators

# Conflicts:
#	airflow/providers/amazon/aws/sensors/redshift.py
#	docs/apache-airflow-providers-amazon/operators/redshift.rst
    dbarrundia3 committed Dec 7, 2021
    Configuration menu
    Copy the full SHA
    d20662a View commit details
    Browse the repository at this point in the history

  3. Rolling back changes to not use enum and move all Operators into one … 

    …unified file.
    dbarrundia3 committed Dec 7, 2021
    Configuration menu
    Copy the full SHA
    e72bec0 View commit details
    Browse the repository at this point in the history

  4. Rolling back changes to not use enum and move all Operators into one … 

    …unified file.
    dbarrundia3 committed Dec 7, 2021
    Configuration menu
    Copy the full SHA
    e659253 View commit details
    Browse the repository at this point in the history

  5. Rolling back changes to not use enum and move all Operators into one … 

    …unified file.
    dbarrundia3 committed Dec 7, 2021
    Configuration menu
    Copy the full SHA
    b303dc7 View commit details
    Browse the repository at this point in the history

  6. Update airflow/providers/amazon/aws/operators/redshift.py 

    Co-authored-by: Daniel Standish <15932138+dstandish@users.noreply.github.com>
    @dbarrundiag @dstandish
    dbarrundiag and dstandish authored Dec 7, 2021
    Configuration menu
    Copy the full SHA
    6c63fed View commit details
    Browse the repository at this point in the history

  7. Update airflow/providers/amazon/aws/operators/redshift.py 

    Co-authored-by: Daniel Standish <15932138+dstandish@users.noreply.github.com>
    @dbarrundiag @dstandish
    dbarrundiag and dstandish authored Dec 7, 2021
    Configuration menu
    Copy the full SHA
    3ac8f33 View commit details
    Browse the repository at this point in the history

  8. Removing no longer required subclass

    dbarrundia3 committed Dec 7, 2021
    Configuration menu
    Copy the full SHA
    50becc6 View commit details
    Browse the repository at this point in the history

  9. Merge remote-tracking branch 'origin/redshift-resume-pause-cluster-op… 

    …erators' into redshift-resume-pause-cluster-operators
    dbarrundia3 committed Dec 7, 2021
    Configuration menu
    Copy the full SHA
    653d6b2 View commit details
    Browse the repository at this point in the history

Commits on Dec 9, 2021

  1. Add unittests for TestPauseClusterOperator and TestResumeClusterOperator

    dbarrundia3 committed Dec 9, 2021
    Configuration menu
    Copy the full SHA
    f012c29 View commit details
    Browse the repository at this point in the history

  2. Clean unittest for redshift Operators

    dbarrundia3 committed Dec 9, 2021
    Configuration menu
    Copy the full SHA
    a7ff6ca View commit details
    Browse the repository at this point in the history

  3. Small fixups

    dbarrundia3 committed Dec 9, 2021
    Configuration menu
    Copy the full SHA
    f4ac7a4 View commit details
    Browse the repository at this point in the history